Short Notes - Flink Architecture - Task Execution

Flink within YARN

Application Submission and Component interaction

Application

The application consists of a so-called JobGraph, a logical dataflow graph

Job Manager

Single application control by single Job Manager

For each job client is submitted , the session will spawn its own JobManager.

• All Jobs run under session-user credentials
•  Resource Manager holds on to containers for a certain time
•  Internally, sessions built on the dispatcher component. 
•  TaskManager == YARN Container

Data Flow Graph

Blue Nodes == tasks

Edge == data dependency

Data Exchange Strategies

Forward

Sending data from one task to a receiving task

Broadcast

Send all the data to all parallel tasks

Key-based

Guarantees that data items having the same key will be processed by the same task

Random

Data distribute randomly to tasks

Parallel Task Execution

• C & A operators are data sources while E is a data sink.   

• TaskManager runs only tasks of one application

• Scheduling tasks as slices to slots has the advantage that many tasks are co-located on the TaskManager which means that they can efficiently exchange data without accessing the network.

 

References:

https://www.safaribooksonline.com/library/view/stream-processing-with/9781491974285/

YARN Job Scheduling and Task Execution

Hadoop 1.X Architecture

Hadoop 2.X Architecture

Explanation :
  
    1. Submit job to job client

    2. Job client request a new application id

    3. Check output directory already created and copy resources to HDFS

    4. Submit job to resource manager.

    5. Resource Manager contact Node Manager and to allocate a container and  launch a Application Master.

    6. Application Master create an object for book keeping and task management purposes.

    7. Application Master retrieve inputs splits and create one map per split.

    8. If the application task is small (uber task) it will run on same JVM. If the task is not uber, Application Master requests Resource Manager for computing resources.   
    Scheduler knows where is splits are located by heart beats received from Node Managers. Scheduler will allocate Node for task execution.

    9. Application Master contact Node manager to launch a container, Node Manager launch a container for task execution.

    10. Container sends progress report every 3 seconds to the Application Master.Application Master aggregate and sends update directly to JobClient.