Edge Executor

EdgeExecutor is an option if you want to distribute tasks to workers distributed in different locations. You can use it also in parallel with other executors if needed. Change your airflow.cfg to point the executor parameter to EdgeExecutor and provide the related settings. The EdgeExecutor is the component to schedule tasks to the edge workers. The edge workers need to be set-up separately as described in Edge Worker Deployment.

The configuration parameters of the Edge Executor can be found in the Edge provider’s Configuration Reference.

To understand the setup of the Edge Executor, please also take a look to Edge Provider Architecture.

See more details Airflow documentation

Using Multiple Executors Concurrently.

Queues

When using the EdgeExecutor, the workers that tasks are sent to can be specified. queue is an attribute of BaseOperator, so any task can be assigned to any queue. The default queue for the environment is defined in the airflow.cfg’s operators -> default_queue. This defines the queue that tasks get assigned to when not specified, as well as which queue Airflow workers listen to when started.

Workers can listen to one or multiple queues of tasks. When a worker is started (using command airflow edge worker), a set of comma-delimited queue names (with no whitespace) can be given (e.g. airflow edge worker -q remote,wisconsin_site). This worker will then only pick up tasks wired to the specified queue(s). If the queue attribute is not given then a worker will pick tasks from all queues.

This can be useful if you need specialized workers, either from a resource perspective (for say very lightweight tasks where one worker could take thousands of tasks without a problem), or from an environment perspective (you want a worker running from a specific location where required infrastructure is available).

When using EdgeExecutor in addition to other executors and EdgeExecutor not being the default executor (that is to say the first one in the list of executors), be reminded to also define EdgeExecutor as the executor at task or Dag level in addition to the queues you are targeting. For more details on multiple executors please see Using Multiple Executors Concurrently.

Multi-Team Support

When multiple teams share a single Airflow deployment, each team may need its own set of edge workers — for example, separate on-premise sites, different geographic regions, or isolated execution environments. The EdgeExecutor integrates with Airflow’s Multi-Team mode so that each team’s edge jobs and workers are kept separate.

To use multi-team with the EdgeExecutor, first enable Multi-Team mode in your Airflow deployment and create the teams you need. Then configure the EdgeExecutor for each team in your airflow.cfg:

[core]
multi_team = True
executor = EdgeExecutor;team_a=EdgeExecutor;team_b=EdgeExecutor

With this configuration, the scheduler runs a dedicated EdgeExecutor instance per team. Each instance only schedules and monitors jobs belonging to its own team, and each worker only picks up jobs assigned to its team.

Starting a worker for a specific team:

airflow edge worker --team-name team_a -q queue1,queue2

When --team-name is omitted, the worker operates without team isolation — the same behavior as a single-team deployment. Existing workers continue to work without any changes.

Per-team configuration overrides:

Each team’s EdgeExecutor can have its own settings. Use environment variables with the AIRFLOW__<TEAM_NAME>___<SECTION>__<KEY> pattern (triple underscore between team name and section):

# Set a longer heartbeat interval for team_a's edge workers
export AIRFLOW__TEAM_A___EDGE__HEARTBEAT_INTERVAL=30

# Point team_b's workers to a different API endpoint
export AIRFLOW__TEAM_B___EDGE__API_URL=https://team-b-api.example.com/edge_worker/v1/rpcapi

Warning

Security limitation: Multi-team in the EdgeExecutor provides logical isolation only. Worker management CLI commands (maintenance, shutdown, remove, etc.) operate without team distinction — any administrator can manage any worker regardless of its team. Treat multi-team as an organizational separation for trusted administrators, not as a security boundary. Per-team authentication tokens are planned for a future release.

Concurrency slot handling

Some tasks may need more resources than other tasks, to handle these use case the Edge worker supports concurrency slot handling. The logic behind this is the same as the pool slot feature see Pools. Edge worker reuses pool_slots of task_instance to keep number if task instance parameter as low as possible. The pool_slots value works together with the worker_concurrency value which is defined during start of worker. If a task needs more resources, the pool_slots value can be increased to reduce number of tasks running in parallel. The value can be used to block other tasks from being executed in parallel on the same worker. A pool_slots of 2 and a worker_concurrency of 3 means that a worker which executes this task can only execute a job with a pool_slots of 1 in parallel. If no pool_slots is defined for a task the default value is 1. The pool_slots value only supports integer values.

Here is an example setting pool_slots for a task:

import os

import pendulum

from airflow import DAG
from airflow.decorators import task
from airflow.example_dags.libs.helper import print_stuff
from airflow.settings import AIRFLOW_HOME

with DAG(
    dag_id="example_edge_pool_slots",
    schedule=None,
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
    tags=["example"],
) as dag:

    @task(executor="EdgeExecutor", pool_slots=2)
    def task_with_template():
        print_stuff()

    task_with_template()

Current Limitations Edge Executor

  • Some known limitations

    • Log upload will only work if you use a single api-server / webserver instance or they need to share one log file volume. Logs are uploaded in chunks and are transferred via API. If you use multiple api-servers / webservers w/o a shared log volume the logs will be scattered across the api-server / webserver instances and if you view the logs on UI you will only see fractions of the logs.

    • Performance: No extensive performance assessment and scaling tests have been made. The edge executor package is optimized for stability. This will be incrementally improved in future releases. Setups have reported stable operation with ~80 workers until now. Note that executed tasks require more api-server / webserver API capacity.

    • Multi-team isolation is logical only — all teams share a single authentication secret. A worker administrator could change the team name and access another team’s jobs. See Multi-Team Support for details and planned improvements.

Was this entry helpful?