airflow.providers.openlineage.utils.spark¶
Attributes¶
Functions¶
Inject parent job information into Spark properties if not already present. |
|
Inject transport information into Spark properties if not already present. |
|
Inject parent job information into Glue job arguments if not already present. |
|
Inject transport information into Glue job arguments if not already present. |
Module Contents¶
- airflow.providers.openlineage.utils.spark.inject_parent_job_information_into_spark_properties(properties, context)[source]¶
Inject parent job information into Spark properties if not already present.
- Args:
properties: Spark properties. context: The context containing task instance information.
- Returns:
Modified Spark properties with OpenLineage parent job information properties injected, if applicable.
- airflow.providers.openlineage.utils.spark.inject_transport_information_into_spark_properties(properties, context)[source]¶
Inject transport information into Spark properties if not already present.
- Args:
properties: Spark properties. context: The context containing task instance information.
- Returns:
Modified Spark properties with OpenLineage transport information properties injected, if applicable.
- airflow.providers.openlineage.utils.spark.inject_parent_job_information_into_glue_arguments(script_args, context)[source]¶
Inject parent job information into Glue job arguments if not already present.
Glue jobs pass Spark properties via the
--confkey in the script_args dict. Multiple Spark conf properties are combined into the--confkey value with' --conf 'as separator between each property assignment.- Args:
script_args: Glue job script arguments dict (maps to boto3
Arguments). context: The context containing task instance information.- Returns:
Modified script_args dict with OpenLineage parent job information injected, if applicable.
- airflow.providers.openlineage.utils.spark.inject_transport_information_into_glue_arguments(script_args, context)[source]¶
Inject transport information into Glue job arguments if not already present.
Glue jobs pass Spark properties via the
--confkey in the script_args dict. Multiple Spark conf properties are combined into the--confkey value with' --conf 'as separator between each property assignment.- Args:
script_args: Glue job script arguments dict (maps to boto3
Arguments). context: The context containing task instance information.- Returns:
Modified script_args dict with OpenLineage transport information injected, if applicable.