airflow.providers.openlineage.utils.spark

Attributes

log

Functions

inject_parent_job_information_into_spark_properties(...)

Inject parent job information into Spark properties if not already present.

inject_transport_information_into_spark_properties(...)

Inject transport information into Spark properties if not already present.

inject_parent_job_information_into_glue_arguments(...)

Inject parent job information into Glue job arguments if not already present.

inject_transport_information_into_glue_arguments(...)

Inject transport information into Glue job arguments if not already present.

Module Contents

airflow.providers.openlineage.utils.spark.log[source]
airflow.providers.openlineage.utils.spark.inject_parent_job_information_into_spark_properties(properties, context)[source]

Inject parent job information into Spark properties if not already present.

Args:

properties: Spark properties. context: The context containing task instance information.

Returns:

Modified Spark properties with OpenLineage parent job information properties injected, if applicable.

airflow.providers.openlineage.utils.spark.inject_transport_information_into_spark_properties(properties, context)[source]

Inject transport information into Spark properties if not already present.

Args:

properties: Spark properties. context: The context containing task instance information.

Returns:

Modified Spark properties with OpenLineage transport information properties injected, if applicable.

airflow.providers.openlineage.utils.spark.inject_parent_job_information_into_glue_arguments(script_args, context)[source]

Inject parent job information into Glue job arguments if not already present.

Glue jobs pass Spark properties via the --conf key in the script_args dict. Multiple Spark conf properties are combined into the --conf key value with ' --conf ' as separator between each property assignment.

Args:

script_args: Glue job script arguments dict (maps to boto3 Arguments). context: The context containing task instance information.

Returns:

Modified script_args dict with OpenLineage parent job information injected, if applicable.

airflow.providers.openlineage.utils.spark.inject_transport_information_into_glue_arguments(script_args, context)[source]

Inject transport information into Glue job arguments if not already present.

Glue jobs pass Spark properties via the --conf key in the script_args dict. Multiple Spark conf properties are combined into the --conf key value with ' --conf ' as separator between each property assignment.

Args:

script_args: Glue job script arguments dict (maps to boto3 Arguments). context: The context containing task instance information.

Returns:

Modified script_args dict with OpenLineage transport information injected, if applicable.

Was this entry helpful?