Lably
Skip to content

Apache Airflow

  • 1 min read
  • 2 chapters

Related on Lably

Through Open-source software, User interface and Python (programming language)

Lab · Class 6–12AI & Python Builds (Medha AI)Lab · Class 9–11Python SandboxLab · Class 3–12Coding in Python (Python Lab)Lab · Class 11Python for ML — NumPy, Pandas, scikit-learn

Airflow is written in Python, and workflows are created via Python scripts. Airflow is designed under the principle of "configuration as code". While other "configuration as code" workflow platforms exist using markup languages like XML, using Python allows developers to import libraries and classes to help them create their workflows.

Overview

Airflow uses directed acyclic graphs (DAGs) to manage workflow orchestration. Tasks and dependencies are defined in Python and then Airflow manages the scheduling and execution. DAGs can be run either on a defined schedule (e.g. hourly or daily) or based on external event triggers (e.g. a file appearing in Hive). Previous DAG-based schedulers like Oozie and Azkaban tended to rely on multiple configuration files and file system trees to create a DAG, whereas in Airflow, DAGs can often be written in one Python file.

Sign in to keep reading

One more chapter is free with a Lably account.

More in Engineering

NanotechnologyTechnology with features near one nanometerSoftware engineeringEngineering approach to software developmentChemical engineeringEngineering discipline focused on the design and operation of chemical plantsAerospace engineeringBranch of engineeringASTM InternationalStandards organizationProfessional associationOrganization promoting a specific profession