Kim Pyeong Gwon

Airflow DAG batch scheduling

KT

Moved KT's Crontab-based Spark and stored-procedure batches to Airflow DAGs on Docker with explicit task dependencies and web-based monitoring.

Problem
Crontab-based batches had no dependency control or visibility.
Design
Analyzed Spark, stored-procedure, and Crontab jobs and moved them to Airflow DAGs with explicit task dependencies, on Docker, with web-based execution monitoring.
Result
Per-task failure tracking, web-based log and schedule inspection, and better maintainability on Docker.
Stack
Apache Airflow, Docker, Spark, PostgreSQL

See also