PPAS backup & recovery / data lakehouse
KT
Cut KT's PostgreSQL (PPAS) backup storage from 150TB to 7.53TB (about 95%) with Parquet on HDFS organized as Apache Iceberg tables.
- Problem
- Full PostgreSQL (PPAS) backups consumed 150TB.
- Design
- Extracted with Sqoop, stored as Parquet on HDFS, and organized as Apache Iceberg tables. Tiered backups into hourly, daily, weekly, and monthly retention to balance retention and cost.
- Rationale
- Chose Iceberg over plain Hive tables for snapshot isolation and schema evolution, so history never has to be rebuilt when source schemas change.
- Result
- Backup storage cut from 150TB to 7.53TB (about 95%). Columnar Parquet improved recovery speed and storage efficiency.
- Stack
- Apache Iceberg, Parquet, HDFS, Sqoop, Hive, PostgreSQL
Related projects
- Real-time analytics platform enhancement
- MartDB generative AI data agent (GPT-4o)
- Airflow DAG batch scheduling
- Insurance system migration from on-premises to AWS
- iOS/Android QA automation
- Document platform PoC series — the predecessors of localization.md
- GitHub single-file operation-line challenge