Problem
Tables were checked only once every 13 days, leaving layout and snapshot problems between reviews.
Approach
Scan sizes, file layouts, snapshots and schemas daily. Compact small files, clean up old snapshots and orphaned data, and show each catalog and table’s health in a dashboard.
Measured outcome
Checks every 13 days → daily; queries up to 48% faster after maintenance
Tech stack
- Lake/Storage
- Apache IcebergR2 Data Catalog
- Transform
- DuckDBCloudflare Containers
- Serve
- Cloudflare Workers