Bronze to Gold Without the 3am Pages: Making a Medallion Pipeline Actually Reliable
The architecture was never the problem. Bronze, silver, and gold layers were all in place - but every schema drift, late-arriving file, and partial load ended the same way: a 3am page to the one engineer who understood the whole graph.
What was at riskThe pipeline was one resignation away from nobody understanding how gold tables were produced.
Does this match your situation?
Tick what sounds familiar. This is for you, not us — nothing is submitted.
Nothing ticked yet — read on, or skim the outcomes below.
What we were solving
Enterprise data team whose data footprint roughly doubled in eighteen months while headcount stayed flat. One senior engineer had become the de facto owner of every pipeline failure.
- Failures surfaced downstream as wrong numbers in a dashboard, not upstream as a failed job - so the data team usually heard about breakage from finance, not from monitoring.
- Every recovery was manual and undocumented: re-run the job, hope the partial write did not double-count, reconcile by hand.
- One engineer held the entire dependency graph in their head. Their on-call weeks were unsustainable and their holiday weeks were terrifying.
What we built
- Put explicit data contracts between every layer so a schema change fails loudly at the bronze/silver boundary instead of silently corrupting gold.
- Made every transform idempotent and re-runnable, so recovery became 'run it again' rather than a hand-reconciled repair with a spreadsheet.
- Added freshness and volume SLAs per table with alerting tied to the table, not the job - the team now learns a number is stale before a stakeholder does.
- Wrote runbooks generated from the pipeline definition itself, so the recovery path for each table stays accurate as the pipeline changes.
Architecture notes for your engineers
- Delta Lake time travel used as the rollback mechanism - a bad load is reverted to the prior table version rather than repaired in place.
- Quality gates between bronze/silver/gold quarantine failing records instead of halting the whole run, so one bad source stops being a full-pipeline outage.
- Z-ordering and compaction scheduled off-peak to keep scan costs predictable as volume grew.
What actually changed
- How breakage is found
- A stakeholder notices a wrong number
- Freshness alert fires before anyone downstream looks
- Recovery process
- Manual re-run, hand-reconciled
- Idempotent re-run, or roll back to prior table version
- Schema drift
- Silently corrupts gold tables
- Fails loudly at the bronze/silver boundary
- Who can fix it
- One senior engineer
- Any of four, following a generated runbook
- Overnight pages
- About four a week
- Under one a month
The delivery arc
Catalogued every incident from the previous six months and grouped them - most traced back to three recurring causes, not dozens.
Key results
- Overnight pages fell from roughly four a week to under one a month
- Silver-layer freshness SLA met on 99.2% of runs across a rolling 90 days
- Four engineers can now run recovery, up from one
- Schema drift fails at the boundary instead of corrupting gold tables
- Bad loads roll back to a prior table version instead of being repaired by hand
What it was built on
Representative tools and patterns — exact vendors vary per client environment.
Lakehouse
Orchestration & quality
Reliability
What we'd tell the next team
- Medallion architecture is not a reliability strategy on its own - bronze/silver/gold describes where data sits, not what happens when a load fails at 2am.
- Idempotency is the single highest-leverage change: it converts every incident from a reconciliation project into a re-run.
- Alert on the table, not the job. Stakeholders care whether the number is fresh, not whether a DAG turned green.
- If your runbooks are hand-written they are already out of date; generate them from the pipeline definition.
Questions this engagement anticipated
Does medallion architecture make a data pipeline reliable by itself?
No. Bronze/silver/gold describes where data sits at each stage of refinement - it says nothing about what happens when a load fails halfway, a schema drifts, or a file arrives late. Reliability comes from contracts between the layers, idempotent transforms, and per-table SLAs layered on top of that structure.
What single change reduced the on-call load most?
Idempotency. Once every transform could be safely re-run, recovery stopped being a manual reconciliation exercise and became a re-run - which is also what made it safe to hand recovery to engineers other than the original author.
Why alert on tables instead of on jobs?
A job can succeed while producing stale or partial data, and a job can fail harmlessly on a retry. Stakeholders only care whether the number they are about to use is fresh and complete, so the alert should describe the table's state, not the DAG's.
Go deeper on the thinking
This is one of several case studies on data engineering & legacy migrations.
See the rest of the cluster →Compare your situation to this case.
Bring your constraints - environment, timeline, and budget. We scope before we quote.