Governed ingestion on a UK insurance data platform
Taking over as architect on a UK insurance data platform and halving the time it took to onboard a new data source, through standardised ingestion and a governance layer on Unity Catalog.
- Sector
- Insurance · UK data platform
- Our role
- Architect
Architecture
Sources
Ingest
Lakehouse
Governance
Consumers
- Databricks
- Unity Catalog
- Delta Lake
- Medallion
- Azure DevOps
The situation
The platform worked, but every new data source arrived as its own bespoke pipeline. That is fine for the first few and expensive by the fortieth: each one is a fresh set of decisions, a fresh thing to test, and a fresh thing to maintain. The cost of adding data was rising with every source, which is the opposite of what a platform is supposed to do.
One shape for every source
We standardised the ingestion pattern so that a new source follows a known path from landing through to a curated product, rather than being invented each time. The medallion structure on Delta Lake gave that path a consistent shape, and the repeatable parts stopped being rewritten. New data started arriving through a process rather than a project.
Governance that scales
Underneath it we put a proper governance layer on Unity Catalog: a single catalogue, lineage that shows where data came from, and access control that a team can reason about. That is what lets standardisation stay safe as it scales, because the faster you onboard data the more it matters that you can see and govern all of it.
Building something like this?
Book a call