The challenge
This health benefits company’s business runs on data moving between parties — members, brokers, hospitals, carriers, claims — each generating records in its own system, on its own schedule, in its own shape. There was no consolidated, near-real-time view of these entities. The warehouse held part of the picture, but it was built for analytics, not for arbitrary consumption patterns.
Leadership had already aligned on solving this. What they didn’t have was technical direction. Early drafts pointed toward stream processing engines — a heavier stack the team didn’t know how to run or hire for. Committing to the wrong foundation would be expensive to undo, so they brought in a partner to validate or invalidate their assumptions before they built anything.
What we did
Rather than hand over a slide deck of options, we built toward a decision the team could defend:
- Pressure-tested every proposed use case against the latency it actually required, instead of assuming all of them needed real-time streaming
- Evaluated vendor offerings across 20+ dimensions — HIPAA coverage, cost, operational burden, dbt integration — in a rubric transparent enough for the team to check our work
- Designed an end-to-end architecture covering ingestion, processing, canonical entities, serving, and distribution, built through repeated working sessions with the data engineering lead
How we did it
Four weeks, structured to de-risk the decision before any implementation began:
- 1
Map
worked through use cases with the data engineering lead to establish what actually needed near-real-time data, and what didn't
- 2
Test
pressure-tested each use case's latency requirement and evaluated vendor options across 20+ dimensions, from HIPAA coverage to dbt integration
- 3
Architect
designed the end-to-end system — ingestion, processing, canonical entities, serving and distribution — refined against the team's feedback
- 4
Roadmap
sequenced a multi-phase implementation plan with dependencies, effort estimates, and written assumptions
The results
- Every latency requirement met without a stream processor — using tooling the existing team already understands
- An implementation plan without a platform build — scoped to a few months and prioritized against existing timelines, so the team can start without hiring
- A decision the team can defend — architecture and vendor choices grounded in evidence, not opinion
Best fit
Any organization about to commit to infrastructure it will live with for years — especially when the obvious answer is heavier than the problem requires, and the cost of being wrong is too high to find out later.