Building for Scale: Lessons from Healthcare Data Platforms

In registry-scale healthcare reporting, the performance ceiling sits in the data tier, and statistical correctness constrains the architecture before throughput does. Application-layer caching, output compression, and CDN placement move the numbers at the margin. The structure of the data and the shape of the queries determine everything else.

The work behind this post is SRTR's interactive reporting platform, which publishes transplant program outcomes to two audiences with incompatible requirements: clinicians comparing programs across a set of statistical measures, and patients trying to understand options for their own care. Same registry data, two renderings, two definitions of correct.

Correctness is a hard constraint on the query layer

Healthcare outcome data cannot be displayed uniformly, which pushes statistical logic into the retrieval path rather than the presentation layer.

A transplant program that performed 14 transplants in a reporting period produces outcome statistics with confidence intervals wide enough that the point estimate carries little information. A program that performed 400 produces a usable estimate. Rendering both as the same bar on the same chart misrepresents the second and defames the first. In a regulated publication context, that is a compliance exposure, not a design preference.

The practical consequence is that the system must know, before it renders, which cohorts qualify for display:

• Volume thresholds determine whether a metric is published, suppressed, or published with a qualifier. The threshold is a domain decision and belongs in the data model, not in view logic.
• Confidence intervals are displayed where statistical weight warrants it and suppressed where they would imply precision the sample does not support.
• Small-cell suppression applies where a published figure could identify an individual patient. This is a privacy requirement with a specific rule set, and it interacts with the volume thresholds above.
• Audience-specific rendering means the patient view omits measures that require clinical context to interpret. That omission is a data-access decision, applied server-side, not a hidden CSS class.

Every one of those checks is a predicate that runs against the same result set the chart consumes. Designing them as post-processing over a generic query means running the expensive query first and discarding most of it. Designing them into the aggregation means the retrieval returns display-ready data.

The cost concentrates in cross-sectional queries

For registry data — millions of records, filtered concurrently across organ type, program, geography, and reporting period — the dominant cost is the cross-sectional aggregate, not row retrieval.

The query pattern that breaks naive implementations is the one users most want: filter by four dimensions, aggregate an outcome measure, compare against a national benchmark, and return in under a second while several hundred other users do the same with different filters. Against normalized registry tables, that query scans far more than it returns.

Where the time goes, in the order worth investigating:

• Scans that should be seeks. Multi-dimensional filters against tables indexed for a different access pattern. Covering indexes aligned to the actual filter combinations resolve most of this, at a well-understood cost in write throughput and storage.
• Aggregation over raw records at request time. Computing a five-year survival measure across a cohort on every page load is the single most expensive thing these systems do.
• Parameter sniffing on wide filter surfaces. A plan compiled for a high-volume program performs badly for a low-volume one and vice versa. This surfaces as intermittent slowness that does not reproduce in testing.
• Row-by-row statistical logic. Suppression rules implemented as procedural checks per row rather than as set-based predicates.

Pre-aggregation is the primary lever

Pre-computed result sets for the common view combinations delivered the majority of the performance gain on the SRTR platform, and the same pattern holds for registry reporting generally.

Interactive filtering across multiple dimensions is viable when each filter change queries a pre-computed table rather than raw registry records. The design work is deciding what to pre-compute:

• Enumerate the filter combinations the interface exposes. This is a finite, usually small set — far smaller than the combinatorial space it appears to be, because the interface constrains which dimensions can vary together.
• Materialize aggregates at the grain those combinations require, with the suppression flags and interval bounds computed at build time.
• Treat the aggregation build as a first-class pipeline with its own monitoring, not as a maintenance script. When it fails silently, the site serves stale statistics that look current.

The maintenance burden is real and is the reason teams skip it. It is more work than querying raw tables, and it is the difference between an interface users explore and one they wait on.

Caching rules follow the publication cycle

Cache aggressiveness should be derived from the data's update cadence, and in registry systems that cadence is documented rather than inferred.

• Quarterly or annual publication data can be cached for its full cycle. This covers most of what a registry reporting platform serves. Invalidation is an event tied to publication, not a timer.
• Daily-refreshed operational data cannot use the same policy and generally should not share a cache region with publication data.
• User-parameterized results are cacheable per parameter set, with the caveat that the key must include every dimension that affects suppression. A cache key that omits the audience dimension will eventually serve a clinical view to a patient-facing page.

Invalidation errors in this context are not performance bugs. Serving last cycle's statistics under this cycle's publication date is a data accuracy incident, and it will be found by the people whose programs are being measured.

What to instrument

Instrument at the query grain, because aggregate response-time percentiles hide the failure mode that matters.

• Capture execution statistics per query pattern, not per endpoint. One slow aggregate inside an otherwise fast page is invisible in endpoint timings.
• Alert on aggregation pipeline completion and record count, not just on job success. A build that completes having produced zero rows is the dangerous case.
• Track cache hit ratio segmented by data category. A falling hit ratio on publication data means a key is including something it should not.

Applying this

The sequence that holds for healthcare data platforms is: model the statistical rules into the data tier, pre-aggregate against the interface's real filter surface, cache on the publication cycle, and instrument at the query level.

Performance in this domain is a data architecture problem before it is an application problem. Time spent on table structure, aggregate design, and query shape pays out in the application layer. Time spent optimizing the application layer against a data model that forces per-request aggregation does not.