LunaLabs

Actuarial studies · 2025

Health Index

Discover, enrich, and model a national health-services market.

A health-services market has no single register. The authoritative directory is incomplete, live business data is unstructured, and the variable that actually matters for demand modelling — how many practitioners a site really employs — is published almost nowhere. The pipeline reconstructs it in four stages: ingest, discover, extract, predict.

Status
Production
Domain
Health economics · Geospatial ML
Role
Pipeline architecture, modelling
Year
2025

Stack

  • PostgreSQL
  • Python
  • Playwright
  • LLM extraction
  • scikit-learn
  • Geospatial · SA2

A data pipeline that builds a national index of health businesses from a public services directory and live discovery, enriches it by scraping and language-model extraction, and predicts staffing levels with a supervised model.

01

Four stages, each independently checkable

The national directory is ingested into a normalised schema and enriched with statistical-area codes so every site can be joined to census geography.

Live discovery fills the gaps the directory misses. Site content is then scraped and passed through language-model extraction to recover the unstructured facts — practitioner lists, service mixes, opening patterns. Finally a supervised model predicts full-time-equivalent staffing where it is unobserved.

02

Geography as the join key

Attaching statistical-area codes at ingestion time is what makes the index useful rather than merely large.

Once every location carries its area code, the index joins directly to demographic and demand data, and the questions change from "where are the clinics" to "where is the supply thin relative to the population it serves".

03

Language models for extraction only

The model reads prose and returns structure.

It does not estimate the staffing figure — that is a supervised model with a training set and a measurable error. Keeping the two jobs apart means the uncertain quantity has an error bar rather than a confident sentence.

04

The directory is a starting point, not a source of truth

A national services directory is maintained by the organisations it lists, which means it is incomplete in a biased way: sites that never registered are absent, and sites that closed remain.

Live discovery is not a supplement to it but a correction of it, and reconciling the two is where most of the pipeline’s judgement lives — a match that is too loose merges distinct practices, and one that is too strict double-counts a single site under two trading names.

05

Stages that can be re-run

Each stage writes to the database and can be re-run without corrupting what came before, because a pipeline this long will fail partway through — a rate limit, a layout change, an exhausted quota.

Designing for resumption rather than for a clean single pass is what makes the difference between a pipeline that is operated and one that is nursed.

06

What the index is actually for

The output is not a directory.

It is an input to demand and supply modelling: given the practitioners a catchment has and the population it serves, where is provision thin, where is it saturated, and how does that move with demographics. That downstream purpose is what justifies predicting the staffing variable at all — it is the quantity the model needs and the one nobody publishes.

Pipeline

Four stages, each checkable

Stage Can reject

  1. 01

    Directory

    Baseline register

  2. 02

    Geocode

    Statistical areas

    Census-joinable from here on

  3. 03

    Discover

    Gap-filled index

  4. 04

    Resolve

    Deduplicated sites

    Fuzzy match adjudicated

  5. 05

    Extract

    Structured facts

    LLM reads prose only

  6. 06

    Predict FTE

    Staffing estimate

    Supervised, with error

Skills exercised

What the build
actually demanded.

Against the corpus · 15 systems

  • Stack breadth 6
  • Design decisions 6
  • Pipeline stages 6
  • Decision gates 1

This system Corpus median

Data engineering

  • Normalised schema design over a national directory feed
  • Multi-source reconciliation and deduplication
  • Headless-browser discovery and enrichment at scale
  • Idempotent, restartable ingestion stages

Modelling

  • Supervised FTE prediction from heterogeneous features
  • Language-model extraction constrained to structure
  • Feature engineering from unstructured site content
  • Validation against a held-out labelled subset

Geospatial

  • Statistical-area coding at ingestion time
  • Census joins for supply-versus-population analysis
  • Catchment and coverage measurement
  • Shapefile handling and spatial indexing

What it establishes

  • Directory + live discovery

    Two sources reconciled into one index

  • Census-joinable

    Statistical-area codes attached at ingestion

  • FTE prediction

    Supervised model, not a language-model guess

  • Resumable by design

    Every stage re-runnable after a partial failure

  • Built for supply analysis

    Provision measured against the population served