LunaLabs

Data infrastructure · 2025

Cost Data Pipeline

Structured extraction from an application that publishes no data.

The source is a single-page application: the page HTML contains no data, and there is no export. The pipeline drives a headless browser to reach the internal endpoints the application itself calls, then normalises the responses into flat, analysable tables across every specialty and procedure.

Status
Shipped
Domain
Data acquisition
Role
Pipeline engineering
Year
2025

Stack

  • Python
  • Playwright
  • pandas
  • CSV · JSON · XLSX

An extraction pipeline that recovers national procedure-cost data — specialist fees, out-of-pocket expenses and state-level breakdowns — from a client-rendered application with no published dataset.

01

Reaching the data the application uses

Rather than parsing rendered markup — brittle, and it breaks on every styling change — the pipeline uses a headless browser to exercise the application and capture the structured responses behind it.

The contract it depends on is the data shape, which changes rarely, instead of the DOM, which changes constantly.

02

Flat output, three formats

Nested responses are flattened into one row per service with national and state-level columns, emitted as CSV for pipelines, JSON for programmatic use and a workbook for the analysts who will actually open it.

Every column is documented, because an undocumented column is a column that gets misread.

03

Traversal is the design problem

The catalogue is a graph, not a list: specialties contain services, services carry national and sub-national breakdowns, and the same service appears under more than one parent.

Walking it exactly once, in an order that can be resumed after an interruption, is most of the engineering — a naive traversal either misses branches or fetches the same record repeatedly and takes hours longer than it needs to.

04

Documentation as part of the artifact

A cost dataset that nobody can interpret is worse than no dataset, because it will be interpreted anyway.

Every emitted column carries a written definition — what it measures, which population it covers, what it excludes — shipped with the data rather than held by the person who built the scraper.

Pipeline

Extraction

Stage Can reject

  1. 01

    Headless browser

    Application session

  2. 02

    Endpoint capture

    Structured responses

    Data shape, not markup

  3. 03

    Traverse

    Full catalogue

    Resumable graph walk

  4. 04

    Flatten

    One row per service

  5. 05

    Emit

    CSV · JSON · XLSX

Skills exercised

What the build
actually demanded.

Against the corpus · 15 systems

  • Stack breadth 4
  • Design decisions 4
  • Pipeline stages 5
  • Decision gates 1

This system Corpus median

Acquisition

  • Headless-browser automation against client-rendered apps
  • Internal endpoint discovery and response capture
  • Rate-aware traversal of a full catalogue
  • Resilience to partial failure mid-run

Shaping

  • Flattening nested responses to one row per service
  • National and sub-national column normalisation
  • Multi-format emission for different consumers
  • Column-level documentation as a deliverable

What it establishes

  • Endpoint-level extraction

    Depends on data shape, not markup

  • Full national coverage

    Every specialty, every named procedure

  • Three output formats

    Pipeline, programmatic, and human

  • Resumable traversal

    A graph walked once, restartable mid-run