The Survey

Where each territory stands, and which one we walk into first.

Read the bars and a strategy draws itself. Care is the richest territory by raw fetchable count — twenty of its twenty-three sources came back clean, because CMS runs one of the best open-data operations in the federal government and publishes hospital, nursing-home, home-health, and hospice quality as tidy CSVs. Coverage is close behind at fourteen of twenty, and it wins the slice recommendation for a subtler reason we'll get to. Property sits at ten of fifteen but carries the most landmines — thirteen distinct risks logged, because environmental data lives in a zoo of agencies and formats. And Justice is the hard frontier: eight fetchable against ten gated, the only territory where the gated sources outnumber the open ones, because legal data in America is a patchwork of portals, account walls, and one flagship academic API that shut down mid-survey.

The scout that mattered most for sequencing was Coverage's, and its recommendation was sharper than "this atlas is ready." It was: build exactly the ACA plan-comparison page first — the Plan Attributes PUF crossed with the Rate PUF, flowing into a silver.plan spine and out to a gold comparison surface — because both of those sources are CSV-inside-ZIP, both are CI-fetchable, both were fully sampled, and that single path touches zero of the territory's hazards. It avoids every XLSX file that needs special parsing, every gated complaint database, and every oversized download. It's the cleanest possible first cut: a real, useful public surface (compare every ACA plan's premium by age and rating area) built entirely from sources we have already held in our hands. That is how you choose a vertical slice — not the most impressive page, the most certain one.

The slice winner. Twenty datasets, fourteen fetchable, and the readiest clean path of the four.

Coverage is where the survey first proved its own worth. The scout confirmed real, downloadable bytes for the three spines that matter most: the Plan spine (the ACA Plan Attributes and Rate PUFs, plus the QHP landscape), the Enrollment spine (Medicare monthly enrollment down to the county, MA/Part D contract-level enrollment, Medicaid program information), and — the one that made me grin while I was writing the revenue page — the Denial spine, anchored by the CMS Transparency-in-Coverage PUF. That file carries, as actual named columns, Issuer_Percent_Internal_Appeals_Overturned and its external-appeal twin. The hypothetical I'd invented an hour earlier to make a point about high-intent audiences — "which plans overturn the most appeals" — turned out to be a column header in a file we can download today. The fiction had quietly become fact underneath me.

But the scout's real value was the bad news it was unafraid to write down. Two of the four most consequential files — the QHP Landscape (58MB) and the Transparency PUF — are XLSX inside a ZIP, not CSV, which means the bronze reader cannot use csv.reader and must reach for openpyxl. The MA Medical Loss Ratio PUF is a three-file relational structure that needs a join before it can become a clean silver row. The Part D Formulary is a 2.3GB monster whose inner ZIP filenames contain double spaces that will shatter a naive unzip. And everything in the complaint, SERFF rate-filing, T-MSIS, and prior-authorization world is gated or portal-only, which means the Complaint and PriorAuth spines have no version-one source and must wait. None of that is a surprise to find — but all of it is the kind of thing that, undiscovered, turns a "two-day" atlas into a two-week debugging slog. We found it before writing a line of pipeline code. That is the entire point of surveying first.

The richest territory. Twenty fetchable sources — and one structural trap that would have bitten us on every refresh.

Care came back the strongest, and it should: CMS Care Compare is the closest thing the U.S. has to a national public registry of healthcare quality, and it publishes Hospital General Information, Complications and Deaths, Healthcare-Associated Infections, nursing-home staffing and inspection results, home-health and hospice measures — all as clean, fetchable CSVs with real star ratings and measure scores inside. The scout sampled the actual columns: overall hospital star plus domain stars and ownership; mortality and readmission measure scores; HAI standardized infection ratios. The entity pages this atlas wants to build — a dossier for every one of the roughly six thousand U.S. hospitals, every nursing home, every home-health agency — have their data sources confirmed and in hand for quality, staffing, integrity, and access. This is the territory closest to "press the stamp and go."

Its scout also caught the single most important structural risk in the entire survey, the kind of thing that doesn't fail your first run — it fails your fourth. Every CMS provider-data URL is version-stamped: the hash, the UUID, and the date-folder in the path rotate on every quarterly refresh. Hardcode today's URL and your pipeline materializes perfectly now and silently 404s in three months when CMS republishes. The scout's prescription is exact: resolve the download URL dynamically every run via the DKAN metastore (dataset_id to distribution.downloadURL) or the data.json catalog, and never write a versioned path into the code. That one sentence, learned now from a careful scout instead of later from a broken cron job, is worth more than the data itself. The only genuine gap is hospital price transparency — the machine-readable files exist but require a dedicated crawler or a commercial aggregator decision, so the price pages wait while the quality and safety pages ship.

Ten fetchable, five gated, and thirteen logged risks — the most landmines of any territory.

Property is where the data is real but the terrain is rough. The good news is genuinely good: the entire OpenFEMA surface is a clean JSON API, and the scout confirmed live, fetchable access to FIMA NFIP redacted policies and claims (the actual flood-insurance record, parcel-adjacent), Hazard Mitigation Assistance subapplications, NOAA Storm Events, NOAA sea-level-rise rasters, the NHC HURDAT2 hurricane track database, USGS seismic-hazard and landslide inventories, and the FIO/NAIC insurance-market workbook. That's enough, today, to build community flood profiles, hazard-event records by county, insurance-market profiles, and mitigation-action guides end to end with no manual staging at all. The scout even named the exact five-dataset first slice that does it.

The rough part is the five gated sources, and they're gated in five different ways, which is itself the lesson. FEMA's National Risk Index hides behind a portal redirect. The CDC Social Vulnerability Index sits behind a dynamic form. FEMA's flood hazard layer has no national flat file at all — it's tiled. CAL FIRE's fire-hazard severity zones throw an Akamai 403 on bulk download, the same bot-block that stopped BLS earlier. And the U.S. Forest Service wildfire-risk raster sits behind a web-application-firewall challenge. Each one is solvable, but each needs a different key, and the scout's thirteen risk notes are mostly a careful inventory of exactly which door needs which key. The reassuring footnote: the L5 boundary here is crisp across all four zones, and nothing about ingesting flood or fire data creates regulated-advice exposure — the danger is purely engineering, not legal.

The hard frontier. The only territory where gated sources outnumber the open ones.

Justice is the territory that proves why we surveyed instead of assuming. Eight fetchable against ten gated — the inverse of every other atlas — because civil-legal data in America is genuinely, structurally hard to get, and a scaffold that listed these sources as available would have walked the build straight into a wall. The good news the scout confirmed is that the federal layer is fully buildable today with no manual staging: the Federal Judicial Center's Integrated Database of civil cases (a real ZIP), the Administrative Office's C-3 statistical tables, the official U.S. Courts PDF forms, CourtListener's bulk opinion CSVs sitting on public S3, a clean statute-of-limitations CSV, and the Legal Services Corporation's grant-allocation JSON API and eviction dataset. That's a working federal-procedure-plus-deadlines-plus-legal-aid pipeline, in hand, runnable.

The bad news is where the human need is highest. The two most valuable datasets for self-represented people facing eviction or debt collection — the LawAtlas policy-surveillance databases and the LSC Eviction Laws Database — are gated behind free-account and Box-portal walls that require a one-time manual download before any pipeline can touch them. The scout also caught two things a list never would: the Harvard Caselaw Access Project API, long the flagship of open legal data, is effectively shut down — every call now redirects to documentation — so CourtListener's bulk files are the substitute. And the National Center for State Courts caseload data, the only public source for small-claims filing volumes by state, has no bulk download at all. The honest read: a version-one Justice slice scoped to federal civil procedure plus the legal-aid directory is fully CI-runnable now, and state-level procedure — which is what most ordinary people actually face — requires staging those gated databases first. We know that today, in detail, instead of discovering it the night we try to ship eviction-defense pages. That foreknowledge is the whole gift of the survey.

Step back from the four territories and the same two shapes appear in all of them: what opens to a plain fetch, and what fights you.

The most valuable thing a survey of four unrelated domains gives you isn't the four answers — it's the pattern across them, because the pattern is what makes the fifth, sixth, and tenth atlas fast. Lay all 76 datasets side by side and they sort cleanly into two maps. The fetchable map has a signature: federal open-data programs that took their own mandate seriously. CMS, OpenFEMA, NOAA, USGS, the federal courts, CourtListener — these publish direct, stable, documented files and APIs, and they account for the overwhelming majority of our 52 wins. When a U.S. agency decided to be a real data publisher, the data is there for the taking, clean, and free. That is the public record doing exactly what the public paid for it to do.

The gated map is where the survey earns its keep, because every entry on it is a specific, named, solvable problem instead of a vague "this might be hard." And the solution to almost all of it is one piece of infrastructure we already sketched in the build manifest: an atlas-{domain}-raw R2 bucket that the gated sources get stocked into once, by hand or by headless browser, after which CI syncs from the bucket instead of fighting the portal. The Akamai blocks, the Box walls, the form gates, the WAF challenges — they all collapse into the same move: stage it once into our own storage, then treat our storage as the source. Twenty-four gated datasets sounds like twenty-four problems. It's really one problem with twenty-four instances, and we already know its name.

Four domains, one pattern. Federal open data opens to a fetch; everything else stages once into our own bucket — and then it opens too.

A survey you can trust is one that's allowed to come back and say: not this, not yet. Here is exactly what we cannot build — and why that sentence is a feature, not a failure.

It would be easy to write this page as a victory lap — fifty-two open doors, four territories mapped, on to the build. But the thing that makes a field report worth more than a pitch deck is that it tells you where the ground gives way, and the no-overclaim law that governs this whole project demands it: the free public core only has value if it never fakes a number. An atlas that confidently displays a denial rate it doesn't actually have a source for is worse than no atlas at all, because it launders a guess into the authority of a citation. So the scouts were told to be just as precise about the gaps as about the wins, and they were. Here is what we genuinely cannot stand up in version one.

In Coverage, the two spines with no fetchable v1 source are the Complaint spine and the PriorAuthRequest spine. The NAIC complaint index, the SERFF rate filings, the T-MSIS Medicaid encounter data, and the prior-authorization datasets are all gated or portal-only — which means the page a frustrated person most wants ("which insurer in my state generates the most complaints, and how often do they make you fight for care you already paid for") is exactly the page we cannot honestly publish until those sources are staged. We will ship plan comparison and appeal-overturn rates, because we have those bytes; we will not ship a complaint ranking we'd have to fabricate. In Care, the gap is hospital prices: the machine-readable transparency files exist by law, but they're scattered across thousands of hospital websites in inconsistent shapes, so the price pages wait behind a crawler-or-aggregator decision while the quality, safety, staffing, and integrity pages — all fully sourced — go first.

In Justice the gap is the cruelest, because it falls exactly where ordinary people need the most help. The federal layer is buildable today, but federal civil procedure is not what a tenant facing eviction or a person being chased by a debt collector actually encounters — that's state procedure, and the two databases that encode it (LawAtlas and the LSC Eviction Laws DB) sit behind account and portal walls that demand a one-time manual stage before any pipeline can run. Worse, the only public source for small-claims filing volumes by state has no bulk download at all and may never be cleanly ingestable. In Property, the headline multi-hazard risk scores (FEMA's National Risk Index) need a single manual stage before they're stable, and the flood hazard layer is published as map tiles rather than a national flat file, which means flood-zone-by-parcel lookups need a fundamentally different ingestion shape than a CSV download. None of these gaps is a dead end. Every one of them is a known, named, sequenced piece of work — and knowing precisely where the ground is soft is what lets us walk across it without falling through. We will build the certain pages now and stage the gated sources in parallel, and we will never publish a number we cannot trace back to a file we actually held.