A persistent VPS? A container with block storage? Modal? The honest answer: there are three different problems wearing one coat — where state lives, where compute runs, where the control plane lives. Solve each with the cheapest thing that fits.
Atlas Commons is a stateful batch pipeline running on stateless ephemeral compute (GitHub Actions runners that are born, run once, and die). That single mismatch causes everything:
1 · Input state has nowhere to live. The singulariki atlas was built on a curated 4.6 GB local corpus (data/staging/ — O*NET dumps, BLS workbooks, crosswalks, Lightcast, academic datasets). It's git-ignored (correctly — big + partly licensed). Ephemeral runners don't have it → FileNotFoundError. This is a state-location problem, not a compute problem.
2 · Re-fetch cost & fragility. Even URL-fetched assets re-download every run (nothing persists). Today we landed O*NET — 46 tables — but it re-pulls the whole O*NET DB each time.
3 · No always-on control plane. No live Dagster UI, schedules, or asset-freshness memory. A cron prompt pokes a workflow. Fine for a sprint; thin for a forever-system.
Decouple (A) where state lives · (B) where compute runs · (C) where the control plane lives. Mix vendors freely across the three. The trap is buying a server to fix (1) when (1) is about where the bytes live, not where the CPU is.
An excellent executor. Not a state store, not a control plane. Keep it — just stop asking it to hold state.
Upload the 4.6 GB corpus to a private atlas-staging R2 bucket; re-point bronze readers from Path("data/staging/…") to an R2 read.
Not a compute option — the keystone fix. The one change that unblocks singulariki on the free infra you already have. Do it regardless of everything else.
Always-on Linux box (Hetzner/DO/Linode) + attached volume; Dagster daemon + UI resident; corpus on the volume.
Best only if you want an always-on control plane + a heavy box to dev on. For a cron loop it's more than needed — but it's the most tangible "runs itself" upgrade. If you go here, do it as a self-hosted runner (Option 6), not a parallel install.
Compelling for serving + light/medium jobs next to the data. Don't make it the heavy-batch executor yet — re-verify limits.
Serverless Python with persistent Volumes mounted into ephemeral runs, scale-to-zero billing, huge on-demand memory.
If the priority is persistent state + serverless + no ops, arguably more optimal than a VPS — the persistent volume without the pet. Strongest alternative to "GitHub + R2 staging."
The control-plane answer once you want to see the estate breathe instead of reading it from a spec. Pairs with any executor.
The most pragmatic way to get persistence + power without rewriting anything. If you want a server, make it a runner.
The data's home is R2. Compute in AWS/GCP writing to R2 means cross-cloud egress on every materialization — the one architecture that turns a $0.07/mo bill into a real one. Stay in Cloudflare + GitHub + (Modal, which peers cheaply).
Tier 1 — now · ~$0 · unblocks singulariki this week
Tier 2 — when you want it to watch itself · ~$10–40/mo
Tier 3 — elegant alternative to Tier 1's re-pointing
Move the source location; do not redo the parse logic. The bronze parse code is good; silver/gold are unaffected (they read bronze from the catalog, not disk). Only ~8 modules read data/staging/….
• O*NET — already URL-fetched; 46 tables landed today in singulariki_bronze on a plain runner in 4m19s. Proves the path.
• Staging-bound modules — add one helper, staging_open(relpath), that resolves to a local path if present else an R2 GET from atlas-staging. Re-point ~25 path constants through it. Runs both locally and remotely; no parse changes; no silver/gold changes.
• Then materialize in layers (bronze light→heavy, silver, gold) — the pattern that just worked for O*NET. Same indirection serves money-pipelines and any future staging atlas.
The 4.6 GB upload is a one-time push (files are local, wrangler is authed — I can run it). The one thing I'll confirm with you first: whether any specific licensed file is okay in even a private bucket.
Once staging lives in R2, "the atlases I can build remotely" and "the atlas that only built on Ryan's laptop" become the same thing. All eight atlases — plus singulariki's 172 tables and money-pipelines' 74 — become rebuildable by any runner, anywhere, into one catalog, queryable by one SQL surface, for about seven cents a month of input storage. That's the difference between a pipeline that ran once on a particular machine and a public utility that regenerates itself on cheap, disposable compute — the whole thesis. The compute can be GitHub today, Modal tomorrow, a VPS if you want a pet; the architecture stops caring, because the state finally lives in the one place that outlives every executor: R2.