The Pipeline That Runs While I Sleep (and While I Game)
Series: Building a Side Project That Runs Itself
- The Origin Story
- The Static Site Bet
- Kotlin for a CLI Backend
- Multi-Cloud Without the Drama
- The Pipeline That Runs While I Game (this post)
- Three Months In: Retrospective
This is the post where everything clicks together. The static site, the Kotlin CLI, the multi-cloud setup — those are all just components sitting on a shelf until something orchestrates them. That something is a set of GitHub Actions workflows running on a schedule, and it’s the reason I can spend a Saturday afternoon deep in a game without giving Stockadora a single thought.
GitHub Actions as an Orchestrator (and Why I Didn’t Use Airflow)
Let me address the obvious question first: why GitHub Actions for scheduling and orchestration? There are purpose-built tools for this. Airflow is the enterprise classic. Argo Workflows runs beautifully on Kubernetes and has a proper DAG UI. Prefect and Dagster are the modern alternatives with nicer developer experiences. They all exist for good reasons.
For a personal project, they’re all overkill.
Every one of those tools requires infrastructure to host. Airflow needs a scheduler process, a webserver, a metadata database, and ideally a proper executor — that’s at minimum a few containers running 24/7 before you’ve scheduled a single job. Argo runs on Kubernetes, which you either already have (great, but now you’re maintaining a cluster for a side project) or you don’t (not great). Even managed versions of these tools have operational overhead that exceeds what I want to deal with for something that runs once a day.
GitHub Actions costs me nothing for this workload, runs on infrastructure I’m not responsible for, and I was already using it for CI. The configuration lives in the same repository as the code. There’s no separate system to log into when something fails — the workflow history is right there in the repo.
What GitHub Actions lacks in compared to dedicated pipeline tools — a visual DAG editor, rich observability, backfill support — I don’t actually need for a daily batch job that I can monitor through failure emails. The 80% solution that requires zero ops is almost always the right call for a project like this.
The one thing that surprises people: GitHub Actions can express a DAG. Job dependencies with needs: give you fan-out parallelism and fan-in synchronization. It’s not as expressive as Argo or Airflow, but it’s enough.
The Pipeline Is Actually a DAG
Here’s what the daily backend workflow actually looks like, which I think illustrates why “scheduled jobs” undersells it:
┌─────────────────────────────────────────────────────────────┐
│ 8:42 AM UTC daily │
└─────────────────────────┬───────────────────────────────────┘
│
┌────────────┴────────────┐
▼ ▼
[crawl-sec-tickers] [crawl-sec-filings]
(metadata, facts) (3-day lookback)
│ │
│ ┌────────────┼────────────┐
│ ▼ ▼ ▼
│ [ai-summary] [insider-trading] [news-digest]
│ (IPO, 10-K, (Form 4 analysis) (daily brief)
│ 8-K events)
│ │ │ │
└────────────┴────────────┴────────────┘
│
▼
[company-data-aggregation]
(merges all outputs, writes S3)
│
┌────────┘
▼
[website-publish]
(Astro build → S3 → CloudFront)Seven jobs. Some run in parallel, some wait for their dependencies to finish. The three AI processing jobs — filing summaries, insider trading analysis, news digest — all fan out from the filing crawl and run simultaneously. They fan back in to the aggregation job, which writes the final JSON to S3. Then the site build kicks off.
This DAG runs start-to-finish every single day. Mostly while I’m asleep or, failing that, while I’m playing something.
Two Cron Schedules, One Purpose
I actually split the pipeline into a backend workflow (the DAG above) and a separate site publish workflow that the backend triggers at the end. But I also run the website publish on its own schedule as a fallback — if the backend job fails partway through, the site still rebuilds with whatever data made it to S3.
on:
schedule:
- cron: '42 8 * * *'Why 8:42 and not 8:00? Because 0 8 * * * is when roughly every other scheduled job in the world fires. GitHub Actions queues back up at round numbers because everyone writes 0 * * * * or 0 8 * * *. Running at :42 means the jobs start within a minute of their scheduled time, reliably. For a pipeline where I care about whether the site updates within a predictable daily window, that minute of queue time actually matters.
The Self-Hosted Runner
This is my favorite part of the whole setup, and I want to give it the attention it deserves.
I’ve worked with self-hosted runners in professional environments before — setting them up, scaling them, migrating CI infrastructure across platforms. The thing I kept taking away from those experiences is that the concept is beautifully simple. You install an agent. You give the agent labels. Workflows target those labels. Whether the agent is running on a bare-metal server in a rack, a VM in a Kubernetes cluster, or a mini PC in your living room — the interface is identical.
That’s exactly what I have here. I have a rack in my living room with several mini computers. One of them runs the Stockadora pipeline. It registers itself as a GitHub Actions self-hosted runner, it’s online 24/7, and when the cron fires it picks up the job immediately with no cold start.
The practical advantages over GitHub-hosted runners for this workload:
Persistent caches. The Kotlin CLI uses Gradle. Gradle build caches are valuable. On a GitHub-hosted runner, the cache uploads and downloads happen over the network on every run — it helps, but it’s slower than a cache that just lives on disk. On my self-hosted machine, the Gradle cache and the pnpm store persist between runs natively. The builds are genuinely faster.
Docker, already running. One of the pipeline jobs relies on a Python-based SEC filing service that I package as a Docker container (cerrorism/sec-filing-service). The workflow starts the container, waits for the health check, runs the work, and tears it down. On a hosted runner you’d be pulling the image on every run. On my machine, Docker is already there and the image is already cached.
No per-minute billing anxiety. A full run of the 7-job DAG takes a while. On GitHub-hosted runners, that’s metered. On my own hardware, the compute is already paid for.
The machine runs 24/7 because it also handles other homelab workloads. Adding the Stockadora runner was just an apt install and a registration token.
I wrote more about the homelab setup in my self-hosted runner post if you want the full picture. For this project, the short version is: the runner is always on, always available, and has never been the thing that caused a pipeline failure.
OIDC Auth in Practice
I covered the “why” of keyless auth in the previous post. Here’s what it actually looks like in the workflow.
The pipeline uses a composite action I called setup-env that consolidates all the environment setup — Java, AWS credentials, GCP credentials, and downloading the pre-built Stockadora CLI from S3 — into a single reusable step:
permissions:
id-token: write
contents: read
steps:
- name: Setup environment
uses: ./.github/composite/setup-env
- name: Run filing summaries
run: stockadora summarize --lookback 3Inside that composite action, the credential setup looks like this:
- name: Authenticate to AWS
uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: arn:aws:iam::ACCOUNT_ID:role/github-actions
aws-region: us-east-1
role-duration-seconds: 14400
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: projects/PROJECT/locations/global/workloadIdentityPools/github-actions-pool/providers/github-actions-provider
service_account: vertex-ai-service@PROJECT_ID.iam.gserviceaccount.comGitHub generates the OIDC token automatically because of the id-token: write permission. Both cloud providers exchange it for temporary credentials scoped to exactly what the pipeline needs. The Kotlin CLI picks up AWS credentials from the environment automatically. The Gemini SDK does the same for GCP.
No secrets stored in the repository. No rotation schedule in my calendar. The credentials expire after four hours — longer than any pipeline run, shorter than anything that matters if somehow leaked.
The setup-env composite action also downloads the pre-built Stockadora CLI binary from S3 and adds it to PATH, so every job that needs it can call stockadora <command> directly. One definition, used in all seven jobs.
What Failure Actually Looks Like
Most posts about pipelines describe the happy path. Here’s the realistic failure landscape for Stockadora.
Most common: transient Gemini errors. The API occasionally returns 429 (rate limited) or 503 during high-load periods. I handle these with exponential backoff retries inside the Kotlin CLI, and they resolve on re-run around 85% of the time. When the batch job fails, I get an email from GitHub. I look at it sometime in the next few hours, hit re-run, it succeeds. Total intervention time: about 30 seconds.
Less common: EDGAR slowness. The SEC’s servers occasionally time out during peak filing periods — earnings season, quarterly index updates. The filing crawl is the root dependency in the DAG, so if it fails, nothing else runs. Re-run resolves it.
Rare: actual bugs. Once, a Gemini model update changed output formatting in a way that broke my JSON parsing. That required a real fix — about an hour to diagnose and patch, then a manual re-run of the affected days. This is the only category that requires me to actually think.
Monitoring is minimal by design: GitHub Actions sends failure emails. No Datadog, no PagerDuty, no Slack alerts. For a project with no SLA, a failure email I see within a few hours is sufficient. A day of stale content is unfortunate, not catastrophic. I have calibrated my anxiety accordingly.
The Real Test
A few months after launch, I went on a week-long trip. I didn’t check the pipeline or the site once.
When I got back, I pulled up the GitHub Actions history. Seven successful backend runs. Seven successful site rebuilds. One backend job had failed on day three — a Gemini rate limit issue — auto-retried once, and succeeded. The site had updated every single day.
That’s what “runs itself” looks like in practice. Not zero failures, because nothing that touches external APIs runs with zero failures. But resilient enough that a week away produces nothing that requires human intervention.
I unpacked, made coffee, and sat down for three hours of games before I even thought to open a browser. The site was fine.
Next up: the honest retrospective. What this costs, what actually broke, and whether any of this was worth building.