Building a CI/CD pipeline from commit to production is one problem for a single project. A monorepo’s CI pipeline usually starts out simple the same way and gets slow in a very specific, predictable way: every commit, regardless of what it actually touched, triggers a full build and test run of everything in the repository. That’s tolerable at ten projects and painful at fifty, and the fix isn’t a faster CI runner — it’s making the pipeline actually know what changed.
The Naive Pipeline, and Why It Doesn’t Scale
# .github/workflows/ci.yml — builds and tests everything, every time
jobs:
build:
steps:
- run: npm run build --workspaces
- run: npm run test --workspacesThis is correct in the sense that it never misses a broken build — everything gets checked, always. It’s also wasteful in a way that compounds: a one-line fix to a shared README triggers the exact same full build-and-test cycle as a change touching every service in the repo, and as project count grows, so does the tax every single commit pays regardless of its actual size. The fix has two independent halves that get conflated but solve different problems: caching (don’t redo work whose inputs haven’t changed) and affected-project detection (don’t even attempt work for projects nothing touched).
Half One: Content-Addressed Caching
Tools built for monorepos — Turborepo, Nx — replace “did the source file’s timestamp change” with “did the actual content that feeds this task change,” computed as a hash over the relevant source files, their dependencies, and the task’s own configuration:
$ turbo run build
api:build: cache hit, replaying output
web:build: cache miss, running build
worker:build: cache hit, replaying output
Tasks: 3 successful, 3 total
Cached: 2 cached, 3 total
Time: 4.2s (>>> FULL TURBO)“Cache hit, replaying output” means the tool recognized this exact combination of inputs has already been built somewhere before — possibly on a completely different CI run, or another engineer’s machine, if a remote cache is configured — and simply replays the previous build’s logs and artifacts instead of rerunning anything. This is what content-addressing buys over a naive file-timestamp check: a file can be re-saved with no actual content change (a formatter running, a git checkout touching mtimes) without invalidating the cache, because the hash of the actual bytes didn’t change even though the timestamp did.
Half Two: Only Testing What Could Actually Be Affected
Caching skips redoing unchanged work; affected-project detection skips even attempting work for projects that couldn’t possibly be affected by a given change, based on the dependency graph between projects in the repo:
$ nx affected --target=test --base=main
Affected projects: api, shared-auth
(web, worker, docs: unaffected, skipped entirely)The tool builds a dependency graph from actual import statements (or, in less automated setups, from an explicitly declared graph) — api imports shared-auth, so a change to shared-auth marks both as affected; worker imports neither, so it’s skipped without even attempting a cache lookup. This is the difference between a change to a leaf project running in isolation, and a change to a widely-depended-on shared library correctly triggering every consumer — the graph, not a guess based on which folder changed, decides the blast radius.
The Trap: an Incomplete Dependency Graph Is Worse Than No Graph
This is the failure mode worth naming explicitly, because it’s the one that erodes trust in the whole system quietly. If the affected-project graph is built from static import analysis but something in the repo depends on another project in a way the analyzer can’t see — a shared database schema two “unrelated” services both read, a config file one project generates that another consumes at runtime, a dynamic require() the static analyzer doesn’t follow — a change to that dependency won’t mark the real consumers as affected, their tests won’t run, and a real breakage ships with a fully green pipeline. A team that hits this once tends to overcorrect by manually forcing broad test runs “to be safe,” which quietly recreates the exact problem affected-detection was meant to solve. The actual fix is narrower: declare implicit dependencies the tool explicitly (most support a manual override for exactly this case) rather than distrusting the whole mechanism.
Remote Caching: Sharing the Win Across Machines
Local caching only helps a given machine rebuild something it already built once. Remote caching — a shared cache store (S3, or the tool vendor’s hosted service) that every CI runner and every developer’s machine reads from and writes to — means a build one engineer already ran locally can be skipped entirely on CI, and a build one CI job already ran can be skipped by a parallel job in the same pipeline needing the same artifact:
# turbo.json
{
"remoteCache": { "signature": true }
}The practical effect on a busy team: the first person to touch a given change pays the full build cost once, and everyone else — CI included — gets a cache hit for that exact same input. This is usually where the largest wall-clock time savings actually come from, more than local caching alone, because it eliminates redundant work across an entire team instead of just across one person’s repeated runs.
The Checklist
- Add content-addressed caching before affected-detection if you can only do one first — it’s lower-risk (correctness doesn’t depend on a dependency graph being complete) and often the bigger initial win.
- Build affected-project detection from real import analysis, and declare any dependency the analyzer genuinely can’t see rather than falling back to running everything “to be safe.”
- Set up remote caching once local caching is proven — team-wide cache sharing is usually where the largest time savings actually show up.
- Treat a suspiciously fast green pipeline after a shared-dependency change as worth double-checking, not celebrating — it’s the specific symptom of an incomplete affected-project graph.
