How Package Managers Actually Ruin Your Day
Contents
I’ve broken a working Python environment by installing one new package often enough that I now treat every pip install on a project I care about as a small, contained emergency. The problem pip solves is much harder than the one-line command makes it look, even though pip itself is written perfectly well. “Install this package” sounds like copying a file. What it actually means is “find a version of this package, and a version of every package it depends on, and a version of every package those depend on, such that every constraint every one of them has expressed about every other one is simultaneously satisfied” — and that is a constraint-satisfaction problem, not a file copy, with all the computational nastiness that implies.
Understanding it as a constraint problem rather than a plumbing problem changes how you debug it. When apt, npm or pip throws a wall of red text about conflicting version requirements, the tool isn’t malfunctioning. It’s telling you, in its own limited vocabulary, that the constraints you’ve asked it to satisfy don’t have a solution, and that it needs a human to relax one of them, because no amount of cleverness in the solver invents compatibility that doesn’t exist between the actual pieces of software.
The diamond that breaks everything
The canonical shape of the problem is the diamond dependency: your project depends on library A and library B, and both A and B depend on library C, but A needs C version 1.x and B needs C version 2.x. There is no single version of C on disk that satisfies both, because most package ecosystems (with important exceptions, below) assume one project uses one version of any given library at a time. Your options are narrow: convince one of A or B to accept the other’s required version of C (only possible if a compatible release actually exists), find alternate versions of A or B that both agree on a version of C, or accept that A and B simply cannot coexist in this project as things stand.
| |
This is a real limitation of the shared-dependency model, and it’s precisely why some ecosystems have moved away from it. Node’s npm sidesteps the diamond almost entirely by allowing nested, independent copies of the same package at different versions inside node_modules — A gets its own private copy of C at 1.x, B gets its own private copy at 2.x, and neither knows the other exists. That solves the conflict at the cost of disk space and, more subtly, at the cost of correctness whenever the two copies of C need to share global state (a database connection pool, a singleton cache) that the duplication silently breaks, because “the same library” is now two unrelated instances that happen to share a name.
Semver is a promise, not a guarantee
Most of the version ranges a resolver works with — ^2.0.0, >=1.4,<2.0 — exist because of semantic versioning, the convention that a major version bump signals a breaking change, a minor bump signals new backward-compatible functionality, and a patch bump signals a bug fix with no behaviour change at all. The entire dependency-resolution ecosystem is built on trusting that convention, because a resolver deciding it’s safe to pick library C version 2.4.1 instead of 2.3.0 to satisfy a ^2.0.0 constraint is relying entirely on the maintainer of C having correctly classified 2.4.1 as non-breaking.
That trust is regularly, if usually accidentally, violated. A maintainer fixes what they consider a bug — a function that used to silently accept a malformed argument now throws — and ships it as a patch release, because from their perspective they fixed something rather than changed a public contract. From your perspective, code that depended on the old, technically-buggy behaviour now throws in production, and the resolver did exactly what it was told: it picked a version inside the allowed range, in good faith, based on a promise the actual release didn’t keep. This is why pinning exact versions in a lockfile matters even when your manifest file specifies a generous range — the manifest describes what you’re willing to accept in principle, the lockfile records what you actually tested against, and the two are allowed to diverge the moment you regenerate the lockfile against a new “compatible” release that turns out not to be.
System packages are the same problem with higher stakes
Everything above describes language-level package managers — pip, npm, cargo — where a broken resolution mostly costs you a failed build or a broken virtual environment you can delete and rebuild. System package managers like apt and dpkg solve the identical constraint problem against your operating system’s shared libraries, where the failure mode is worse, because there’s usually only one copy of a system library on the machine, shared by every program that links against it, with no equivalent of npm’s nested-copies trick available. Upgrade OpenSSL to satisfy one package’s requirement and you’ve potentially just changed the behaviour of every other program on the box that links against the same library, for better or worse, whether or not you tested any of them against the new version.
This is the reason apt refuses upgrades that would require removing an unrelated package to satisfy a conflict, rather than resolving it silently the way a language-level resolver might — the blast radius of getting it wrong is the entire system rather than one project’s virtual environment. It’s also the underlying motivation behind approaches like Nix flakes, which sidestep the shared-library diamond entirely by giving every package its own isolated dependency closure, hashed and addressed independently, so two programs on the same machine can depend on genuinely incompatible versions of the same library without either one being aware the other exists — the same nested-copy trick npm uses for JavaScript packages, generalised to the entire operating system.
Why solving it is provably hard
Dependency resolution belongs to a family of problems computer science already understands to be, in the general case, NP-complete — the same theoretical territory as boolean satisfiability, because a system of “package X requires version range R of package Y” constraints is exactly a boolean satisfiability problem in disguise. That doesn’t mean every resolution is slow in practice — most real dependency graphs are small and well-behaved enough to solve in milliseconds — but it does mean there’s no shortcut algorithm that’s guaranteed to find an answer quickly for every possible graph, and it explains why resolvers occasionally spend tens of seconds (or, in genuinely pathological cases, get stuck entirely) chewing through a large, poorly-versioned dependency tree before giving up and reporting a conflict.
Modern resolvers use SAT-solving techniques directly rather than the naive backtracking that older tools relied on — pip’s resolver, rewritten in 2020 specifically to fix a long history of installing genuinely broken combinations, is a real constraint solver now, not a script that tries the first plausible version of everything and hopes. That rewrite is a good illustration of the underlying point: making the solver smarter reduces how often you hit a genuinely unsolvable diamond, but it cannot eliminate the diamond itself when one of your dependencies has a hard, incompatible requirement on another. No amount of solver sophistication invents a version of library C that satisfies both A and B if no such version was ever published.
Why lockfiles exist, and what they actually fix
A lockfile — package-lock.json, poetry.lock, Cargo.lock, Gemfile.lock — isn’t a performance optimisation, even though it does make installs faster by skipping re-resolution. It’s a record of one specific, verified solution to the constraint problem, pinned so that every machine that installs from it gets the exact same versions rather than re-running the solver against whatever’s newest on the registry at install time. Without a lockfile, npm install run today and the same command run in six months can produce genuinely different dependency trees, because “the latest version satisfying ^2.0.0” keeps moving as new releases land, and a new release two levels down your dependency tree can silently introduce a bug or a breaking change that nobody at your project reviewed.
| |
That integrity hash matters as much as the version pin — it’s a content hash of the exact package tarball, and it’s the mechanism that stops a compromised or tampered registry from silently swapping in different bytes under a version number you already trusted, which is precisely the class of problem behind supply-chain attacks from typosquatting to poisoned container images. A lockfile without integrity hashes only pins the version number; a lockfile with them pins the actual bytes, and the difference matters enormously the day a package’s maintainer account gets compromised and a malicious release gets pushed under an existing version.
Troubleshooting a broken dependency tree
The first move when a resolver reports a conflict is reading the actual conflicting constraints rather than the summary line, because most tools bury the useful information (“A wants C>=1.0,<2.0, B wants C>=2.0”) several lines below a more alarming-looking top-level error. pip install --report or npm ls library-c (which shows every place in the tree that resolved a copy of that package, and at what version) turns a vague failure into a specific, fixable pair of constraints.
The second is checking whether the conflict is real or just outdated version ranges — often one of the two libraries specified an upper bound (<2.0) defensively, before the newer major version of its dependency existed, and a newer release of that same library has since loosened the bound to <3.0 without you having upgraded to it yet. Bumping the outer library, not the conflicting dependency directly, frequently resolves diamonds that look unsolvable at the version you’re currently pinned to.
The third, and the one that costs the most time when missed, is a phantom conflict caused by a stale lockfile that no longer matches the manifest file it’s supposed to describe — someone hand-edited package.json without regenerating package-lock.json, and the two disagree about what’s actually required. npm ci (which refuses to proceed if the lockfile and manifest don’t match, rather than silently reconciling them the way npm install will) is the right tool for catching this in CI specifically because it fails loudly instead of quietly resolving a drift nobody asked for.
Is it worth the pain
Yes, because the alternative — vendoring every dependency’s exact source into your own repository and never updating anything — trades a solvable, visible problem for an invisible one: security patches you’ll never receive and bugs you’ll never know were already fixed upstream. The actual fix is respecting what package managers are doing rather than avoiding them: commit the lockfile, review dependency updates the same way you’d review any other code change rather than rubber-stamping automated updates from a bot, and accept that occasionally the constraint solver is going to hand you a genuine, unsolvable diamond that only a human decision — drop a library, fork it, or wait for an upstream fix — can actually resolve. The tool isn’t ruining your day out of malice. It’s reporting, accurately, that the day was already unsolvable before you ran the command.
The one habit that pays for itself repeatedly is separating “can I upgrade this” from “should I upgrade this right now.” Just because a resolver can find a satisfying combination of newer versions doesn’t mean every one of those newer versions has been exercised against your actual code paths — a successful resolution is a statement about compatible version ranges, not a statement about tested behaviour. Treat a green install as permission to run your test suite, not as the test itself, and the diamonds that do slip through tend to surface in CI rather than in front of a customer.
Related Content
