The npm Debacle Was Partly Your Fault (and here’s what you can do about it)

“You muppets. Do you have any idea what you’ve done?” [1]

On February 27th, the Node.js community lit up in anger. Deployments were failing around the globe, in some cases to devastating effect.

“this just fucked my launch. I was first page of Hacker News and then BAM the whole site crashed. FML” [2]

It was a major screw up. One that can be laid firmly at the feet of npm, Inc... and you.

What Happened

“npm,” if you’re not familiar with it, is the package manager people use to manage their Node.js dependencies. The npm client uses SSL for secure communication with the npm registry. For a long time, the npm maintainers used their own, non-standard certificate authority (CA) to sign the certificates the registry used. Support for this non-standard authority was hard-coded into the npm client. By default, it’s the only authority the client trusted.

After the npm maintainers founded npm, Inc., they updated their servers to use a standard certificate signed by DigiCert. Presumably they thought of this as a routine task—just part of establishing their new business. But npm clients were hardcoded to only trust the non-standard CA. The new DigiCert certificate was seen as fraudulent, causing npm to error out whenever it tried to talk to the npm registry.

Annoying, but not world-shaking, right? Right, except for one thing: almost everyone runs npm install when they deploy. And npm install talks to the npm registry, so it failed... along with everyone’s deployments.

Why You’re to Blame

Okay, maybe not you you. But for anyone who was affected by this issue, then yes, it was partly their fault.

Buried away in the npm FAQ is this bit of advice:

Use npm to manage dependencies in your dev environment, but not in your deployment scripts.

Some vendors, such as Heroku, misappropriate npm’s package.json file—which is meant for npm packages—and use it for deployment of production apps instead. They run npm install to automatically download your dependencies when you deploy. It’s mildly convenient, kinda cool, and it puts you at the mercy of all sorts of issues.

1. You’re reliant on the npm infrastructure being up and working when you need it (it won’t always be, as was so clearly demonstrated on Feburary 27th... and in January... and November... and etc...).

2. You’re assuming nobody will subvert the repository and insert malicious code (not as far-fetched as it sounds—it happened to Ruby).

3. You’re relying on npm to serve the exact bug-for-bug compatible version of the code that you originally developed and tested against, including all subdependencies (which it doesn’t, not without special effort).

4. You’re trusting that the online repository will remain available and serving the exact packages you need for the rest of your application’s life, which could be decades.

Unfortunately, it seems that the majority of the Node community chose the expedience of deploying from package.json over the rigor advised by npm. And here we are.

“I can’t believe how shoddily this happened. My QA team is sitting around being paid to do nothing because I have completely screwed their environment, and I cannot deploy a new version... What a debacle.” [3]

When the excrement is flying—and eventually, it always does—your code needs to keep working. The buck stops with you. Should have npm, Inc., exercised more care in updating their certificate? Absolutely. Are you ultimately responsible for your code? Most definitely.

What You Can Do About It

The good news is that this problem is trivially easy to fix.

Don’t use npm for deployment.

Tools like npm are development tools. They’re a convenient way to download and update modules. They’re not deployment tools, have never been deployment tools, and should not be used for deployment!

You have two options.

Option 1: Check your npm modules into your repository.

Really.

Your repository is an ongoing history of everything you’ve done. You should be able to go back to any commit, at any point in history, and see your software work exactly as it did then. If it doesn’t, you’ve effectively lost that code.

Sure, you still have ASCII in a bunch of files, but you no longer have the ability to make it run. And, as anybody who’s been in this situation knows, it can be a major effort, or even impossible, to get it working again. (Just think about how hard it can be to get a development environment up and running when you start a new job. Now imagine going through that pain without anybody around to help.)

The easiest way by far to keep your code working is to store everything in your repository. You should be able to check out an old version, disconnect from the network, and see your build work perfectly. This isn’t always practical for big-ticket items, such as your OS or Node itself, but it’s completely feasible for npm modules.

(At this point, some people get concerned about bloating the size of their repository. Don’t worry about it. First, disk space is cheap. Second, you’re not going to be checking in binaries, as I’ll explain in a moment. Honestly, if this is an issue for you, you should rethink your version control strategy.)

How to do it

In theory, checking in your npm modules should be as easy as committing your node_modules directory. Unfortunately, modules put their build artifacts in node_modules, too. You have to tell your version control software to ignore those artifacts.

Here’s how. (This recipe assumes your dependencies are already in package.json.)

1. Make sure you’ve checked in any recent changes, then delete your node_modules directory (so you’re starting fresh) and remove it from your .gitignore file or equivalent (so you can check it in).

2. Run these commands from your project root (replace the git commands if you’re using a different tool):

   npm install --ignore-scripts    # Download modules without building them
   git add . && git commit -a      # Check in modules
   npm rebuild                     # Build the modules
   git status                      # Show the files created by the build (there might not be any)
   

3. Add the new files to .gitignore (or equivalent) and check it in.

The same approach also works when installing or updating a new dependency, except you don’t delete your node_modules directory.

Option 2: Check your releases into a deployment repository.

If you can’t check your npm modules into your regular repository, you can set up a second repository just for deployment. Then, when you’re ready to release:

1. Run npm shrinkwrap in the development repository and check in the result. This will lock your module versions in place, including sub-dependencies.

2. Copy the development files to the deployment repository, run npm install, and check in the result. You might want to tag it or something, too.

3. To deploy, check out the version you want and copy the files to your production machine(s), then run npm rebuild on the production machine.

This approach isn’t as good as option 1. Its main advantage is that you don’t have to futz around with finding and ignoring build artifacts. It’s less effective overall because it doesn’t protect you against the npm registry going down or getting corrupted. It does ensure that you can roll back to previous releases if you need to, though, and that’s valuable.

One caveat

Some modules are particularly weird and run npm install as part of their post-install build script. These will screw up whichever approach you use. The best solution is to open an issue, fork the module, and fix your copy.

If you use option 1 (checking modules into your development repository), you can identify these “weird” modules because dependencies in their node_modules subdirectory (or the whole node_modules subdirectory itself) will show up as files to ignore.

It’s Your Responsibility

Vendor screwups happen. That’s guaranteed. What’s not guaranteed is how they affect you. Will you take the expedient way, and make yourself vulnerable to every little hiccup? Or will you spend a bit more time, be a little more rigorous, and armor yourself against these problems?

For npm, the answer is clear. It’s not a deployment tool and shouldn’t be used as one. Checking dependencies into source control isn’t just the right thing to do, it’s easy. Save yourself from the next debacle and do it now.