highlight.js: install 9.15.1 fail

incident report

What happened?

About a week ago we merged #1951, which made changes to the tools/build.js utility by excluding certain files (docs, demos, and other things). The PR’s intention was to have a way to build a highlight bundle from the command line, but to not build the docs and demos that are not needed on CI environments.

Although we’d been thoroughly testing this binary in production in another project for over a year, we’d been installing highlight.js over a github: URL. Unknown to me, the highlight.js npm publish script only included certain files: The script is on an ex-maintainer’s private server, so I’d incorrectly assumed that it was publishing the repository as is. This assumption was wrong and led to things breaking.

When 9.15.1 release went live, it was missing tools/build.js and possibly other things causing depedents to break.

What we did in response

At first, I thought the problem was that the bin path was maybe wrong. However, thanks to helpful https://github.com/highlightjs/highlight.js/issues/1984#issuecomment-466938694 from @netanelgilad, it was pointed out that the problem was with how the package was being published: it was missing directories/files.

As a result, I reverted #1951. However, this was insufficient because we’d changed our release script to work with the new tools/build.js. We then needed to revert some changes to the release script.

After a few bad releases, we then successfully published 9.15.5.

What we learned

Keeping in mind that the maintainers are new to the project:

• the published package does not reflect what’s in the repository.

What could we do to prevent this from happening again

We need to figure out a way to pre-publish (or publish as “beta”) a package, and then try to install the package in a mock project to make sure things still work as expected.

If folks have suggestion for how to do that - or what a good setup might be to test what’s actually published before it goes live, I’d be open to suggestions.

I would like if you revert the change. It is currently blocking a lot of build from other people

On a personal note, let me take this as an opportunity to explain something about the current sorry state of relationship between businesses and open source projects. (Yeah, I know, but people still don’t get it.)

highlight.js is not a business, it’s a hobby.

It means that whatever gets pushed to this repository or npm should be assumed to be the result of someone having fooled around and gone away for a weekend with their family. Or for a busy working day at their job.

If a business has made a decision to rely on this artifact for anything requiring any sort of stability (i.e. “blocking a lot of build from other people”), it made a stupid and uninformed decision. Or more realistically, it simply relies on maintainers feeling ashamed enough to quickly fix problems when they happen. Even more realistically, it just accept the fact that their engineers are going to deal with maintainers by soliciting free support, because it has always worked this way. I, for one, don’t feel any urge at all supporting someone’s misplaced expectations 😃

So, dear fellow engineers, please take this build hiccup as an opportunity to explain your particular business people that their entire intellectual property is a thin layer on top of a shaky foundation of open-source code lazily maintained by hobbyists or paid for by other businesses having their own goals in mind. Mention the leftpad story for more effect. So if they really want stability they have to invest in it. By, for example, hiring engineers to deal with myriad of dependencies, maintain local stable forks, contribute patches upstream, or whatever — the key point is that it should not look like it “just works” on fairy dust.

@isagalaev is right. We maintainers do the best we can, and it’s definitely not part of our day job. There are a few solutions people can use to protect themselves from these kinds of events:

• lock dependency to a particular version number.
• rely on a service like https://greenkeeper.io - which will check that an updates doesn’t break your build.
• always test before updating, don’t blindly just go for “latest”.

Ok, seems we are good… closing.

@marcoscaceres I would like if you revert the change. It is currently blocking a lot of build from other people 😢

Ok, testing fix on Travis right now: https://github.com/highlightjs/highlight.js/pull/1985/files#diff-b9cfc7f2cdf78a7f4b91a753d10865a2R25

Apologies, will fix in ~2 hours unless someone wants to put together a pull request quickly.

It cause by https://github.com/highlightjs/highlight.js/commit/ee2ae804e6a39864c0284280bd827d8e78ac81bf#diff-b9cfc7f2cdf78a7f4b91a753d10865a2R25

@ljharb, that’s a good suggestion. I’d also like to write a better check on TravisCI. I wasn’t expecting the package to be stripped of some of its contents.

I’ll write an incident report and publish it here describing what happened and what we will try to improve to prevent this from happening again.

Again, this is a call for the community to help us (we literally run this project on a skeleton crew… putting in maybe 1 hour a month to keeping it running). I’m definitely not an expert on NPM - so help from folks with more experience would be greatly appreciated to keep this project healthy.

@marcoscaceres Seems like this is still not going to work, and installing from a branch is not a good test. The problem is that the tools folder is missing from the published package to npm (check https://unpkg.com/highlight.js@9.15.1/) but it’s needed by bin in the package.json. When installing a branch from github, you get all the files in the repo, and you get the tools folder and that’s why it works. When installing from npm it doesn’t.

I think you need to change in your build/publish scripts to add the tools folder.

And thanks for the quick responses 😃

@marcoscaceres Our build system is all green again - got the latest 9.15.5 version as npm lists that version now too. @isagalaev & @marcoscaceres Thanks for your efforts, appreciated very much 😃

@netanelgilad, yes, you are probably right. Having a look if there is some quick way to do that. Otherwise, I’ll revert the offending commit.

+1

+1

+1

If folks have suggestion for how to do that - or what a good setup might be to test what’s actually published before it goes live, I’d be open to suggestions.

There are a few ways to go about this -

• Publish each release, potentially each time master builds, with a prerelease version indicated by a dash at the end: x.x.x-0 these can be individually tested just like regular versions, but users won’t pick them up by default, and it’s clear that they may have potential bugs. When it’s ready to release, test the latest prerelease and if it works, trim the prerelease part from the version and do a release.
• Another approach (which could be done alongside the latter) is to make use of npm tags. When you release a new version, tag with “next” rather than “latest” and test it. If it works, only then change the tag to “latest”. That way even if 9.16.0 is released after 9.15.5, as long as it’s marked “next”, people depending on “^9.0.0” won’t pick it up.

+1

+1

+1

+1

+1

+1

Could you change latest tag in npm to fix this problem quickly?

+1

I also met this issue yesterday, thanks for resolving it

Huh – npm i highlight.js@9.15.2 doesn’t work for me, same error – and npm hasn’t updated the version (yet?)

Edit: But npm i highlight.js@9.15.3 or any other future version gives a different error – so a release is available, just not tagged on npmjs.com and it’s broken 😦

npx hljs 

I have tried it, It works for me.

+1

it cause error in npm install.

I can tell you it is working right now. I have a project that depends of typedoc, hence this project.

Being honest, you give me a lot of headaches today. For me today was a Murphy moment that ended with this incident today.

I’m not blaming you. This is an open source project and we have to be aware this things are going to happen.

And personally, I PREFER, 100x times, an open source project like this one that a private one with no documents.

success+1

@loooSwc @sushantkarki @yurytolochko He is doing his best. The build finished a minute ago https://travis-ci.org/highlightjs/highlight.js/builds/498080876

Yep, reverting ee2ae804e6a39864c0284280bd827d8e78ac81bf now.

One suggestion that might make it easier for maintainers would be to use a prepublish script, rather than having a separate release script - that way, npm publish can’t proceed until that script passes.

+1

Ok, 9.15.2 released… hopefully good now 😅

Release in progress… https://highlightjs.org/api/release/

Ok, can a few folks try:

npm install github:highlightjs/highlight.js#fix_build
npx hljs 

Let me know if that works…

It works! 👍

Waiting for a fix…

Also a work around for dependent packages is to install 9.15.0 as a peer dependency via npm i highlight.js@9.15.0 -D

For those in a hurry (Like I was). You can download the repo and install it locally with npm and it works fine

@marcoscaceres My system can accomplish npm install now. The installed highlight.js version is 9.15.5. Thanks for you fix!

Folks, I’ve published 9.15.5… any better? 🤞

Starting to work on the fix. I’m muting the bug, as it’s getting spammy. If you’d like to subscribe to the bug, to be notified when it’s fixed, please use the subscribe button.