andycarrell

Blog

Publishing an NPM package on GitHub

This is a guide to publishing and consuming a node package / library to the GitHub package registry, specifically for NPM. The combination of GitHub packages and actions makes sharing code between repositories easier than ever before. I encourage you to have a good read of both sets of documentation— all the information you need is in the official documentation, or associated forums. This post distills down that information, focuses in on NPM, and outlines the process that's working for the frontend team at Jasper.

Overview

The output of this blog post should be a working GitHub action. As we progress through the post we will:

Create a GitHub action that builds and publishes your library on push to your main branch.
Ensure the action only runs if the library files have changed.
Extend the action to perform a "dry run" of publishing the package, to confirm everything will run as expected before you commit.
Discuss how to access the GitHub NPM registry locally and in continuous integration.

Mention some of the stumbling blocks we encountered along the way.

The complete GitHub action code is as follows. Read on for a detailed explanation of what we've done and why.

1name: Library build & publish
2on:
3  pull_request:
4    paths:
5      - "library/**"
6  push:
7    branches:
8      - main
9    paths:
10      - "library/**"
11
12jobs:
13  build:
14    name: Build & publish
15    runs-on: ubuntu-latest
16    steps:
17      - name: Checkout code
18        uses: actions/checkout@v1
19      - name: Authenticate GitHub package registry
20        run: echo '//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}' > ~/.npmrc
21      - name: Set short sha as environment variable
22        run: echo ::set-env name=sha_short::$(git rev-parse --short=7 ${{ github.sha }})
23      - name: Setup node
24        uses: actions/setup-node@v1
25      - name: Install
26        run: npm install
27      - name: Verify
28        run: npm run lint && npm run test
29      - name: Build
30        run: npm run build-library -- --version-suffix ${{ env.sha_short }}
31      - name: Publish - dry run
32        run: npm publish output -- --dry-run
33      - name: Publish
34        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
35        run: npm publish output

Assumptions

This blog post assumes:

You're using GitHub!
GitHub packages is enabled for your respository.
You're already bundling / building your library code.

GitHub packages

Check GitHub packages are available for your repository. From what I can tell, if you navigate to https://github.com/<user-name>/<repo-name>/packages and you see the following screen, you're good to go:

If not, you may have to dive into your plan and even reach out to GitHub for help. Some legacy plans may be excluded from using packages. On that note, if you're developing a package in a private repository double check prices and usage for your current plan, otherwise if public, at the time of writing it's all free!

Library build

I assume you're already able to bundle / build your library code into a form that you're happy to publish. How we configure and test our library warrants a blog post itself, so for demonstration purposes I'll reference a custom NPM script:

npm run build-library

Our build uses Rollup and outputs to output/*. I recommend wrapping your build process in a custom script and referencing it in GitHub actions in a similar way.

You'll also need to update your library's package.json to include the following configuration:

1{
2  // ...
3  publishConfig: { registry: "https://npm.pkg.github.com/" },
4  repository: {
5    type: "git",
6    url: "ssh://git@github.com:<user-name>/<library-name>.git",
7    directory: "output"
8  }
9  // ...
10}

Create a GitHub action

If you're familiar with GitHub actions, or you understand what's going on in the example code then you can probably skip the following. More information on the syntax mentioned can be found in the documentation.

We start by defining a YAML file in the GitHub workflows directory .github/workflows From the docs:

You must store workflow files in the .github/workflows directory of your repository.

The name of the file doesn't matter, we'll call it something like publish-library.yml. Similarly, the name of the action isn't too significant, although it does appear in the GitHub actions UI, so an identifiable name will help:

1name: Library build & publish
2on:
3  push:
4    branches:
5      # or your 'default' branch
6      - main
7    paths:
8      - "library/**"

The "on" configuration defines which events will trigger your workflow. We further refine the conditions in which workflow is triggered— in this example, any git push, to the main branch, where file(s) in the library or directory have changed.

A GitHub action needs at least one "job" (with steps) to run anything. Again, the job id and name aren't so important, but will identify the job in the GitHub UI:

1# ...
2# on: ...
3
4jobs:
5  # job 'id'
6  build:
7    name: Build & publish
8    runs-on: ubuntu-latest
9    steps:
10      - name: Checkout code
11        uses: actions/checkout@v1

If we want to run an action with or against our code we need to check it out first. Whilst GitHub doesn't do that by default, we can simply reference the checkout action.

This action checks-out your repository under $GITHUB_WORKSPACE, so your workflow can access it.

Also note the runs-on property— this is required and specifies the type of machine to run the job on. There's a few options, but for a node / npm based action ubuntu-latest will work fine.

To publish a package, we need to authenticate our action for the GitHub NPM registry. The official documentation recommends a couple of ways of doing this, neither of which were practical for use in an continuous integration (CI) context. The solution we found involves appending the GITHUB_TOKEN to the .npmrc file after checkout, which avoids having to commit sensitive tokens. We can acheive this using bash syntax to overwrite a file:

echo '...' > ~/.npmrc

We run this immediately after checking out our code:

1# ...
2# on: ...
3
4jobs:
5  build:
6    # name: ...
7    steps:
8      - name: Checkout code
9        uses: actions/checkout@v1
10      - name: Authenticate GitHub package registry
11        run: echo '//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}' > ~/.npmrc

For reference, the recommended approaches for authenticating are:

npm login— requires CLI input so won't work in CI.
Editing your per-user ~/.npmrc file— requires committing sensitive auth tokens in the project's ~/.npmrc file.

To run node in our GitHub action, we need to set it up— there's an action for that too. Now we can install, run verification scripts, build the library and finally publish!

1name: Library build & publish
2on:
3  push:
4    branches:
5      - main
6    paths:
7      - "library/**"
8
9jobs:
10  build:
11    name: Build & publish
12    runs-on: ubuntu-latest
13    steps:
14      - name: Checkout code
15        uses: actions/checkout@v1
16      - name: Authenticate GitHub package registry
17        run: echo '//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}' > ~/.npmrc
18        # official action to setup node
19      - name: Setup node
20        uses: actions/setup-node@v1
21      - name: Install
22        run: npm install
23      - name: Verify
24        # these are custom scripts to run eslint and tests
25        run: npm run lint && npm run test
26      - name: Build
27        # we use rollup, so our script is 'rollup -c rollup.config.js'
28        run: npm run build-library
29      - name: Publish
30        run: npm publish output

At this point you should be publishing every time you make changes and push to main— provided you update the library version with each change. Carefully consider if this is an appropriate publishing frequency for your library.

Our motivation for publishing in this way is to minimise the time (and effort) taken to make a change and consume it in another project. The cost of this is we have many versions each containing small changes, which are also often undocumented. This doesn't make for a good consumer experience. However, in this case it's a library for internal use only, and generally developers making the changes are also consuming them. It's also something we can definitely iterate on.

To improve the developer experience of contributing to our library— we added the following features:

Early feedback— if a developer is making a change, we can let them know if the library will publish successfully before committing to main.
Automated unique versioning— for a given change, we assume the developer wants a new version released, without the chore of releasing it themselves.

Pull request dry run

We can update our GitHub action so that on every pull request we run all the steps, except for actually publishing a new version.

First, we need to trigger the action on pull request as well as push, then we need to only run the publish step when a push (to main) event happens:

1name: Library build & publish
2on:
3  pull_request:
4    paths:
5      - "library/**"
6  # push: ...
7
8jobs:
9  build:
10    # name: ...
11    steps:
12      # Checkout, authenticate & setup node ...
13      # Install, verify & build ...
14      - name: Publish
15        # only run this step on commit to main
16        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
17        run: npm publish output

Finally, on pull requests we can perform a "dry run" of publishing:

[--dry-run] As of npm@6, does everything publish would do except actually publishing to the registry.

This is useful as it shows whether the change will actually build and publish successfully.

Our action with pull request feedback:

1name: Library build & publish
2on:
3  pull_request:
4    paths:
5      - "library/**"
6  push:
7    branches:
8      - main
9    paths:
10      - "library/**"
11
12jobs:
13  build:
14    name: Build & publish
15    runs-on: ubuntu-latest
16    steps:
17      - name: Checkout code
18        uses: actions/checkout@v1
19      - name: Authenticate GitHub package registry
20        run: echo '//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}' > ~/.npmrc
21      - name: Setup node
22        uses: actions/setup-node@v1
23      - name: Install
24        run: npm install
25      - name: Verify
26        run: npm run lint && npm run test
27      - name: Build
28        run: npm run build-library
29      - name: Publish - dry run
30        # this step runs every time, but only takes a few seconds
31        run: npm publish output -- --dry-run
32      - name: Publish
33        # only run this step on commit to main
34        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
35        run: npm publish output

For even earlier feedback, take a look at nektos/act. It allows you to run actions locally— be careful you don't accidentally publish a new version!

Versioning

As the action stands, the publish step will fail unless the developer remembers to update the version with each change.

Once a package is published with a given name and version, that specific name and version combination can never be used again, even if it is removed with npm-unpublish.

Additionally, if multiple developers are working on the library at the same time, they'll need to coordinate to ensure that they don't land on the same new version, or that a later version lands before an earlier one.

These issues highlight a manual process that could be improved by automation— both the choice of version, and applying a new version with each change. To achieve this, we apply the constraint that each of our versions consists of a standard semantic version and a unqiue suffix, which is the (short) Git SHA.

"@<user-name>/<library>": "0.1.0-3b4c0a0"

Our build script takes a version suffix as a command line argument and appends that to a semantic version defined in code. Prior to that, we parse the first 7 digits of the Git SHA and assign it to an environment variable using:

echo ::set-env name=sha_short::$(git rev-parse --short=7 ${{ github.sha }})

So, the complete action including GitHub SHA versioning is as follows:

1# .github/workflows/publish-library.yml
2name: Library build & publish
3on:
4  pull_request:
5    paths:
6      - "library/**"
7  push:
8    branches:
9      # or your 'default' branch
10      - main
11    paths:
12      - "library/**"
13
14jobs:
15  build:
16    name: Build & publish
17    runs-on: ubuntu-latest
18    steps:
19      - name: Checkout code
20        uses: actions/checkout@v1
21      - name: Authenticate GitHub package registry
22        run: echo '//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}' > ~/.npmrc
23      - name: Set short sha as environment variable
24        run: echo ::set-env name=sha_short::$(git rev-parse --short=7 ${{ github.sha }})
25      - name: Setup node
26        uses: actions/setup-node@v1
27      - name: Install
28        run: npm install
29      - name: Verify
30        # these are custom scripts to run eslint and tests
31        run: npm run lint && npm run test
32      - name: Build
33        run: npm run build-library -- --version-suffix ${{ env.sha_short }}
34      - name: Publish - dry run
35        run: npm publish output -- --dry-run
36      - name: Publish
37        # only run this step on commit to main
38        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
39        run: npm publish output

If you're using Rollup and are extracting a library from a larger project, or need to manipulate the final package.json, I recommend the rollup plugin generate-package-json.
It allows you to overwrite the contents of the current file, as well as automatically populating dependencies that are used by your build.

Consuming your GitHub package

Consuming your NPM package requires adding the GitHub registry URL to your .npmrc:

1registry=https://npm.pkg.github.com/<user-name>

Find the latest version of your package under the packages tab of your projects's repository:

https://github.com/<user-name>/<repo-name>/packages

Add this version to the package.json of the project where you want to consume it.

If your project's repository is private, then the package will be too. To give yourself access to install locally, you'll need to authenticate an NPM registry user account.

First, create a new GitHub token with the read:packages scope. Adding write:packages and delete:packages will be helpful if you plan on managing packages from the command line. Keep the access token handy for the next step.

Next, enter the following commands into your terminal:

1# cd into your root folder
2cd ~
3
4# authenticate for the GitHub package registry
5npm adduser --registry=https://npm.pkg.github.com/ --scope=<user-name>
6
7# command line will prompt you for your details
8Username: <your github username>
9Password: <paste github access token>
10Email: (this IS public) <your github public email>

You should then have access to install and use your package locally.

To authenticate GitHub actions in other repositories, you'll need to authenticate in the same way as we did in our publish action, after we checkout, and before we install:

1# ...
2jobs:
3  another-action:
4    # name: ...
5    steps:
6      - name: Checkout code
7        uses: actions/checkout@v1
8      - name: Authenticate GitHub package registry
9        run: echo '//npm.pkg.github.com/:_authToken=${{ secrets.NPM_TOKEN }}' > ~/.npmrc
10      # Setup node ...
11      # Install, verify ...

Note that we use a different secret key (here called NPM_TOKEN) because:

GITHUB_TOKEN cannot install packages from any private repository besides the repository where the action runs.

You can create another token or use the same one you created for local usage. Storing secrets against the repo (https://github.com/<user-name>/<repo-name>/settings/secrets), makes them available to GitHub actions.

Private packages

If you publish a private package (which happens by default for private repositories), then you may run into the following issues.

NPM audit

npm audit is a useful tool that scans your project for vulnerabilities, but it will fail with the following error:

Your configured registry (https://npm.pkg.github.com/<user-name>) does not support audit requests.

Dependabot automated dependency updates

Similarly, at the time of writing, Dependabot doesn't support updating dependency files that use private package registries. There is an outstanding GitHub community issue for this issue.

Conclusion

Hopefully this clarifies some of the process required for GitHub actions and the NPM package registry, and in doing so enables other teams to use this functionality too. If I can explain the process further, or clarify why we did what we did please reach out. I'm open to feedback, and if you have a completely different way to do what we're doing, I'd love to hear that too.