2022.11 Vol.1

Developing In JavaScript

Been struggling again recently with the TypeScript/JavaScript build tools ecosystem. Can never take for granted how much the go command handles in go-land. But I think I have found a happy combo of tools for now.

I work in a monorepo environment and would like to optimize the developer experience working in JavaScript. While the goal is to get something close to a go-based monorepo, it’s helpful to keep in mind that the tooling in the JavaScript and TypeScript ecosystem is almost always package-based, not repository like go tooling. So the big challenge for JavaScript is selecting a set of tools to accomplish staple developer needs (formatting, linting, testing) while not having each tool re-build the “state of the repository” every time.

This is how we accomplished that at Mash. tldr; The mash-js monorepo is a clean implementation of the following. TypeScript and ESLint LSPs play very well with the setup making it pretty enjoyable to develop in.

A simple eight step plan.

1. Use Yarn v2+ (which is confusingly also called “berry”) to manage dependencies across the repository.

A Yarn workspace is 1:1 with a JavaScript package and provides a managed environment to develop that package. While its possible to declare 3rd party at the root of a repository and share they them with the packages in a repository (similar to a go module), this is not the “yarn way” and fights the system. It is best to declare dependencies explicitly in each package. Yarn will do its best to only keep one version of a 3rd party dependency across all workspaces (packages). You can try to lock a version with a root-level overrides rule, but this pattern appears to cause more issues than it solves. We are trusting Yarn to minimize our dependency headaches and so far it is working out.

Yarn has special syntax and smarts for internal package dependencies. It is necessary to declare these dependencies for a package to build. Yarn can also use this information to optimize the build performance (e.g. package A and B depend on C, so I’ll build C first and share it with A and B…performance!). At Mash though we have decided this is not the tool for build optimizations, we instead leverage TypeScript project references described more below.

Note: npm can also be used instead of Yarn, its workspace interface is almost the same.

2. Lock the development version of node.

Whether or not the packages being developed are intended for a node environment, it is pretty impossible to not use node to lint, test, and build the code. To minimize surprises and allow assumptions to be made of the stack, the development version of node should be locked. This is specified in the engines key of the root-level package.json

Yarn v1 enforced the engines configuration by default. Strangely, v2+ (berry) requires a plugin. Its pretty minimal overhead though and definitely worth it. At Mash we use the devoto13/yarn-plugin-engines plugin.

3. Make all packages ESM modules only.

It has taken an extremely long time for JavaScript to define some sort of code loading system. ESM was established back in 2015. As of today, the browser environment is very ESM friendly while node is just getting there. Before ESM, the most common loader for node was its own CJS. From a 10,000 foot perspective, CJS is dynamic where ESM is static. Similar to the benefits of a statically-typed language, static ESM allows build tools to make assumptions and optimize performance and use.

There are varying levels of ESM usage, but the most gung-ho and clean is just setting the type key to module in a packages package.json. Mash is lucky enough to only be targeting the browser runtime at the moment, so its a pretty easy decision to be “ESM Only”. But even if a package is targeting the node runtime these days, it is still a valid call.

4. Make all packages TypeScript.

When optimizing developer productivity at scale, statically-typed languages are a necessity. TypeScript brings that to the JavaScript world. It also brings one more layer of complexity, but it is well worth it.

5. Use TypeScript project references.

TypeScript code is transpiled to JavaScript before being run or published. This can be an expensive operation since a package must also transpile its internal package dependencies. Luckily the TypeScript compiler, tsc, exposes an incremental build system. The catch is that tsc must be told what a package’s internal dependencies are and where they live on the file system. This is defined in a package’s tsconfig file with the references key.

6. Sync the internal dependency graph.

TypeScript project references is the layer which Mash is leveraging the most for build optimizations. Many other tools in the JavaScript ecosystem, including Yarn, attempt a similar strategy to optimize performance and also ask to be told the internal dependency graph. Sadly, there is no standard way to share this information in the ecosystem (something that happens automatically in go-land). We want to avoid asking developers to manually keep the internal dependency graph sync’d between X number of tools, this will just lead to confusing bugs as they drift. So when we choose a tool for our JavaScript, be it the test runner or dependency manager, it needs to either be able to leverage the TypeScript project reference definitions or automatically sync them.

It is pretty easy to sync the references from Yarn => TypeScript with a tool like @monorepo-utils/workspaces-to-typescript-project-references. So at Mash we define all internal dependencies in Yarn and have them synced to TypeScript. A CI step ensures everything is in sync.

7. Use ESLint.

ESLint is a powerful TypeScript/JavaScript linting tool. And in the wild west world of JavaScript, this is very important in order to ship quality code. We have found though that ESlint tooling doesn’t handle “repository”-scope very well. ESLint processes can quickly bog down leading to a bad developer experience since it is common for IDE’s to make many requests to the linter. To avoid bad performance, keep ESLint package-scoped by defining a .eslintrc file per package. Configuration can still easily be centralized in a repository by importing a shared file. It might even be worth looking in to defining the configuration as a package itself (tbd).

8. Use node’s builtin test runner.

Originally we were using the jest test runner. Jest brings a lot of functionality with it (e.g. mocking, snapshots, coverage reports), but it struggles with ESM (step #3) and leveraging TypeScript project references (step #6). Due to these deficiencies we switched to the node’s new, as of node v18, builtin test runner.

There are some key points to make this work which we can go through by breaking down this test script defined in each package.

"scripts": {
    ...
    "test": "tsc --build && c8 --all --reporter=text --reporter=lcov node --enable-source-maps --test dist/"
    ...
  }

all packages test script definition

TypeScript’s project references are leverage by calling tsc --build first. This also compiles a package’s code and tests. Tests are then ran on the compiled JavaScript with node --enable-source-maps --test dist/. It is a little confusing to run tests on the emitted code instead of the TypeScript itself because by default failed test reports will point to lines in the compiled JavaScript. A developer would then have to map that back to TypeScript in order to decipher and fix the issue. Luckily, source maps are very well supported these days. tsc is told to emit source with the sourceMap compiler setting and node is told to read them with the --enable-source-maps flag. Now test reports will print the original TypeScript lines. One side effect of this strategy is the need to filter these map files for published packages as they are of no use to consumers. This can be done with a "!dist/**/*.map" line added to a package’s files key in package.json.

An alternative to source maps is using an ESM loader like tsx to load a package’s TypeScript into node and transpile it on the fly. We would still have to first run tsc --build to build a package’s dependencies though, so tsx would be doing extra work transpiling the package’s TypeScript again. Source maps let us avoid the extra work.

c8 --all --reporter=text --reporter=lcov is simply providing test coverage reports based on the output. The --all flag ensures files which are not covered by tests are still included in the report (no loop holes here).

And there you have it. Easy.