2022.06 Vol.1

Oh JavaScript

Our monorepo has two major directories. One for our backend services written in Go and one for our frontend clients written in TypeScript. For the past month, I have been putting a lot of hours in to trying to raise the quality of the CI/CD process of the TypeScript to that of Go.

If you have worked with Go, it shouldn’t be too stunning that the our CI/CD process for our backend services is solid. Go comes with a pre-built and highly opinionated toolchain which covers many of the responsibilities a CI/CD system attempts to enforce: code linting, running tests, and consistent builds (dependencies, executables). go fmt handles linting. go test is a sufficient test runner. And while Go’s new module system is still ironing out the kinks as of Go 1.18 (like this annoying go install limitation to install executables which use replace statements) it is able to get the job done and is fairly straight forward. It also has great cross platform compilation builtin. All of these tools also work great in a monorepo. The stability allows other tools (like IDE’s) to easily build on top and provide value for developers. It’s just really great.

So about that TypeScript. First up, TypeScript is still relatively new in the sense that its having to build on the existing JavaScript ecosystem. Since its not greenfield its going to move slowly, dragging the baggage of previous design mistakes. This means the toolchain is still JavaScript focused and is slowly being ported to take advantage of the benefits of static typing. And given JavaScripts history, it shouldn’t be too stunning that the equivalent of Go’s toolchain is a disjoint hodge-podge of tools clobbered together, sometimes playing nice. There is the extra challenge that many of the tools were not designed with monorepos in mind.

Our TypeScript tooling isn’t at Go’s level yet, but its a little better than when we first started.

Dependencies

We roll with yarn for our dependency management. As with all things JavaScript, there are many tools to accomplish this (e.g. npm, learna) and its hard to say which one is the best. But yarn has worked for us and is probably the least painful aspect of our tooling. yarn keeps our 3rd party dependency management maintainable, locking things in for builds and guaranteeing what you would expect. yarn’s workspace feature makes supporting monorepo development possible. And while some things are left very open ended (like script naming convention), it gets the job done.

Testing

There are many test runners out there. We are using jest which appears to be the most popular, but who is to say. This is where we gotta sink in to the disjoint issues in the toolchain. So jest doesn’t understand TypeScript, but it does have support for compiling TypeScript and then running tests. This support is jest specific and requires its own configuration. Compare this to Go’s toolchain, where each step (e.g. fmt, test, build) can leverage to work done in other steps like compilation. It doesn’t matter if we just linted our TypeScript codebase and did a lot of compilation work, jest is gonna do it all over again cause there is no way (currently) to share that info with it. This obviously leads to slow performance and a maintenance burden.

jest has a “multi-project” feature which is sold as monorepo friendly and sounds cool on paper, like offering test coverage aggregation across projects. However, its really a much more abstract feature that allows for different configurations of any kind (e.g. different code, different runners, different settings) to be run concurrently. Its actually really difficult to share configuration between these “projects”. So our monorepo contains ~20 packages and would require the same number of hand configured jest configurations. Not maintainable or robust.

Maybe there is another runner out there which would better support our monorepo and TypeScript needs.

Linting

Linting in TypeScript and JavaScript is a big deal. In Go, a lot of issues are sidestepped by the static typing and limited interface scope (its a simple language). That leaves the go fmt tool to just cover stuff like “how many spaces here”, which is still valuable cause developers never talk about it now, but very small scope compared to its TypeScript equivalent.

We use ESLint for our linting and it covers a lot of ground. It attempts to reign in the chaos of JavaScript, giving developers warnings when they are doing something they probably shouldn’t. ESLint is highly configurable and projects can inject all sorts of logic into it. From what I have seen, it is pretty much an essential tool for developer productivity.

Ideally, a lint process can be constantly giving a developer feedback. If you type some Go code in an IDE it will almost instantly give you feedback if you misstep. ESLint has similar integrations with IDEs, however, it is not nearly as performant as the Go equivalent. This became obvious quickly even in our modestly sized monorepo. ESLint processes were sucking up over eight gigs of memory. The information it reports is incredibly valuable, but we had to do some tuning to make it more actionable. We ended up adding an .eslintrc configuration per package. This keeps the memory usage down. And luckily, unlike jest, the configuration is almost 100% shareable so its easy to maintain across the repository.

One other pain point with ESLint is TypeScript dependencies. ESLint is able to use type information to provide more valuable feedback, it is not integrated into the TypeScript build process though. What this means is that in order to get the feedback, the linter is dependent on an out of band tool (like tsc) to compile internal dependencies. This is another issue of the disjoint toolchain which I took for granted not dealing with in Go.

Bundling

As I mentioned up top, Go has great cross platform compilation support. Its easy to target different platforms and trust that things will “just work”. We use it all the time internally building tools which run on linux machines as well as macos boxes.

This is arguably a more complicated problem in the JavaScript world. Instead of targeting a handful of platforms, JavaScript programs need to target totally different runtimes (browser vs. node) and an endless number of different degrees of support in different browsers. To tackle this, yet another complicated tool is added to the toolchain. These bundlers usually transpile code to match a target’s requirements.

We are currently using webpack to get the job done, but are looking in to switching to the more performant esbuild. No matter which we end up with, as with the testing and linting tools, these bundlers run in isolation from the other tools. Often re-doing work in their own special way. There doesn’t seem to be much work into changing this anytime soon. But other than webpack being slow as shit, it hasn’t caused much issues. As of today, the most developer productivity pain comes from the linting and testing tools. Hopefully it will change some day.