2021.02 Vol.1

I fight for the user.

I recently had to update the ol’ resume since I am finally leaving Fitbit. I thought it turned out to be a pretty good story, so I’ll capture it here too since the resume will change with time.

There is just one section under my experience header on the resume: Fitbit. I joined fresh out of school back in 2011 and had a crazy experience which ended with Google acquiring Fitbit in 2021. I never wanted to work for a company the size of Google and decided it was time to go, but let’s go back to the beginning.

On my first day, my first task was to get a keyboard from the Radio Shack down the street (you can tell that time has passed). After that my manager drew up a picture of the current backend architecture at Fitbit. It was composed of a handful of tomcat webapp servers and some MySQL databases. Maybe 10 boxes total on the whiteboard.

For the first few years at Fitbit I was on the only backend engineering team. We wore a lot of hats as we dealt with the incredible scaling challenges of a successful product. Boxes were added to the whiteboard, new technologies were introduced, corners were cut, lessons were learned, technologies were removed. Repositories were created, repositories were consolidated (monorepo!), kafka and micro-services were introduced (sad to report that as of 2021 the original “monolith” java app was still running despite many heroic efforts), NoSQL (Cassandra…:shakes-fist:) found its way on to the whiteboard. I think I personally only took down production once when I removed an index on the friend invitation table (whoops).

During the middle years of my time at Fitbit, I held IC roles on the initial SRE and Developer Productivity teams which were created out of necessity. SRE was not really my thing, but I did develop a soft spot for dev/prod. I took that with me when I joined the Data Platform team which I stayed on until the Google acquisition. It was on Data Platform that I found my passion for simplifying tools and interfaces.

Fitbit was in the unique position where it had spent almost a decade collecting data and doing little more than presenting it back to the user (e.g. you walked x steps today!). All the data storage and encodings were optimized for these user point queries.

Another challenge was the Big Data tooling at the time was (and for the most part, still is) geared towards serving ads, a domain where it is OK to get things wrong once or twice as long it works in general. Fitbit researchers intended to analyze the Fitbit data and serve health recommendations, a domain where you cannot be wrong.

Lastly, initial algorithm creation attempts at Fitbit were painfully slow. Researchers developed algorithms in python on their laptops. They would hand these off to a backend feature teams who translated them into java and deployed them to production. If there was a bug (there is always a bug) the process started over. This would take months.

The Data Platform team’s mission was to “unlock” this existing data and enable our researchers to quickly develop quality models and algorithms. Over four years, three data warehouses, two datalakes, and many smaller iterations, we delivered.

I believe the team’s greatest accomplishment was a platform which enabled our researchers to create an FDA approved atrial fibrillation detection algorithm (and the potential to create more). Data Platform designed and delivered tools which had our researchers (who desperately wanted to sling python notebooks on their laptops) writing high quality scala code, which they deployed to production with a CI/CD process. The iteration time for algorithm development was measured in minutes instead of months.

How did we do that? Not sure exactly, but I do remember one whiteboarding session early on where we drew up everything required to develop, deploy, and maintain an algorithm. There were a lot of boxes. We set out to find how we could hide them from the researchers. The answer was not immediately obvious, and this explanation makes it sound easier than it was, but by shifting things around we designed a “harness” for our researchers code to live. This wasn’t revolutionary computer science, but the simplicity sure was satisfying.

My thoughts on my years at Fitbit can be summed up in small story from 2020. When Covid hit, Pria and I decided that was the sign for us to move back to San Diego. We bought a house one freeway exit away from where my Grandma lived, which was nice because her health had been deteriorating and this made it easy to see her more often. One night after she had dinner with us, I was driving her home and telling her about how our afib algorithm development was coming along. When I dropped her off she told me in her thick Dutch accent “I should wear the Fitbit”. Less than a month later she passed away from a stroke. This isn’t a story about how I wished we had shipped the algorithm sooner, just that I know for sure that the work will help someone out there in a big way.