Our tests used to be full of hundreds of random ids like 5233ac4c7220e9000000000d, 642c65770f6aa00887d97974, 53c450afac801fe5b8000019 that all had relationships to one another. 5233ac4c7220e9000000000d was a school, right? Was 642c65770f6aa00887d97974 a parent in that school? What if I needed a teacher and student in that school too—how did I find appropriate test entities? It wasn't an insurmountable problem—you could either poke around the CSVs that generated our fixtures or query the local database—but it slowed down writing and maintaining tests. Some engineers even had a few of these ids memorized; they'd say thing like "Oh, but 000d is parent4! Let's use 002f instead. They're in a school." or "I quite like 9797, it's a solid class for story-related tests."

To make navigating our fixture IDs and their relationships a bit simpler, I wrote a simple script to query our database and decorate names for these fixture ids (e.g., student2Id, school5Id) with the most common relationships for that entity. For a school, we show teachers, parents, students, and classes. For a parent, we show children, teachers, schools, classes, and message threads.

/**
 * - name: Student TwoParents
 * - **1 current classes**: classroom3Id
 * - **0 past classes**:
 * - **2 parents**: parent10Id, parent11Id
 * - **1 current teachers**: teacher1Id
 * - **schoolId**: none
 */
export const student5Id = "57875a885eb4ec6cb0184d68";

Being able to write a line like import { student5Id } from "../fixtureIds"; and then hover over it and see that if we wanted a parent for that student, we could use parent10Id, makes writing tests a bit more pleasant. The script to make generate this fixture-id file was pretty straightforward:

1. Get ordered lists of all of the entities in our system and assign them names like parent22Id or teacher13Id[^1]
2. Set up a map between an id like 57875a885eb4ec6cb0184d68 and student5Id.
3. For each entity type, write model queries to get all of the relationship IDs that we're interested in.
4. Use JS template strings to create nicely formatted JSDoc-decorated strings and write those strings to a file.

One small pain point I ran into was migrating all of our existing test code to reference these new fixture IDs. I wrote a script to find lines like const X = /[0-9a-f]{24}/;, delete those lines, update the variable name with the fixture-id name from the file, and then add an appropriate import statement to the top of the file. (Shell patterns for easy automated code migrations talks through patterns I use to do migrations like this one.)

After setting up these initial fixtureId JSDoc comments, we've added JSDoc comments for more and more of our collections; it's proven to be a useful tool. We also set up a complementary fixtureEntities file that exports the same information as TS documents so that it's straightforward to programmatically find appropriate entities. All in all, it's made our test code nicer to work with—I just wish we'd made the change sooner!

[^1]: Whenever we add any new IDs to our fixtures, we need to make sure that they come after the most recent fixtureID. Otherwise we'll end up with fixtureID conflicts!

When the ClassDojo engineering team was in the office, we loved our information radiators: we had multiple huge monitors showing broken jenkins builds, alerts, and important performance statistics. They worked amazingly well for helping us keep our CI/CD pipelines fast & unblocked, helped us keep the site up & fast, and helped us build an engineering culture that prioritized the things we showed on the info radiators. They worked well while the whole team was in the office, but when we went fully remote, our initial attempt of moving that same information into a slack channel failed completely, and we had to find a different way to get the same value.

Most teams have an #engineering-bots channel of some sort: it's a channel that quickly becomes full of alerts & broken builds, and that everyone quickly learns to ignore. For most of these things, knowing that something was broken isn't particularly interesting: we want to know what the current state of the world is, and that's impossible to glean from a slack channel (unless everyone on the team has inhuman discipline around claiming & updating these alerts).

We had, and still have, an #engineering-bots channel that has 100s of messages in it per day. As far as I know, every engineer on the team has that channel muted because the signal to noise ratio in it is far too low. This meant that we occasionally had alerts that we completely missed because they quickly scrolled out of view in the channel, and that we'd have important builds that'd stay broken for weeks. This made any fixes to builds expensive, allowed some small production issues to stay broken, and slowed down our teams.

After about a year of frustration, we decided that we needed to figure out a way to give people a way to set up in-home info-radiators. We had a few requirements for a remote-work info-radiator:

1. It needed to be configurable: teams needed a way to see only their broken builds & the alerts that they cared about. Most of the time, the info-radiator shouldn't show anything at all!
2. It needed be on an external display: not everyone had an office setup with enough monitor real-estate to support a page and keep it open
3. It needed to display broken builds from multiple Jenkins locations, broken builds from GitHub Actions, and triggered alerts from Datadog and Pagerduty on a single display

We set up a script that fetches data from Jenkins, Github Actions, Datadog, Pagerduty, and Prowler, transforms that data into an easily consumable JSON file, and finally uploads that file to S3. We then have a simple progressive web app that we installed on small, cheap Android displays that fetches that JSON file regularly, filters it for the builds that each person cares about, and renders them nicely.

These remote info-radiators have made it much simpler to stay on top of alerts & broken builds, and have sped us up as an engineering organization. There's been a lot written about how valuable info-radiators can be for a team, but I never appreciated their value until we didn't have them, and the work we put into making sure we had remote ones has already more than paid for itself.

ClassDojo occasionally has a few containers get into bad states that they're not able to recover from. This normally happens when a connection for a database gets into a bad state -- we've seen this with Redis, MySQL, MongoDB, and RabbitMQ connections. We do our best to fix these problems, but we also want to make it so that our containers have a chance of recovering on their own without manual intervention. We don't want to wake people up at night if we don't need to! Our main strategy to make that happen is having our containers decide whether they should try restarting themselves.

The algorithm we use for this is straightforward: every ten seconds, the container checks if it's seen an excessive number of errors. If it has, it tries to claim a token from our shutdown bucket. If it's able to claim a token, it starts reporting that it's down to our load balancer and container manager (in this case, nomad). Our container manager will take care of shutting down the container and bringing up a new one.

On every container, we keep a record of how many errors we've seen over the past minute. Here's a simplified version of what we're doing:

let recentErrorTimes: number[] = [];
function serverError(...args: things[]) {
  recentErrorTimes.push(Date.now());
}

export function getPastMinuteErrorCount () {
  return recentErrorTimes.count((t) => t >= Date.now() - 60_000);
}

Check out ERROR, WARN, and INFO aren't actionable logging levels for some more details on ClassDojo's approach to logging and counting errors.

After tracking our errors, we can then check whether we've seen an excessive number of errors on an interval. If we've seen an excessive number of errors we'll use a leaky token bucket to decide whether or not we should shut down. Having a leaky token bucket for deciding whether or not we should try to shut down the container is essential: if we don't have that, a widespread issue that's impacting all of our containers would cause ALL of our containers to shut down and we'd bring the entire site down. We only want to cull a container when we're sure that we're leaving enough other containers to handle the load. For us, that means we're comfortable letting up to 10 containers shut themselves down without any manual intervention. After that point, something is going wrong and we'll want an engineer in the loop.

let isUp = true;
const EXCESSIVE_ERROR_COUNT = 5;
const delay = (ms: Number) => new Promise((resolve) => setTimeout(resolve, ms));

export async function check () {
  if (!isUp) return;
  if (getPastMinuteErrorCount() >= EXCESSIVE_ERROR_COUNT && await canHaveShutdownToken()) {
    isUp = false;
    return;
  }

  await delay(10_000);
  check();
}

export function getIsUp () {
  return isUp;
}

At this point, we can use getIsUp to start reporting that we're down to our load balancer and to our container manager. We'll go through our regular graceful server shutdown logic and when our container manager brings up a new container, starting from scratch should make us likely to avoid whatever issue caused the problem in the first place.

router.get("/api/haproxy", () => {
  if (getIsUp()) return 200;
  return 400;
});

We use redis for our leaky token bucket. If something goes wrong with the connection to Redis, our culling algorithm won't work and we're OK with that. We don't need our algorithm to be perfect -- we just want it to be good enough to increase the chance that a container is able to recover from a problem on its own.

For our leaky token bucket, we decided to do the bare minimum: we wanted to have something simple to understand and test. For our use case, it's OK to have the leaky token bucket fully refill every ten minutes.

/**
 * returns errorWatcher:0, errorWatcher:1,... errorWatcher:5
 * based on the current minute past the hour
 */
export function makeKey(now: Date) {
  const minutes = Math.floor(now.getMinutes() / 10);
  return `errorWatcher:${minutes}`;
}

const TEN_MINUTES_IN_SECONDS = 10 * 60;
const BUCKET_CAPACITY = 10;
export async function canHaveShutdownToken(now = new Date()): Promise<boolean> {
  const key = makeKey(now);
  const multi = client.multi();
  multi.incr(key);
  multi.expire(key, TEN_MINUTES_IN_SECONDS);
  try {
    const results = await multi.execAsync<[number, number]>();
    return results[0] <= BUCKET_CAPACITY;
  } catch (err) {
    // if we fail here, we want to know about it
    // but we don't want our error watcher to cause more errors
    sampleLog("errorWatcher.token_fetch_error", err);
    return false;
  }
}

See Even better rate-limiting for a description of how to set up a leaky token bucket that incorporates data from the previous time period to avoid sharp discontinuities between time periods.

Our container culling code has been running in production for several months now, and has been working quite well! Over the past two weeks, it successfully shut down 14 containers that weren't going to be able to recover on their own and saved a few engineers from needing to do any manual interventions. The one drawback has been that it makes it easier to ignore some of these issues causing these containers to get into these bad states to begin with, but it's a tradeoff we're happy to make.