Debugging a database that only sings on Fridays

The bug report said the database was “humming through the vents.” That was not actionable, but it was at least reproducible after lunch.

First check

npm run diagnostics -- --tone=minor --weekday=friday

The logs were unhelpful in a way that felt almost elegant:

12:04:18 database: connected
12:04:19 database: warmed up
12:04:20 database: la la la

Query

select
  table_name,
  count(*) as suspicious_rows
from information_schema.tables
where table_name like '%chorus%'
group by table_name
order by suspicious_rows desc;

Findings

1. The read replica had perfect pitch.
2. The migration name included the word anthem.
3. Nobody had touched the cron job, which means the cron job was obviously involved.

The fix was to remove one unused environment variable and pretend that explained the whole thing.