Status: Accepted, 2025. Supersedes nothing. Revisit if the warehouse ever carries fewer than about thirty datasets, where hand tuning stays cheaper.
Freshness monitoring in most warehouses is a fixed rule per table: "alert if this has not loaded by 07:00". The rule is right for whichever table it was written for and wrong for every other table it was copied onto. Two failure modes follow, and teams pick one:
- Set it tight and drown. On the 53 dataset demo warehouse in this repo, a fixed 07:00 rule raises 423 alerts over a 45 day window, of which 272 land on cycles that were never late. That is 6.04 false alarms a day for one team, and a precision of 0.357. Within a month nobody reads the channel.
- Set it loose and miss. Loosening the same rule until it stops crying wolf pushes it past the arrival time of the early tables, and the outage on the 01:10 inventory snapshot goes unnoticed until someone spots a flat chart.
The specific things a single fixed number cannot represent, all of which exist in the demo catalog and all of which exist in real warehouses:
- A vendor feed whose median arrival is 07:48 UTC. Every single load is "late" under a 07:00 rule. Measured here: 144 of 144 undisturbed cycles flagged.
- Weekday-only and business-day-only feeds. A daily rule fires every Saturday.
- Feeds scheduled in a local timezone. The UTC arrival moves an hour twice a year, and a threshold tuned in January starts firing in March.
- Fat right tails. The 99th percentile of a lognormal arrival is four times the median, so a threshold set from the median pages constantly.
- Permanent schedule moves. A vendor brings the export forward by two hours and the old threshold quietly stops meaning anything.
A. Keep fixed thresholds, hand tuned per dataset. Rejected. It is not that fixed thresholds cannot work, it is that they only work while somebody maintains 53 of them. The maintenance is invisible until it is not done, and nothing tells you a threshold has gone stale. We did measure the strongest fair version of this (one grace period per cadence, swept for best F1 on the training window, and separately handed the correct delivery calendar for free) because a comparison against a strawman proves nothing. Even that version loses: see the numbers below.
B. A forecasting library (Prophet, ARIMA, a gradient boosted regressor). Rejected on three grounds. First, explainability: an on-call engineer woken at 06:00 has to be able to ask "why did this page" and get an answer shorter than a paragraph. "The 99th percentile of the 21 Thursdays we have on record in summer time is 517 minutes and this load is at 690" is an answer. A tree ensemble's feature attributions are not. Second, these are point forecast tools and the quantity we need is an upper quantile with an uncertainty band, which they give awkwardly. Third, dependency weight: Prophet drags in a compiler toolchain, and this has to run in a slim container with no build step.
C. A learned conditional quantile per dataset. Chosen.
For each dataset, learn the distribution of arrival offset (minutes from the cycle anchor to the load landing) conditioned on calendar context, and publish the alert threshold as an upper quantile of that distribution.
Concretely, per dataset:
- Infer the delivery calendar from the arrival days themselves. Candidates are every hour, every calendar day, Mon to Fri, business days with US federal holidays removed, weekly on a given weekday, and monthly on business day k for k in 1 to 10. Each is scored by F1 against the days that actually saw a delivery, and the best wins. Nothing is read from a config file, so a feed that quietly stops running at weekends is picked up rather than alerted on.
- Detect a regime change with a segmented model that fits the split point and the calendar cell effects together, then refit on the most recent segment only. Fitting them together rather than sequentially is what stops a daylight saving move being mistaken for a permanent schedule change (see the war story in the README).
- Trim the tail before estimating the tail. History contains incidents, and they sit in exactly the quantile the SLA is estimated from. An iterative outlier fence at median + 10 x (q75 - q50) removes them. Without this the threshold lands inside the incident mass and detects nothing.
- Estimate quantiles per calendar cell in a shallow, hand-chosen hierarchy (for daily data: DST state and day of week, then DST state and weekend flag, then DST state, then global). A sparse cell is shrunk toward its parent with weight n / (n + 12). Every level above global keeps the DST flag, so a winter arrival time is never blended with a summer one.
- Publish threshold = q99 + max(10 minutes, 0.5 x (q99 - q50)), with the expected window as (q10, q90) and a percentile bootstrap band on the threshold. The margin term is the part that buys the low false alarm rate on heavy tailed feeds: it costs a little detection latency and removes most of the noise floor.
Everything published is a quantile, a median or a count. A status line can always be traced to a cell name and an observation count, and it is.
45 day holdout, models fitted only on data before it, 12,465 cycles from 47
hourly and daily grain datasets, 171 of them genuinely late. Full output in
benchmark/results/ is separate; this table comes from docs/evaluation.json,
a committed copy of what freshness evaluate writes to artifacts/evaluation.json.
| Detector | False alarm rate | Detection rate | Precision | F1 | False alarms/day |
|---|---|---|---|---|---|
| Fixed clock, alert if not loaded by 07:00 | 2.212% | 88.30% | 0.357 | 0.508 | 6.04 |
| Fixed, one grace per cadence, tuned for best F1 | 1.009% | 90.06% | 0.554 | 0.686 | 2.76 |
| Fixed, tuned and handed the correct calendar | 0.569% | 90.06% | 0.688 | 0.780 | 1.56 |
| Learned per dataset (this decision) | 0.309% | 99.42% | 0.817 | 0.897 | 0.84 |
Read the third row carefully, because it is the honest comparison. That baseline was given the tuned grace period and the correct delivery calendar, which no real fixed rule has. The learned model still raises 45.7% fewer false alarms and detects 9.4 percentage points more genuine lateness. Against the rule teams actually run, it is 86.0% fewer false alarms at 11.1 points more detection.
Monthly grain datasets are excluded from the comparison. A 45 day holdout holds at most one monthly cycle each, which cannot support a rate estimate, and including them would have flattered the learned model by letting the fixed rule fire every day on a table that loads once a month.
Good. False alarm volume drops by roughly a factor of seven against the rule in common use, at higher detection. Thresholds maintain themselves: a vendor who moves an export window is picked up by the regime detector, and a feed that stops running at weekends is picked up by the calendar inference. Every alert carries its own justification.
Bad, and accepted.
- Cold start is real and cannot be modelled away. A dataset with five loads
has no distribution. The policy is explicit rather than silent: below a
grain-scaled floor there is no learned SLA at all, the dataset gets a loose
cadence default, and it can never raise a severity above advisory or fail
freshness check. Between the floors the SLA is provisional and widened. This means brand new tables are genuinely less protected for their first month, and the console says so rather than pretending otherwise. - History that contains a long standing problem teaches the problem. If a table has been arriving two hours late for six months, the learned SLA says two hours late is normal. This is a real limitation of learning from behaviour rather than from a contract. The mitigation shipped here is that the expected window is published next to the threshold, so a human reviewing the console can see that "expected 07:20 to 09:40" is not what they agreed to. Comparing learned SLAs against contractual ones is future work.
- A shift too recent to fit is invisible to the changepoint scan. Found during the build: a schedule move with fewer post-move cycles than the minimum segment length produced 42 false alarms in the holdout. The fix is a separate shift watch that widens the threshold by the observed move and flags the model as provisional, rather than paging every night until enough history accrues.
- More moving parts than a number in a YAML file. Six steps, five tunables that matter. Mitigated by making every tunable an environment variable with a documented default, and by covering the fitter at 98% statement coverage.