The sample rate and the retention sweep.

Part one: the sampling rate

A press stroke takes somewhere between eight and thirty seconds. We were persisting a power reading every thirty seconds.

Written down like that the problem is obvious. Running, it is not: a stream like this can go weeks without ever looking wrong. The charts draw. The totals accumulate. Nobody’s page throws an error. What surfaced it here was a day view — twenty-four hours of power on one chart — which is the point at which somebody does the arithmetic against the stroke rather than against the screen.

At a thirty-second sample against a twelve-second cycle, you are sampling below Nyquist. Every reading lands wherever it lands in the cycle: sometimes on the shot, more often on the idle floor, occasionally halfway up the ramp. The stored stream is not a coarse version of the truth. It is an aliased version — a different signal, with peaks that never happened and troughs that were never that deep, and no relationship to the original that can be inverted.

The important consequence is the one that catches people out downstream: you cannot fix this later. Min/max banding, smoothing, interpolation, a nicer chart library — none of them recover a peak that was never taken. The information left the building at the sampler. Everything after that is decoration on a signal that is already wrong.

So the sampling rate is not a storage-cost question dressed up as an engineering one. It is a question about whether the data means anything at all, and it has to be answered against the physical thing being measured, not against a disk budget. Almost nothing in a monitoring stack asks that question for you, which is why it is the first one we ask of any deployment, ours included.

The interval we set, and why that number

The persist interval is two seconds. At two seconds a twelve-second cycle contributes six readings, and even the shortest stroke we see contributes four, and both the shot peak and the idle floor are in the data rather than being sampled past. That is the threshold at which the stored stream starts describing the machine again.

It is roughly fifteen times the row count for the recent window. That was accepted deliberately, with a retention sweep as the thing that stops it being the cost of all history — see part two.

The day view we originally wanted is now offered, with its own honest caption: twenty-four hours across 720 columns, two minutes per column. That carries true extremes all day and cannot separate one cycle from the next, which the chart says out loud rather than implying otherwise by drawing a confident line. Resolving individual cycles across a full day would need 21,600 columns. No point budget buys that, and pretending otherwise on a 1,200-pixel screen is how a chart starts lying politely.

There is a ceiling as well as a floor

Sampling faster was not free of thought either. The meter refreshes its measured values on its own internal measuring cycle — for this device class, family-documented at around 200 ms. Polling faster than the instrument’s refresh buys nothing for analog values: it hands back the same measurement until its next internal cycle, and we would have been paying network round trips for duplicate rows.

That sets the useful analog ceiling at roughly 5 Hz for this instrument, which is a genuinely different regime from the storage rate: fifty to a hundred and fifty points per cycle, enough to segment a stroke into its phases and compare its shape against previous ones. Whether those points are worth storing is a separate decision from whether they are worth taking, and conflating the two is how monitoring systems end up either blind or bloated.

Sub-refresh polling only ever mattered for digital-input edge detection, where the thing being watched is a contact rather than a measurement, and there the answer is different again — there is more on that in instrumenting a press that will not talk to you.

Part two: the roll-up that overstates energy by 2.8×

Full-resolution data cannot be kept at full resolution forever. Every time-series system faces this and almost all of them answer it the same way: bucket the old rows and keep each bucket’s minimum and maximum, so the shape of the history survives even as the resolution drops.

It is the standard answer, and it is what our own retention design specified until somebody put numbers through it. On a duty-cycling machine it is badly wrong, and here is why.

Energy is the integral of power over time. Every integrator in every charting and reporting stack does the same thing between two stored readings: it draws a straight line and takes the area under it. That is correct when the readings are a fair sample of what happened in between.

Now feed it a min/max roll-up. What you have stored is an alternating sequence of every bucket’s lowest trough and highest peak — a stream that never spends any time near the middle. The integrator, dutifully drawing straight lines between them, computes the area of a sawtooth that spends half its life at the peak.

Put numbers on it. These are illustrative rather than measurements from the press above — a machine idling at 400 W with a 9 kW shot and a fifteen per cent duty cycle:

Method	Power figure	Where it comes from
Truth	1.7 kW average	0.15 × 9 kW + 0.85 × 0.4 kW
Min/max roll-up, integrated	4.7 kW average	the trough-peak-trough sawtooth

About 2.8 times the truth, silently, on every kWh figure and every pound figure derived from aged history. Nothing errors. The chart looks better than the raw data did — that is the trap, because peak preservation genuinely is the right instinct for a chart. It is the wrong rule for a ledger, and an energy total is a ledger.

Worse, it is a discontinuity in the record. The same period reports one number this week and a different number next week, when the sweep gets to it. Anyone comparing a quarter against the quarter before is comparing full-resolution data against thinned data and reading the difference as a change in the factory.

What we do instead: decimation

Aged rows are decimated, not aggregated: keep the reading nearest each thirty-second mark, discard the rest. No new values are synthesised. Every row that remains is a real measurement that the meter actually reported at the timestamp it carries.

The property that makes this the right answer is a boring one, and it is the point: decimation reproduces exactly the stream this product stored before the interval changed. So no historical energy figure, cost figure or peak-load figure moves when the sweep runs. The record is stable across time, which is the thing a ledger has to be.

The cost is real and worth stating: peaks in aged history are lost. A shot that happened between two retained samples is gone, and no chart drawn over thinned data can show it. That is exactly what the product always did before, so nothing has regressed — but a reader looking at a three-month-old trace needs to know they are looking at coarse history, so the chart says so in its caption rather than presenting thinned data with the same authority as live data.

The retention policy that came out of this:

• Full resolution for a configurable fortnight. Long enough to investigate anything anyone is actually investigating.
• Then thinned to thirty seconds by decimation. Resolution ages.
• Raw rows are never dropped outright. History is not deleted; only its resolution changes.

Steady state comes out at roughly 1.5× the old row count per device-year, not 15×. The expensive window is recent and finite; the long tail is the same size it always was.

What we took from it

Three rules came out of this. They govern how we build this part of the product, and they are what we look for when we are handed someone else’s.

One: sample against the physics, not the storage budget. The interval is set by the fastest thing the data has to show, and the cost of getting it wrong is not a coarse chart — it is a wrong chart, unrecoverably, from the moment the sample is taken.

Two: know what the integrator assumes. Every energy, volume, throughput and cost figure we report is an integral over stored points, and it assumes those points are a fair sample of the interval between them. Any transformation that breaks that assumption — min/max banding being the common one — changes every derived number without changing a single stored value. We now check that before changing anything about how readings are stored.

Three: an aggregation is a claim about what happened between the readings. Decimation makes a modest claim — these particular readings are true, and about the gaps it says nothing. Min/max banding makes a much stronger one, and on a duty-cycling load that claim is false. Faced with a method that loses information honestly and one that invents shape convincingly, we take the honest loss, and we have yet to find a case where it was also the better-looking option on screen.

Which is the real lesson. Both of these traps produce better-looking output than the correct behaviour does. A chart with preserved peaks looks more informative than one without. A thirty-second stream draws perfectly readable lines. No monitoring system flags either, because nothing is broken in any sense a test can assert. They are caught only by someone doing the arithmetic on the physical process and asking whether the number could possibly be right.

That is not a tooling gap. It is a habit, and it is the one worth having.