Era history is fetched once at startup, so a long-running node silently stops accepting Plutus transactions with future validity bounds

[hadelive]

Summary

Hydra.Node.Run.getGlobalsForChain queries the era history once at node startup
and never refreshes it. As the chain advances past that forecast horizon, the
ledger can no longer convert a future slot to a POSIXTime, which only matters
when building a ScriptContext. The result is that Plutus transactions carrying
a future validity bound begin failing with OutsideForecast while plain
transfers continue to work, so the head looks healthy.

Evidence

Identical transaction shapes, same head, differing only in node uptime:

transaction	no bound	future upper bound
key-only transfer	accepted	accepted, +1 … +300 slots
Plutus script spend, node up 9h10m	accepted	OutsideForecast at every offset
Plutus script spend, node freshly restarted	accepted	accepted, +5 … +300 slots

(+0 and +1 fail earlier with OutsideValidityIntervalUTxO, which is expected and
unrelated.)

Source

Cardano ... ->
  -- Online mode: query era history from the chain for correct
  -- slot-to-time conversions in Plutus script evaluation.
  ... queryEraHistory QueryTip >>= newGlobalsWithEraHistory

Fetched once; the horizon is anchored to the slot at which it was queried.

Not a devnet artifact

The obvious objection is that a devnet's short epochLength exhausts the horizon
unrealistically fast. Re-tested on preprod (epochLength = 432,000,
production sizing) against a node running 6 days 20 hours (~590,000 slots,
about 1.4 epochs). The failure reproduces.

Why it matters

The degradation is silent, time-dependent and asymmetric. Plain transfers keep
working, so monitoring based on liveness sees nothing. Only deadline-enforcing
contracts fail, and they fail with a ledger error that does not resemble "your
node is stale". A head that works on day one stops honouring time-bounded
contracts on day two with no warning and no state corruption to point at.

For any application using validity_range — order expiry, liquidation windows,
withdrawal timeouts — those paths degrade while everything else appears fine.

Suggested fix

Re-query the era history periodically, or on OutsideForecast. Small relative to
the failure mode.

Workaround

Restart hydra-node periodically, or before relying on deadline-enforcing
contracts. A synthetic script transaction with a small future bound makes a cheap
liveness probe — it fails long before anything else does.

Note on how this was found

Our first conclusion was "scripts cannot carry validity bounds in a head", which
was wrong and would have led to redesigning contracts around a limitation that
does not exist. It was corrected by reading how globals is constructed and
re-running the identical probe against a restarted node. Recording that in case
others hit the same false trail.