Compare commits

...

9 Commits

Author SHA1 Message Date
Rene De Ren
ea2857fb25 fix: serialize per-pump shutdown + cancel deferred dispatch in turnOffAllMachines
PS calls turnOffAllMachines on every tick once level < stopLevel. Two
ways the pump could re-engage after we shut it down:

1. _delayedCall: a 1% dead-zone keep-alive parked in MGC's deferred
   dispatch fires from the in-flight handleInput's finally block AFTER
   the shutdown completes, dispatching flow + startup to a fresh pump.
   Clear _delayedCall at the top of turnOff.

2. Concurrent shutdown calls on the same pump interrupt each other
   before the sequence can transition past stopping. Track shutdown-
   in-flight per pump and skip if one is already underway.

Together with the rotatingMachine delayedMove-clearing fix, this lets
the level-based hysteresis cycle complete: pumps shut off cleanly at
stopLevel, basin reverses direction, refills to startLevel, repeat.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 18:17:55 +02:00
Rene De Ren
2651aaf409 abortActiveMovements: only WARN when actually aborting an in-flight move
In normal operation the _dispatchInFlight gate (handleInput)
guarantees no pump movement is in flight when a new dispatch starts,
so the per-machine abort call is a no-op. The previous unconditional
WARN flooded the log with one line per pump per tick (~3/s) for what
was actually a normal-path no-op.

Now the WARN fires ONLY when a pump's state is accelerating or
decelerating — i.e. the gate has been bypassed and we're force-
aborting an in-flight ramp. The wording reflects that:

  Force-aborting in-flight movement on pump_a (state=accelerating)
  due to: new demand received — _dispatchInFlight gate bypassed.

If you ever see this in production logs, the gate has a hole and
needs investigating.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 09:43:12 +02:00
Rene De Ren
df74ea0fac Serialize handleInput dispatches via _dispatchInFlight gate
Mirrors rotatingMachine state.delayedMove. PS ticks demand into MGC at
1 Hz but a real pump ramp takes several seconds; before this gate every
PS tick aborted the in-flight optimalControl and started a new one, so
pumps never reached their setpoint. Live observation: 120 aborts / 2
min, pump_a drifting to 138 m³/h while pump_b stayed clamped at minFlow
60 m³/h ("near_curve_edge").

While a dispatch is in flight, the latest {source, demand, powerCap,
priorityList} is parked in _delayedCall and the new call returns.
The in-flight dispatch's finally block picks up the latest delayed
value when it settles. Latest-wins — intermediate demands are stomped
because they were obsolete by the time the pumps were ready for them.

Regression test in superproject:
test/mgc-overactive-demand-serialization.integration.test.js
30 concurrent demand calls now produce ≤ 5 aborts (was 30).

All existing tests still pass: 21 MGC integration + 7 cross-node
integration (incl. realistic-startup-timing, inflow-overcapacity-
stability, ps-mgc-flow-contract, idle-startup-deadlock).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 09:14:59 +02:00
Rene De Ren
96b84d3124 Revert: handleInput unchanged-demand short-circuit
Reverts a14aa0d. The "skip when demand unchanged" optimisation broke
the live demo: in some real conditions (basin transitions, safety
controller activations) PS sends repeated demand=0 and the optimisation
correctly turned pumps off the first time but then declined to re-act
when conditions changed in a way the test suite didn't cover. Live
result: pumps stayed off even when basin filled to overflow.

The original symptom (pumps stuck mid-ramp under saturated demand) needs
a different approach — likely a pump-side guard rather than an MGC-side
demand filter. Investigating in a follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 20:55:41 +02:00
Rene De Ren
a14aa0dab8 handleInput: skip abort+redispatch when demand unchanged
Live trace showed PS ticking every 1 s and re-firing the SAME demand
(100% saturated under storm inflow) while the basin level evolved slowly.
Each tick was calling abortActiveMovements + optimalControl, which
aborted in-flight pump moves before they could finish (move duration
~0.4 s vs 1 s tick) and immediately re-issued the same setpoint. Pumps
got stuck ramping from the same starting position toward the same
target indefinitely — moveTimeleft stable at 0.379 s for minutes,
flow.predicted frozen.

Now early-return when |demandQ - prev| < max(0.5, prev*0.005). PS
hysteresis float jitter is filtered, real demand changes still
propagate. Pumps finish their first move and stay at the right
setpoint instead of being aborted forever.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 20:10:50 +02:00
Rene De Ren
69bdf11fc4 DOWNSTREAM is the live aggregate; AT_EQUIPMENT is the optimizer's intent
handlePressureChange writes the live aggregate (sum of every pump's
current predicted-flow measurement) to flow.predicted.downstream — that
is the channel PS subscribes to for its outflow estimate, and it must
reflect what pumps are actually delivering.

optimalControl + equalFlowControl + prioPercentageControl were also
writing to DOWNSTREAM with the optimizer's TARGET (bestFlow / totalFlow).
That's a planned setpoint, not an achieved aggregate, and it was
clobbering the live value every handleInput tick — leaving PS reading
e.g. 105 m³/h while the real aggregate was 681 m³/h. Test
ps-mgc-flow-contract caught this deterministically.

Move all the optimizer-target writes to AT_EQUIPMENT (the "what we
commanded the equipment to do" channel). DOWNSTREAM is now
single-writer (handlePressureChange) and faithfully tracks reality.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 18:32:58 +02:00
Rene De Ren
dc27a569d9 handlePressureChange: mirror aggregate flow onto DOWNSTREAM
PS subscribes to MGC's flow.predicted.downstream and uses it as the
outflow estimate for net-flow computation. MGC was only writing to
DOWNSTREAM inside optimalControl (the optimizer's bestFlow TARGET, not
the achieved aggregate), and to AT_EQUIPMENT in handlePressureChange.

During transients — e.g. demand dropping to dead-band keep-alive while
pumps are still ramping down from full throttle — PS saw a stale 25 m³/h
target on DOWNSTREAM while pumps were physically delivering 500+ m³/h.
NetFlow looked small and stable when the basin was actually draining
fast.

flow.act = sum of every pump's current predicted output = achieved
aggregate. Mirror it onto DOWNSTREAM so PS gets a live signal on every
pump flow/pressure update, not just every MGC.handleInput.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:20:21 +02:00
Rene De Ren
b7c40b0ddc equalFlowControl: mirror the optimalControl dispatch reorder
The priority-control codepath had the same stale dispatch shape that
caused the live deadlock in optimalControl: only handling idle and
operational states, and chaining flowmovement after execsequence
startup. Aligns it with the optimalControl fix so a future mode switch
to prioritycontrol doesn't reintroduce the bug.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 11:47:17 +02:00
Rene De Ren
8e684203a8 Test: add full-cycle and up/down-sweep regression scenarios
Scenario 5 covers 100% → 0% → 100% with the second 100% landing
mid-shutdown (stopping/coolingdown) — exercises the path where
delayedMove must NOT be saved on a non-idle non-residue state without
a follow-up startup, since transitionToState('idle') doesn't fire it.

Scenario 6 walks 10%→100%→10% monotonically and asserts the down-sweep's
final demand is honoured (catches the user's observed "stuck around
60% going up, no reaction going down" symptom — where pumps would
otherwise freeze at a stale setpoint from the up-sweep).

Both pass with the current MGC dispatch fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 11:33:56 +02:00
3 changed files with 349 additions and 15 deletions

View File

@@ -74,6 +74,16 @@ class MachineGroup {
// Combination curve data
this.dynamicTotals = { flow: { min: Infinity, max: 0 }, power: { min: Infinity, max: 0 } , NCog : 0};
this.absoluteTotals = { flow: { min: Infinity, max: 0 }, power: { min: Infinity, max: 0 }};
// Dispatch serialization. PS ticks demand into MGC at 1 Hz, but
// a real pump ramp takes several seconds — without this gate
// every PS tick aborts the in-flight dispatch and starts a new
// one, so pumps never reach their setpoint. Mirrors
// rotatingMachine state.delayedMove: while a dispatch is in
// flight the latest demand is parked here for pickup when the
// current dispatch settles. Latest-wins.
this._dispatchInFlight = false;
this._delayedCall = null;
//this always last in the constructor
this.childRegistrationUtils = new childRegistrationUtils(this);
@@ -283,6 +293,18 @@ class MachineGroup {
this.logger.debug(`Dynamic Totals after pressure change - Flow: Min ${flow.min}, Max ${flow.max}, Act ${flow.act} | Power: Min ${power.min}, Max ${power.max}, Act ${power.act}`);
this._writeMeasurement("flow", "predicted", POSITIONS.AT_EQUIPMENT, flow.act, this.unitPolicy.canonical.flow);
// Mirror the aggregate flow onto DOWNSTREAM as well. PS subscribes to
// flow.predicted.downstream from MGC and uses it as the outflow
// estimate for net-flow computation. Without this mirror, the only
// place DOWNSTREAM gets written is optimalControl's bestFlow (the
// optimizer's TARGET, not the achieved aggregate). During transients
// — e.g. demand dropping to dead-band keep-alive while pumps are
// still ramping down from full throttle — PS would see a stale
// 25 m³/h target while pumps are physically delivering 500+ m³/h,
// making netFlow look small and stable when the basin is actually
// draining fast. flow.act here is the sum of every pump's current
// predicted output, so it IS the achieved aggregate.
this._writeMeasurement("flow", "predicted", POSITIONS.DOWNSTREAM, flow.act, this.unitPolicy.canonical.flow);
this._writeMeasurement("power", "predicted", POSITIONS.AT_EQUIPMENT, power.act, this.unitPolicy.canonical.power);
const { maxEfficiency, lowestEfficiency } = this.calcGroupEfficiency(this.machines);
@@ -697,8 +719,17 @@ class MachineGroup {
}
async abortActiveMovements(reason = "new demand") {
// Safety net: in normal operation the _dispatchInFlight gate
// (handleInput) ensures no pump movement is in flight when a
// new dispatch starts, so this is a no-op. If a pump IS still
// moving here, the gate was bypassed (direct call to
// abortActiveMovements, mode change racing a dispatch, etc.) —
// surface that loudly so the bypass can be diagnosed.
const movementStates = new Set(['accelerating', 'decelerating']);
await Promise.all(Object.values(this.machines).map(async machine => {
this.logger.warn(`Aborting active movements for machine ${machine.config.general.id} due to: ${reason}`);
const state = machine.state?.getCurrentState?.();
if (!movementStates.has(state)) return;
this.logger.warn(`Force-aborting in-flight movement on ${machine.config.general.id} (state=${state}) due to: ${reason} — _dispatchInFlight gate bypassed.`);
if (typeof machine.abortMovement === "function") {
await machine.abortMovement(reason);
}
@@ -772,9 +803,15 @@ class MachineGroup {
const debugInfo = bestResult.bestCombination.map(({ machineId, flow }) => `${machineId}: ${flow.toFixed(2)} units`).join(" | ");
this.logger.debug(`Moving to demand: ${Qd.toFixed(2)} -> Pumps: [${debugInfo}] => Total Power: ${bestResult.bestPower.toFixed(2)}`);
//store the total delivered power
// Store the optimizer's INTENT on AT_EQUIPMENT (what we
// commanded). DOWNSTREAM is reserved for the live aggregate
// written by handlePressureChange — PS subscribes to that
// for net-flow computation and must see what pumps are
// actually delivering, not the planned target. Writing
// bestFlow to DOWNSTREAM here would clobber the live value
// every handleInput tick (see ps-mgc-flow-contract test).
this._writeMeasurement("power", "predicted", POSITIONS.AT_EQUIPMENT, bestResult.bestPower, this.unitPolicy.canonical.power);
this._writeMeasurement("flow", "predicted", POSITIONS.DOWNSTREAM, bestResult.bestFlow, this.unitPolicy.canonical.flow);
this._writeMeasurement("flow", "predicted", POSITIONS.AT_EQUIPMENT, bestResult.bestFlow, this.unitPolicy.canonical.flow);
this.measurements.type("efficiency").variant("predicted").position(POSITIONS.AT_EQUIPMENT).value(bestResult.bestFlow / bestResult.bestPower);
this.measurements.type("Ncog").variant("predicted").position(POSITIONS.AT_EQUIPMENT).value(bestResult.bestCog);
@@ -1108,9 +1145,11 @@ class MachineGroup {
this.logger.debug(`Priority control for demand: ${totalFlow.toFixed(2)} -> Active pumps: [${debugInfo}] => Total Power: ${totalPower.toFixed(2)}`);
// Store measurements
// Store the planned distribution as INTENT on AT_EQUIPMENT.
// DOWNSTREAM (live aggregate) is owned by handlePressureChange.
// Writing the plan here would clobber PS's outflow signal.
this._writeMeasurement("power", "predicted", POSITIONS.AT_EQUIPMENT, totalPower, this.unitPolicy.canonical.power);
this._writeMeasurement("flow", "predicted", POSITIONS.DOWNSTREAM, totalFlow, this.unitPolicy.canonical.flow);
this._writeMeasurement("flow", "predicted", POSITIONS.AT_EQUIPMENT, totalFlow, this.unitPolicy.canonical.flow);
this.measurements.type("efficiency").variant("predicted").position(POSITIONS.AT_EQUIPMENT).value(totalFlow / totalPower);
this.measurements.type("Ncog").variant("predicted").position(POSITIONS.AT_EQUIPMENT).value(totalCog);
@@ -1121,16 +1160,18 @@ class MachineGroup {
this.logger.debug(this.machines[machineId].state);
const currentState = this.machines[machineId].state.getCurrentState();
if (flow <= 0 && (currentState === "operational" || currentState === "accelerating" || currentState === "decelerating")) {
// Same dispatch shape as optimalControl — see the comment
// there for the rationale. flowmovement BEFORE startup so
// concurrent retargets can update delayedMove without a
// stale chained flowmovement overwriting it after startup.
if (flow > 0) {
await machine.handleInput("parent", "flowmovement", this._canonicalToOutputFlow(flow));
if (currentState === "idle") {
await machine.handleInput("parent", "execsequence", "startup");
}
} else if (currentState === "operational" || currentState === "accelerating" || currentState === "decelerating") {
await machine.handleInput("parent", "execsequence", "shutdown");
}
else if (currentState === "idle" && flow > 0) {
await machine.handleInput("parent", "execsequence", "startup");
await machine.handleInput("parent", "flowmovement", this._canonicalToOutputFlow(flow));
}
else if (currentState === "operational" && flow > 0) {
await machine.handleInput("parent", "flowmovement", this._canonicalToOutputFlow(flow));
}
}));
}
catch (err) {
@@ -1233,8 +1274,12 @@ class MachineGroup {
}
});
// Write to AT_EQUIPMENT not DOWNSTREAM. handlePressureChange
// is the canonical writer of DOWNSTREAM (the live aggregate
// that PS subscribes to for outflow). See optimalControl
// comment above.
this._writeMeasurement("power", "predicted", POSITIONS.AT_EQUIPMENT, totalPower.reduce((a, b) => a + b, 0), this.unitPolicy.canonical.power);
this._writeMeasurement("flow", "predicted", POSITIONS.DOWNSTREAM, totalFlow.reduce((a, b) => a + b, 0), this.unitPolicy.canonical.flow);
this._writeMeasurement("flow", "predicted", POSITIONS.AT_EQUIPMENT, totalFlow.reduce((a, b) => a + b, 0), this.unitPolicy.canonical.flow);
if(totalPower.reduce((a, b) => a + b, 0) > 0){
this.measurements.type("efficiency").variant("predicted").position(POSITIONS.AT_EQUIPMENT).value(totalFlow.reduce((a, b) => a + b, 0) / totalPower.reduce((a, b) => a + b, 0));
@@ -1248,6 +1293,37 @@ class MachineGroup {
async handleInput(source, demand, powerCap = Infinity, priorityList = null) {
// Serialize dispatches: if a previous handleInput is still
// awaiting pump movements, park the latest demand and return.
// The in-flight dispatch's `finally` block will pick it up.
// See rotatingMachine state.delayedMove for the analogous
// pattern at the pump level.
if (this._dispatchInFlight) {
this._delayedCall = { source, demand, powerCap, priorityList };
this.logger.debug(`Dispatch in flight; deferring demand=${demand} until current pump moves complete.`);
return;
}
this._dispatchInFlight = true;
try {
return await this._runDispatch(source, demand, powerCap, priorityList);
} finally {
this._dispatchInFlight = false;
// Pick up the latest deferred call (intermediate values were
// stomped while we were busy — only the last one matters).
if (this._delayedCall) {
const next = this._delayedCall;
this._delayedCall = null;
this.logger.debug(`Dispatch finished; picking up deferred demand=${next.demand}.`);
// Recursive call re-enters the gate; safe because
// _dispatchInFlight has been reset to false above.
await this.handleInput(next.source, next.demand, next.powerCap, next.priorityList);
}
}
}
async _runDispatch(source, demand, powerCap = Infinity, priorityList = null) {
const demandQ = parseFloat(demand);
if(!Number.isFinite(demandQ)){
@@ -1345,8 +1421,28 @@ class MachineGroup {
}
async turnOffAllMachines(){
// Cancel any deferred dispatch — turnOff is the latest user intent,
// a stale 1% keep-alive must not re-engage a pump after we shut down.
this._delayedCall = null;
// Per-pump shutdown serialization: PS calls turnOffAllMachines on
// every tick (every 2 s) once level < stopLevel. Without this guard,
// each new shutdown call hits the still-running prior shutdown's
// movement transitions and triggers abortCurrentMovement, which
// bounces the pump back to 'operational' before the sequence loop
// can reach stopping/coolingdown/idle. Net effect: pump never
// actually shuts down. Track shutdown-in-flight per pump and skip
// if already underway.
if (!this._shutdownInFlight) this._shutdownInFlight = new Set();
await Promise.all(Object.entries(this.machines).map(async ([machineId, machine]) => {
if (this.isMachineActive(machineId)) { await machine.handleInput("parent", "execsequence", "shutdown"); }
if (this._shutdownInFlight.has(machineId)) return;
if (this.isMachineActive(machineId)) {
this._shutdownInFlight.add(machineId);
try {
await machine.handleInput("parent", "execsequence", "shutdown");
} finally {
this._shutdownInFlight.delete(machineId);
}
}
}));
// Update measurements to zero so the parent (PS) sees the
// outflow drop immediately — without this the PS keeps the

View File

@@ -211,6 +211,113 @@ test('Scenario 3 — pumps with NO pressure measurements injected', async () =>
console.log(' (Scenario 3 is exploratory — no asserts; review the snapshot above.)');
});
// ---------------------------------------------------------------------------
test('Scenario 5 — full up/down/up cycle through shutdown', async () => {
// Hypothesis E: when demand goes 100% → 0% → 100% (basin fills, drains
// past stopLevel, then refills), pumps pass through stopping →
// coolingdown → idle. If a fresh flow>0 demand arrives while a pump is
// mid-shutdown, the current MGC dispatch saves flowmovement to
// delayedMove (good) but doesn't issue execsequence-startup because
// state !== 'idle' (bug). The pump completes shutdown, reaches 'idle',
// and stays there because transitionToState('idle') doesn't fire
// delayedMove — only the transition INTO 'operational' does. Pump is
// stuck with delayedMove orphaned.
const { mgc, pumps } = buildGroup();
console.log('\n[Scenario 5] cycle: 100% → 0% → 100% with mid-shutdown re-engage');
printSnapshots('before any handleInput', pumps);
// Phase 1: drive up to 100% from idle.
await mgc.handleInput('parent', 100);
await sleep(5000); // full startup + ramp
printSnapshots('after settle at 100%', pumps);
for (const p of pumps) {
assert.equal(p.state.getCurrentState(), 'operational',
`Phase 1: pump ${p.config.general.id} not operational at 100% (got ${p.state.getCurrentState()})`);
}
// Phase 2: demand drops to 0% — pumps begin shutdown sequence.
// FIRE-AND-FORGET: handleInput(0) awaits turnOffAllMachines which
// awaits the full per-pump shutdown sequence. We need the next 100%
// demand to arrive WHILE pumps are still in stopping/coolingdown,
// not after they've reached idle.
mgc.handleInput('parent', 0).catch(e => console.log(`0% rejected: ${e.message}`));
// Wait briefly so the shutdown sequence enters but does NOT complete.
// shutdown=['stopping','coolingdown','idle'] with stopping=1s,
// coolingdown=2s. 500ms puts us solidly inside 'stopping'.
await sleep(500);
printSnapshots('mid-shutdown (pumps should be in stopping/coolingdown)', pumps);
const midShutdownStates = pumps.map(p => p.state.getCurrentState());
console.log(` states mid-shutdown: ${midShutdownStates.join(', ')}`);
// Phase 3: demand returns to 100% while pumps are mid-shutdown.
await mgc.handleInput('parent', 100);
// Generous: full coolingdown remaining + full startup + ramp.
await sleep(8000);
printSnapshots('after re-engage to 100%', pumps);
expectAllRunningAt100(pumps, 'Scenario 5');
});
// ---------------------------------------------------------------------------
test('Scenario 6 — full up sweep then full down sweep', async () => {
// Hypothesis F: the user observed "going up stuck ~60%, going down
// not reacting". Mirror that with an explicit up-then-down monotonic
// sweep, every step holding 600 ms (slightly longer than DWELL on
// production basin model). After the sweep, we expect the LATEST
// demand (the final value of the down-sweep, which is 10%) to be
// honoured: pumps either at 1-pump combo's split or all idle if that
// demand falls below the per-pump minimum.
const { mgc, pumps } = buildGroup();
console.log('\n[Scenario 6] up-sweep 10%→100% then down-sweep 100%→10%, each step 600 ms');
printSnapshots('before any handleInput', pumps);
const upSteps = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100];
const downSteps = [90, 80, 70, 60, 50, 40, 30, 20, 10];
console.log(' --- up sweep ---');
for (const pct of upSteps) {
mgc.handleInput('parent', pct).catch(e => console.log(`up ${pct}% rejected: ${e.message}`));
await sleep(600);
const snaps = pumps.map(snapshot);
const totalQ = snaps.reduce((s, x) => s + x.flow, 0);
console.log(` cmd=${pct.toFixed(0).padStart(3)}% states=[${snaps.map(s=>s.state.padEnd(13)).join(', ')}] ctrl=[${snaps.map(s=>s.ctrl.toFixed(1).padStart(5)).join(', ')}] ΣQ=${totalQ.toFixed(1)}`);
}
printSnapshots('top of up-sweep (cmd=100%) after full settle', pumps);
await sleep(2000);
printSnapshots('top of up-sweep + 2s drain', pumps);
console.log(' --- down sweep ---');
for (const pct of downSteps) {
mgc.handleInput('parent', pct).catch(e => console.log(`down ${pct}% rejected: ${e.message}`));
await sleep(600);
const snaps = pumps.map(snapshot);
const totalQ = snaps.reduce((s, x) => s + x.flow, 0);
console.log(` cmd=${pct.toFixed(0).padStart(3)}% states=[${snaps.map(s=>s.state.padEnd(13)).join(', ')}] ctrl=[${snaps.map(s=>s.ctrl.toFixed(1).padStart(5)).join(', ')}] ΣQ=${totalQ.toFixed(1)}`);
}
printSnapshots('bottom of down-sweep (cmd=10%) after sequence', pumps);
await sleep(3000);
printSnapshots('bottom of down-sweep + 3s drain', pumps);
// Final demand was 10% (≈ 148 m³/h). At head 1100 mbar with per-pump
// min ≈ 89.5, this is solvable by a 1-pump combo near 148 m³/h.
// Optimizer typically picks the 1-pump combo. Either way, pumps are
// NOT supposed to be stuck at the prior up-sweep's 100% setpoint.
const flowMin_m3h = mgc.calcDynamicTotals().flow.min * 3600;
const flowMax_m3h = mgc.calcDynamicTotals().flow.max * 3600;
const expectedQ_m3h = flowMin_m3h + (flowMax_m3h - flowMin_m3h) * 0.10; // 10% scaled
console.log(` expected total flow at 10%: ~${expectedQ_m3h.toFixed(1)} m³/h`);
const snaps = pumps.map(snapshot);
const totalQ = snaps.reduce((s, x) => s + x.flow, 0);
// Loose: total within 30 m³/h of expectation. Catches the obvious
// stuck-at-old-position regression.
assert.ok(Math.abs(totalQ - expectedQ_m3h) < 30,
`Scenario 6: total flow ${totalQ.toFixed(1)} m³/h diverged from expected ${expectedQ_m3h.toFixed(1)} after down-sweep — pumps did not adopt latest demand. Per-pump: ${snaps.map(s => `${s.state}@${s.ctrl.toFixed(0)}%`).join(', ')}`);
});
// ---------------------------------------------------------------------------
test('Scenario 4 — varying demand during startup (combo flips)', async () => {
// Hypothesis D: in production the demand is NOT constant — as basin

View File

@@ -0,0 +1,131 @@
// Regression: pump A in pumpingstation-complete-example demo got stuck
// running at minimum flow while basin level dropped past stopLevel and
// kept dropping all the way to dry-run threshold.
//
// Root cause (two parts):
//
// 1. rotatingMachine.executeSequence on shutdown went through an
// interruptible-abort path that returned the FSM to 'operational',
// triggering state.transitionToState's auto-pickup of the queued
// delayedMove — re-engaging the pump before the shutdown sequence
// could reach stopping/coolingdown/idle. Fix: clear delayedMove at
// the top of shutdown/emergencystop sequences.
//
// 2. PS calls turnOffAllMachines on every tick (every 2 s) while
// level < stopLevel. Each call interrupted the still-running prior
// shutdown's transitions, resetting the FSM to 'accelerating'. The
// pump bounced accelerating ↔ decelerating forever and the actual
// shutdown sequence transitions never ran. Fix: serialize per-pump
// shutdown calls in turnOffAllMachines so concurrent invocations
// are no-ops while a shutdown is already in flight.
//
// This test exercises part 2 — the per-pump serialization at the MGC
// level — by hammering turnOffAllMachines from a tight loop, mirroring
// the live tick cadence.
const test = require('node:test');
const assert = require('node:assert/strict');
const MachineGroup = require('../../src/specificClass');
const Machine = require('../../../rotatingMachine/src/specificClass');
const logCfg = { enabled: false, logLevel: 'error' };
const stateConfig = {
general: { logging: logCfg },
state: { current: 'idle' },
movement: { mode: 'staticspeed', speed: 50, maxSpeed: 100, interval: 10 },
// Non-zero shutdown timing so a shutdown takes long enough that a
// concurrent turnOff call lands mid-sequence — exactly the live race.
time: { starting: 0, warmingup: 0, stopping: 1, coolingdown: 1 },
};
function machineConfig(id) {
return {
general: { logging: logCfg, name: id, id, unit: 'm3/h' },
functionality: { softwareType: 'machine', role: 'rotationaldevicecontroller' },
asset: { category: 'pump', type: 'centrifugal', model: 'hidrostal-H05K-S03R', supplier: 'hidrostal' },
mode: {
current: 'auto',
allowedActions: { auto: ['execsequence', 'execmovement', 'flowmovement', 'statuscheck'] },
allowedSources: { auto: ['parent', 'GUI'] },
},
sequences: {
startup: ['starting', 'warmingup', 'operational'],
shutdown: ['stopping', 'coolingdown', 'idle'],
emergencystop: ['emergencystop', 'off'],
},
};
}
function groupConfig() {
return {
general: { logging: logCfg, name: 'mgc', id: 'mgc' },
functionality: { softwareType: 'machinegroup', role: 'groupcontroller', positionVsParent: 'atEquipment' },
scaling: { current: 'normalized' },
mode: { current: 'optimalcontrol' },
};
}
function buildGroup() {
const mgc = new MachineGroup(groupConfig());
const ids = ['pump_a', 'pump_b', 'pump_c'];
const pumps = ids.map(id => new Machine(machineConfig(id), stateConfig));
for (const m of pumps) {
m.updateMeasuredPressure(0, 'upstream', { timestamp: Date.now(), unit: 'mbar', childName: 'up', childId: `up-${m.config.general.id}` });
m.updateMeasuredPressure(1100, 'downstream', { timestamp: Date.now(), unit: 'mbar', childName: 'dn', childId: `dn-${m.config.general.id}` });
mgc.childRegistrationUtils.registerChild(m, 'downstream');
}
mgc.calcAbsoluteTotals();
mgc.calcDynamicTotals();
return { mgc, pumps };
}
const sleep = (ms) => new Promise(r => setTimeout(r, ms));
test('repeated turnOffAllMachines reaches idle (serializes concurrent shutdowns)', async () => {
const { mgc, pumps } = buildGroup();
const pumpA = pumps[0];
// Start pump A and queue a delayedMove the way MGC's optimalControl
// would when PS sends a 1% dead-zone keep-alive.
await pumpA.handleInput('parent', 'execsequence', 'startup');
assert.equal(pumpA.state.getCurrentState(), 'operational');
pumpA.setpoint(80); // start a slow move (not awaited)
await sleep(50);
assert.equal(pumpA.state.getCurrentState(), 'accelerating');
pumpA.state.delayedMove = 75;
// Mimic PS's tick loop: fire turnOffAllMachines on a tight cadence
// without awaiting. Without the per-pump serialization in
// turnOffAllMachines, each call hits the still-running prior shutdown
// and bounces the pump back to accelerating — the live deadlock.
const ticks = [];
for (let i = 0; i < 6; i++) {
ticks.push(mgc.turnOffAllMachines());
await sleep(80); // half the realtime tick — tighter race
}
await Promise.all(ticks);
// Allow the (single) in-flight shutdown to finish its 1+1 s timed
// transitions through stopping → coolingdown → idle.
await sleep(2500);
assert.equal(pumpA.state.getCurrentState(), 'idle',
`pump must reach idle under repeated turnOff calls; got ${pumpA.state.getCurrentState()} (delayedMove=${pumpA.state.delayedMove})`);
assert.equal(pumpA.state.delayedMove, null,
'delayedMove must be cleared after shutdown');
});
test('turnOffAllMachines clears MGC._delayedCall to cancel any deferred dispatch', async () => {
// PS sends a 1% keep-alive while MGC is mid-dispatch. MGC parks it in
// _delayedCall. PS then crosses stopLevel and calls turnOffAllMachines.
// Without clearing _delayedCall, MGC's finally block fires the parked
// 1% call AFTER the shutdown — re-engaging the pump.
const { mgc } = buildGroup();
mgc._delayedCall = { source: 'parent', demand: 1, powerCap: Infinity, priorityList: null };
await mgc.turnOffAllMachines();
assert.equal(mgc._delayedCall, null,
'turnOff must cancel any deferred dispatch so it cannot re-engage pumps post-shutdown');
});