RnD/EVOLV
1
0
Fork 0
Code Issues 10 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: measurement trial-ready — digital mode + dispatcher fix + 71 tests
Some checks failed
CI / lint-and-test (push) Has been cancelled
Details

Browse Source 
Bumps:
- nodes/generalFunctions  75d16c6 -> e50be2e  (permissive unit check + measurement schema additions)
- nodes/measurement       f7c3dc2 -> 495b4cf  (digital mode + dispatcher fix + 59 new tests + rewritten README + UI)

Wiki:
- wiki/manuals/nodes/measurement.md — new user manual covering analog and
  digital modes, topic reference, smoothing/outlier methods, unit policy,
  and the pre-fix dispatcher bug advisory.
- wiki/sessions/2026-04-13-measurement-digital-mode.md — session note with
  findings, fix scope, test additions, and dual-mode E2E results.
- wiki/index.md — links both pages and adds the missing 2026-04-13
  rotatingMachine session entry that was omitted from the earlier commit.

Status: measurement is now trial-ready in both analog and digital modes.
71/71 unit tests green (was 12), dual-mode E2E on live Dockerized
Node-RED verifies analog regression and a three-channel MQTT-style
payload (temperature/humidity/pressure) dispatching independently with
per-channel smoothing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
znetsixe
2026-04-13 13:46:00 +02:00
parent a1aa44f6ca
commit 0300a76ae8
5 changed files with 317 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
nodes/generalFunctions

Submodule nodes/generalFunctions updated: 75d16c620a...e50be2ee66

2
nodes/measurement

Submodule nodes/measurement updated: 0918be7705...495b4cf400

3
wiki/index.md
View File

@@ -38,6 +38,7 @@ updated: 2026-04-13
## Manuals
- [rotatingMachine User Manual](manuals/nodes/rotatingMachine.md) — inputs, outputs, state machine, examples
- [measurement User Manual](manuals/nodes/measurement.md) — analog + digital modes, smoothing, outlier filtering
- [FlowFuse Dashboard Layout](manuals/node-red/flowfuse-dashboard-layout-manual.md)
- [FlowFuse Widget Catalog](manuals/node-red/flowfuse-widgets-catalog.md)
- [Node-RED Function Patterns](manuals/node-red/function-node-patterns.md)
@@ -46,6 +47,8 @@ updated: 2026-04-13
## Sessions
- [2026-04-07: Production Hardening](sessions/2026-04-07-production-hardening.md) — rotatingMachine + machineGroupControl
- [2026-04-13: rotatingMachine Trial-Ready](sessions/2026-04-13-rotatingMachine-trial-ready.md) — FSM interruptibility, config schema sync, UX polish, dual-curve tests
- [2026-04-13: measurement Digital Mode](sessions/2026-04-13-measurement-digital-mode.md) — silent dispatcher bug fix, 59 new tests, MQTT-style multi-channel input mode
## Other Documentation (outside wiki)
- `CLAUDE.md` — Claude Code project guide (root)

203
wiki/manuals/nodes/measurement.md Normal file
View File

@@ -0,0 +1,203 @@
---
title: measurement — User Manual
node: measurement
updated: 2026-04-13
status: trial-ready
---
# measurement — User Manual
The `measurement` node is the sensor-side of every EVOLV flow. It takes raw signal data, applies offset / scaling / smoothing / outlier rejection, and publishes a conditioned value into the shared `MeasurementContainer`. A parent equipment node (rotatingMachine, pumpingStation, reactor, ...) subscribes automatically via the child-registration handshake on port 2.
## At a glance
| Item | Value |
|---|---|
| Node category | EVOLV |
| Inputs | 1 (message-driven) |
| Outputs | 3 — `process` / `dbase` / `parent` |
| Tick period | 1 s |
| Input modes | `analog` (default) — one scalar per msg. `digital` — object payload with many keys. |
| Smoothing methods | 12 (`none`, `mean`, `min`, `max`, `sd`, `lowPass`, `highPass`, `weightedMovingAverage`, `bandPass`, `median`, `kalman`, `savitzkyGolay`) |
| Outlier methods | 3 (`zScore`, `iqr`, `modifiedZScore`) |
## Choosing a mode
### Analog — one scalar per message (PLC / 4-20 mA)
The classic pattern — what the node did before v1.1. `msg.payload` is a single number. The node runs one offset → scaling → smoothing → outlier pipeline and emits exactly one MeasurementContainer slot keyed by the asset's type + position.
```json
{ "topic": "measurement", "payload": 12.34 }
```
Use when one Node-RED `measurement` node represents one physical sensor.
### Digital — object payload, many channels (MQTT / IoT / JSON)
Use when one Node-RED `measurement` node represents one physical **device** that publishes multiple readings. Common shapes:
```json
{ "topic": "measurement",
"payload": { "temperature": 22.5, "humidity": 45, "pressure": 1013 } }
```
```json
{ "topic": "measurement",
"payload": { "co2": 618, "voc": 122, "pm25": 8 } }
```
Each top-level key maps to a **channel** with its own `type`, `position`, `unit`, and pipeline parameters. Unknown keys are ignored (logged at debug).
## Configuration
### Common (both modes)
- **Asset** (menu): supplier, category, asset type (`assetType`), model, unit.
- **Logger** (menu): log level + enable flag.
- **Position** (menu): `upstream` / `atEquipment` / `downstream`, optional distance offset.
### Analog fields
| Field | Meaning |
|---|---|
| **Scaling** | enables linear interpolation from source range to process range |
| **Source Min / Max** | raw input bounds (e.g. `4` / `20` for mA) |
| **Input Offset** | additive bias applied before scaling |
| **Process Min / Max** | mapped output bounds (e.g. `0` / `3000` for mbar) |
| **Simulator** | internal random-walk source for testing |
| **Smoothing** | method (dropdown) |
| **Window** | smoothing window size |
### Digital fields
- **Input Mode**: set to `digital` in the dropdown.
- **Channels (JSON)**: array of channel definitions.
Each channel:
```json
{
"key": "temperature",
"type": "temperature",
"position": "atEquipment",
"unit": "C",
"scaling": { "enabled": false, "inputMin": 0, "inputMax": 1, "absMin": -50, "absMax": 150, "offset": 0 },
"smoothing": { "smoothWindow": 5, "smoothMethod": "mean" },
"outlierDetection": { "enabled": true, "method": "zScore", "threshold": 3 }
}
```
`scaling` / `smoothing` / `outlierDetection` are optional — missing sections inherit the top-level analog-mode fields. `key` is the JSON field name inside `msg.payload`; `type` is the MeasurementContainer axis — any string works, not just the physical-unit-backed defaults.
## Input topics
| Topic | Payload | Effect |
|---|---|---|
| `measurement` | number (analog) / object (digital) | drives the pipeline |
| `simulator` | — | toggle the internal random-walk simulator |
| `outlierDetection` | — | toggle outlier rejection |
| `calibrate` | — | set the offset so the current output matches `Source Min` (scaling on) or `Process Min` (scaling off). Requires a stable window — aborts if the signal is fluctuating. |
## Output ports
### Port 0 — process
Delta-compressed payload.
**Analog** shape:
```json
{ "mAbs": 4.2, "mPercent": 42, "totalMinValue": 0, "totalMaxValue": 100,
"totalMinSmooth": 0, "totalMaxSmooth": 4.2 }
```
**Digital** shape:
```json
{ "channels": {
"temperature": { "key": "temperature", "type": "temperature", "position": "atEquipment",
"unit": "C", "mAbs": 24, "mPercent": 37,
"totalMinValue": 22.5, "totalMaxValue": 25.5,
"totalMinSmooth": 22.5, "totalMaxSmooth": 24 },
"humidity": { ... },
"pressure": { ... }
} }
```
### Port 1 — dbase
InfluxDB line-protocol telemetry. Tags = asset metadata; fields = measurements. See [InfluxDB Schema Design](../../concepts/influxdb-schema-design.md).
### Port 2 — parent
`{ topic: "registerChild", payload: <nodeId>, positionVsParent, distance }` — emitted once ~200 ms after deploy so the parent equipment node registers this sensor.
## Pipeline per value
1. **Outlier check** (if enabled) — rejects via zScore / IQR / modifiedZScore. Rejected values never advance, don't update min/max, don't emit.
2. **Offset** — `value + scaling.offset`.
3. **Scaling** (if enabled) — linear interpolation from `[inputMin, inputMax]` to `[absMin, absMax]` with boundary clamping.
4. **Smoothing** — current value pushed into the rolling window; the configured method produces the smoothed output.
5. **Min/Max tracking** — both raw (pre-smoothing) and smoothed min/max tracked for display.
6. **Constrain** — smoothed value clamped to `[absMin, absMax]`.
7. **Emit** — `MeasurementContainer.type(...).variant('measured').position(...).distance(...).value(out, ts, unit)` triggers the event `<type>.measured.<position>` (lowercase) that the parent equipment subscribes to.
In digital mode, each channel runs this pipeline independently.
## Smoothing methods — quick reference
| Method | Use case |
|---|---|
| `none` | pass raw value through — useful for testing |
| `mean` | simple arithmetic average over window |
| `min` / `max` | worst-case / peak reporting |
| `sd` | outputs standard deviation (noise indicator) |
| `median` | outlier-resistant central tendency |
| `weightedMovingAverage` | later samples weighted higher |
| `lowPass` | EMA-style attenuation of high-frequency noise |
| `highPass` | emphasises rapid changes (step detection) |
| `bandPass` | `lowPass + highPass - raw` — band-of-interest filtering |
| `kalman` | recursive noise filter, converges to steady value |
| `savitzkyGolay` | polynomial smoothing over 5-point window |
## Outlier methods — quick reference
| Method | Best when |
|---|---|
| `zScore` | signal is approximately normal; threshold = # of SDs |
| `iqr` | signal is non-normal; robust to skewed distributions |
| `modifiedZScore` | small samples; uses median / MAD instead of mean / SD |
> **Historical bug fixed 2026-04-13:** The dispatcher compared against camelCase keys (`lowPass`, `zScore`, ...) but the validator lowercases enum values. Result: 4 smoothing methods and 2 outlier methods were silently no-ops when chosen from the editor — they fell through to the "unknown" branch and emitted the raw last value. Review any flow deployed before 2026-04-13 that relied on these methods.
## Unit policy
Unknown measurement types (anything not in the container's built-in `measureMap`: `pressure`, `flow`, `power`, `temperature`, `volume`, `length`, `mass`, `energy`) are accepted without unit compatibility checks. This lets digital channels use `humidity` (`%`), `co2` (`ppm`), arbitrary IoT units. Known types still validate strictly.
## Example flow (digital)
```json
[
{ "id": "dig", "type": "measurement",
"mode": "digital",
"channels": "[{\"key\":\"temperature\",\"type\":\"temperature\",\"position\":\"atEquipment\",\"unit\":\"C\",\"scaling\":{\"enabled\":false,\"absMin\":-50,\"absMax\":150},\"smoothing\":{\"smoothWindow\":5,\"smoothMethod\":\"mean\"}},{\"key\":\"humidity\",\"type\":\"humidity\",\"position\":\"atEquipment\",\"unit\":\"%\",\"scaling\":{\"enabled\":false,\"absMin\":0,\"absMax\":100},\"smoothing\":{\"smoothWindow\":5,\"smoothMethod\":\"mean\"}}]",
...
}
]
```
## Testing
```bash
cd nodes/measurement
npm test
```
71 tests — coverage includes every smoothing method, every outlier strategy, scaling, interpolation, constrain, calibration, stability, simulation, per-channel pipelines, digital-mode dispatch, malformed-channel handling, event emits.
End-to-end benchmark scripts live in the superproject at `/tmp/m_e2e_baseline.py` (analog) and `/tmp/m_digital_e2e.py` (digital). Run against a Dockerized Node-RED stack (`docker compose up -d nodered`).
## Production status
Trial-ready as of 2026-04-13 after the session that fixed the silent dispatcher bug and added digital mode. See [session 2026-04-13](../../sessions/2026-04-13-measurement-digital-mode.md) and the memory file `node_measurement.md`.

109
wiki/sessions/2026-04-13-measurement-digital-mode.md Normal file
View File

@@ -0,0 +1,109 @@
---
title: "Session: measurement node — dispatcher bug fix + digital/MQTT mode"
created: 2026-04-13
updated: 2026-04-13
status: proven
tags: [session, measurement, smoothing, outlier, mqtt, iot]
---
# 2026-04-13 — measurement trial-ready + digital mode
## Scope
Honest review of the `measurement` node. Benchmark every method, reason about keeping the node agnostic across analog and digital sources, add a digital (MQTT/IoT) mode without breaking analog.
## Findings
### Silent dispatcher bug (critical)
`validateEnum` in `generalFunctions` lowercases enum values (`zScore` → `zscore`, `lowPass` → `lowpass`). But `specificClass.outlierDetection` and `specificClass.applySmoothing` compared against camelCase keys. Effect:
- 5 of 11 smoothing methods silently fell through to a no-op: `lowPass`, `highPass`, `weightedMovingAverage`, `bandPass`, `savitzkyGolay`.
- 2 of 3 outlier methods silently disabled: `zScore`, `modifiedZScore`.
- Only `mean`, `median`, `sd`, `min`, `max`, `none`, `kalman`, `iqr` (the already-lowercase ones) actually worked.
Users who picked any camelCase method from the dropdown got the raw last value or no outlier filtering, with no error. Flows deployed before this session that relied on these filters got no filtering at all.
### Test coverage was thin
Pre-session: **12 tests** — 1 for scaling, 1 for outlier toggle, 1 for event emit, 3 for example flow shape, 1 constructor, 1 routing, 1 invalid payload, 2 other. Every smoothing method beyond `mean` and every outlier method beyond a toggle-flip was untested. The dispatcher bug would have been caught immediately by per-method unit tests.
### Analog-only input shape
The node only accepted scalar `msg.payload`. MQTT / IoT devices commonly publish a single JSON blob with many readings per message. Every user wanting that pattern had to fan out into N measurement nodes — ugly, and the device's shared timestamp is lost.
## Fixes + additions
### Dispatcher normalization (`specificClass.js`)
Both `outlierDetection()` and `applySmoothing()` now lowercase the configured method and the lookup table keys. Legacy camelCase config values and normalized lowercase config values both work.
### `MeasurementContainer.isUnitCompatible` permissive short-circuit
Previously: if the unit couldn't be described by the convert module, compatibility returned false regardless of type. This blocked user-defined types like `humidity` with unit `%`. Now: when `measureMap[type]` is undefined (unknown type), accept any unit. Known types still validate strictly.
### Digital mode (new)
`config.mode.current === 'digital'` opts into a new input shape. `config.channels` declares one entry per JSON key. The new `Channel` class (`src/channel.js`) is a self-contained per-channel pipeline — outlier → offset → scaling → smoothing → min/max → constrain → emit. Analog behaviour is preserved exactly; flows built before this session work unchanged.
## Test additions
Before → after: **12 → 71** tests.
New files:
- `test/basic/smoothing-methods.basic.test.js` — every smoothing method covered, 16 tests.
- `test/basic/outlier-detection.basic.test.js` — every outlier method + toggle + fall-through, 10 tests.
- `test/basic/scaling-and-interpolation.basic.test.js` — offset / interpolateLinear / constrain / handleScaling / updateMinMaxValues / updateOutputPercent / updateOutputAbs / getOutput, 10 tests.
- `test/basic/calibration-and-stability.basic.test.js` — calibrate / isStable / evaluateRepeatability / toggleSimulation / tick / simulateInput, 11 tests.
- `test/integration/digital-mode.integration.test.js` — 12 tests covering channel build, payload dispatch, multi-channel emit, unknown keys, per-channel scaling / smoothing / outlier, empty channels, malformed entries, non-numeric values, digital-output shape.
## E2E verification (Dockerized Node-RED)
### Analog baseline — `/tmp/m_e2e_baseline.py`
Deploys `examples/basic.flow.json`, fires `{topic:"measurement", payload:42}` repeatedly. Observed port-0 output: `mAbs` climbed 0 → 2.1 → 2.8 → 3.15 → 3.36 → 4.2 across five ticks as the mean window filled with 42s (scaling 0..100 → 0..10). Tick cadence 909–1001 ms (avg 981 ms). Registration at t=0.22 s.
### Digital end-to-end — `/tmp/m_digital_e2e.py`
Deploys a single measurement node in digital mode with three channels (`temperature` / `humidity` / `pressure`) and fires two MQTT-shaped payloads.
| Tick | Channel | mAbs | totalMinSmooth | totalMaxSmooth |
|---|---|---:|---:|---:|
| after inject 1 | temperature | 22.5 | 22.5 | 22.5 |
| after inject 1 | humidity | 45 | 45 | 45 |
| after inject 1 | pressure | 1013 | 1013 | 1013 |
| after inject 2 | temperature | 24 | 22.5 | 24 |
| after inject 2 | humidity | 42.5 | 42.5 | 45 |
| after inject 2 | pressure | 1014 | 1013 | 1014 |
Mean smoothing across a window of 3 computed per-channel, the `unknown` key in the payload ignored, all three events emitted on `<type>.measured.atequipment`.
## Files changed
```
nodes/generalFunctions/src/measurements/MeasurementContainer.js # permissive unit check for user-defined types
nodes/generalFunctions/src/configs/measurement.json # mode + channels schema
nodes/measurement/src/channel.js # new per-channel pipeline class
nodes/measurement/src/specificClass.js # dispatcher fix + digital dispatch
nodes/measurement/src/nodeClass.js # mode-aware input handler + tick
nodes/measurement/measurement.html # Mode dropdown + Channels JSON + help panel
nodes/measurement/README.md # rewrite
nodes/measurement/test/basic/smoothing-methods.basic.test.js # +16 tests
nodes/measurement/test/basic/outlier-detection.basic.test.js # +10 tests
nodes/measurement/test/basic/scaling-and-interpolation.basic.test.js # +10 tests
nodes/measurement/test/basic/calibration-and-stability.basic.test.js # +11 tests
nodes/measurement/test/integration/digital-mode.integration.test.js # +12 tests
```
## Production status
Trial-ready for both modes. Supervised trial recommended for digital-mode deployments until the channels-editor UI (currently a JSON textarea) lands.
## Follow-ups
- Repeatable-row editor widget for channels.
- `validateArray.minLength=0` evaluates as falsy; pre-existing generalFunctions bug affecting this node's `channels` and also `measurement.assetRegistration.childAssets`. Harmless warn at deploy time.
- Per-channel calibration + simulation for digital mode.
- Runtime channel reconfiguration via a dedicated topic (`addChannel` / `removeChannel`).