RnD/EVOLV
3
0
Fork 0
Code Issues 22 Pull Requests Actions Packages Projects Releases Wiki Activity

release: palette redesign + CoreSync scaffolding + dashboardAPI MODULE_NOT_FOUND fix

Browse Source 
PALETTE REDESIGN (2026-05-21)
  Sidebar swatches switched from S88 level (all blue) to domain-hue per node.
  Family hue = function (rotating=orange, valves=teal, biology=green/olive,
  sampling=violet, sensor=amber, aeration=sky-blue, infrastructure=slate);
  within a family, darker = higher S88 / "more controller-ish."
  Editor-group rectangles in flow.json still follow S88 — only the
  registerType colour changed.

  Submodule bumps for palette: rotatingMachine, machineGroupControl,
  pumpingStation, valve, valveGroupControl, reactor, settler, monster,
  measurement, diffuser, dashboardAPI.

  Docs touched:
    - CLAUDE.md: palette swatch vs. editor-group bullets split out.
    - .claude/rules/node-red-flow-layout.md: new §10.0 introduces the two
      color systems, full 12-row palette table, and explicit warning not to
      mix the two hexes.
    - .claude/refactor/MODULE_SPLIT.md: per-node headers annotated with
      both `group #XXX` and `palette #XXX`.
    - .claude/refactor/WIKI_HOME_TEMPLATE.md + WIKI_TEMPLATE.md: clarify
      Mermaid classDefs visualize hierarchy, not palette swatches.
    - .claude/refactor/OPEN_QUESTIONS.md: dated decision entry with
      rationale, file list, and follow-ups.

CORESYNC SUBMODULE (new)
  nodes/coresync added pointing at https://gitea.wbd-rd.nl/RnD/coresync.
  FROST/SensorThings handoff path — first version forwards FROST-ready HTTP
  request messages on the dbase output; a downstream http-request node
  performs the POST and feeds responses back on msg.topic = "frost.response".
  Lazy stream resolver, latest-wins queue (keep first + latest, drop middle),
  knot-emit on slope change, provenance preserved in Observation parameters.

    - .gitmodules: add nodes/coresync entry.
    - package.json: register coresync as a Node-RED node.
    - generalFunctions bump: new frostFormatter + 4 node config schemas
      expose the dbase format option.
    - measurement bump: "frost" option added to dbaseOutputFormat dropdown
      (plus the in-flight data.measurement unit-handling work).
    - machineGroupControl bump: small editor compact-fields tweak alongside
      the palette change.
    - CORESYNC_FROST_INTERVIEW_HANDOFF.md added at root with interview state
      (Q20 open: slope angle vs. relative delta comparison).

DASHBOARDAPI MODULE_NOT_FOUND FIX
  package.json: dashboardapi entry path corrected to
  nodes/dashboardAPI/dashboardAPI.js. Commit e04c4a1 renamed the files to
  camelCase but missed package.json; on case-sensitive filesystems
  (Linux/Docker, where the tarball lands) the require resolved to nothing
  and the node showed MODULE_NOT_FOUND in the Node-RED palette.

MISC CLEANUP
  - examples/README.md + examples/pumpingstation-complete-example/ removal
    (build_flow.py, flow.json, README.md superseded by per-node examples).
  - jest.config.js: in-progress tweak.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
znetsixe
2026-05-21 15:09:33 +02:00
parent 1a9d0477bf
commit 025bdb4c7e
27 changed files with 495 additions and 7643 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
examples/README.md
View File

@@ -1,53 +0,0 @@
# EVOLV — End-to-End Example Flows
> **Working with these examples?** See [`WORKFLOW.md`](WORKFLOW.md) — the canonical guide for editing, switching projects, persistence, and debugging.
Demo flows that show how multiple EVOLV nodes work together in a realistic wastewater-automation scenario. Each example is self-contained: its folder has a `flow.json` you can import directly into Node-RED plus a `README.md` that walks through the topology, control modes, and dashboard layout.
These flows complement the per-node example flows under `nodes/<name>/examples/` (which exercise a single node in isolation). Use the per-node flows for smoke tests during development; use the flows here when you want to see how a real plant section behaves end-to-end.
## Catalogue
| Folder | What it shows |
|---|---|
| [`pumpingstation-complete-example/`](pumpingstation-complete-example/) | End-to-end stack: pumpingStation + MGC + 3 pumps + 12 measurement nodes (4 per pump, physics-coupled), operator-driven inflow with scenario buttons (Constant / Sine / Diurnal / Storm), FlowFuse dashboard (realtime + 1h trends), and provisioned Grafana dashboard backed by InfluxDB. |
## How it loads
Each subfolder here is a **Node-RED project**. The Docker stack has Node-RED's Projects feature enabled and bootstraps each `examples/<name>/` into `/data/projects/<name>/` on first container start.
To run:
1. `docker compose up -d` from the EVOLV root.
2. Open Node-RED at `http://localhost:1880`.
3. Menu → **Projects** → **Open Project** → pick one.
4. Open the FlowFuse dashboard at `http://localhost:1880/dashboard`.
The default active project is `pumpingstation-complete-example` (override via `DEFAULT_PROJECT` env var on the nodered service). Switching is two clicks; persistence is handled by the `evolv_nodered_data` named volume — `docker compose down && up` doesn't lose the active flow.
Each example uses a unique dashboard `path` so they can coexist if you load multiple in the same runtime.
## Adding new examples
When you create a new end-to-end example:
1. Make a subfolder under `examples/` named `<scenario>-<focus>`.
2. Include at least `flow.json` and `README.md`. A `build_flow.py` (or equivalent generator) is recommended so the JSON stays diff-friendly.
3. `docker compose restart nodered` — the entrypoint will bootstrap your new folder as a Node-RED project (synthesizes `package.json`, `git init`, initial commit) under `/data/projects/<name>/`.
4. Editor → Projects → Open Project → pick your new one.
5. Add a row to the catalogue table above.
The bootstrap skips folders that already exist in the volume. To force a refresh of an existing project from the repo source (e.g. after editing `build_flow.py`), use `./scripts/sync-example.sh <name>`.
## Wishlist for future examples
These are scenarios worth building when there's a session for it:
- **Pump failure + MGC re-routing** — kill pump 2 mid-run, watch MGC redistribute to pumps 1 and 3.
- **Energy-optimal vs equal-flow control** — same demand profile run through `optimalcontrol` and `prioritycontrol` modes side-by-side, energy comparison chart.
- **Schedule-driven demand** — diurnal flow pattern (low at night, peak at 7 am), MGC auto-tuning over 24 simulated hours.
- **Reactor + clarifier loop** — `reactor` upstream feeding `settler`, return sludge controlled by a small `pumpingStation`.
- **Diffuser + DO control** — aeration grid driven by a PID controller from a dissolved-oxygen sensor.
- **Digital sensor bundle** — MQTT-style sensor (BME280, ATAS, etc.) feeding a `measurement` node in digital mode + parent equipment node.
- **Maintenance window** — entermaintenance / exitmaintenance cycle with operator handover dashboard.
- **Calibration walk-through** — measurement node calibrate cycle with stable / unstable input demonstrations.

195
examples/pumpingstation-complete-example/README.md
View File

@@ -1,195 +0,0 @@
# Pumping Station — Complete Example
End-to-end EVOLV stack: 1 pumpingStation + 1 machineGroupControl + 3 rotatingMachine pumps + 12 measurement nodes (4 per pump), wired through Node-RED to InfluxDB and Grafana.
This is the canonical "everything works together" demo. After any cross-node refactor, run this and verify the Node-RED dashboard, the InfluxDB writes, and the Grafana dashboard all populate.
## Quick start
```bash
cd /home/znetsixe/EVOLV
docker compose up -d
# Wait for http://localhost:1880/nodes to return 200, then:
curl -s -X POST http://localhost:1880/flows \
-H "Content-Type: application/json" \
-H "Node-RED-Deployment-Type: full" \
--data-binary @examples/pumpingstation-complete-example/flow.json
```
Then open:
- Node-RED dashboard (realtime + 1h trends): <http://localhost:1880/dashboard>
- Grafana dashboard (realtime gauges + historic graphs): <http://localhost:3000> (anonymous viewer is on; the dashboard is `EVOLV / Pumping Station (complete)`)
- InfluxDB UI: <http://localhost:8086> (user `evolv` / password `evolv-dev-pw`)
## What the flow contains
| Layer | Node(s) | Role |
|---|---|---|
| Process Cell | `pumpingStation` "Pumping Station" | Wet-well basin model. Levelbased control: drives MGC by basin level. Inflow comes from the Drivers tab; outflow is computed from the pumps. |
| Unit | `machineGroupControl` "MGC — Pump Group" | Distributes flow across the 3 pumps via `optimalcontrol`. |
| Equipment | `rotatingMachine` × 3 — Pump A / B / C | Hidrostal H05K-S03R curve. Auto by default; manual setpoint slider per pump when in `virtualControl`. |
| Control Modules | `measurement` × 12 (4 per pump) | Upstream pressure, downstream pressure, flow, power. Each pump's 4 sensors are driven by a per-pump physics function — values are physically coupled to plant state, not random. |
| Telemetry | shared `evt:tlm` link channel → http POST → InfluxDB | Every EVOLV node's port-1 payload is converted to v2 line protocol and POSTed to `telemetry` bucket. |
## Tabs
The flow is split across 5 tabs, by **concern**:
| Tab | Lives here | Why |
|---|---|---|
| 🏭 **Process Plant** | EVOLV nodes (PS, MGC, 3 pumps, 12 sensors) + per-node output formatters + per-pump physics feeders | The deployable plant model. |
| 📊 **Dashboard UI** | All `ui-*` widgets, button/setpoint wrappers, dispatch functions | Display + operator inputs. No business logic. |
| 🎛️ **Demo Drivers** | Inflow generator (Constant / Sine / Diurnal / Storm) + 1Hz tick | Inflow is operator-driven via slider + scenario buttons. Outflow is implicit (the pumps drain the basin). |
| ⚙️ **Setup & Init** | One-shot `once: true` injects (MGC scaling/mode, pumps mode, initial inflow scenario) | Runs at deploy time only. |
| 📈 **Telemetry** | link-in `evt:tlm` → line-protocol function → http POST | InfluxDB writer. |
Cross-tab wiring uses **named link-out / link-in pairs**, never direct cross-tab wires.
### Channel contract
| Channel | Direction | What it carries |
|---|---|---|
| `cmd:inflow-baseline` | UI → Drivers | numeric m³/h baseline |
| `cmd:inflow-scenario` | UI → Drivers | `'constant' \| 'sine' \| 'diurnal' \| 'storm'` |
| `cmd:q_in` | Drivers → process | computed inflow in m³/s |
| `cmd:Qd` | UI → process | manual demand m³/h (manual mode only) |
| `cmd:ps-mode` | UI → process | `'levelbased' \| 'manual'` |
| `cmd:mode` | Setup → process | per-pump `setMode` broadcast |
| `cmd:station-startup / -shutdown / -estop` | UI → process | station-wide command, fanned to all 3 pumps |
| `cmd:setpoint-A / -B / -C` | UI → process | per-pump setpoint slider value |
| `cmd:pump-A-seq / -B-seq / -C-seq` | UI → process | per-pump start/stop |
| `evt:pump-A / -B / -C` | process → UI | formatted per-pump status |
| `evt:mgc` | process → UI | MGC totals |
| `evt:ps` | process → UI | basin state, level, fill |
| `evt:inflow` | Drivers → UI | live inflow value + active scenario |
| `evt:tlm` | every EVOLV node → Telemetry | port-1 payload in `{measurement, fields, tags}` shape |
| `setup:to-mgc` | Setup → process | one-shot MGC scaling/mode init |
## Per-pump physics feeder
Each pump has a `physics_<pump>` function node on the Process Plant tab. It receives:
1. The pump's own port-0 stream (state, predicted flow, predicted power).
2. PS port-0 stream (basin level), fanned out by `ps_to_physics`.
It computes physically-coupled values for each sensor and emits them to the 4 measurement nodes:
| Sensor | Computation |
|---|---|
| Upstream pressure | `ρ g h` where `h = max(0, basinLevel − outflowLevel)`; pump suction sees the basin's hydrostatic head. |
| Downstream pressure | Idle → static head only (12 m → 1177 mbar). Running → static + flow²-scaled dynamic head (up to ~2354 mbar at q=200 m³/h). |
| Flow | Mirrors rotatingMachine's predicted flow with 1% Gaussian noise. Zero when the pump is idle. |
| Power | Mirrors rotatingMachine's predicted power with 0.5% Gaussian noise. Zero when the pump is idle. |
Gaussian noise uses a 12-uniform-sum approximation (no external libs).
## Inflow scenarios
Pick a scenario on the **Realtime** dashboard page (group "Inflow"):
| Scenario | Behaviour |
|---|---|
| Constant | `q_h = baseline` (no modulation) |
| Sine | `baseline · (1 + 0.5 · sin(2πt/240))` — period 4 min |
| Diurnal | `baseline · (1 + 0.6 · sin(2πt/480 − π/2))` — period 8 min, peak offset |
| Storm | 4-min cycle: rapid 5× ramp, then linear decay back to baseline |
Slider sets `baseline` in m³/h (0–250). The generator emits `q_in` to PS every second.
## Dashboard map
### Node-RED — `/dashboard`
Realtime page (`/dashboard/realtime`):
1. Inflow — slider, 4 scenario buttons, live value + active scenario label
2. Station mode + commands — Auto/Manual switch, manual Qd slider, Start All / Stop All / Emergency Stop
3. Basin realtime — direction, level, volume, fill %, net flow, time-to-full/empty, inflow, outflow, safety state, gauges (level + fill)
4. MGC — total flow + power (text + gauges), efficiency
5. Pump A / B / C — state, mode, controller %, flow, power, up/dn pressure (text), setpoint slider, Startup / Shutdown buttons
Trends page (`/dashboard/trends`) — 1-hour rolling windows:
- Basin level + fill %
- Inflow / Outflow / Per-pump flow (one chart, multi-series)
- Per-pump power
- Per-pump up/dn pressure
### Grafana — `EVOLV / Pumping Station (complete)`
Two rows:
- **Realtime** — gauges for basin level + fill, stat panels for total flow / total power / per-pump state.
- **Historic** — line charts for level + fill, inflow/outflow/net, per-pump flow + power (predicted), per-pump pressure, per-pump sensor flow + power (measured).
Default time range: last 15 minutes. Adjust with the Grafana picker for longer history.
## Verification
```bash
# 1. Bring up the stack
docker compose up -d
sleep 10 # wait for Node-RED ready
# 2. Deploy the flow
curl -s -X POST http://localhost:1880/flows \
-H 'Content-Type: application/json' \
-H 'Node-RED-Deployment-Type: full' \
--data-binary @examples/pumpingstation-complete-example/flow.json | jq .
# 3. Quick sanity check on Influx writes
curl -s -X POST 'http://localhost:8086/api/v2/query?org=evolv' \
-H 'Authorization: Token evolv-dev-token' \
-H 'Accept: application/csv' \
-H 'Content-type: application/vnd.flux' \
--data 'from(bucket:"telemetry") |> range(start: -1m) |> count() |> group(columns: ["_measurement"])'
```
You should see counts per measurement (`Pumping Station`, `Pump A`, `MGC — Pump Group`, the per-pump sensors, …) growing in real time.
## Regenerating `flow.json`
`flow.json` is generated from `build_flow.py`. Edit the Python (cleaner diff) and regenerate:
```bash
cd examples/pumpingstation-complete-example
python3 build_flow.py > flow.json
```
The Python is the source of truth.
After regenerating, push the new flow into the running runtime:
```bash
./scripts/sync-example.sh pumpingstation-complete-example
```
## Projects + persistence (Node-RED)
The Docker stack uses a named volume (`evolv_nodered_data`) for `/data`, and Node-RED's **Projects** feature is enabled. Each folder under `examples/` is bootstrapped into `/data/projects/<name>/` on first container start with its own `git init` and a synthesized `package.json`. Switching between projects is two clicks in the editor: **menu → Projects → Open Project**.
| What you do | Where it lives | What persists |
|---|---|---|
| `docker compose down && up` | Container is recreated; named volume survives | Active flow + project list survive |
| Edit a flow in the Node-RED editor | `/data/projects/<name>/flow.json` (in volume) | Until `docker compose down -v` |
| Edit `examples/<name>/build_flow.py` then regenerate | `examples/<name>/flow.json` (in repo) | Always — it's in Git |
| Run `scripts/sync-example.sh <name>` | Copies repo's `flow.json` → volume's project + reloads | Volume copy now matches repo |
### Adding a new example as a project
1. Create `examples/<your-name>/flow.json` (build it however you like — `build_flow.py` is one way).
2. Restart the Node-RED container: `docker compose restart nodered`.
3. Editor → Projects → Open Project → pick `<your-name>`.
The bootstrap is idempotent: existing projects in the volume aren't overwritten. To force a refresh from the repo: delete the project in the volume (`docker exec evolv-nodered rm -rf /data/projects/<name>`) and restart, or use `scripts/sync-example.sh` for a flow-only refresh.
To start fresh (wipe all volume state including flows, sessions, project history): `docker compose down -v`.
## Notable design choices
- **PS in `levelbased` mode** with `manual` mode toggleable from the UI. Levelbased = PS commands MGC by basin level; manual = operator drives MGC via the Qd slider.
- **Inflow is operator-driven**, outflow is implicit (computed from pump activity). Single steerable knob (the Inflow group) keeps the demo focused.
- **Sensors driven externally**, not by the measurement node's built-in simulator. The physics feeder is a function node on the Process Plant tab — disable it and sensors freeze, which is a useful failure mode to demonstrate.
- **All EVOLV port 1 → one shared telemetry channel** (`evt:tlm`) → one writer. Adding a new EVOLV node anywhere in the flow only needs a new `lout_tlm_<id>` link-out + appending the id to `_all_tlm_lout_ids()` in `build_flow.py`.
- **Dashboard pages split by concern, not data**: realtime widgets never share a page with historical charts.

1910
examples/pumpingstation-complete-example/build_flow.py
View File

File diff suppressed because it is too large Load Diff

5462
examples/pumpingstation-complete-example/flow.json
View File

File diff suppressed because it is too large Load Diff