docs: bump valve submodule pointer — wiki Home.md FSM + config rewrite
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
znetsixe
89
wiki/Archive/SCHEMA.md
Normal file
89
wiki/Archive/SCHEMA.md
Normal file
@@ -0,0 +1,89 @@
|
||||
# Project Wiki Schema
|
||||
|
||||
## Purpose
|
||||
LLM-maintained knowledge base for this project. The LLM writes and maintains everything. You read it (ideally in Obsidian). Knowledge compounds across sessions instead of being lost in chat history.
|
||||
|
||||
## Directory Structure
|
||||
```
|
||||
wiki/
|
||||
SCHEMA.md — this file (how to maintain the wiki)
|
||||
index.md — catalog of all pages with one-line summaries
|
||||
log.md — chronological record of updates
|
||||
overview.md — project overview and current status
|
||||
metrics.md — all numbers with provenance
|
||||
knowledge-graph.yaml — structured data, machine-queryable
|
||||
tools/ — search, lint, query scripts
|
||||
concepts/ — core ideas and mechanisms
|
||||
architecture/ — design decisions, system internals
|
||||
findings/ — honest results (what worked AND what didn't)
|
||||
sessions/ — per-session summaries
|
||||
```
|
||||
|
||||
## Page Conventions
|
||||
|
||||
### Frontmatter
|
||||
Every page starts with YAML frontmatter:
|
||||
```yaml
|
||||
---
|
||||
title: Page Title
|
||||
created: YYYY-MM-DD
|
||||
updated: YYYY-MM-DD
|
||||
status: proven | disproven | evolving | speculative
|
||||
tags: [tag1, tag2]
|
||||
sources: [path/to/file.py, commit abc1234]
|
||||
---
|
||||
```
|
||||
|
||||
### Status values
|
||||
- **proven**: tested and verified with evidence
|
||||
- **disproven**: tested and honestly shown NOT to work (document WHY)
|
||||
- **evolving**: partially working, boundary not fully mapped
|
||||
- **speculative**: proposed but not yet tested
|
||||
|
||||
### Cross-references
|
||||
Use `[[Page Name]]` Obsidian-style wikilinks.
|
||||
|
||||
### Contradictions
|
||||
When new evidence contradicts a prior claim, DON'T delete the old claim. Add:
|
||||
```
|
||||
> [!warning] Superseded
|
||||
> This was shown to be incorrect on YYYY-MM-DD. See [[New Finding]].
|
||||
```
|
||||
|
||||
### Honesty rule
|
||||
If something doesn't work, say so. If a result was a false positive, document how it was discovered. The wiki must be trustworthy.
|
||||
|
||||
## Operations
|
||||
|
||||
### Ingest (after a session or new source)
|
||||
1. Read outputs, commits, findings
|
||||
2. Update relevant pages
|
||||
3. Create new pages for new concepts
|
||||
4. Update `index.md`, `log.md`, `knowledge-graph.yaml`
|
||||
5. Check for contradictions with existing pages
|
||||
|
||||
### Query
|
||||
1. Use `python3 wiki/tools/query.py` for structured lookup
|
||||
2. Use `wiki/tools/search.sh` for full-text
|
||||
3. Read `index.md` to find relevant pages
|
||||
4. File valuable answers back into the wiki
|
||||
|
||||
### Lint (periodically)
|
||||
```bash
|
||||
bash wiki/tools/lint.sh
|
||||
```
|
||||
Checks: orphan pages, broken wikilinks, missing frontmatter, index completeness.
|
||||
|
||||
## Data Layer
|
||||
|
||||
- `knowledge-graph.yaml` — structured YAML with every metric and data point
|
||||
- `metrics.md` — human-readable dashboard
|
||||
- When adding new results, update BOTH the wiki page AND the knowledge graph
|
||||
- The knowledge graph is the single source of truth for numbers
|
||||
|
||||
## Source of Truth Hierarchy
|
||||
1. **Test results** (actual outputs) — highest authority
|
||||
2. **Code** (current state) — second authority
|
||||
3. **Knowledge graph** (knowledge-graph.yaml) — structured metrics
|
||||
4. **Wiki pages** — synthesis, may lag
|
||||
5. **Chat/memory** — ephemeral, may be stale
|
||||
56
wiki/Archive/architecture-3d-pump-curves.md
Normal file
56
wiki/Archive/architecture-3d-pump-curves.md
Normal file
@@ -0,0 +1,56 @@
|
||||
---
|
||||
title: 3D Pump Curve Architecture
|
||||
created: 2026-04-07
|
||||
updated: 2026-04-07
|
||||
status: proven
|
||||
tags: [predict, curves, interpolation, rotatingMachine]
|
||||
sources: [nodes/generalFunctions/src/predict/predict_class.js, nodes/rotatingMachine/src/specificClass.js]
|
||||
---
|
||||
|
||||
# 3D Pump Curve Prediction
|
||||
|
||||
## Data Structure
|
||||
A family of 2D curves indexed by pressure (f-dimension):
|
||||
- **X-axis**: control position (0-100%)
|
||||
- **Y-axis**: flow (nq) or power (np) in canonical units
|
||||
- **F-dimension**: pressure (Pa) — the 3rd dimension
|
||||
|
||||
Raw curves are in curve units (m3/h, kW, mbar). `_normalizeMachineCurve()` converts to canonical (m3/s, W, Pa).
|
||||
|
||||
## Interpolation
|
||||
Monotonic cubic spline (Fritsch-Carlson) in both dimensions:
|
||||
- **X-Y splines**: at each discrete pressure level
|
||||
- **F-splines**: across pressure levels for intermediate pressure interpolation
|
||||
|
||||
## Prediction Flow
|
||||
```
|
||||
predict.y(x):
|
||||
1. Clamp x to [currentFxyXMin, currentFxyXMax]
|
||||
2. Normalize x to [normMin, normMax]
|
||||
3. Evaluate spline at normalized x for current fDimension
|
||||
4. Return y in canonical units (m3/s or W)
|
||||
```
|
||||
|
||||
## Unit Conversion Chain
|
||||
```
|
||||
Raw curve (m3/h, kW, mbar)
|
||||
→ _normalizeMachineCurve → canonical (m3/s, W, Pa)
|
||||
→ predict class → canonical output
|
||||
→ MeasurementContainer.getCurrentValue(outputUnit) → output units
|
||||
```
|
||||
|
||||
No double-conversion. Clean separation: specificClass handles units, predict handles normalization/interpolation.
|
||||
|
||||
## Three Predict Instances per Machine
|
||||
- `predictFlow`: control % → flow (nq curve)
|
||||
- `predictPower`: control % → power (np curve)
|
||||
- `predictCtrl`: flow → control % (reversed nq curve)
|
||||
|
||||
## Boundary Behavior
|
||||
- Below/above curve X range: flat extrapolation (clamped)
|
||||
- Below/above f-dimension range: clamped to min/max pressure level
|
||||
|
||||
## Performance
|
||||
- `y(x)`: O(log n), effectively O(1) for 5-10 data points
|
||||
- `buildAllFxyCurves`: sub-10ms for typical curves
|
||||
- Full caching of normalized curves, splines, and calculated curves
|
||||
278
wiki/Archive/architecture-deployment-blueprint.md
Normal file
278
wiki/Archive/architecture-deployment-blueprint.md
Normal file
@@ -0,0 +1,278 @@
|
||||
---
|
||||
title: EVOLV Deployment Blueprint
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [deployment, docker, edge, site, central]
|
||||
---
|
||||
|
||||
# EVOLV Deployment Blueprint
|
||||
|
||||
## Purpose
|
||||
|
||||
This document turns the current EVOLV architecture into a concrete deployment model.
|
||||
|
||||
It focuses on:
|
||||
|
||||
- target infrastructure layout
|
||||
- container/service topology
|
||||
- environment and secret boundaries
|
||||
- rollout order from edge to site to central
|
||||
|
||||
It is the local source document behind the wiki deployment pages.
|
||||
|
||||
## 1. Deployment Principles
|
||||
|
||||
- edge-first operation: plant logic must continue when central is unavailable
|
||||
- site mediation: site services protect field systems and absorb plant-specific complexity
|
||||
- central governance: external APIs, analytics, IAM, CI/CD, and shared dashboards terminate centrally
|
||||
- layered telemetry: InfluxDB exists where operationally justified at edge, site, and central
|
||||
- configuration authority: `tagcodering` should become the source of truth for configuration
|
||||
- secrets hygiene: tracked manifests contain variables only; secrets live in server-side env or secret stores
|
||||
|
||||
## 2. Layered Deployment Model
|
||||
|
||||
### 2.1 Edge node
|
||||
|
||||
Purpose:
|
||||
|
||||
- interface with PLCs and field assets
|
||||
- execute local Node-RED logic
|
||||
- retain local telemetry for resilience and digital-twin use cases
|
||||
|
||||
Recommended services:
|
||||
|
||||
- `evolv-edge-nodered`
|
||||
- `evolv-edge-influxdb`
|
||||
- optional `evolv-edge-grafana`
|
||||
- optional `evolv-edge-broker`
|
||||
|
||||
Should not host:
|
||||
|
||||
- public API ingress
|
||||
- central IAM
|
||||
- source control or CI/CD
|
||||
|
||||
### 2.2 Site node
|
||||
|
||||
Purpose:
|
||||
|
||||
- aggregate one or more edge nodes
|
||||
- host plant-local dashboards and engineering visibility
|
||||
- mediate traffic between edge and central
|
||||
|
||||
Recommended services:
|
||||
|
||||
- `evolv-site-nodered` or `coresync-site`
|
||||
- `evolv-site-influxdb`
|
||||
- `evolv-site-grafana`
|
||||
- optional `evolv-site-broker`
|
||||
|
||||
### 2.3 Central platform
|
||||
|
||||
Purpose:
|
||||
|
||||
- fleet-wide analytics
|
||||
- API and integration ingress
|
||||
- engineering lifecycle and releases
|
||||
- identity and governance
|
||||
|
||||
Recommended services:
|
||||
|
||||
- reverse proxy / ingress
|
||||
- API gateway
|
||||
- IAM
|
||||
- central InfluxDB
|
||||
- central Grafana
|
||||
- Gitea
|
||||
- CI/CD runner/controller
|
||||
- optional broker for asynchronous site/central workflows
|
||||
- configuration services over `tagcodering`
|
||||
|
||||
## 3. Target Container Topology
|
||||
|
||||
### 3.1 Edge host
|
||||
|
||||
Minimum viable edge stack:
|
||||
|
||||
```text
|
||||
edge-host-01
|
||||
- Node-RED
|
||||
- InfluxDB
|
||||
- optional Grafana
|
||||
```
|
||||
|
||||
Preferred production edge stack:
|
||||
|
||||
```text
|
||||
edge-host-01
|
||||
- Node-RED
|
||||
- InfluxDB
|
||||
- local health/export service
|
||||
- optional local broker
|
||||
- optional local dashboard service
|
||||
```
|
||||
|
||||
### 3.2 Site host
|
||||
|
||||
Minimum viable site stack:
|
||||
|
||||
```text
|
||||
site-host-01
|
||||
- Site Node-RED / CoreSync
|
||||
- Site InfluxDB
|
||||
- Site Grafana
|
||||
```
|
||||
|
||||
Preferred production site stack:
|
||||
|
||||
```text
|
||||
site-host-01
|
||||
- Site Node-RED / CoreSync
|
||||
- Site InfluxDB
|
||||
- Site Grafana
|
||||
- API relay / sync service
|
||||
- optional site broker
|
||||
```
|
||||
|
||||
### 3.3 Central host group
|
||||
|
||||
Central should not be one giant undifferentiated host forever. It should trend toward at least these responsibility groups:
|
||||
|
||||
```text
|
||||
central-ingress
|
||||
- reverse proxy
|
||||
- API gateway
|
||||
- IAM
|
||||
|
||||
central-observability
|
||||
- central InfluxDB
|
||||
- Grafana
|
||||
|
||||
central-engineering
|
||||
- Gitea
|
||||
- CI/CD
|
||||
- deployment orchestration
|
||||
|
||||
central-config
|
||||
- tagcodering-backed config services
|
||||
```
|
||||
|
||||
For early rollout these may be colocated, but the responsibility split should remain clear.
|
||||
|
||||
## 4. Compose Strategy
|
||||
|
||||
The current repository shows:
|
||||
|
||||
- `docker-compose.yml` as a development stack
|
||||
- `temp/cloud.yml` as a broad central-stack example
|
||||
|
||||
For production, EVOLV should not rely on one flat compose file for every layer.
|
||||
|
||||
Recommended split:
|
||||
|
||||
- `compose.edge.yml`
|
||||
- `compose.site.yml`
|
||||
- `compose.central.yml`
|
||||
- optional overlay files for site-specific differences
|
||||
|
||||
Benefits:
|
||||
|
||||
- clearer ownership per layer
|
||||
- smaller blast radius during updates
|
||||
- easier secret and env separation
|
||||
- easier rollout per site
|
||||
|
||||
## 5. Environment And Secrets Strategy
|
||||
|
||||
### 5.1 Current baseline
|
||||
|
||||
`temp/cloud.yml` now uses environment variables instead of inline credentials. That is the minimum acceptable baseline.
|
||||
|
||||
### 5.2 Recommended production rule
|
||||
|
||||
- tracked compose files contain `${VARIABLE}` placeholders only
|
||||
- real secrets live in server-local `.env` files or a managed secret store
|
||||
- no shared default production passwords in git
|
||||
- separate env files per layer and per environment
|
||||
|
||||
Suggested structure:
|
||||
|
||||
```text
|
||||
/opt/evolv/
|
||||
compose.edge.yml
|
||||
compose.site.yml
|
||||
compose.central.yml
|
||||
env/
|
||||
edge.env
|
||||
site.env
|
||||
central.env
|
||||
```
|
||||
|
||||
## 6. Recommended Network Flow
|
||||
|
||||
### 6.1 Northbound
|
||||
|
||||
- edge publishes or syncs upward to site
|
||||
- site aggregates and forwards selected data to central
|
||||
- central exposes APIs and dashboards to approved consumers
|
||||
|
||||
### 6.2 Southbound
|
||||
|
||||
- central issues advice, approved config, or mediated requests
|
||||
- site validates and relays to edge where appropriate
|
||||
- edge remains the execution point near PLCs
|
||||
|
||||
### 6.3 Forbidden direct path
|
||||
|
||||
- enterprise or internet clients should not directly query PLC-connected edge runtimes
|
||||
|
||||
## 7. Rollout Order
|
||||
|
||||
### Phase 1: Edge baseline
|
||||
|
||||
- deploy edge Node-RED
|
||||
- deploy local InfluxDB
|
||||
- validate PLC connectivity
|
||||
- validate local telemetry and resilience
|
||||
|
||||
### Phase 2: Site mediation
|
||||
|
||||
- deploy site Node-RED / CoreSync
|
||||
- connect one or more edge nodes
|
||||
- validate site-local dashboards and outage behavior
|
||||
|
||||
### Phase 3: Central services
|
||||
|
||||
- deploy ingress, IAM, API, Grafana, central InfluxDB
|
||||
- deploy Gitea and CI/CD services
|
||||
- validate controlled northbound access
|
||||
|
||||
### Phase 4: Configuration backbone
|
||||
|
||||
- connect runtime layers to `tagcodering`
|
||||
- reduce config duplication in flows
|
||||
- formalize config promotion and rollback
|
||||
|
||||
### Phase 5: Smart telemetry policy
|
||||
|
||||
- classify signals
|
||||
- define reconstruction rules
|
||||
- define authoritative layer per horizon
|
||||
- validate analytics and auditability
|
||||
|
||||
## 8. Immediate Technical Recommendations
|
||||
|
||||
- treat `docker/settings.js` as development-only and create hardened production settings separately
|
||||
- split deployment manifests by layer
|
||||
- define env files per layer and environment
|
||||
- formalize healthchecks and backup procedures for every persistent service
|
||||
- define whether broker usage is required at edge, site, central, or only selectively
|
||||
|
||||
## 9. Next Technical Work Items
|
||||
|
||||
1. create draft `compose.edge.yml`, `compose.site.yml`, and `compose.central.yml`
|
||||
2. define server directory layout and env-file conventions
|
||||
3. define production Node-RED settings profile
|
||||
4. define site-to-central sync path
|
||||
5. define deployment and rollback runbook
|
||||
45
wiki/Archive/architecture-group-optimization.md
Normal file
45
wiki/Archive/architecture-group-optimization.md
Normal file
@@ -0,0 +1,45 @@
|
||||
---
|
||||
title: Group Optimization Architecture
|
||||
created: 2026-04-07
|
||||
updated: 2026-04-07
|
||||
status: proven
|
||||
tags: [machineGroupControl, optimization, BEP-Gravitation]
|
||||
sources: [nodes/machineGroupControl/src/specificClass.js]
|
||||
---
|
||||
|
||||
# machineGroupControl Optimization
|
||||
|
||||
## Algorithm: BEP-Gravitation + Marginal-Cost Refinement
|
||||
|
||||
### Step 1 — Pressure Equalization
|
||||
Sets all non-operational pumps to the group's max downstream / min upstream pressure. Ensures fair curve evaluation across combinations.
|
||||
|
||||
### Step 2 — Combination Enumeration
|
||||
Generates all 2^n pump subsets (n = number of machines). Filters by:
|
||||
- Machine state (excludes off, cooling, stopping, emergency)
|
||||
- Mode compatibility (`execsequence` allowed in auto)
|
||||
- Flow bounds: `sumMinFlow ≤ Qd ≤ sumMaxFlow`
|
||||
- Optional power cap
|
||||
|
||||
### Step 3 — BEP-Gravitation Distribution (per combination)
|
||||
1. **BEP seed**: `estimatedBEP = minFlow + span * NCog` per pump
|
||||
2. **Slope estimation**: samples dP/dQ at BEP ± delta (directional: slopeLeft, slopeRight)
|
||||
3. **Slope redistribution**: iteratively shifts flow from steep to flat curves (weight = 1/slope)
|
||||
4. **Marginal-cost refinement**: after slope redistribution, shifts flow from highest actual dP/dQ to lowest using real `inputFlowCalcPower` evaluations. Converges regardless of curve convexity. Max 50 iterations, typically 5-15.
|
||||
|
||||
### Step 4 — Best Selection
|
||||
Pick combination with lowest total power. Tiebreak by deviation from BEP.
|
||||
|
||||
### Step 5 — Execution
|
||||
Start/stop pumps as needed, send `flowmovement` commands in output units via `_canonicalToOutputFlow()`.
|
||||
|
||||
## Three Control Modes
|
||||
|
||||
| Mode | Distribution | Combination Selection |
|
||||
|------|-------------|----------------------|
|
||||
| optimalControl | BEP-Gravitation + refinement | exhaustive 2^n |
|
||||
| priorityControl | equal split, priority-ordered | sequential add/remove |
|
||||
| priorityPercentageControl | percentage-based, normalized | count-based |
|
||||
|
||||
## Key Design Decision
|
||||
The `flowmovement` command sends flow in the **machine's output units** (m3/h), not canonical (m3/s). The `_canonicalToOutputFlow()` helper converts before sending. Without this conversion, every pump stays at minimum flow (the critical bug fixed on 2026-04-07).
|
||||
426
wiki/Archive/architecture-node-architecture.md
Normal file
426
wiki/Archive/architecture-node-architecture.md
Normal file
@@ -0,0 +1,426 @@
|
||||
---
|
||||
title: EVOLV Architecture
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [architecture, node-red, three-layer]
|
||||
---
|
||||
|
||||
# EVOLV Architecture
|
||||
|
||||
## 1. System Overview
|
||||
|
||||
High-level view of how EVOLV fits into the wastewater treatment automation stack.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
NR[Node-RED Runtime] <-->|msg objects| EVOLV[EVOLV Nodes]
|
||||
EVOLV -->|InfluxDB line protocol| INFLUX[(InfluxDB)]
|
||||
INFLUX -->|queries| GRAFANA[Grafana Dashboards]
|
||||
EVOLV -->|process output| NR
|
||||
EVOLV -->|parent output| NR
|
||||
|
||||
style NR fill:#b22222,color:#fff
|
||||
style EVOLV fill:#0f52a5,color:#fff
|
||||
style INFLUX fill:#0c99d9,color:#fff
|
||||
style GRAFANA fill:#50a8d9,color:#fff
|
||||
```
|
||||
|
||||
Each EVOLV node produces three outputs:
|
||||
| Port | Name | Purpose |
|
||||
|------|------|---------|
|
||||
| 0 | process | Process data forwarded to downstream nodes |
|
||||
| 1 | dbase | InfluxDB-formatted measurement data |
|
||||
| 2 | parent | Control messages to parent nodes (e.g. registerChild) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Node Architecture (Three-Layer Pattern)
|
||||
|
||||
Every node follows a consistent three-layer design that separates Node-RED wiring from domain logic.
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Node-RED Runtime"
|
||||
REG["RED.nodes.registerType()"]
|
||||
end
|
||||
|
||||
subgraph "Layer 1 — Wrapper (valve.js)"
|
||||
W[wrapper .js]
|
||||
W -->|"new nodeClass(config, RED, this, name)"| NC
|
||||
W -->|MenuManager| MENU[HTTP /name/menu.js]
|
||||
W -->|configManager| CFG[HTTP /name/configData.js]
|
||||
end
|
||||
|
||||
subgraph "Layer 2 — Node Adapter (src/nodeClass.js)"
|
||||
NC[nodeClass]
|
||||
NC -->|_loadConfig| CFGM[configManager]
|
||||
NC -->|_setupSpecificClass| SC
|
||||
NC -->|_attachInputHandler| INPUT[onInput routing]
|
||||
NC -->|_startTickLoop| TICK[1s tick loop]
|
||||
NC -->|_tick → outputUtils| OUT[formatMsg]
|
||||
end
|
||||
|
||||
subgraph "Layer 3 — Domain Logic (src/specificClass.js)"
|
||||
SC[specificClass]
|
||||
SC -->|measurements| MC[MeasurementContainer]
|
||||
SC -->|state machine| ST[state]
|
||||
SC -->|hydraulics / biology| DOMAIN[domain models]
|
||||
end
|
||||
|
||||
subgraph "generalFunctions"
|
||||
GF[shared library]
|
||||
end
|
||||
|
||||
REG --> W
|
||||
GF -.->|logger, outputUtils, configManager,\nMeasurementContainer, validation, ...| NC
|
||||
GF -.->|MeasurementContainer, state,\nconvert, predict, ...| SC
|
||||
|
||||
style W fill:#0f52a5,color:#fff
|
||||
style NC fill:#0c99d9,color:#fff
|
||||
style SC fill:#50a8d9,color:#fff
|
||||
style GF fill:#86bbdd,color:#000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. generalFunctions Module Map
|
||||
|
||||
The shared library (`nodes/generalFunctions/`) provides all cross-cutting concerns.
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
GF[generalFunctions/index.js]
|
||||
|
||||
subgraph "Core Helpers (src/helper/)"
|
||||
LOGGER[logger]
|
||||
OUTPUT[outputUtils]
|
||||
CHILD[childRegistrationUtils]
|
||||
CFGUTIL[configUtils]
|
||||
ASSERT[assertionUtils]
|
||||
VALID[validationUtils]
|
||||
end
|
||||
|
||||
subgraph "Validators (src/helper/validators/)"
|
||||
TV[typeValidators]
|
||||
CV[collectionValidators]
|
||||
CURV[curveValidator]
|
||||
end
|
||||
|
||||
subgraph "Domain Modules (src/)"
|
||||
MC[MeasurementContainer]
|
||||
CFGMGR[configManager]
|
||||
MENUMGR[MenuManager]
|
||||
STATE[state]
|
||||
CONVERT[convert / Fysics]
|
||||
PREDICT[predict / interpolation]
|
||||
NRMSE[nrmse / errorMetrics]
|
||||
COOLPROP[coolprop]
|
||||
end
|
||||
|
||||
subgraph "Data (datasets/)"
|
||||
CURVES[assetData/curves]
|
||||
ASSETS[assetData/assetData.json]
|
||||
UNITS[unitData.json]
|
||||
end
|
||||
|
||||
subgraph "Constants (src/constants/)"
|
||||
POS[POSITIONS / POSITION_VALUES]
|
||||
end
|
||||
|
||||
GF --> LOGGER
|
||||
GF --> OUTPUT
|
||||
GF --> CHILD
|
||||
GF --> CFGUTIL
|
||||
GF --> ASSERT
|
||||
GF --> VALID
|
||||
VALID --> TV
|
||||
VALID --> CV
|
||||
VALID --> CURV
|
||||
GF --> MC
|
||||
GF --> CFGMGR
|
||||
GF --> MENUMGR
|
||||
GF --> STATE
|
||||
GF --> CONVERT
|
||||
GF --> PREDICT
|
||||
GF --> NRMSE
|
||||
GF --> COOLPROP
|
||||
GF --> CURVES
|
||||
GF --> POS
|
||||
|
||||
style GF fill:#0f52a5,color:#fff
|
||||
style LOGGER fill:#86bbdd,color:#000
|
||||
style OUTPUT fill:#86bbdd,color:#000
|
||||
style VALID fill:#86bbdd,color:#000
|
||||
style MC fill:#50a8d9,color:#fff
|
||||
style CFGMGR fill:#50a8d9,color:#fff
|
||||
style MENUMGR fill:#50a8d9,color:#fff
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Data Flow (Message Lifecycle)
|
||||
|
||||
Sequence diagram showing a typical input message and the periodic tick output cycle.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant NR as Node-RED
|
||||
participant W as wrapper.js
|
||||
participant NC as nodeClass
|
||||
participant SC as specificClass
|
||||
participant OU as outputUtils
|
||||
|
||||
Note over W: Node startup
|
||||
W->>NC: new nodeClass(config, RED, node, name)
|
||||
NC->>NC: _loadConfig (configManager.buildConfig)
|
||||
NC->>SC: new specificClass(config, stateConfig, options)
|
||||
NC->>NR: send([null, null, {topic: registerChild}])
|
||||
|
||||
Note over NC: Every 1 second (tick loop)
|
||||
NC->>SC: getOutput()
|
||||
SC-->>NC: raw measurement data
|
||||
NC->>OU: formatMsg(raw, config, 'process')
|
||||
NC->>OU: formatMsg(raw, config, 'influxdb')
|
||||
NC->>NR: send([processMsg, influxMsg])
|
||||
|
||||
Note over NR: Incoming control message
|
||||
NR->>W: msg {topic: 'execMovement', payload: {...}}
|
||||
W->>NC: onInput(msg)
|
||||
NC->>SC: handleInput(source, action, setpoint)
|
||||
SC->>SC: update state machine & measurements
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Node Types
|
||||
|
||||
| Node | S88 Level | Purpose |
|
||||
|------|-----------|---------|
|
||||
| **measurement** | Control Module | Generic measurement point — reads, validates, and stores sensor values |
|
||||
| **valve** | Control Module | Valve simulation with hydraulic model, position control, flow/pressure prediction |
|
||||
| **rotatingMachine** | Control Module | Pumps, blowers, mixers — rotating equipment with speed control and efficiency curves |
|
||||
| **diffuser** | Control Module | Aeration diffuser — models oxygen transfer and pressure drop |
|
||||
| **settler** | Equipment | Sludge settler — models settling behavior and sludge blanket |
|
||||
| **reactor** | Equipment | Hydraulic tank and biological process simulator (activated sludge, digestion) |
|
||||
| **monster** | Equipment | MONitoring and STrEam Routing — complex measurement aggregation |
|
||||
| **pumpingStation** | Unit | Coordinates multiple pumps as a pumping station |
|
||||
| **valveGroupControl** | Unit | Manages multiple valves as a coordinated group — distributes flow, monitors pressure |
|
||||
| **machineGroupControl** | Unit | Group control for rotating machines — load balancing and sequencing |
|
||||
| **dashboardAPI** | Utility | Exposes data and unit conversion endpoints for external dashboards |
|
||||
# EVOLV Architecture
|
||||
|
||||
## Node Hierarchy (S88)
|
||||
|
||||
EVOLV follows the ISA-88 (S88) batch control standard. Each node maps to an S88 level and uses a consistent color scheme in the Node-RED editor.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
classDef area fill:#0f52a5,color:#fff,stroke:#0a3d7a
|
||||
classDef processCell fill:#0c99d9,color:#fff,stroke:#0977aa
|
||||
classDef unit fill:#50a8d9,color:#fff,stroke:#3d89b3
|
||||
classDef equipment fill:#86bbdd,color:#000,stroke:#6a9bb8
|
||||
classDef controlModule fill:#a9daee,color:#000,stroke:#87b8cc
|
||||
classDef standalone fill:#f0f0f0,color:#000,stroke:#999
|
||||
|
||||
%% S88 Levels
|
||||
subgraph "S88: Area"
|
||||
PS[pumpingStation]
|
||||
end
|
||||
|
||||
subgraph "S88: Equipment"
|
||||
MGC[machineGroupControl]
|
||||
VGC[valveGroupControl]
|
||||
end
|
||||
|
||||
subgraph "S88: Control Module"
|
||||
RM[rotatingMachine]
|
||||
V[valve]
|
||||
M[measurement]
|
||||
R[reactor]
|
||||
S[settler]
|
||||
end
|
||||
|
||||
subgraph "Standalone"
|
||||
MON[monster]
|
||||
DASH[dashboardAPI]
|
||||
DIFF[diffuser - not implemented]
|
||||
end
|
||||
|
||||
%% Parent-child registration relationships
|
||||
PS -->|"accepts: measurement"| M
|
||||
PS -->|"accepts: machine"| RM
|
||||
PS -->|"accepts: machineGroup"| MGC
|
||||
PS -->|"accepts: pumpingStation"| PS2[pumpingStation]
|
||||
|
||||
MGC -->|"accepts: machine"| RM
|
||||
|
||||
RM -->|"accepts: measurement"| M2[measurement]
|
||||
RM -->|"accepts: reactor"| R
|
||||
|
||||
VGC -->|"accepts: valve"| V
|
||||
VGC -->|"accepts: machine / rotatingmachine"| RM2[rotatingMachine]
|
||||
VGC -->|"accepts: machinegroup / machinegroupcontrol"| MGC2[machineGroupControl]
|
||||
VGC -->|"accepts: pumpingstation / valvegroupcontrol"| PS3["pumpingStation / valveGroupControl"]
|
||||
|
||||
R -->|"accepts: measurement"| M3[measurement]
|
||||
R -->|"accepts: reactor"| R2[reactor]
|
||||
|
||||
S -->|"accepts: measurement"| M4[measurement]
|
||||
S -->|"accepts: reactor"| R3[reactor]
|
||||
S -->|"accepts: machine"| RM3[rotatingMachine]
|
||||
|
||||
%% Styling
|
||||
class PS,PS2,PS3 area
|
||||
class MGC,MGC2 equipment
|
||||
class VGC equipment
|
||||
class RM,RM2,RM3 controlModule
|
||||
class V controlModule
|
||||
class M,M2,M3,M4 controlModule
|
||||
class R,R2,R3 controlModule
|
||||
class S controlModule
|
||||
class MON,DASH,DIFF standalone
|
||||
```
|
||||
|
||||
### Registration Summary
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
classDef parent fill:#0c99d9,color:#fff
|
||||
classDef child fill:#a9daee,color:#000
|
||||
|
||||
PS[pumpingStation] -->|measurement| LEAF1((leaf))
|
||||
PS -->|machine| RM1[rotatingMachine]
|
||||
PS -->|machineGroup| MGC1[machineGroupControl]
|
||||
PS -->|pumpingStation| PS1[pumpingStation]
|
||||
|
||||
MGC[machineGroupControl] -->|machine| RM2[rotatingMachine]
|
||||
|
||||
VGC[valveGroupControl] -->|valve| V1[valve]
|
||||
VGC -->|source| SRC["machine, machinegroup,<br/>pumpingstation, valvegroupcontrol"]
|
||||
|
||||
RM[rotatingMachine] -->|measurement| LEAF2((leaf))
|
||||
RM -->|reactor| R1[reactor]
|
||||
|
||||
R[reactor] -->|measurement| LEAF3((leaf))
|
||||
R -->|reactor| R2[reactor]
|
||||
|
||||
S[settler] -->|measurement| LEAF4((leaf))
|
||||
S -->|reactor| R3[reactor]
|
||||
S -->|machine| RM3[rotatingMachine]
|
||||
|
||||
class PS,MGC,VGC,RM,R,S parent
|
||||
class LEAF1,LEAF2,LEAF3,LEAF4,RM1,RM2,RM3,MGC1,PS1,V1,SRC,R1,R2,R3 child
|
||||
```
|
||||
|
||||
## Node Types
|
||||
|
||||
| Node | S88 Level | softwareType | role | Accepts Children | Outputs |
|
||||
|------|-----------|-------------|------|-----------------|---------|
|
||||
| **pumpingStation** | Area | `pumpingstation` | StationController | measurement, machine (rotatingMachine), machineGroup, pumpingStation | [process, dbase, parent] |
|
||||
| **machineGroupControl** | Equipment | `machinegroupcontrol` | GroupController | machine (rotatingMachine) | [process, dbase, parent] |
|
||||
| **valveGroupControl** | Equipment | `valvegroupcontrol` | ValveGroupController | valve, machine, rotatingmachine, machinegroup, machinegroupcontrol, pumpingstation, valvegroupcontrol | [process, dbase, parent] |
|
||||
| **rotatingMachine** | Control Module | `rotatingmachine` | RotationalDeviceController | measurement, reactor | [process, dbase, parent] |
|
||||
| **valve** | Control Module | `valve` | controller | _(leaf node, no children)_ | [process, dbase, parent] |
|
||||
| **measurement** | Control Module | `measurement` | Sensor | _(leaf node, no children)_ | [process, dbase, parent] |
|
||||
| **reactor** | Control Module | `reactor` | Biological reactor | measurement, reactor (upstream chaining) | [process, dbase, parent] |
|
||||
| **settler** | Control Module | `settler` | Secondary settler | measurement, reactor (upstream), machine (return pump) | [process, dbase, parent] |
|
||||
| **monster** | Standalone | - | - | dual-parent, standalone | - |
|
||||
| **dashboardAPI** | Standalone | - | - | accepts any child (Grafana integration) | - |
|
||||
| **diffuser** | Standalone | - | - | _(not implemented)_ | - |
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Measurement Data Flow (upstream to downstream)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Sensor as measurement (sensor)
|
||||
participant Machine as rotatingMachine
|
||||
participant Group as machineGroupControl
|
||||
participant Station as pumpingStation
|
||||
|
||||
Note over Sensor: Sensor reads value<br/>(pressure, flow, level, temp)
|
||||
|
||||
Sensor->>Sensor: measurements.type(t).variant("measured").position(p).value(v)
|
||||
Sensor->>Sensor: emitter.emit("type.measured.position", eventData)
|
||||
|
||||
Sensor->>Machine: Event: "pressure.measured.upstream"
|
||||
Machine->>Machine: Store in own MeasurementContainer
|
||||
Machine->>Machine: getMeasuredPressure() -> calcFlow() -> calcPower()
|
||||
Machine->>Machine: emitter.emit("flow.predicted.downstream", eventData)
|
||||
|
||||
Machine->>Group: Event: "flow.predicted.downstream"
|
||||
Group->>Group: handlePressureChange()
|
||||
Group->>Group: Aggregate flows across all machines
|
||||
Group->>Group: Calculate group totals and efficiency
|
||||
|
||||
Machine->>Station: Event: "flow.predicted.downstream"
|
||||
Station->>Station: Store predicted flow in/out
|
||||
Station->>Station: _updateVolumePrediction()
|
||||
Station->>Station: _calcNetFlow(), _calcTimeRemaining()
|
||||
```
|
||||
|
||||
### Control Command Flow (downstream to upstream)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Station as pumpingStation
|
||||
participant Group as machineGroupControl
|
||||
participant Machine as rotatingMachine
|
||||
participant Machine2 as rotatingMachine (2)
|
||||
|
||||
Station->>Group: handleInput("parent", action, param)
|
||||
|
||||
Group->>Group: Determine scaling strategy
|
||||
Group->>Group: Calculate setpoints per machine
|
||||
|
||||
Group->>Machine: handleInput("parent", "execMovement", setpoint)
|
||||
Group->>Machine2: handleInput("parent", "execMovement", setpoint)
|
||||
|
||||
Machine->>Machine: setpoint() -> state.moveTo(pos)
|
||||
Machine->>Machine: updatePosition() -> calcFlow(), calcPower()
|
||||
Machine->>Machine: emitter.emit("flow.predicted.downstream")
|
||||
|
||||
Machine2->>Machine2: setpoint() -> state.moveTo(pos)
|
||||
Machine2->>Machine2: updatePosition() -> calcFlow(), calcPower()
|
||||
Machine2->>Machine2: emitter.emit("flow.predicted.downstream")
|
||||
```
|
||||
|
||||
### Wastewater Treatment Process Flow
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
classDef process fill:#50a8d9,color:#fff
|
||||
classDef equipment fill:#86bbdd,color:#000
|
||||
|
||||
PS_IN[pumpingStation<br/>Influent] -->|flow| R1[reactor<br/>Anoxic]
|
||||
R1 -->|effluent| R2[reactor<br/>Aerated]
|
||||
R2 -->|effluent| SET[settler]
|
||||
SET -->|effluent out| PS_OUT[pumpingStation<br/>Effluent]
|
||||
SET -->|sludge return| RM_RET[rotatingMachine<br/>Return pump]
|
||||
RM_RET -->|recirculation| R1
|
||||
|
||||
PS_IN --- MGC_IN[machineGroupControl]
|
||||
MGC_IN --- RM_IN[rotatingMachine<br/>Influent pumps]
|
||||
|
||||
class PS_IN,PS_OUT process
|
||||
class R1,R2,SET process
|
||||
class MGC_IN,RM_IN,RM_RET equipment
|
||||
```
|
||||
|
||||
### Event-Driven Communication Pattern
|
||||
|
||||
All parent-child communication uses Node.js `EventEmitter`:
|
||||
|
||||
1. **Registration**: Parent calls `childRegistrationUtils.registerChild(child, position)` which stores the child and calls the parent's `registerChild(child, softwareType)` method.
|
||||
2. **Event binding**: The parent's `registerChild()` subscribes to the child's `measurements.emitter` events (e.g., `"flow.predicted.downstream"`).
|
||||
3. **Data propagation**: When a child updates a measurement, it emits an event. The parent's listener stores the value in its own `MeasurementContainer` and runs its domain logic.
|
||||
4. **Three outputs**: Every node sends data to three Node-RED outputs: `[process, dbase, parent]` -- process data for downstream nodes, InfluxDB for persistence, and parent aggregation data.
|
||||
|
||||
### Position Convention
|
||||
|
||||
Children register with a position relative to their parent:
|
||||
- `upstream` -- before the parent in the flow direction
|
||||
- `downstream` -- after the parent in the flow direction
|
||||
- `atEquipment` -- physically located at/on the parent equipment
|
||||
158
wiki/Archive/architecture-platform-overview.md
Normal file
158
wiki/Archive/architecture-platform-overview.md
Normal file
@@ -0,0 +1,158 @@
|
||||
---
|
||||
title: EVOLV Platform Architecture
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [architecture, platform, edge-first]
|
||||
---
|
||||
|
||||
# EVOLV Platform Architecture
|
||||
|
||||
## At A Glance
|
||||
|
||||
EVOLV is not only a Node-RED package. It is a layered automation platform:
|
||||
|
||||
- edge for plant-side execution
|
||||
- site for local aggregation and resilience
|
||||
- central for coordination, analytics, APIs, and governance
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph EDGE["Edge"]
|
||||
PLC["PLC / IO"]
|
||||
ENR["Node-RED"]
|
||||
EDB["Local InfluxDB"]
|
||||
EUI["Local Monitoring"]
|
||||
end
|
||||
|
||||
subgraph SITE["Site"]
|
||||
SNR["CoreSync / Site Node-RED"]
|
||||
SDB["Site InfluxDB"]
|
||||
SUI["Site Dashboards"]
|
||||
end
|
||||
|
||||
subgraph CENTRAL["Central"]
|
||||
API["API Gateway"]
|
||||
CFG["Tagcodering"]
|
||||
CDB["Central InfluxDB"]
|
||||
CGR["Grafana"]
|
||||
INTEL["Overview Intelligence"]
|
||||
GIT["Gitea + CI/CD"]
|
||||
end
|
||||
|
||||
PLC --> ENR
|
||||
ENR --> EDB
|
||||
ENR --> EUI
|
||||
ENR <--> SNR
|
||||
EDB <--> SDB
|
||||
SNR --> SUI
|
||||
SNR <--> API
|
||||
API <--> CFG
|
||||
API --> INTEL
|
||||
SDB <--> CDB
|
||||
CDB --> CGR
|
||||
GIT --> ENR
|
||||
GIT --> SNR
|
||||
```
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. Edge-first operation
|
||||
|
||||
The edge layer must remain useful and safe when central systems are down.
|
||||
|
||||
That means:
|
||||
|
||||
- local logic remains operational
|
||||
- local telemetry remains queryable
|
||||
- local dashboards can keep working
|
||||
|
||||
### 2. Multi-level telemetry
|
||||
|
||||
InfluxDB is expected on multiple levels:
|
||||
|
||||
- local for resilience and digital-twin use
|
||||
- site for plant diagnostics
|
||||
- central for fleet analytics and advisory logic
|
||||
|
||||
### 3. Smart storage
|
||||
|
||||
Telemetry should not be stored only with naive deadband rules.
|
||||
|
||||
The target model is signal-aware:
|
||||
|
||||
- preserve critical change points
|
||||
- reduce low-information flat sections
|
||||
- allow downstream reconstruction where justified
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
SIG["Process Signal"] --> EVAL["Slope / Event Evaluation"]
|
||||
EVAL --> KEEP["Keep critical points"]
|
||||
EVAL --> REDUCE["Reduce reconstructable points"]
|
||||
KEEP --> L0["Local InfluxDB"]
|
||||
REDUCE --> L0
|
||||
L0 --> L1["Site InfluxDB"]
|
||||
L1 --> L2["Central InfluxDB"]
|
||||
```
|
||||
|
||||
### 4. Central is the safe entry point
|
||||
|
||||
External systems should enter through central APIs, not by directly calling field-edge systems.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
EXT["External Request"] --> API["Central API Gateway"]
|
||||
API --> AUTH["Auth / Policy"]
|
||||
AUTH --> SITE["Site Layer"]
|
||||
SITE --> EDGE["Edge Layer"]
|
||||
EDGE --> PLC["Field Assets"]
|
||||
|
||||
EXT -. blocked .-> EDGE
|
||||
EXT -. blocked .-> PLC
|
||||
```
|
||||
|
||||
### 5. Configuration belongs in `tagcodering`
|
||||
|
||||
The intended configuration source of truth is the database-backed `tagcodering` model:
|
||||
|
||||
- machine metadata
|
||||
- asset configuration
|
||||
- runtime-consumable configuration
|
||||
- future central/site configuration services
|
||||
|
||||
This already exists partially but still needs more work before it fully serves that role.
|
||||
|
||||
## Layer Roles
|
||||
|
||||
### Edge
|
||||
|
||||
- PLC connectivity
|
||||
- local logic
|
||||
- protocol translation
|
||||
- local telemetry buffering
|
||||
- local monitoring and digital-twin support
|
||||
|
||||
### Site
|
||||
|
||||
- aggregation of edge systems
|
||||
- local dashboards and diagnostics
|
||||
- mediation between OT and central
|
||||
- protected handoff for central requests
|
||||
|
||||
### Central
|
||||
|
||||
- enterprise/API gateway
|
||||
- fleet dashboards
|
||||
- analytics and intelligence
|
||||
- source control and CI/CD
|
||||
- configuration governance through `tagcodering`
|
||||
|
||||
## Why This Matters
|
||||
|
||||
This architecture gives EVOLV:
|
||||
|
||||
- better resilience
|
||||
- safer external integration
|
||||
- better data quality for analytics
|
||||
- a path from Node-RED package to platform
|
||||
632
wiki/Archive/architecture-stack-review.md
Normal file
632
wiki/Archive/architecture-stack-review.md
Normal file
@@ -0,0 +1,632 @@
|
||||
---
|
||||
title: EVOLV Architecture Review
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [architecture, stack, review]
|
||||
---
|
||||
|
||||
# EVOLV Architecture Review
|
||||
|
||||
## Purpose
|
||||
|
||||
This document captures:
|
||||
|
||||
- the architecture implemented in this repository today
|
||||
- the broader edge/site/central architecture shown in the drawings under `temp/`
|
||||
- the key strengths and weaknesses of that direction
|
||||
- the currently preferred target stack based on owner decisions from this review
|
||||
|
||||
It is the local staging document for a later wiki update.
|
||||
|
||||
## Evidence Used
|
||||
|
||||
Implemented stack evidence:
|
||||
|
||||
- `docker-compose.yml`
|
||||
- `docker/settings.js`
|
||||
- `docker/grafana/provisioning/datasources/influxdb.yaml`
|
||||
- `package.json`
|
||||
- `nodes/*`
|
||||
|
||||
Target-state evidence:
|
||||
|
||||
- `temp/fullStack.pdf`
|
||||
- `temp/edge.pdf`
|
||||
- `temp/CoreSync.drawio.pdf`
|
||||
- `temp/cloud.yml`
|
||||
|
||||
Owner decisions from this review:
|
||||
|
||||
- local InfluxDB is required for operational resilience
|
||||
- central acts as the advisory/intelligence and API-entry layer, not as a direct field caller
|
||||
- intended configuration authority is the database-backed `tagcodering` model
|
||||
- architecture wiki pages should be visual, not text-only
|
||||
|
||||
## 1. What Exists Today
|
||||
|
||||
### 1.1 Product/runtime layer
|
||||
|
||||
The codebase is currently a modular Node-RED package for wastewater/process automation:
|
||||
|
||||
- EVOLV ships custom Node-RED nodes for plant assets and process logic
|
||||
- nodes emit both process/control messages and telemetry-oriented outputs
|
||||
- shared helper logic lives in `nodes/generalFunctions/`
|
||||
- Grafana-facing integration exists through `dashboardAPI` and Influx-oriented outputs
|
||||
|
||||
### 1.2 Implemented development stack
|
||||
|
||||
The concrete development stack in this repository is:
|
||||
|
||||
- Node-RED
|
||||
- InfluxDB 2.x
|
||||
- Grafana
|
||||
|
||||
That gives a clear local flow:
|
||||
|
||||
1. EVOLV logic runs in Node-RED.
|
||||
2. Telemetry is emitted in a time-series-oriented shape.
|
||||
3. InfluxDB stores the telemetry.
|
||||
4. Grafana renders operational dashboards.
|
||||
|
||||
### 1.3 Existing runtime pattern in the nodes
|
||||
|
||||
A recurring EVOLV pattern is:
|
||||
|
||||
- output 0: process/control message
|
||||
- output 1: Influx/telemetry message
|
||||
- output 2: registration/control plumbing where relevant
|
||||
|
||||
So even in its current implemented form, EVOLV is not only a Node-RED project. It is already a control-plus-observability platform, with Node-RED as orchestration/runtime and InfluxDB/Grafana as telemetry and visualization services.
|
||||
|
||||
## 2. What The Drawings Describe
|
||||
|
||||
Across `temp/fullStack.pdf` and `temp/CoreSync.drawio.pdf`, the intended platform is broader and layered.
|
||||
|
||||
### 2.1 Edge / OT layer
|
||||
|
||||
The drawings consistently place these capabilities at the edge:
|
||||
|
||||
- PLC / OPC UA connectivity
|
||||
- Node-RED container as protocol translator and logic runtime
|
||||
- local broker in some variants
|
||||
- local InfluxDB / Prometheus style storage in some variants
|
||||
- local Grafana/SCADA in some variants
|
||||
|
||||
This is the plant-side operational layer.
|
||||
|
||||
### 2.2 Site / local server layer
|
||||
|
||||
The CoreSync drawings also show a site aggregation layer:
|
||||
|
||||
- RWZI-local server
|
||||
- Node-RED / CoreSync services
|
||||
- site-local broker
|
||||
- site-local database
|
||||
- upward API-based synchronization
|
||||
|
||||
This layer decouples field assets from central services and absorbs plant-specific complexity.
|
||||
|
||||
### 2.3 Central / cloud layer
|
||||
|
||||
The broader stack drawings and `temp/cloud.yml` show a central platform layer with:
|
||||
|
||||
- Gitea
|
||||
- Jenkins
|
||||
- reverse proxy / ingress
|
||||
- Grafana
|
||||
- InfluxDB
|
||||
- Node-RED
|
||||
- RabbitMQ / messaging
|
||||
- VPN / tunnel concepts
|
||||
- Keycloak in the drawing
|
||||
- Portainer in the drawing
|
||||
|
||||
This is a platform-services layer, not just an application runtime.
|
||||
|
||||
## 3. Architecture Decisions From This Review
|
||||
|
||||
These decisions now shape the preferred EVOLV target architecture.
|
||||
|
||||
### 3.1 Local telemetry is mandatory for resilience
|
||||
|
||||
Local InfluxDB is not optional. It is required so that:
|
||||
|
||||
- operations continue when central SCADA or central services are down
|
||||
- local dashboards and advanced digital-twin workflows can still consume recent and relevant process history
|
||||
- local edge/site layers can make smarter decisions without depending on round-trips to central
|
||||
|
||||
### 3.2 Multi-level InfluxDB is part of the architecture
|
||||
|
||||
InfluxDB should exist on multiple levels where it adds operational value:
|
||||
|
||||
- edge/local for resilience and near-real-time replay
|
||||
- site for plant-level history, diagnostics, and resilience
|
||||
- central for fleet-wide analytics, benchmarking, and advisory intelligence
|
||||
|
||||
This is not just copy-paste storage at each level. The design intent is event-driven and selective.
|
||||
|
||||
### 3.3 Storage should be smart, not only deadband-driven
|
||||
|
||||
The target is not simple "store every point" or only a fixed deadband rule such as 1%.
|
||||
|
||||
The desired storage approach is:
|
||||
|
||||
- observe signal slope and change behavior
|
||||
- preserve points where state is changing materially
|
||||
- store fewer points where the signal can be reconstructed downstream with sufficient fidelity
|
||||
- carry enough metadata or conventions so reconstruction quality is auditable
|
||||
|
||||
This implies EVOLV should evolve toward smart storage and signal-aware retention rather than naive event dumping.
|
||||
|
||||
### 3.4 Central is the intelligence and API-entry layer
|
||||
|
||||
Central may advise and coordinate edge/site layers, but external API requests should not hit field-edge systems directly.
|
||||
|
||||
The intended pattern is:
|
||||
|
||||
- external and enterprise integrations terminate centrally
|
||||
- central evaluates, aggregates, authorizes, and advises
|
||||
- site/edge layers receive mediated requests, policies, or setpoints
|
||||
- field-edge remains protected behind an intermediate layer
|
||||
|
||||
This aligns with the stated security direction.
|
||||
|
||||
### 3.5 Configuration source of truth should be database-backed
|
||||
|
||||
The intended configuration authority is the database-backed `tagcodering` model, which already exists but is not yet complete enough to serve as the fully realized source of truth.
|
||||
|
||||
That means the architecture should assume:
|
||||
|
||||
- asset and machine metadata belong in `tagcodering`
|
||||
- Node-RED flows should consume configuration rather than silently becoming the only configuration store
|
||||
- more work is still needed before this behaves as the intended central configuration backbone
|
||||
|
||||
## 4. Visual Model
|
||||
|
||||
### 4.1 Platform topology
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph OT["OT / Field"]
|
||||
PLC["PLC / IO"]
|
||||
DEV["Sensors / Machines"]
|
||||
end
|
||||
|
||||
subgraph EDGE["Edge Layer"]
|
||||
ENR["Edge Node-RED"]
|
||||
EDB["Local InfluxDB"]
|
||||
EUI["Local Grafana / Local Monitoring"]
|
||||
EBR["Optional Local Broker"]
|
||||
end
|
||||
|
||||
subgraph SITE["Site Layer"]
|
||||
SNR["Site Node-RED / CoreSync"]
|
||||
SDB["Site InfluxDB"]
|
||||
SUI["Site Grafana / SCADA Support"]
|
||||
SBR["Site Broker"]
|
||||
end
|
||||
|
||||
subgraph CENTRAL["Central Layer"]
|
||||
API["API / Integration Gateway"]
|
||||
INTEL["Overview Intelligence / Advisory Logic"]
|
||||
CDB["Central InfluxDB"]
|
||||
CGR["Central Grafana"]
|
||||
CFG["Tagcodering Config Model"]
|
||||
GIT["Gitea"]
|
||||
CI["CI/CD"]
|
||||
IAM["IAM / Keycloak"]
|
||||
end
|
||||
|
||||
DEV --> PLC
|
||||
PLC --> ENR
|
||||
ENR --> EDB
|
||||
ENR --> EUI
|
||||
ENR --> EBR
|
||||
ENR <--> SNR
|
||||
EDB <--> SDB
|
||||
SNR --> SDB
|
||||
SNR --> SUI
|
||||
SNR --> SBR
|
||||
SNR <--> API
|
||||
API --> INTEL
|
||||
API <--> CFG
|
||||
SDB <--> CDB
|
||||
INTEL --> SNR
|
||||
CGR --> CDB
|
||||
CI --> GIT
|
||||
IAM --> API
|
||||
IAM --> CGR
|
||||
```
|
||||
|
||||
### 4.2 Command and access boundary
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
EXT["External APIs / Enterprise Requests"] --> API["Central API Gateway"]
|
||||
API --> AUTH["AuthN/AuthZ / Policy Checks"]
|
||||
AUTH --> INTEL["Central Advisory / Decision Support"]
|
||||
INTEL --> SITE["Site Integration Layer"]
|
||||
SITE --> EDGE["Edge Runtime"]
|
||||
EDGE --> PLC["PLC / Field Assets"]
|
||||
|
||||
EXT -. no direct access .-> EDGE
|
||||
EXT -. no direct access .-> PLC
|
||||
```
|
||||
|
||||
### 4.3 Smart telemetry flow
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
RAW["Raw Signal"] --> EDGELOGIC["Edge Signal Evaluation"]
|
||||
EDGELOGIC --> KEEP["Keep Critical Change Points"]
|
||||
EDGELOGIC --> SKIP["Skip Reconstructable Flat Points"]
|
||||
EDGELOGIC --> LOCAL["Local InfluxDB"]
|
||||
LOCAL --> SITE["Site InfluxDB"]
|
||||
SITE --> CENTRAL["Central InfluxDB"]
|
||||
KEEP --> LOCAL
|
||||
SKIP -. reconstruction assumptions / metadata .-> SITE
|
||||
CENTRAL --> DASH["Fleet Dashboards / Analytics"]
|
||||
```
|
||||
|
||||
## 5. Upsides Of This Direction
|
||||
|
||||
### 5.1 Strong separation between control and observability
|
||||
|
||||
Node-RED for runtime/orchestration and InfluxDB/Grafana for telemetry is still the right structural split:
|
||||
|
||||
- control stays close to the process
|
||||
- telemetry storage/querying stays in time-series-native tooling
|
||||
- dashboards do not need to overload Node-RED itself
|
||||
|
||||
### 5.2 Edge-first matches operational reality
|
||||
|
||||
For wastewater/process systems, edge-first remains correct:
|
||||
|
||||
- lower latency
|
||||
- better degraded-mode behavior
|
||||
- less dependence on WAN or central platform uptime
|
||||
- clearer OT trust boundary
|
||||
|
||||
### 5.3 Site mediation improves safety and security
|
||||
|
||||
Using central as the enterprise/API entry point and site as the mediator improves posture:
|
||||
|
||||
- field systems are less exposed
|
||||
- policy decisions can be centralized
|
||||
- external integrations do not probe the edge directly
|
||||
- site can continue operating even when upstream is degraded
|
||||
|
||||
### 5.4 Multi-level storage enables better analytics
|
||||
|
||||
Multiple Influx layers can support:
|
||||
|
||||
- local resilience
|
||||
- site diagnostics
|
||||
- fleet benchmarking
|
||||
- smarter retention and reconstruction strategies
|
||||
|
||||
That is substantially more capable than a single central historian model.
|
||||
|
||||
### 5.5 `tagcodering` is the right long-term direction
|
||||
|
||||
A database-backed configuration authority is stronger than embedding configuration only in flows because it supports:
|
||||
|
||||
- machine metadata management
|
||||
- controlled rollout of configuration changes
|
||||
- clearer versioning and provenance
|
||||
- future API-driven configuration services
|
||||
|
||||
## 6. Downsides And Risks
|
||||
|
||||
### 6.1 Smart storage raises algorithmic and governance complexity
|
||||
|
||||
Signal-aware storage and reconstruction is promising, but it creates architectural obligations:
|
||||
|
||||
- reconstruction rules must be explicit
|
||||
- acceptable reconstruction error must be defined per signal type
|
||||
- operators must know whether they see raw or reconstructed history
|
||||
- compliance-relevant data may need stricter retention than operational convenience data
|
||||
|
||||
Without those rules, smart storage can become opaque and hard to trust.
|
||||
|
||||
### 6.2 Multi-level databases can create ownership confusion
|
||||
|
||||
If edge, site, and central all store telemetry, you must define:
|
||||
|
||||
- which layer is authoritative for which time horizon
|
||||
- when backfill is allowed
|
||||
- when data is summarized vs copied
|
||||
- how duplicates or gaps are detected
|
||||
|
||||
Otherwise operations will argue over which trend is "the real one."
|
||||
|
||||
### 6.3 Central intelligence must remain advisory-first
|
||||
|
||||
Central guidance can become valuable, but direct closed-loop dependency on central would be risky.
|
||||
|
||||
The architecture should therefore preserve:
|
||||
|
||||
- local control authority at edge/site
|
||||
- bounded and explicit central advice
|
||||
- safe behavior if central recommendations stop arriving
|
||||
|
||||
### 6.4 `tagcodering` is not yet complete enough to lean on blindly
|
||||
|
||||
It is the right target, but its current partial state means there is still architecture debt:
|
||||
|
||||
- incomplete config workflows
|
||||
- likely mismatch between desired and implemented schema behavior
|
||||
- temporary duplication between flows, node config, and database-held metadata
|
||||
|
||||
This should be treated as a core platform workstream, not a side issue.
|
||||
|
||||
### 6.5 Broker responsibilities are still not crisp enough
|
||||
|
||||
The materials still reference MQTT/AMQP/RabbitMQ/brokers without one stable responsibility split. That needs to be resolved before large-scale deployment.
|
||||
|
||||
Questions still open:
|
||||
|
||||
- command bus or event bus?
|
||||
- site-only or cross-site?
|
||||
- telemetry transport or only synchronization/eventing?
|
||||
- durability expectations and replay behavior?
|
||||
|
||||
## 7. Security And Regulatory Positioning
|
||||
|
||||
### 7.1 Purdue-style layering is a good fit
|
||||
|
||||
EVOLV's preferred structure aligns well with a Purdue-style OT/IT layering approach:
|
||||
|
||||
- PLCs and field assets stay at the operational edge
|
||||
- edge runtimes stay close to the process
|
||||
- site systems mediate between OT and broader enterprise concerns
|
||||
- central services host APIs, identity, analytics, and engineering workflows
|
||||
|
||||
That is important because it supports segmented trust boundaries instead of direct enterprise-to-field reach-through.
|
||||
|
||||
### 7.2 NIS2 alignment
|
||||
|
||||
Directive (EU) 2022/2555 (NIS2) requires cybersecurity risk-management measures, incident handling, and stronger governance for covered entities.
|
||||
|
||||
This architecture supports that by:
|
||||
|
||||
- limiting direct exposure of field systems
|
||||
- separating operational layers
|
||||
- enabling central policy and oversight
|
||||
- preserving local operation during upstream failure
|
||||
|
||||
### 7.3 CER alignment
|
||||
|
||||
Directive (EU) 2022/2557 (Critical Entities Resilience Directive) focuses on resilience of essential services.
|
||||
|
||||
The edge-plus-site approach supports that direction because:
|
||||
|
||||
- local/site layers can continue during central disruption
|
||||
- essential service continuity does not depend on one central runtime
|
||||
- degraded-mode behavior can be explicitly designed per layer
|
||||
|
||||
### 7.4 Cyber Resilience Act alignment
|
||||
|
||||
Regulation (EU) 2024/2847 (Cyber Resilience Act) creates cybersecurity requirements for products with digital elements.
|
||||
|
||||
For EVOLV, that means the platform should keep strengthening:
|
||||
|
||||
- secure configuration handling
|
||||
- vulnerability and update management
|
||||
- release traceability
|
||||
- lifecycle ownership of components and dependencies
|
||||
|
||||
### 7.5 GDPR alignment where personal data is present
|
||||
|
||||
Regulation (EU) 2016/679 (GDPR) applies whenever EVOLV processes personal data.
|
||||
|
||||
The architecture helps by:
|
||||
|
||||
- centralizing ingress
|
||||
- reducing unnecessary propagation of data to field layers
|
||||
- making access, retention, and audit boundaries easier to define
|
||||
|
||||
### 7.6 What can and cannot be claimed
|
||||
|
||||
The defensible claim is that EVOLV can be deployed in a way that supports compliance with strict European cybersecurity and resilience expectations.
|
||||
|
||||
The non-defensible claim is that EVOLV is automatically compliant purely because of the architecture diagram.
|
||||
|
||||
Actual compliance still depends on implementation and operations, including:
|
||||
|
||||
- access control
|
||||
- patch and vulnerability management
|
||||
- incident response
|
||||
- logging and audit evidence
|
||||
- retention policy
|
||||
- data classification
|
||||
|
||||
## 8. Recommended Ideal Stack
|
||||
|
||||
The ideal EVOLV stack should be layered around operational boundaries, not around tools.
|
||||
|
||||
### 7.1 Layer A: Edge execution
|
||||
|
||||
Purpose:
|
||||
|
||||
- connect to PLCs and field assets
|
||||
- execute time-sensitive local logic
|
||||
- preserve operation during WAN/central loss
|
||||
- provide local telemetry access for resilience and digital-twin use cases
|
||||
|
||||
Recommended components:
|
||||
|
||||
- Node-RED runtime for EVOLV edge flows
|
||||
- OPC UA and protocol adapters
|
||||
- local InfluxDB
|
||||
- optional local Grafana for local engineering/monitoring
|
||||
- optional local broker only when multiple participants need decoupling
|
||||
|
||||
Principle:
|
||||
|
||||
- edge remains safe and useful when disconnected
|
||||
|
||||
### 7.2 Layer B: Site integration
|
||||
|
||||
Purpose:
|
||||
|
||||
- aggregate multiple edge systems at plant/site level
|
||||
- host plant-local dashboards and diagnostics
|
||||
- mediate between raw OT detail and central standardization
|
||||
- serve as the protected step between field systems and central requests
|
||||
|
||||
Recommended components:
|
||||
|
||||
- site Node-RED / CoreSync services
|
||||
- site InfluxDB
|
||||
- site Grafana / SCADA-supporting dashboards
|
||||
- site broker where asynchronous eventing is justified
|
||||
|
||||
Principle:
|
||||
|
||||
- site absorbs plant complexity and protects field assets
|
||||
|
||||
### 7.3 Layer C: Central platform
|
||||
|
||||
Purpose:
|
||||
|
||||
- fleet-wide analytics
|
||||
- shared dashboards
|
||||
- engineering lifecycle
|
||||
- enterprise/API entry point
|
||||
- overview intelligence and advisory logic
|
||||
|
||||
Recommended components:
|
||||
|
||||
- Gitea
|
||||
- CI/CD
|
||||
- central InfluxDB
|
||||
- central Grafana
|
||||
- API/integration gateway
|
||||
- IAM
|
||||
- VPN/private connectivity
|
||||
- `tagcodering`-backed configuration services
|
||||
|
||||
Principle:
|
||||
|
||||
- central coordinates, advises, and governs; it is not the direct field caller
|
||||
|
||||
### 7.4 Cross-cutting platform services
|
||||
|
||||
These should be explicit architecture elements:
|
||||
|
||||
- secrets management
|
||||
- certificate management
|
||||
- backup/restore
|
||||
- audit logging
|
||||
- monitoring/alerting of the platform itself
|
||||
- versioned configuration and schema management
|
||||
- rollout/rollback strategy
|
||||
|
||||
## 9. Recommended Opinionated Choices
|
||||
|
||||
### 8.1 Keep Node-RED as the orchestration layer, not the whole platform
|
||||
|
||||
Node-RED should own:
|
||||
|
||||
- process orchestration
|
||||
- protocol mediation
|
||||
- edge/site logic
|
||||
- KPI production
|
||||
|
||||
It should not become the sole owner of:
|
||||
|
||||
- identity
|
||||
- long-term configuration authority
|
||||
- secret management
|
||||
- compliance/audit authority
|
||||
|
||||
### 8.2 Use InfluxDB by function and horizon
|
||||
|
||||
Recommended split:
|
||||
|
||||
- edge: resilience, local replay, digital-twin input
|
||||
- site: plant diagnostics and local continuity
|
||||
- central: fleet analytics, advisory intelligence, benchmarking, and long-term cross-site views
|
||||
|
||||
### 8.3 Prefer smart telemetry retention over naive point dumping
|
||||
|
||||
Recommended rule:
|
||||
|
||||
- keep information-rich points
|
||||
- reduce information-poor flat spans
|
||||
- document reconstruction assumptions
|
||||
- define signal-class-specific fidelity expectations
|
||||
|
||||
This needs design discipline, but it is a real differentiator if executed well.
|
||||
|
||||
### 8.4 Put enterprise/API ingress at central, not at edge
|
||||
|
||||
This should become a hard architectural rule:
|
||||
|
||||
- external requests land centrally
|
||||
- central authenticates and authorizes
|
||||
- central or site mediates downward
|
||||
- edge never becomes the exposed public integration surface
|
||||
|
||||
### 8.5 Make `tagcodering` the target configuration backbone
|
||||
|
||||
The architecture should be designed so that `tagcodering` can mature into:
|
||||
|
||||
- machine and asset registry
|
||||
- configuration source of truth
|
||||
- site/central configuration exchange point
|
||||
- API-served configuration source for runtime layers
|
||||
|
||||
## 10. Suggested Phasing
|
||||
|
||||
### Phase 1: Stabilize contracts
|
||||
|
||||
- define topic and payload contracts
|
||||
- define telemetry classes and reconstruction policy
|
||||
- define asset, machine, and site identity model
|
||||
- define `tagcodering` scope and schema ownership
|
||||
|
||||
### Phase 2: Harden local/site resilience
|
||||
|
||||
- formalize edge and site runtime patterns
|
||||
- define local telemetry retention and replay behavior
|
||||
- define central-loss behavior
|
||||
- define dashboard behavior during isolation
|
||||
|
||||
### Phase 3: Harden central platform
|
||||
|
||||
- IAM
|
||||
- API gateway
|
||||
- central observability
|
||||
- CI/CD
|
||||
- backup and disaster recovery
|
||||
- config services over `tagcodering`
|
||||
|
||||
### Phase 4: Introduce selective synchronization and intelligence
|
||||
|
||||
- event-driven telemetry propagation rules
|
||||
- smart-storage promotion/backfill policies
|
||||
- advisory services from central
|
||||
- auditability of downward recommendations and configuration changes
|
||||
|
||||
## 11. Immediate Open Questions Before Wiki Finalization
|
||||
|
||||
1. Which signals are allowed to use reconstruction-aware smart storage, and which must remain raw or near-raw for audit/compliance reasons?
|
||||
2. How should `tagcodering` be exposed to runtime layers: direct database access, a dedicated API, or both?
|
||||
3. What exact responsibility split should EVOLV use between API synchronization and broker-based eventing?
|
||||
|
||||
## 12. Recommended Wiki Structure
|
||||
|
||||
The wiki should not be one long page. It should be split into:
|
||||
|
||||
1. platform overview with the main topology diagram
|
||||
2. edge-site-central runtime model
|
||||
3. telemetry and smart storage model
|
||||
4. security and access-boundary model
|
||||
5. configuration architecture centered on `tagcodering`
|
||||
|
||||
## 13. Next Step
|
||||
|
||||
Use this document as the architecture baseline. The companion markdown page in `architecture/` can then be shaped into a wiki-ready visual overview page with Mermaid diagrams and shorter human-readable sections.
|
||||
454
wiki/Archive/concepts-generalfunctions-api.md
Normal file
454
wiki/Archive/concepts-generalfunctions-api.md
Normal file
@@ -0,0 +1,454 @@
|
||||
---
|
||||
title: generalFunctions API Reference
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [api, generalFunctions, reference]
|
||||
---
|
||||
|
||||
# generalFunctions API Reference
|
||||
|
||||
Shared library (`nodes/generalFunctions/`) used across all EVOLV Node-RED nodes.
|
||||
|
||||
```js
|
||||
const { logger, outputUtils, MeasurementContainer, ... } = require('generalFunctions');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Logger](#logger)
|
||||
2. [OutputUtils](#outpututils)
|
||||
3. [ValidationUtils](#validationutils)
|
||||
4. [MeasurementContainer](#measurementcontainer)
|
||||
5. [ConfigManager](#configmanager)
|
||||
6. [ChildRegistrationUtils](#childregistrationutils)
|
||||
7. [MenuUtils](#menuutils)
|
||||
8. [EndpointUtils](#endpointutils)
|
||||
9. [Positions](#positions)
|
||||
10. [AssetLoader / loadCurve](#assetloader--loadcurve)
|
||||
|
||||
---
|
||||
|
||||
## Logger
|
||||
|
||||
Structured, level-filtered console logger.
|
||||
|
||||
**File:** `src/helper/logger.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new Logger(logging = true, logLevel = 'debug', nameModule = 'N/A')
|
||||
```
|
||||
|
||||
| Param | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `logging` | `boolean` | `true` | Enable/disable all output |
|
||||
| `logLevel` | `string` | `'debug'` | Minimum severity: `'debug'` \| `'info'` \| `'warn'` \| `'error'` |
|
||||
| `nameModule` | `string` | `'N/A'` | Label prefixed to every message |
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Description |
|
||||
|---|---|---|
|
||||
| `debug` | `(message: string): void` | Log at DEBUG level |
|
||||
| `info` | `(message: string): void` | Log at INFO level |
|
||||
| `warn` | `(message: string): void` | Log at WARN level |
|
||||
| `error` | `(message: string): void` | Log at ERROR level |
|
||||
| `setLogLevel` | `(level: string): void` | Change minimum level at runtime |
|
||||
| `toggleLogging` | `(): void` | Flip logging on/off |
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const Logger = require('generalFunctions').logger;
|
||||
const log = new Logger(true, 'info', 'MyNode');
|
||||
log.info('Node started'); // [INFO] -> MyNode: Node started
|
||||
log.debug('ignored'); // silent (below 'info')
|
||||
log.setLogLevel('debug');
|
||||
log.debug('now visible'); // [DEBUG] -> MyNode: now visible
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## OutputUtils
|
||||
|
||||
Tracks output state and formats messages for InfluxDB or process outputs. Only emits changed fields.
|
||||
|
||||
**File:** `src/helper/outputUtils.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new OutputUtils() // no parameters
|
||||
```
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `formatMsg` | `(output, config, format)` | `object \| undefined` | Diff against last output; returns formatted msg or `undefined` if nothing changed |
|
||||
| `checkForChanges` | `(output, format)` | `object` | Returns only the key/value pairs that changed since last call |
|
||||
|
||||
**`format`** must be `'influxdb'` or `'process'`.
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const out = new OutputUtils();
|
||||
const msg = out.formatMsg(
|
||||
{ temperature: 22.5, pressure: 1013 },
|
||||
config,
|
||||
'influxdb'
|
||||
);
|
||||
// msg = { topic: 'nodeName', payload: { measurement, fields, tags, timestamp } }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ValidationUtils
|
||||
|
||||
Schema-driven config validation with type coercion, range clamping, and nested object support.
|
||||
|
||||
**File:** `src/helper/validationUtils.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new ValidationUtils(loggerEnabled = true, loggerLevel = 'warn')
|
||||
```
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `validateSchema` | `(config, schema, name)` | `object` | Walk the schema, validate every field, return a clean config. Unknown keys are stripped. Missing keys get their schema default. |
|
||||
| `constrain` | `(value, min, max)` | `number` | Clamp a numeric value to `[min, max]` |
|
||||
| `removeUnwantedKeys` | `(obj)` | `object` | Strip `rules`/`description` metadata, collapse `default` values |
|
||||
|
||||
**Supported `rules.type` values:** `number`, `integer`, `boolean`, `string`, `enum`, `array`, `set`, `object`, `curve`, `machineCurve`.
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const ValidationUtils = require('generalFunctions').validation;
|
||||
const v = new ValidationUtils(true, 'warn');
|
||||
|
||||
const schema = {
|
||||
temperature: { default: 20, rules: { type: 'number', min: -40, max: 100 } },
|
||||
unit: { default: 'C', rules: { type: 'enum', values: [{ value: 'C' }, { value: 'F' }] } }
|
||||
};
|
||||
|
||||
const validated = v.validateSchema({ temperature: 999 }, schema, 'myNode');
|
||||
// validated.temperature === 100 (clamped)
|
||||
// validated.unit === 'C' (default applied)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MeasurementContainer
|
||||
|
||||
Chainable measurement storage organised by **type / variant / position**. Supports auto unit conversion, windowed statistics, events, and positional difference calculations.
|
||||
|
||||
**File:** `src/measurements/MeasurementContainer.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new MeasurementContainer(options = {}, logger)
|
||||
```
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `windowSize` | `number` | `10` | Rolling window for statistics |
|
||||
| `defaultUnits` | `object` | `{ pressure:'mbar', flow:'m3/h', ... }` | Default unit per measurement type |
|
||||
| `autoConvert` | `boolean` | `true` | Auto-convert values to target unit |
|
||||
| `preferredUnits` | `object` | `{}` | Per-type unit overrides |
|
||||
|
||||
### Chainable Setters
|
||||
|
||||
All return `this` for chaining.
|
||||
|
||||
```js
|
||||
container
|
||||
.type('pressure')
|
||||
.variant('static')
|
||||
.position('upstream')
|
||||
.distance(5)
|
||||
.unit('bar')
|
||||
.value(3.2, Date.now(), 'bar');
|
||||
```
|
||||
|
||||
| Method | Signature | Description |
|
||||
|---|---|---|
|
||||
| `type` | `(typeName): this` | Set measurement type (e.g. `'pressure'`) |
|
||||
| `variant` | `(variantName): this` | Set variant (e.g. `'static'`, `'differential'`) |
|
||||
| `position` | `(positionValue): this` | Set position (e.g. `'upstream'`, `'downstream'`) |
|
||||
| `distance` | `(distance): this` | Set physical distance from parent |
|
||||
| `unit` | `(unitName): this` | Set unit on the underlying measurement |
|
||||
| `value` | `(val, timestamp?, sourceUnit?): this` | Store a value; auto-converts if `sourceUnit` differs from target |
|
||||
|
||||
### Terminal / Query Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `get` | `()` | `Measurement \| null` | Get the raw measurement object |
|
||||
| `getCurrentValue` | `(requestedUnit?)` | `number \| null` | Latest value, optionally converted |
|
||||
| `getAverage` | `(requestedUnit?)` | `number \| null` | Windowed average |
|
||||
| `getMin` | `()` | `number \| null` | Window minimum |
|
||||
| `getMax` | `()` | `number \| null` | Window maximum |
|
||||
| `getAllValues` | `()` | `array \| null` | All stored samples |
|
||||
| `getLaggedValue` | `(lag?, requestedUnit?)` | `number \| null` | Value from `lag` samples ago |
|
||||
| `getLaggedSample` | `(lag?, requestedUnit?)` | `object \| null` | Full sample `{ value, timestamp, unit }` from `lag` samples ago |
|
||||
| `exists` | `({ type?, variant?, position?, requireValues? })` | `boolean` | Check if a measurement series exists |
|
||||
| `difference` | `({ from?, to?, unit? })` | `object \| null` | Compute `{ value, avgDiff, unit }` between two positions |
|
||||
|
||||
### Introspection / Lifecycle
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `getTypes` | `()` | `string[]` | All registered measurement types |
|
||||
| `getVariants` | `()` | `string[]` | Variants under current type |
|
||||
| `getPositions` | `()` | `string[]` | Positions under current type+variant |
|
||||
| `getAvailableUnits` | `(measurementType?)` | `string[]` | Units available for a type |
|
||||
| `getBestUnit` | `(excludeUnits?)` | `object \| null` | Best human-readable unit for current value |
|
||||
| `setPreferredUnit` | `(type, unit)` | `this` | Override default unit for a type |
|
||||
| `setChildId` | `(id)` | `this` | Tag container with a child node ID |
|
||||
| `setChildName` | `(name)` | `this` | Tag container with a child node name |
|
||||
| `setParentRef` | `(parent)` | `this` | Store reference to parent node |
|
||||
| `clear` | `()` | `void` | Reset all measurements and chain state |
|
||||
|
||||
### Events
|
||||
|
||||
The internal `emitter` fires `"type.variant.position"` on every `value()` call with:
|
||||
|
||||
```js
|
||||
{ value, originalValue, unit, sourceUnit, timestamp, position, distance, variant, type, childId, childName, parentRef }
|
||||
```
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const { MeasurementContainer } = require('generalFunctions');
|
||||
const mc = new MeasurementContainer({ windowSize: 5 });
|
||||
|
||||
mc.type('pressure').variant('static').position('upstream').value(3.2);
|
||||
mc.type('pressure').variant('static').position('downstream').value(2.8);
|
||||
|
||||
const diff = mc.type('pressure').variant('static').difference();
|
||||
// diff = { value: -0.4, avgDiff: -0.4, unit: 'mbar', from: 'downstream', to: 'upstream' }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ConfigManager
|
||||
|
||||
Loads JSON config files from disk and builds merged runtime configs.
|
||||
|
||||
**File:** `src/configs/index.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new ConfigManager(relPath = '.')
|
||||
```
|
||||
|
||||
`relPath` is resolved relative to the configs directory.
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `getConfig` | `(configName)` | `object` | Load and parse `<configName>.json` |
|
||||
| `getAvailableConfigs` | `()` | `string[]` | List config names (without `.json`) |
|
||||
| `hasConfig` | `(configName)` | `boolean` | Check existence |
|
||||
| `getBaseConfig` | `()` | `object` | Shortcut for `getConfig('baseConfig')` |
|
||||
| `buildConfig` | `(nodeName, uiConfig, nodeId, domainConfig?)` | `object` | Merge base schema + UI overrides into a runtime config |
|
||||
| `createEndpoint` | `(nodeName)` | `string` | Generate browser JS that injects config into `window.EVOLV.nodes` |
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const { configManager } = require('generalFunctions');
|
||||
const cfg = configManager.buildConfig('measurement', uiConfig, node.id, {
|
||||
scaling: { enabled: true, inputMin: 0, inputMax: 100 }
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ChildRegistrationUtils
|
||||
|
||||
Manages parent-child node relationships: registration, lookup, and structure storage.
|
||||
|
||||
**File:** `src/helper/childRegistrationUtils.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new ChildRegistrationUtils(mainClass)
|
||||
```
|
||||
|
||||
`mainClass` is the parent node instance (must expose `.logger` and optionally `.registerChild()`).
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `registerChild` | `(child, positionVsParent, distance?)` | `Promise<any>` | Register a child node under the parent. Sets up parent refs, measurement context, and stores by softwareType/category. |
|
||||
| `getChildrenOfType` | `(softwareType, category?)` | `array` | Get children filtered by software type and optional category |
|
||||
| `getChildById` | `(childId)` | `object \| null` | Lookup a single child by its ID |
|
||||
| `getAllChildren` | `()` | `array` | All registered children |
|
||||
| `logChildStructure` | `()` | `void` | Debug-print the full child tree |
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const { childRegistrationUtils: CRU } = require('generalFunctions');
|
||||
const cru = new CRU(parentNode);
|
||||
await cru.registerChild(sensorNode, 'upstream');
|
||||
cru.getChildrenOfType('measurement'); // [sensorNode]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MenuUtils
|
||||
|
||||
Browser-side UI helper for Node-RED editor. Methods are mixed in from separate modules: toggles, data fetching, URL utils, dropdown population, and HTML generation.
|
||||
|
||||
**File:** `src/helper/menuUtils.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new MenuUtils() // no parameters; sets isCloud=false, configData=null
|
||||
```
|
||||
|
||||
### Key Methods
|
||||
|
||||
**Toggles** -- control UI element visibility:
|
||||
|
||||
| Method | Signature | Description |
|
||||
|---|---|---|
|
||||
| `initBasicToggles` | `(elements)` | Bind log-level row visibility to log checkbox |
|
||||
| `initMeasurementToggles` | `(elements)` | Bind scaling input rows to scaling checkbox |
|
||||
| `initTensionToggles` | `(elements, node)` | Show/hide tension row based on interpolation method |
|
||||
|
||||
**Data Fetching:**
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `fetchData` | `(url, fallbackUrl)` | `Promise<array>` | Fetch JSON from primary URL; fall back on failure |
|
||||
| `fetchProjectData` | `(url)` | `Promise<object>` | Fetch project-level data |
|
||||
| `apiCall` | `(node)` | `Promise<object>` | POST to asset-register API |
|
||||
|
||||
**URL Construction:**
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `getSpecificConfigUrl` | `(nodeName, cloudAPI)` | `{ cloudConfigURL, localConfigURL }` | Build cloud + local config URLs |
|
||||
| `constructUrl` | `(base, ...paths)` | `string` | Join URL segments safely |
|
||||
| `constructCloudURL` | `(base, ...paths)` | `string` | Same as `constructUrl`, for cloud endpoints |
|
||||
|
||||
**Dropdown Population:**
|
||||
|
||||
| Method | Signature | Description |
|
||||
|---|---|---|
|
||||
| `fetchAndPopulateDropdowns` | `(configUrls, elements, node)` | Cascading supplier > subType > model > unit dropdowns |
|
||||
| `populateDropdown` | `(htmlElement, options, node, property, callback?)` | Fill a `<select>` with options and wire change events |
|
||||
| `populateLogLevelOptions` | `(logLevelSelect, configData, node)` | Populate log-level dropdown from config |
|
||||
| `populateSmoothingMethods` | `(configUrls, elements, node)` | Populate smoothing method dropdown |
|
||||
| `populateInterpolationMethods` | `(configUrls, elements, node)` | Populate interpolation method dropdown |
|
||||
| `generateHtml` | `(htmlElement, options, savedValue)` | Write `<option>` HTML into an element |
|
||||
|
||||
---
|
||||
|
||||
## EndpointUtils
|
||||
|
||||
Server-side helper that serves `MenuUtils` as browser JavaScript via Node-RED HTTP endpoints.
|
||||
|
||||
**File:** `src/helper/endpointUtils.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new EndpointUtils({ MenuUtilsClass? })
|
||||
```
|
||||
|
||||
| Param | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `MenuUtilsClass` | `class` | `MenuUtils` | The MenuUtils constructor to introspect |
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `createMenuUtilsEndpoint` | `(RED, nodeName, customHelpers?)` | `void` | Register `GET /<nodeName>/resources/menuUtils.js` |
|
||||
| `generateMenuUtilsCode` | `(nodeName, customHelpers?)` | `string` | Produce the browser JS string (introspects `MenuUtils.prototype`) |
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const EndpointUtils = require('generalFunctions/src/helper/endpointUtils');
|
||||
const ep = new EndpointUtils();
|
||||
ep.createMenuUtilsEndpoint(RED, 'valve');
|
||||
// Browser can now load: GET /valve/resources/menuUtils.js
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Positions
|
||||
|
||||
Canonical constants for parent-child spatial relationships.
|
||||
|
||||
**File:** `src/constants/positions.js`
|
||||
|
||||
### Exports
|
||||
|
||||
```js
|
||||
const { POSITIONS, POSITION_VALUES, isValidPosition } = require('generalFunctions');
|
||||
```
|
||||
|
||||
| Export | Type | Value |
|
||||
|---|---|---|
|
||||
| `POSITIONS` | `object` | `{ UPSTREAM: 'upstream', DOWNSTREAM: 'downstream', AT_EQUIPMENT: 'atEquipment', DELTA: 'delta' }` |
|
||||
| `POSITION_VALUES` | `string[]` | `['upstream', 'downstream', 'atEquipment', 'delta']` |
|
||||
| `isValidPosition` | `(pos: string): boolean` | Returns `true` if `pos` is one of the four values |
|
||||
|
||||
---
|
||||
|
||||
## AssetLoader / loadCurve
|
||||
|
||||
Loads JSON asset files (machine curves, etc.) from the datasets directory with LRU caching.
|
||||
|
||||
**File:** `datasets/assetData/curves/index.js`
|
||||
|
||||
### Singleton convenience functions
|
||||
|
||||
```js
|
||||
const { loadCurve } = require('generalFunctions');
|
||||
```
|
||||
|
||||
| Function | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `loadCurve` | `(curveType: string)` | `object \| null` | Load `<curveType>.json` from the curves directory |
|
||||
| `loadAsset` | `(datasetType, assetId)` | `object \| null` | Load any JSON asset by dataset folder and ID |
|
||||
| `getAvailableAssets` | `(datasetType)` | `string[]` | List asset IDs in a dataset folder |
|
||||
|
||||
### AssetLoader class
|
||||
|
||||
```js
|
||||
new AssetLoader(maxCacheSize = 100)
|
||||
```
|
||||
|
||||
Same methods as above (`loadCurve`, `loadAsset`, `getAvailableAssets`), plus `clearCache()`.
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const { loadCurve } = require('generalFunctions');
|
||||
const curve = loadCurve('hidrostal-H05K-S03R');
|
||||
// curve = { flow: [...], head: [...], ... } or null
|
||||
```
|
||||
21
wiki/Archive/concepts-sources-readme.md
Normal file
21
wiki/Archive/concepts-sources-readme.md
Normal file
@@ -0,0 +1,21 @@
|
||||
# Source Documents
|
||||
|
||||
Place actual scientific papers, standards, and technical manuals here. Reference them from the summary files in the parent directory.
|
||||
|
||||
## Suggested Sources to Add
|
||||
|
||||
- IWA Scientific and Technical Report No. 1 — ASM1 (Henze et al., 1987)
|
||||
- IWA Scientific and Technical Report No. 3 — ASM2d (Henze et al., 1999)
|
||||
- IWA Scientific and Technical Report No. 9 — ASM3 (Gujer et al., 1999)
|
||||
- Takacs et al. (1991) "A dynamic model of the clarification-thickening process" Water Res. 25(10), 1263-1271
|
||||
- Astrom & Hagglund (2006) "Advanced PID Control" ISA
|
||||
- Karassik et al. "Pump Handbook" McGraw-Hill
|
||||
- Europump/Hydraulic Institute "Pump Life Cycle Costs"
|
||||
- IEC 62443 series (OT security)
|
||||
- IEC 61298 series (process measurement)
|
||||
- EU Directive 91/271/EEC (Urban Waste Water Treatment)
|
||||
- NIST SP 800-82 Rev 3 (Guide to ICS Security)
|
||||
|
||||
## File Naming Convention
|
||||
|
||||
`<author-year>-<short-title>.pdf` — e.g., `takacs-1991-clarification-thickening.pdf`
|
||||
88
wiki/Archive/findings-open-issues-2026-03.md
Normal file
88
wiki/Archive/findings-open-issues-2026-03.md
Normal file
@@ -0,0 +1,88 @@
|
||||
---
|
||||
title: Open Issues — EVOLV Codebase
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [issues, backlog]
|
||||
---
|
||||
|
||||
# Open Issues — EVOLV Codebase
|
||||
|
||||
Issues identified during codebase scan (2026-03-12). Create these on Gitea when ready.
|
||||
|
||||
---
|
||||
|
||||
## Issue 1: Restore diffuser node implementation
|
||||
|
||||
**Labels:** `enhancement`, `node`
|
||||
**Priority:** Medium
|
||||
|
||||
The `nodes/diffuser/` directory contains only `.git`, `LICENSE`, and `README.md` — no implementation. There was a previous experimental version. Needs:
|
||||
|
||||
- Retrieve original diffuser logic from user/backup
|
||||
- Rebuild to current three-layer architecture (wrapper `.js` + `src/nodeClass.js` + `src/specificClass.js`)
|
||||
- Use `require('generalFunctions')` barrel imports
|
||||
- Add config JSON in `generalFunctions/src/configs/diffuser.json`
|
||||
- Register under category `'EVOLV'` with appropriate S88 color
|
||||
- Add tests
|
||||
|
||||
**Blocked on:** User providing original diffuser logic/requirements.
|
||||
|
||||
---
|
||||
|
||||
## Issue 2: Relocate prediction/ML modules to external service
|
||||
|
||||
**Labels:** `enhancement`, `architecture`
|
||||
**Priority:** Medium
|
||||
|
||||
TensorFlow-based influent prediction code was removed from monster node (was broken/incomplete). The prediction functionality needs a new home:
|
||||
|
||||
- LSTM model for 24-hour flow prediction based on precipitation data
|
||||
- Standardization constants: hours `(mean=11.504, std=6.922)`, precipitation `(mean=0.090, std=0.439)`, response `(mean=1188.01, std=1024.19)`
|
||||
- Model was served from `http://127.0.0.1:1880/generalFunctions/datasets/lstmData/tfjs_model/`
|
||||
- Consider: separate microservice, Python-based inference, or ONNX runtime
|
||||
- Monster node should accept predictions via `model_prediction` message topic from external service
|
||||
|
||||
**Related files removed:** `monster_class.js` methods `get_model_prediction()`, `model_loader()`
|
||||
|
||||
---
|
||||
|
||||
## Issue 3: Modernize monster node to three-layer architecture
|
||||
|
||||
**Labels:** `refactor`, `node`
|
||||
**Priority:** Low
|
||||
|
||||
Monster node uses old-style structure (`dependencies/monster/` instead of `src/`). Should be refactored:
|
||||
|
||||
- Move `dependencies/monster/monster_class.js` → `src/specificClass.js`
|
||||
- Create `src/nodeClass.js` adapter (extract from `monster.js`)
|
||||
- Slim down `monster.js` to standard wrapper pattern
|
||||
- Move `monsterConfig.json` → `generalFunctions/src/configs/monster.json`
|
||||
- Remove `modelLoader.js` (TF dependency removed)
|
||||
- Add unit tests
|
||||
|
||||
**Note:** monster_class.js is ~500 lines of domain logic. Keep sampling_program(), aggregation, AQUON integration intact.
|
||||
|
||||
---
|
||||
|
||||
## Issue 4: Clean up inline test/demo code in specificClass files
|
||||
|
||||
**Labels:** `cleanup`
|
||||
**Priority:** Low
|
||||
|
||||
Several specificClass files have test/demo code after `module.exports`:
|
||||
|
||||
- `pumpingStation/src/specificClass.js` (lines 478-697): Demo code guarded with `require.main === module` — acceptable but could move to `test/` or `examples/`
|
||||
- `machineGroupControl/src/specificClass.js` (lines 969-1158): Block-commented test code with `makeMachines()` — dead code, could be removed or moved to test file
|
||||
|
||||
---
|
||||
|
||||
## Issue 5: DashboardAPI node improvements
|
||||
|
||||
**Labels:** `enhancement`, `security`
|
||||
**Priority:** Low
|
||||
|
||||
- Bearer token now relies on `GRAFANA_TOKEN` env var (hardcoded token was removed for security)
|
||||
- Ensure deployment docs mention setting `GRAFANA_TOKEN`
|
||||
- `dashboardapi_class.js` still has `console.log` calls (lines 154, 178) — should use logger
|
||||
- Node doesn't follow three-layer architecture (older style)
|
||||
57
wiki/Archive/index.md
Normal file
57
wiki/Archive/index.md
Normal file
@@ -0,0 +1,57 @@
|
||||
---
|
||||
title: Wiki Index
|
||||
updated: 2026-04-13
|
||||
---
|
||||
|
||||
# EVOLV Project Wiki Index
|
||||
|
||||
## Overview
|
||||
- [Project Overview](overview.md) — what works, what doesn't, node inventory
|
||||
- [Metrics Dashboard](metrics.md) — test counts, power comparison, performance
|
||||
- [Knowledge Graph](knowledge-graph.yaml) — structured data, machine-queryable
|
||||
|
||||
## Architecture
|
||||
- [Node Architecture](architecture/node-architecture.md) — three-layer pattern, ports, mermaid diagrams
|
||||
- [3D Pump Curves](architecture/3d-pump-curves.md) — predict class, spline interpolation, unit chain
|
||||
- [Group Optimization](architecture/group-optimization.md) — BEP-Gravitation, combination selection, marginal-cost refinement
|
||||
- [Platform Overview](architecture/platform-overview.md) — edge/site/central layering, telemetry model
|
||||
- [Deployment Blueprint](architecture/deployment-blueprint.md) — Docker topology, rollout order
|
||||
- [Stack Review](architecture/stack-review.md) — full stack architecture assessment
|
||||
|
||||
## Core Concepts
|
||||
- [generalFunctions API](concepts/generalfunctions-api.md) — logger, MeasurementContainer, configManager, etc.
|
||||
- [Pump Affinity Laws](concepts/pump-affinity-laws.md) — Q ∝ N, H ∝ N², P ∝ N³
|
||||
- [ASM Models](concepts/asm-models.md) — activated sludge model kinetics
|
||||
- [PID Control Theory](concepts/pid-control-theory.md) — proportional-integral-derivative control
|
||||
- [Settling Models](concepts/settling-models.md) — secondary clarifier sludge settling
|
||||
- [Signal Processing for Sensors](concepts/signal-processing-sensors.md) — sensor conditioning
|
||||
- [InfluxDB Schema Design](concepts/influxdb-schema-design.md) — telemetry data model
|
||||
- [OT Security (IEC 62443)](concepts/ot-security-iec62443.md) — industrial security standard
|
||||
- [Wastewater Compliance NL](concepts/wastewater-compliance-nl.md) — Dutch regulatory requirements
|
||||
|
||||
## Findings
|
||||
- [BEP-Gravitation Proof](findings/bep-gravitation-proof.md) — within 0.1% of brute-force optimum (proven)
|
||||
- [NCog Behavior](findings/ncog-behavior.md) — when NCog works, when it's zero, how it's used (evolving)
|
||||
- [Curve Non-Convexity](findings/curve-non-convexity.md) — C5 sparse data artifacts (proven)
|
||||
- [Pump Switching Stability](findings/pump-switching-stability.md) — 1-2 transitions, no hysteresis (proven)
|
||||
- [Open Issues (2026-03)](findings/open-issues-2026-03.md) — diffuser, monster refactor, ML relocation, etc.
|
||||
|
||||
## Manuals
|
||||
- [rotatingMachine User Manual](manuals/nodes/rotatingMachine.md) — inputs, outputs, state machine, examples
|
||||
- [measurement User Manual](manuals/nodes/measurement.md) — analog + digital modes, smoothing, outlier filtering
|
||||
- [FlowFuse Dashboard Layout](manuals/node-red/flowfuse-dashboard-layout-manual.md)
|
||||
- [FlowFuse Widget Catalog](manuals/node-red/flowfuse-widgets-catalog.md)
|
||||
- [Node-RED Function Patterns](manuals/node-red/function-node-patterns.md)
|
||||
- [Node-RED Runtime](manuals/node-red/runtime-node-js.md)
|
||||
- [Messages and Editor Structure](manuals/node-red/messages-and-editor-structure.md)
|
||||
|
||||
## Sessions
|
||||
- [2026-04-07: Production Hardening](sessions/2026-04-07-production-hardening.md) — rotatingMachine + machineGroupControl
|
||||
- [2026-04-13: rotatingMachine Trial-Ready](sessions/2026-04-13-rotatingMachine-trial-ready.md) — FSM interruptibility, config schema sync, UX polish, dual-curve tests
|
||||
- [2026-04-13: measurement Digital Mode](sessions/2026-04-13-measurement-digital-mode.md) — silent dispatcher bug fix, 59 new tests, MQTT-style multi-channel input mode
|
||||
|
||||
## Other Documentation (outside wiki)
|
||||
- `CLAUDE.md` — Claude Code project guide (root)
|
||||
- `.agents/AGENTS.md` — agent routing table, orchestrator policy
|
||||
- `.agents/` — skills, decisions, function-anchors, improvements
|
||||
- `.claude/` — Claude Code agents and rules
|
||||
161
wiki/Archive/knowledge-graph.yaml
Normal file
161
wiki/Archive/knowledge-graph.yaml
Normal file
@@ -0,0 +1,161 @@
|
||||
# Knowledge Graph — structured data with provenance
|
||||
# Every claim has: value, source (file/commit), date, status
|
||||
|
||||
# ── TESTS ──
|
||||
tests:
|
||||
rotatingMachine:
|
||||
basic:
|
||||
count: 10
|
||||
passing: 10
|
||||
file: nodes/rotatingMachine/test/basic/
|
||||
date: 2026-04-07
|
||||
integration:
|
||||
count: 16
|
||||
passing: 16
|
||||
file: nodes/rotatingMachine/test/integration/
|
||||
date: 2026-04-07
|
||||
edge:
|
||||
count: 17
|
||||
passing: 17
|
||||
file: nodes/rotatingMachine/test/edge/
|
||||
date: 2026-04-07
|
||||
machineGroupControl:
|
||||
basic:
|
||||
count: 1
|
||||
passing: 1
|
||||
file: nodes/machineGroupControl/test/basic/
|
||||
date: 2026-04-07
|
||||
integration:
|
||||
count: 3
|
||||
passing: 3
|
||||
file: nodes/machineGroupControl/test/integration/
|
||||
date: 2026-04-07
|
||||
edge:
|
||||
count: 1
|
||||
passing: 1
|
||||
file: nodes/machineGroupControl/test/edge/
|
||||
date: 2026-04-07
|
||||
|
||||
# ── METRICS ──
|
||||
metrics:
|
||||
optimization_gap_to_brute_force:
|
||||
value: "0.1% max"
|
||||
source: distribution-power-table.integration.test.js
|
||||
date: 2026-04-07
|
||||
conditions: "3 pumps, 1000-step brute force, 0.05% flow tolerance"
|
||||
optimization_time_median:
|
||||
value: "0.027-0.153ms"
|
||||
source: benchmark script
|
||||
date: 2026-04-07
|
||||
conditions: "3 pumps, 6 combinations, BEP-Gravitation + refinement"
|
||||
pump_switching_stability:
|
||||
value: "1-2 transitions across 5-95% demand"
|
||||
source: stability sweep
|
||||
date: 2026-04-07
|
||||
conditions: "2% demand steps, both ascending and descending"
|
||||
pump_curves:
|
||||
H05K-S03R:
|
||||
pressure_levels: 33
|
||||
pressure_range: "700-3900 mbar"
|
||||
flow_range: "28-227 m3/h (at 2000 mbar)"
|
||||
data_points_per_level: 5
|
||||
anomalies_fixed: 3
|
||||
date: 2026-04-07
|
||||
C5-D03R-SHN1:
|
||||
pressure_levels: 26
|
||||
pressure_range: "400-2900 mbar"
|
||||
flow_range: "6-53 m3/h"
|
||||
data_points_per_level: 5
|
||||
non_convex: true
|
||||
date: 2026-04-07
|
||||
|
||||
# ── DISPROVEN CLAIMS ──
|
||||
disproven:
|
||||
ncog_proportional_weight:
|
||||
claimed: "Distributing flow proportional to NCog weights is optimal"
|
||||
claimed_date: 2026-04-07
|
||||
disproven_date: 2026-04-07
|
||||
evidence_for: "Simple implementation in calcBestCombination"
|
||||
evidence_against: "Starves small pumps (NCog=0 gets zero flow), overloads large pumps at high demand. BEP-target + scale is correct approach."
|
||||
root_cause: "NCog is a position indicator (0-1 on flow range), not a distribution weight"
|
||||
efficiency_rounding:
|
||||
claimed: "Math.round(flow/power * 100) / 100 preserves BEP signal"
|
||||
claimed_date: pre-2026-04-07
|
||||
disproven_date: 2026-04-07
|
||||
evidence_for: "Removes floating point noise"
|
||||
evidence_against: "In canonical units (m3/s and W), Q/P ratio is ~1e-6. Rounding to 2 decimals produces 0 for all points. NCog, cog, BEP all became 0."
|
||||
root_cause: "Canonical units make the ratio very small — rounding destroys the signal"
|
||||
equal_marginal_cost_optimal:
|
||||
claimed: "Equal dP/dQ across pumps guarantees global power minimum"
|
||||
claimed_date: 2026-04-07
|
||||
disproven_date: 2026-04-07
|
||||
evidence_for: "KKT conditions for convex functions"
|
||||
evidence_against: "C5 pump curve is non-convex (dP/dQ dips from 1.3M to 453K then rises). Sparse data (5 points) causes spline artifacts."
|
||||
root_cause: "Convexity assumption fails with interpolated curves from sparse data"
|
||||
|
||||
# ── PERFORMANCE ──
|
||||
performance:
|
||||
mgc_optimization:
|
||||
median_ms: 0.09
|
||||
p99_ms: 0.5
|
||||
tick_budget_pct: 0.015
|
||||
source: benchmark script
|
||||
date: 2026-04-07
|
||||
predict_y_call:
|
||||
complexity: "O(log n), ~O(1) for 5-10 data points"
|
||||
source: predict_class.js
|
||||
|
||||
# ── ARCHITECTURE ──
|
||||
architecture:
|
||||
canonical_units:
|
||||
pressure: Pa
|
||||
flow: "m3/s"
|
||||
power: W
|
||||
temperature: K
|
||||
output_units:
|
||||
pressure: mbar
|
||||
flow: "m3/h"
|
||||
power: kW
|
||||
temperature: C
|
||||
node_count: 13
|
||||
submodules: 12
|
||||
|
||||
# ── BUGS FIXED ──
|
||||
bugs_fixed:
|
||||
flowmovement_unit_mismatch:
|
||||
severity: critical
|
||||
description: "machineGroupControl sent flow in canonical (m3/s) but rotatingMachine flowmovement expected output units (m3/h). Every pump stayed at minimum."
|
||||
fix: "_canonicalToOutputFlow() conversion before all flowmovement calls"
|
||||
commit: d55f401
|
||||
date: 2026-04-07
|
||||
emergencystop_case:
|
||||
severity: critical
|
||||
description: "specificClass called executeSequence('emergencyStop') but config key was 'emergencystop'"
|
||||
fix: "Lowercase to match config"
|
||||
commit: 07af7ce
|
||||
date: 2026-04-07
|
||||
curve_data_anomalies:
|
||||
severity: high
|
||||
description: "3 flow values leaked into power column in hidrostal-H05K-S03R.json at pressures 1600, 3200, 3300 mbar"
|
||||
fix: "Linearly interpolated correct values from adjacent levels"
|
||||
commit: 024db55
|
||||
date: 2026-04-07
|
||||
efficiency_rounding:
|
||||
severity: high
|
||||
description: "Math.round(Q/P * 100) / 100 destroyed all NCog/BEP calculations"
|
||||
fix: "Removed rounding, use raw ratio"
|
||||
commit: 07af7ce
|
||||
date: 2026-04-07
|
||||
absolute_scaling_bug:
|
||||
severity: high
|
||||
description: "handleInput compared demandQout (always 0) instead of demandQ for max cap"
|
||||
fix: "Reordered conditions, use demandQ throughout"
|
||||
commit: d55f401
|
||||
date: 2026-04-07
|
||||
|
||||
# ── TIMELINE ──
|
||||
timeline:
|
||||
- {date: 2026-04-07, commit: 024db55, desc: "Fix 3 anomalous power values in hidrostal curve"}
|
||||
- {date: 2026-04-07, commit: 07af7ce, desc: "rotatingMachine production hardening: safety + prediction + 43 tests"}
|
||||
- {date: 2026-04-07, commit: d55f401, desc: "machineGroupControl: unit fix + refinement + stability tests"}
|
||||
- {date: 2026-04-07, commit: fd9d167, desc: "Update EVOLV submodule refs"}
|
||||
11
wiki/Archive/log.md
Normal file
11
wiki/Archive/log.md
Normal file
@@ -0,0 +1,11 @@
|
||||
---
|
||||
title: Wiki Log
|
||||
---
|
||||
|
||||
# Wiki Log
|
||||
|
||||
## [2026-04-07] Wiki initialized | Full codebase scan + session findings
|
||||
- Created overview, metrics, knowledge graph from production hardening session
|
||||
- Architecture pages: 3D pump curves, group optimization
|
||||
- Findings: BEP-Gravitation proof, NCog behavior, curve non-convexity, switching stability
|
||||
- Session log: 2026-04-07 production hardening
|
||||
203
wiki/Archive/manuals-nodes-measurement.md
Normal file
203
wiki/Archive/manuals-nodes-measurement.md
Normal file
@@ -0,0 +1,203 @@
|
||||
---
|
||||
title: measurement — User Manual
|
||||
node: measurement
|
||||
updated: 2026-04-13
|
||||
status: trial-ready
|
||||
---
|
||||
|
||||
# measurement — User Manual
|
||||
|
||||
The `measurement` node is the sensor-side of every EVOLV flow. It takes raw signal data, applies offset / scaling / smoothing / outlier rejection, and publishes a conditioned value into the shared `MeasurementContainer`. A parent equipment node (rotatingMachine, pumpingStation, reactor, ...) subscribes automatically via the child-registration handshake on port 2.
|
||||
|
||||
## At a glance
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Node category | EVOLV |
|
||||
| Inputs | 1 (message-driven) |
|
||||
| Outputs | 3 — `process` / `dbase` / `parent` |
|
||||
| Tick period | 1 s |
|
||||
| Input modes | `analog` (default) — one scalar per msg. `digital` — object payload with many keys. |
|
||||
| Smoothing methods | 12 (`none`, `mean`, `min`, `max`, `sd`, `lowPass`, `highPass`, `weightedMovingAverage`, `bandPass`, `median`, `kalman`, `savitzkyGolay`) |
|
||||
| Outlier methods | 3 (`zScore`, `iqr`, `modifiedZScore`) |
|
||||
|
||||
## Choosing a mode
|
||||
|
||||
### Analog — one scalar per message (PLC / 4-20 mA)
|
||||
|
||||
The classic pattern — what the node did before v1.1. `msg.payload` is a single number. The node runs one offset → scaling → smoothing → outlier pipeline and emits exactly one MeasurementContainer slot keyed by the asset's type + position.
|
||||
|
||||
```json
|
||||
{ "topic": "measurement", "payload": 12.34 }
|
||||
```
|
||||
|
||||
Use when one Node-RED `measurement` node represents one physical sensor.
|
||||
|
||||
### Digital — object payload, many channels (MQTT / IoT / JSON)
|
||||
|
||||
Use when one Node-RED `measurement` node represents one physical **device** that publishes multiple readings. Common shapes:
|
||||
|
||||
```json
|
||||
{ "topic": "measurement",
|
||||
"payload": { "temperature": 22.5, "humidity": 45, "pressure": 1013 } }
|
||||
```
|
||||
|
||||
```json
|
||||
{ "topic": "measurement",
|
||||
"payload": { "co2": 618, "voc": 122, "pm25": 8 } }
|
||||
```
|
||||
|
||||
Each top-level key maps to a **channel** with its own `type`, `position`, `unit`, and pipeline parameters. Unknown keys are ignored (logged at debug).
|
||||
|
||||
## Configuration
|
||||
|
||||
### Common (both modes)
|
||||
|
||||
- **Asset** (menu): supplier, category, asset type (`assetType`), model, unit.
|
||||
- **Logger** (menu): log level + enable flag.
|
||||
- **Position** (menu): `upstream` / `atEquipment` / `downstream`, optional distance offset.
|
||||
|
||||
### Analog fields
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| **Scaling** | enables linear interpolation from source range to process range |
|
||||
| **Source Min / Max** | raw input bounds (e.g. `4` / `20` for mA) |
|
||||
| **Input Offset** | additive bias applied before scaling |
|
||||
| **Process Min / Max** | mapped output bounds (e.g. `0` / `3000` for mbar) |
|
||||
| **Simulator** | internal random-walk source for testing |
|
||||
| **Smoothing** | method (dropdown) |
|
||||
| **Window** | smoothing window size |
|
||||
|
||||
### Digital fields
|
||||
|
||||
- **Input Mode**: set to `digital` in the dropdown.
|
||||
- **Channels (JSON)**: array of channel definitions.
|
||||
|
||||
Each channel:
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "temperature",
|
||||
"type": "temperature",
|
||||
"position": "atEquipment",
|
||||
"unit": "C",
|
||||
"scaling": { "enabled": false, "inputMin": 0, "inputMax": 1, "absMin": -50, "absMax": 150, "offset": 0 },
|
||||
"smoothing": { "smoothWindow": 5, "smoothMethod": "mean" },
|
||||
"outlierDetection": { "enabled": true, "method": "zScore", "threshold": 3 }
|
||||
}
|
||||
```
|
||||
|
||||
`scaling` / `smoothing` / `outlierDetection` are optional — missing sections inherit the top-level analog-mode fields. `key` is the JSON field name inside `msg.payload`; `type` is the MeasurementContainer axis — any string works, not just the physical-unit-backed defaults.
|
||||
|
||||
## Input topics
|
||||
|
||||
| Topic | Payload | Effect |
|
||||
|---|---|---|
|
||||
| `measurement` | number (analog) / object (digital) | drives the pipeline |
|
||||
| `simulator` | — | toggle the internal random-walk simulator |
|
||||
| `outlierDetection` | — | toggle outlier rejection |
|
||||
| `calibrate` | — | set the offset so the current output matches `Source Min` (scaling on) or `Process Min` (scaling off). Requires a stable window — aborts if the signal is fluctuating. |
|
||||
|
||||
## Output ports
|
||||
|
||||
### Port 0 — process
|
||||
|
||||
Delta-compressed payload.
|
||||
|
||||
**Analog** shape:
|
||||
|
||||
```json
|
||||
{ "mAbs": 4.2, "mPercent": 42, "totalMinValue": 0, "totalMaxValue": 100,
|
||||
"totalMinSmooth": 0, "totalMaxSmooth": 4.2 }
|
||||
```
|
||||
|
||||
**Digital** shape:
|
||||
|
||||
```json
|
||||
{ "channels": {
|
||||
"temperature": { "key": "temperature", "type": "temperature", "position": "atEquipment",
|
||||
"unit": "C", "mAbs": 24, "mPercent": 37,
|
||||
"totalMinValue": 22.5, "totalMaxValue": 25.5,
|
||||
"totalMinSmooth": 22.5, "totalMaxSmooth": 24 },
|
||||
"humidity": { ... },
|
||||
"pressure": { ... }
|
||||
} }
|
||||
```
|
||||
|
||||
### Port 1 — dbase
|
||||
|
||||
InfluxDB line-protocol telemetry. Tags = asset metadata; fields = measurements. See [InfluxDB Schema Design](../../concepts/influxdb-schema-design.md).
|
||||
|
||||
### Port 2 — parent
|
||||
|
||||
`{ topic: "registerChild", payload: <nodeId>, positionVsParent, distance }` — emitted once ~200 ms after deploy so the parent equipment node registers this sensor.
|
||||
|
||||
## Pipeline per value
|
||||
|
||||
1. **Outlier check** (if enabled) — rejects via zScore / IQR / modifiedZScore. Rejected values never advance, don't update min/max, don't emit.
|
||||
2. **Offset** — `value + scaling.offset`.
|
||||
3. **Scaling** (if enabled) — linear interpolation from `[inputMin, inputMax]` to `[absMin, absMax]` with boundary clamping.
|
||||
4. **Smoothing** — current value pushed into the rolling window; the configured method produces the smoothed output.
|
||||
5. **Min/Max tracking** — both raw (pre-smoothing) and smoothed min/max tracked for display.
|
||||
6. **Constrain** — smoothed value clamped to `[absMin, absMax]`.
|
||||
7. **Emit** — `MeasurementContainer.type(...).variant('measured').position(...).distance(...).value(out, ts, unit)` triggers the event `<type>.measured.<position>` (lowercase) that the parent equipment subscribes to.
|
||||
|
||||
In digital mode, each channel runs this pipeline independently.
|
||||
|
||||
## Smoothing methods — quick reference
|
||||
|
||||
| Method | Use case |
|
||||
|---|---|
|
||||
| `none` | pass raw value through — useful for testing |
|
||||
| `mean` | simple arithmetic average over window |
|
||||
| `min` / `max` | worst-case / peak reporting |
|
||||
| `sd` | outputs standard deviation (noise indicator) |
|
||||
| `median` | outlier-resistant central tendency |
|
||||
| `weightedMovingAverage` | later samples weighted higher |
|
||||
| `lowPass` | EMA-style attenuation of high-frequency noise |
|
||||
| `highPass` | emphasises rapid changes (step detection) |
|
||||
| `bandPass` | `lowPass + highPass - raw` — band-of-interest filtering |
|
||||
| `kalman` | recursive noise filter, converges to steady value |
|
||||
| `savitzkyGolay` | polynomial smoothing over 5-point window |
|
||||
|
||||
## Outlier methods — quick reference
|
||||
|
||||
| Method | Best when |
|
||||
|---|---|
|
||||
| `zScore` | signal is approximately normal; threshold = # of SDs |
|
||||
| `iqr` | signal is non-normal; robust to skewed distributions |
|
||||
| `modifiedZScore` | small samples; uses median / MAD instead of mean / SD |
|
||||
|
||||
> **Historical bug fixed 2026-04-13:** The dispatcher compared against camelCase keys (`lowPass`, `zScore`, ...) but the validator lowercases enum values. Result: 4 smoothing methods and 2 outlier methods were silently no-ops when chosen from the editor — they fell through to the "unknown" branch and emitted the raw last value. Review any flow deployed before 2026-04-13 that relied on these methods.
|
||||
|
||||
## Unit policy
|
||||
|
||||
Unknown measurement types (anything not in the container's built-in `measureMap`: `pressure`, `flow`, `power`, `temperature`, `volume`, `length`, `mass`, `energy`) are accepted without unit compatibility checks. This lets digital channels use `humidity` (`%`), `co2` (`ppm`), arbitrary IoT units. Known types still validate strictly.
|
||||
|
||||
## Example flow (digital)
|
||||
|
||||
```json
|
||||
[
|
||||
{ "id": "dig", "type": "measurement",
|
||||
"mode": "digital",
|
||||
"channels": "[{\"key\":\"temperature\",\"type\":\"temperature\",\"position\":\"atEquipment\",\"unit\":\"C\",\"scaling\":{\"enabled\":false,\"absMin\":-50,\"absMax\":150},\"smoothing\":{\"smoothWindow\":5,\"smoothMethod\":\"mean\"}},{\"key\":\"humidity\",\"type\":\"humidity\",\"position\":\"atEquipment\",\"unit\":\"%\",\"scaling\":{\"enabled\":false,\"absMin\":0,\"absMax\":100},\"smoothing\":{\"smoothWindow\":5,\"smoothMethod\":\"mean\"}}]",
|
||||
...
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
cd nodes/measurement
|
||||
npm test
|
||||
```
|
||||
|
||||
71 tests — coverage includes every smoothing method, every outlier strategy, scaling, interpolation, constrain, calibration, stability, simulation, per-channel pipelines, digital-mode dispatch, malformed-channel handling, event emits.
|
||||
|
||||
End-to-end benchmark scripts live in the superproject at `/tmp/m_e2e_baseline.py` (analog) and `/tmp/m_digital_e2e.py` (digital). Run against a Dockerized Node-RED stack (`docker compose up -d nodered`).
|
||||
|
||||
## Production status
|
||||
|
||||
Trial-ready as of 2026-04-13 after the session that fixed the silent dispatcher bug and added digital mode. See [session 2026-04-13](../../sessions/2026-04-13-measurement-digital-mode.md) and the memory file `node_measurement.md`.
|
||||
247
wiki/Archive/manuals-nodes-rotatingMachine.md
Normal file
247
wiki/Archive/manuals-nodes-rotatingMachine.md
Normal file
@@ -0,0 +1,247 @@
|
||||
---
|
||||
title: rotatingMachine — User Manual
|
||||
node: rotatingMachine
|
||||
updated: 2026-04-13
|
||||
status: trial-ready
|
||||
---
|
||||
|
||||
# rotatingMachine — User Manual
|
||||
|
||||
The `rotatingMachine` node models a single pump, compressor, or blower. It runs an S88-style state machine, predicts flow and power from a supplier curve, and publishes process and telemetry data every second. It is the atomic control module beneath `machineGroupControl` and `pumpingStation`.
|
||||
|
||||
This manual is the operator-facing reference. For architecture and the 3-tier code layout see [Node Architecture](../../architecture/node-architecture.md); for curve theory see [3D Pump Curves](../../architecture/3d-pump-curves.md).
|
||||
|
||||
## At a glance
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Node category | EVOLV |
|
||||
| Inputs | 1 (message-driven) |
|
||||
| Outputs | 3 — `process` / `dbase` / `parent` |
|
||||
| Tick period | 1 s |
|
||||
| State machine | 10 states (S88) |
|
||||
| Predictions | curve-backed (nq flow, np power, reversed nq for ctrl) |
|
||||
| Canonical units | Pa, m³/s, W, K |
|
||||
|
||||
## Editor configuration
|
||||
|
||||
| Field | Default | Meaning |
|
||||
|---|---|---|
|
||||
| **Reaction Speed** | `1` | Ramp rate in controller-position units per second. `1` = 1 %/s. |
|
||||
| **Startup Time** | `0` | Seconds in the `starting` state. |
|
||||
| **Warmup Time** | `0` | Seconds in the protected `warmingup` state. |
|
||||
| **Shutdown Time** | `0` | Seconds in the `stopping` state. |
|
||||
| **Cooldown Time** | `0` | Seconds in the protected `coolingdown` state. |
|
||||
| **Movement Mode** | `staticspeed` | `staticspeed` = linear ramp; `dynspeed` = ease-in/out. |
|
||||
| **Process Output** | `process` | Port 0 payload format: `process` (delta-compressed) / `json` / `csv`. |
|
||||
| **Database Output** | `influxdb` | Port 1 payload format: `influxdb` line protocol / `json` / `csv`. |
|
||||
| **Asset** (menu) | — | Supplier, category, model (must match a curve file in `generalFunctions/datasets`), output flow unit, curve units. |
|
||||
| **Logger** (menu) | `info`, enabled | Log level and toggle. |
|
||||
| **Position** (menu) | `atEquipment` | `upstream` / `atEquipment` / `downstream` relative to parent. Icon and optional distance offset. |
|
||||
|
||||
> **Tip.** With `Reaction Speed = 1` and `Set 60%` from idle, the controller takes ~60 s to reach 60 %. Scale `Reaction Speed` up to emulate a faster actuator (e.g. `20` gives 1 second per 20 % = 3 s to reach 60 %).
|
||||
|
||||
## Input topics
|
||||
|
||||
Every command enters on the single input port. `msg.topic` selects the handler; `msg.payload` carries the arguments.
|
||||
|
||||
### `setMode`
|
||||
|
||||
```json
|
||||
{ "topic": "setMode", "payload": "virtualControl" }
|
||||
```
|
||||
|
||||
Valid values: `auto`, `virtualControl`, `fysicalControl`. The current mode gates *which source* may issue *which action* (mode/action/source policy lives in `generalFunctions/src/configs/rotatingMachine.json`).
|
||||
|
||||
### `execSequence`
|
||||
|
||||
```json
|
||||
{ "topic": "execSequence",
|
||||
"payload": { "source": "GUI", "action": "execSequence", "parameter": "startup" } }
|
||||
```
|
||||
|
||||
`parameter` values: `startup`, `shutdown`, `entermaintenance`, `exitmaintenance`. Case is normalized.
|
||||
|
||||
If a `shutdown` is issued while the machine is mid-ramp (`accelerating` / `decelerating`), the active movement is aborted and the shutdown proceeds as soon as the FSM has returned to `operational`.
|
||||
|
||||
### `execMovement`
|
||||
|
||||
```json
|
||||
{ "topic": "execMovement",
|
||||
"payload": { "source": "GUI", "action": "execMovement", "setpoint": 60 } }
|
||||
```
|
||||
|
||||
`setpoint` is expressed in controller units (0–100 %).
|
||||
|
||||
### `flowMovement`
|
||||
|
||||
```json
|
||||
{ "topic": "flowMovement",
|
||||
"payload": { "source": "parent", "action": "flowMovement", "setpoint": 150 } }
|
||||
```
|
||||
|
||||
`setpoint` is expressed in the configured **output flow unit** (e.g. m³/h). The node converts flow → controller-% via the reversed nq curve and then drives `execMovement`.
|
||||
|
||||
### `emergencystop`
|
||||
|
||||
```json
|
||||
{ "topic": "emergencystop",
|
||||
"payload": { "source": "GUI", "action": "emergencystop" } }
|
||||
```
|
||||
|
||||
Aborts any active movement, runs the `emergencystop` → `off` transition. Allowed from every active state. Case-insensitive.
|
||||
|
||||
### `simulateMeasurement`
|
||||
|
||||
Inject a dashboard-side measurement without wiring a sensor child. Useful for validation, smoke tests, demo flows.
|
||||
|
||||
```json
|
||||
{ "topic": "simulateMeasurement",
|
||||
"payload": { "type": "pressure", "position": "upstream", "value": 200, "unit": "mbar" } }
|
||||
```
|
||||
|
||||
`type`: `pressure` / `flow` / `temperature` / `power`. `unit` is required and must be convertible to the canonical unit for the type.
|
||||
|
||||
### Diagnostics
|
||||
|
||||
- `showWorkingCurves` — snapshot of current curve slices + computed metrics; reply on port 0.
|
||||
- `CoG` — current centre-of-gravity (peak efficiency point) indicators; reply on port 0.
|
||||
|
||||
### `registerChild`
|
||||
|
||||
Internal. Sensor children (typically `measurement` nodes) send this to bind themselves to the machine. The machine also emits one on port 2 shortly after deploy so a parent group/station can register it.
|
||||
|
||||
## Output ports
|
||||
|
||||
### Port 0 — process data
|
||||
|
||||
Delta-compressed payload. Only *changed* fields are emitted each tick. Keys use a **4-segment** format:
|
||||
|
||||
```
|
||||
<type>.<variant>.<position>.<childId>
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| `flow.predicted.downstream.default` | predicted flow at discharge |
|
||||
| `flow.predicted.atequipment.default` | predicted flow at equipment |
|
||||
| `power.predicted.atequipment.default` | predicted electrical power draw |
|
||||
| `pressure.measured.downstream.dashboard-sim-downstream` | simulated discharge pressure |
|
||||
| `pressure.measured.upstream.<childId>` | real upstream sensor reading |
|
||||
| `state` | current FSM state |
|
||||
| `mode` | current mode |
|
||||
| `ctrl` | current controller position (0–100 %) |
|
||||
| `NCog` / `cog` | normalized / absolute centre-of-gravity |
|
||||
| `runtime` | cumulative operational hours |
|
||||
|
||||
Consumers must cache and merge deltas. The example flow `01 - Basic Manual Control.json` includes a function node that does exactly this — reuse its logic in your own flows.
|
||||
|
||||
### Port 1 — dbase (InfluxDB)
|
||||
|
||||
InfluxDB line-protocol payload formatted for the `telemetry` bucket. Tags are low-cardinality fields (node name, machine type); measurements are numeric values. See the [InfluxDB Schema Design](../../concepts/influxdb-schema-design.md) page for the full tag/field contract.
|
||||
|
||||
### Port 2 — parent
|
||||
|
||||
`{ topic: "registerChild", payload: <this-node-id>, positionVsParent }` — emitted once ~180 ms after deploy so a downstream parent group can discover this machine. Subsequent commands and data flow through the parent's input port.
|
||||
|
||||
## State machine
|
||||
|
||||
```
|
||||
┌────────────────────────────┐
|
||||
│ operational │◄────┐
|
||||
└────┬──────────┬────────┬────┘ │
|
||||
│ │ │ │
|
||||
execMovement │ │ │ │
|
||||
execMovement │ │ │ │
|
||||
▼ ▼ ▼ ▼ │
|
||||
accelerating decelerating │ emergencystop ─► off
|
||||
│ │ │
|
||||
└─── (abort)─┘ │
|
||||
│ │
|
||||
┌────▼──────────▼────┐
|
||||
│ stopping │
|
||||
└────────┬─────────────┘
|
||||
│
|
||||
coolingdown
|
||||
│
|
||||
idle
|
||||
│
|
||||
starting
|
||||
│
|
||||
warmingup
|
||||
│
|
||||
(operational)
|
||||
```
|
||||
|
||||
Protected states (cannot be aborted by a new command): `warmingup`, `coolingdown`.
|
||||
|
||||
Interruptible states: `accelerating`, `decelerating`. A `shutdown` or `emergencystop` issued during a ramp aborts the ramp and drives the FSM correctly to `idle` / `off`.
|
||||
|
||||
Active states (contribute to `runtime`): `operational`, `starting`, `warmingup`, `accelerating`, `decelerating`.
|
||||
|
||||
## Predictions and pressure
|
||||
|
||||
Flow and power are curve-backed. The curve set is indexed by the differential pressure across the machine:
|
||||
|
||||
1. Best: both upstream and downstream pressures present → real Δp.
|
||||
2. Degraded: only one side present → falls back to that side with a warn.
|
||||
3. Minimum: no pressure → `fDimension = 0`; flow and power predictions use the lowest curve slice and will look unrealistic.
|
||||
|
||||
Pressure sources are resolved in priority order **real sensor child > virtual dashboard child > aggregated fallback**. Real-child values always win.
|
||||
|
||||
Predictions are only emitted while the FSM is in an active state (`operational`, `starting`, `warmingup`, `accelerating`, `decelerating`). In `idle`, `stopping`, `coolingdown`, `off`, `maintenance` the outputs are clamped to zero.
|
||||
|
||||
### Supported curves and verification
|
||||
|
||||
| Model | Pressure envelope | Flow envelope | Power envelope |
|
||||
|---|---|---|---|
|
||||
| `hidrostal-H05K-S03R` | 700 – 3900 mbar (33 slices) | 9.5 – 227 m³/h | 8.2 – 65.1 kW |
|
||||
| `hidrostal-C5-D03R-SHN1` | 400 – 2900 mbar (26 slices) | 6.4 – 52.5 m³/h | 0.55 – 31.5 kW |
|
||||
|
||||
Both curves are covered by unit tests (`test/integration/curve-prediction.integration.test.js`) and a live E2E benchmark (`test/e2e/curve-prediction-benchmark.py`) that sweeps each pump through its own pressure × controller envelope. Last green run: **2026-04-13** — 12/12 samples per curve inside envelope, ctrl-monotonic, inverse-pressure monotonic.
|
||||
|
||||
> **Pressure out of envelope is not clamped.** If a measured pressure falls *below* the curve's minimum slice, the node extrapolates and may produce implausibly large flow values (e.g. H05K at 400 mbar, ctrl 20 % → flow ≈ 30 000 m³/h; real envelope max is 227). Use realistic sensor ranges on your pressure `measurement` children.
|
||||
|
||||
## Example flows
|
||||
|
||||
In the editor: **Import ▸ Examples ▸ EVOLV ▸ rotatingMachine**.
|
||||
|
||||
- `01 - Basic Manual Control.json` — single machine, inject-only. Good for smoke-testing a node installation.
|
||||
- `02 - Integration with Machine Group.json` — `machineGroupControl` with two pumps as children. Good for verifying registration and parent orchestration.
|
||||
- `03 - Dashboard Visualization.json` — FlowFuse dashboard with live charts. Depends on `@flowfuse/node-red-dashboard`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| Editor says `pressure not initialized`, status ring is yellow | No pressure child wired yet and no simulated pressure injected. | Inject a `simulateMeasurement` of type `pressure` (both sides preferred) or wire a `measurement` child. |
|
||||
| Predictions are enormous at `ctrl = 0 %` | At near-zero controller position with high backpressure, the intercept of the curve gives a nominally-nonzero flow. This is a curve-data artefact, not a runtime bug. | Confirm the curve with Rene / supplier data. For a conservative prediction use a lower `Reaction Speed` or constrain `setpoint` ≥ 10 %. |
|
||||
| "Transition aborted" / "Movement aborted" in logs | Expected during `shutdown` / `emergencystop` issued during a ramp — the fix path intentionally aborts the active move. | None — informational only. |
|
||||
| Status bar shows `pressure not initialized` even after inject | `simulateMeasurement` payload missing `unit` or with a non-convertible value. | Include `unit` (e.g. `"mbar"`) and a finite number in `value`. |
|
||||
| Shutdown does nothing and no error | Machine is in `warmingup` or `coolingdown` (protected). | Wait for the phase to complete (≤ configured seconds) and retry. |
|
||||
|
||||
## Running it locally
|
||||
|
||||
```bash
|
||||
git clone --recurse-submodules https://gitea.wbd-rd.nl/RnD/EVOLV.git
|
||||
cd EVOLV
|
||||
docker compose up -d
|
||||
# Node-RED: http://localhost:1880 InfluxDB: :8086 Grafana: :3000
|
||||
```
|
||||
|
||||
Then in Node-RED: **Import ▸ Examples ▸ EVOLV ▸ rotatingMachine ▸ 01 - Basic Manual Control**.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
cd nodes/rotatingMachine
|
||||
npm test
|
||||
```
|
||||
|
||||
Unit tests (79) cover construction, mode gating, sequences, interruptible movement, emergency stop, shutdown, efficiency/CoG, pressure initialization, output formatting, listener cleanup. See also `examples/README.md` for the flow-level test matrix.
|
||||
|
||||
## Production status
|
||||
|
||||
See the project memory entry `node_rotatingMachine.md` for the latest benchmarks and wishlist. Trial-ready as of 2026-04-13 following the interruptibility + schema-sync fixes documented in [session 2026-04-13](../../sessions/2026-04-13-rotatingMachine-trial-ready.md).
|
||||
56
wiki/Archive/metrics.md
Normal file
56
wiki/Archive/metrics.md
Normal file
@@ -0,0 +1,56 @@
|
||||
---
|
||||
title: Metrics Dashboard
|
||||
updated: 2026-04-07
|
||||
---
|
||||
|
||||
# Metrics Dashboard
|
||||
|
||||
All numbers with provenance. Source of truth: `knowledge-graph.yaml`.
|
||||
|
||||
## Test Results
|
||||
|
||||
| Suite | Pass/Total | File | Date |
|
||||
|---|---|---|---|
|
||||
| rotatingMachine basic | 10/10 | test/basic/*.test.js | 2026-04-07 |
|
||||
| rotatingMachine integration | 16/16 | test/integration/*.test.js | 2026-04-07 |
|
||||
| rotatingMachine edge | 17/17 | test/edge/*.test.js | 2026-04-07 |
|
||||
| machineGroupControl basic | 1/1 | test/basic/*.test.js | 2026-04-07 |
|
||||
| machineGroupControl integration | 3/3 | test/integration/*.test.js | 2026-04-07 |
|
||||
| machineGroupControl edge | 1/1 | test/edge/*.test.js | 2026-04-07 |
|
||||
|
||||
## Performance — machineGroupControl Optimization
|
||||
|
||||
| Metric | Value | Source | Date |
|
||||
|---|---|---|---|
|
||||
| BEP-Gravitation + refinement (3 pumps, 6 combos) | 0.027-0.153ms median | benchmark script | 2026-04-07 |
|
||||
| Tick loop budget used | 0.015% of 1000ms | benchmark script | 2026-04-07 |
|
||||
| Max gap from brute-force optimum (1000 steps) | 0.1% | [[BEP Gravitation Proof]] | 2026-04-07 |
|
||||
| Pump switching stability (5-95% sweep) | 1-2 transitions, no hysteresis | stability sweep | 2026-04-07 |
|
||||
|
||||
## Performance — rotatingMachine Prediction
|
||||
|
||||
| Metric | Value | Source |
|
||||
|---|---|---|
|
||||
| predict.y(x) call | O(log n), effectively O(1) | predict_class.js |
|
||||
| buildAllFxyCurves | sub-10ms for typical curves | predict_class.js |
|
||||
| Curve cache | full caching of splines + calculated curves | predict_class.js |
|
||||
|
||||
## Power Comparison: machineGroupControl vs Baselines
|
||||
|
||||
Station: 2x H05K-S03R + 1x C5-D03R-SHN1 @ ΔP=2000 mbar
|
||||
|
||||
| Demand | Qd (m3/h) | machineGroupControl | Spillover | Equal-all | Gap to optimum |
|
||||
|--------|-----------|--------------------|-----------|-----------|----|
|
||||
| 10% | 71 | 17.6 kW | 22.0 kW (+25%) | 23.9 kW (+36%) | -0.10% |
|
||||
| 25% | 136 | 34.6 kW | 36.3 kW (+5%) | 39.1 kW (+13%) | +0.01% |
|
||||
| 50% | 243 | 62.9 kW | 73.8 kW (+17%) | 64.2 kW (+2%) | -0.00% |
|
||||
| 75% | 351 | 96.8 kW | 102.9 kW (+6%) | 99.6 kW (+3%) | +0.08% |
|
||||
| 90% | 415 | 122.8 kW | 123.0 kW (0%) | 123.0 kW (0%) | +0.07% |
|
||||
|
||||
## Disproven Claims
|
||||
|
||||
| Claim | Evidence For | Evidence Against | Date |
|
||||
|---|---|---|---|
|
||||
| NCog as proportional weight works | Simple implementation | Starves small pumps, overloads large ones at high demand | 2026-04-07 |
|
||||
| Q/P ratio always has mid-range peak | Expected from pump physics | Monotonically decreasing at high ΔP due to affinity laws (P ∝ Q³) | 2026-04-07 |
|
||||
| Equal-marginal-cost solver is optimal | KKT theory for convex curves | C5 curve is non-convex due to sparse data points (5 per pressure) | 2026-04-07 |
|
||||
70
wiki/Archive/overview.md
Normal file
70
wiki/Archive/overview.md
Normal file
@@ -0,0 +1,70 @@
|
||||
---
|
||||
title: EVOLV Project Overview
|
||||
created: 2026-04-07
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [overview, wastewater, node-red, isa-88]
|
||||
---
|
||||
|
||||
# EVOLV — Edge-Layer Evolution for Optimized Virtualization
|
||||
|
||||
Industrial automation platform for wastewater treatment, built as custom Node-RED nodes by Waterschap Brabantse Delta R&D. Follows ISA-88 (S88) batch control standard.
|
||||
|
||||
## Stack
|
||||
|
||||
Node.js, Node-RED, InfluxDB (time-series), TensorFlow.js (prediction), CoolProp (thermodynamics). No build step — pure Node.js.
|
||||
|
||||
## Architecture
|
||||
|
||||
Each node follows a 3-tier pattern:
|
||||
1. **Entry file** — registers with Node-RED, admin HTTP endpoints
|
||||
2. **nodeClass** — Node-RED adapter (tick loop, message routing, status)
|
||||
3. **specificClass** — pure domain logic (physics, state machines, predictions)
|
||||
|
||||
3-port output convention: Port 0 = process data, Port 1 = InfluxDB telemetry, Port 2 = parent-child registration.
|
||||
|
||||
## What Works
|
||||
|
||||
| Capability | Status | Evidence |
|
||||
|---|---|---|
|
||||
| rotatingMachine state machine | proven | 76 tests passing, all sequences verified |
|
||||
| 3D pump curve prediction (flow/power from pressure+control) | proven | Monotonic cubic spline interpolation across 34 pressure levels |
|
||||
| NCog / BEP tracking per pump | proven | Produces meaningful values with differential pressure |
|
||||
| machineGroupControl BEP-Gravitation | proven | Within 0.1% of brute-force global optimum |
|
||||
| Combination selection (2^n exhaustive) | proven | Stable: 1-2 switches across 5-95% demand sweep, no hysteresis |
|
||||
| Prediction health scoring | proven | NRMSE drift, pressure source penalties, edge detection |
|
||||
| Hydraulic efficiency (η = QΔP/P) | proven | CoolProp density, head calculation |
|
||||
| Unit conversion chain | proven | No double-conversion, clean layer separation |
|
||||
|
||||
## What Doesn't Work (honestly)
|
||||
|
||||
| Issue | Status | Evidence |
|
||||
|---|---|---|
|
||||
| C5 curve non-convexity | evolving | 5 raw data points cause spline artifacts, dP/dQ non-monotonic |
|
||||
| NCog = 0 at high ΔP | evolving | At ΔP > 800 mbar for H05K, Q/P is monotonically decreasing |
|
||||
| calcBestCombination (NCog-weight mode) | disproven | Uses NCog as proportional weight instead of BEP target |
|
||||
|
||||
## Current Scale
|
||||
|
||||
- 13 custom Node-RED nodes (12 submodules + generalFunctions)
|
||||
- rotatingMachine: 76 tests, 1563 lines domain logic
|
||||
- machineGroupControl: 90+ tests, 1400+ lines domain logic
|
||||
- 3 real pump curves: H05K-S03R, C5-D03R-SHN1, ECDV
|
||||
- Tick loop: 1000ms interval
|
||||
|
||||
## Node Inventory
|
||||
|
||||
| Node | Purpose | Test Status |
|
||||
|------|---------|-------------|
|
||||
| rotatingMachine | Pump/compressor control | 76 tests (full) |
|
||||
| machineGroupControl | Multi-pump optimization | 90 tests (full) |
|
||||
| pumpingStation | Multi-pump station | needs review |
|
||||
| valve | Valve modeling | needs review |
|
||||
| valveGroupControl | Valve group coordination | needs review |
|
||||
| reactor | Biological reactor (ASM kinetics) | needs review |
|
||||
| settler | Secondary clarifier | needs review |
|
||||
| monster | Multi-parameter bio monitoring | needs review |
|
||||
| measurement | Sensor signal conditioning | needs review |
|
||||
| diffuser | Aeration system control | needs review |
|
||||
| dashboardAPI | InfluxDB + FlowFuse charts | needs review |
|
||||
| generalFunctions | Shared utilities | partial |
|
||||
46
wiki/Archive/sessions-2026-04-07-production-hardening.md
Normal file
46
wiki/Archive/sessions-2026-04-07-production-hardening.md
Normal file
@@ -0,0 +1,46 @@
|
||||
---
|
||||
title: "Session: Production Hardening rotatingMachine + machineGroupControl"
|
||||
created: 2026-04-07
|
||||
updated: 2026-04-07
|
||||
status: proven
|
||||
tags: [session, rotatingMachine, machineGroupControl, testing]
|
||||
---
|
||||
|
||||
# 2026-04-07 — Production Hardening
|
||||
|
||||
## Scope
|
||||
Full code review and hardening of rotatingMachine and machineGroupControl nodes for production readiness.
|
||||
|
||||
## Key Discoveries
|
||||
|
||||
1. **Efficiency rounding destroyed NCog/BEP** — `Math.round(Q/P * 100) / 100` in canonical units (m3/s and W) produces ratios ~1e-6 that all round to 0. All NCog, cog, and BEP calculations were non-functional. Fixed by removing rounding.
|
||||
|
||||
2. **flowmovement unit mismatch** — machineGroupControl computed flow in canonical (m3/s) and sent it directly to rotatingMachine which expected output units (m3/h). Every pump stayed at minimum flow. Fixed with `_canonicalToOutputFlow()`.
|
||||
|
||||
3. **emergencyStop case mismatch** — `"emergencyStop"` vs config key `"emergencystop"`. Emergency stop never worked. Fixed to lowercase.
|
||||
|
||||
4. **Curve data anomalies** — 3 flow values leaked into power columns in hidrostal-H05K-S03R.json at pressures 1600, 3200, 3300 mbar. Fixed with interpolated values.
|
||||
|
||||
5. **C5 pump non-convexity** — 5 data points per pressure level produces non-convex spline. The marginal-cost refinement loop closes the gap to brute-force optimum from 2.1% to 0.1%.
|
||||
|
||||
## Changes Made
|
||||
|
||||
### rotatingMachine (3 files, 7 test files)
|
||||
- Async input handler, null guards, listener cleanup, tick loop race fix
|
||||
- showCoG() implementation, efficiency variant fix, curve anomaly detection
|
||||
- 43 new tests (76 total)
|
||||
|
||||
### machineGroupControl (1 file, 2 test files)
|
||||
- `_canonicalToOutputFlow()` on all flowmovement calls
|
||||
- Absolute scaling bug, empty Qd block, empty-machines guards
|
||||
- Marginal-cost refinement loop in BEP-Gravitation
|
||||
- Missing flowmovement after startup in equalFlowControl
|
||||
|
||||
### generalFunctions (1 file)
|
||||
- 3 curve data fixes in hidrostal-H05K-S03R.json
|
||||
|
||||
## Verification
|
||||
- 90 tests passing across both nodes
|
||||
- machineGroupControl within 0.1% of brute-force global optimum (1000-step search)
|
||||
- Pump switching stable: 1-2 transitions across full demand range, no hysteresis
|
||||
- Optimization cost: 0.03-0.15ms per call (0.015% of tick budget)
|
||||
109
wiki/Archive/sessions-2026-04-13-measurement-digital-mode.md
Normal file
109
wiki/Archive/sessions-2026-04-13-measurement-digital-mode.md
Normal file
@@ -0,0 +1,109 @@
|
||||
---
|
||||
title: "Session: measurement node — dispatcher bug fix + digital/MQTT mode"
|
||||
created: 2026-04-13
|
||||
updated: 2026-04-13
|
||||
status: proven
|
||||
tags: [session, measurement, smoothing, outlier, mqtt, iot]
|
||||
---
|
||||
|
||||
# 2026-04-13 — measurement trial-ready + digital mode
|
||||
|
||||
## Scope
|
||||
|
||||
Honest review of the `measurement` node. Benchmark every method, reason about keeping the node agnostic across analog and digital sources, add a digital (MQTT/IoT) mode without breaking analog.
|
||||
|
||||
## Findings
|
||||
|
||||
### Silent dispatcher bug (critical)
|
||||
|
||||
`validateEnum` in `generalFunctions` lowercases enum values (`zScore` → `zscore`, `lowPass` → `lowpass`). But `specificClass.outlierDetection` and `specificClass.applySmoothing` compared against camelCase keys. Effect:
|
||||
|
||||
- 5 of 11 smoothing methods silently fell through to a no-op: `lowPass`, `highPass`, `weightedMovingAverage`, `bandPass`, `savitzkyGolay`.
|
||||
- 2 of 3 outlier methods silently disabled: `zScore`, `modifiedZScore`.
|
||||
- Only `mean`, `median`, `sd`, `min`, `max`, `none`, `kalman`, `iqr` (the already-lowercase ones) actually worked.
|
||||
|
||||
Users who picked any camelCase method from the dropdown got the raw last value or no outlier filtering, with no error. Flows deployed before this session that relied on these filters got no filtering at all.
|
||||
|
||||
### Test coverage was thin
|
||||
|
||||
Pre-session: **12 tests** — 1 for scaling, 1 for outlier toggle, 1 for event emit, 3 for example flow shape, 1 constructor, 1 routing, 1 invalid payload, 2 other. Every smoothing method beyond `mean` and every outlier method beyond a toggle-flip was untested. The dispatcher bug would have been caught immediately by per-method unit tests.
|
||||
|
||||
### Analog-only input shape
|
||||
|
||||
The node only accepted scalar `msg.payload`. MQTT / IoT devices commonly publish a single JSON blob with many readings per message. Every user wanting that pattern had to fan out into N measurement nodes — ugly, and the device's shared timestamp is lost.
|
||||
|
||||
## Fixes + additions
|
||||
|
||||
### Dispatcher normalization (`specificClass.js`)
|
||||
|
||||
Both `outlierDetection()` and `applySmoothing()` now lowercase the configured method and the lookup table keys. Legacy camelCase config values and normalized lowercase config values both work.
|
||||
|
||||
### `MeasurementContainer.isUnitCompatible` permissive short-circuit
|
||||
|
||||
Previously: if the unit couldn't be described by the convert module, compatibility returned false regardless of type. This blocked user-defined types like `humidity` with unit `%`. Now: when `measureMap[type]` is undefined (unknown type), accept any unit. Known types still validate strictly.
|
||||
|
||||
### Digital mode (new)
|
||||
|
||||
`config.mode.current === 'digital'` opts into a new input shape. `config.channels` declares one entry per JSON key. The new `Channel` class (`src/channel.js`) is a self-contained per-channel pipeline — outlier → offset → scaling → smoothing → min/max → constrain → emit. Analog behaviour is preserved exactly; flows built before this session work unchanged.
|
||||
|
||||
## Test additions
|
||||
|
||||
Before → after: **12 → 71** tests.
|
||||
|
||||
New files:
|
||||
- `test/basic/smoothing-methods.basic.test.js` — every smoothing method covered, 16 tests.
|
||||
- `test/basic/outlier-detection.basic.test.js` — every outlier method + toggle + fall-through, 10 tests.
|
||||
- `test/basic/scaling-and-interpolation.basic.test.js` — offset / interpolateLinear / constrain / handleScaling / updateMinMaxValues / updateOutputPercent / updateOutputAbs / getOutput, 10 tests.
|
||||
- `test/basic/calibration-and-stability.basic.test.js` — calibrate / isStable / evaluateRepeatability / toggleSimulation / tick / simulateInput, 11 tests.
|
||||
- `test/integration/digital-mode.integration.test.js` — 12 tests covering channel build, payload dispatch, multi-channel emit, unknown keys, per-channel scaling / smoothing / outlier, empty channels, malformed entries, non-numeric values, digital-output shape.
|
||||
|
||||
## E2E verification (Dockerized Node-RED)
|
||||
|
||||
### Analog baseline — `/tmp/m_e2e_baseline.py`
|
||||
|
||||
Deploys `examples/basic.flow.json`, fires `{topic:"measurement", payload:42}` repeatedly. Observed port-0 output: `mAbs` climbed 0 → 2.1 → 2.8 → 3.15 → 3.36 → 4.2 across five ticks as the mean window filled with 42s (scaling 0..100 → 0..10). Tick cadence 909–1001 ms (avg 981 ms). Registration at t=0.22 s.
|
||||
|
||||
### Digital end-to-end — `/tmp/m_digital_e2e.py`
|
||||
|
||||
Deploys a single measurement node in digital mode with three channels (`temperature` / `humidity` / `pressure`) and fires two MQTT-shaped payloads.
|
||||
|
||||
| Tick | Channel | mAbs | totalMinSmooth | totalMaxSmooth |
|
||||
|---|---|---:|---:|---:|
|
||||
| after inject 1 | temperature | 22.5 | 22.5 | 22.5 |
|
||||
| after inject 1 | humidity | 45 | 45 | 45 |
|
||||
| after inject 1 | pressure | 1013 | 1013 | 1013 |
|
||||
| after inject 2 | temperature | 24 | 22.5 | 24 |
|
||||
| after inject 2 | humidity | 42.5 | 42.5 | 45 |
|
||||
| after inject 2 | pressure | 1014 | 1013 | 1014 |
|
||||
|
||||
Mean smoothing across a window of 3 computed per-channel, the `unknown` key in the payload ignored, all three events emitted on `<type>.measured.atequipment`.
|
||||
|
||||
## Files changed
|
||||
|
||||
```
|
||||
nodes/generalFunctions/src/measurements/MeasurementContainer.js # permissive unit check for user-defined types
|
||||
nodes/generalFunctions/src/configs/measurement.json # mode + channels schema
|
||||
|
||||
nodes/measurement/src/channel.js # new per-channel pipeline class
|
||||
nodes/measurement/src/specificClass.js # dispatcher fix + digital dispatch
|
||||
nodes/measurement/src/nodeClass.js # mode-aware input handler + tick
|
||||
nodes/measurement/measurement.html # Mode dropdown + Channels JSON + help panel
|
||||
nodes/measurement/README.md # rewrite
|
||||
|
||||
nodes/measurement/test/basic/smoothing-methods.basic.test.js # +16 tests
|
||||
nodes/measurement/test/basic/outlier-detection.basic.test.js # +10 tests
|
||||
nodes/measurement/test/basic/scaling-and-interpolation.basic.test.js # +10 tests
|
||||
nodes/measurement/test/basic/calibration-and-stability.basic.test.js # +11 tests
|
||||
nodes/measurement/test/integration/digital-mode.integration.test.js # +12 tests
|
||||
```
|
||||
|
||||
## Production status
|
||||
|
||||
Trial-ready for both modes. Supervised trial recommended for digital-mode deployments until the channels-editor UI (currently a JSON textarea) lands.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- Repeatable-row editor widget for channels.
|
||||
- `validateArray.minLength=0` evaluates as falsy; pre-existing generalFunctions bug affecting this node's `channels` and also `measurement.assetRegistration.childAssets`. Harmless warn at deploy time.
|
||||
- Per-channel calibration + simulation for digital mode.
|
||||
- Runtime channel reconfiguration via a dedicated topic (`addChannel` / `removeChannel`).
|
||||
134
wiki/Archive/sessions-2026-04-13-rotatingMachine-trial-ready.md
Normal file
134
wiki/Archive/sessions-2026-04-13-rotatingMachine-trial-ready.md
Normal file
@@ -0,0 +1,134 @@
|
||||
---
|
||||
title: "Session: rotatingMachine trial-ready — FSM interruptibility, config schema, UX fixes"
|
||||
created: 2026-04-13
|
||||
updated: 2026-04-13
|
||||
status: proven
|
||||
tags: [session, rotatingMachine, state-machine, docker, e2e]
|
||||
---
|
||||
|
||||
# 2026-04-13 — rotatingMachine trial-ready
|
||||
|
||||
## Scope
|
||||
|
||||
Honest review + production-hardening pass on `rotatingMachine`. Fixes landed on top of the 2026-04-07 hardening and are verified against a Docker-hosted Node-RED stack.
|
||||
|
||||
## Findings (before fixes)
|
||||
|
||||
From a live E2E run captured via the Node-RED debug websocket (`/comms`):
|
||||
|
||||
- **Clean startup→operational→shutdown→idle path** works to spec: 3 s starting + 2 s warmup + 3 s stopping + 2 s cooldown, matching config exactly.
|
||||
- **Tick cadence:** 1000 ms (min 1000, max 1005, avg 1002.5).
|
||||
- **Predictions** gate correctly on pressure injection; at 900 mbar Δp the hidrostal-H05K-S03R curve yields a monotonic flow/power response.
|
||||
- **State machine FSM** *rejects* `stopping`/`coolingdown`/`idle` transitions while the machine is in `accelerating`/`decelerating`, leaving a shutdown command silently dropped. Log symptom: `Invalid transition from accelerating to stopping. Transition not executed.`
|
||||
- **Sequence `emergencyStop` not defined** warn appears when a parent orchestrator with the capital-S casing (e.g. `machineGroupControl` config) forwards the sequence name.
|
||||
- **Config validator strips** `functionality.distance` and top-level `output` that `buildConfig` adds; every deploy prints removal warnings.
|
||||
- Cosmetic: typo "acurate" in single-side pressure warn; editor lacks unit hints for `speed` / `startup` / etc.
|
||||
|
||||
## Fixes
|
||||
|
||||
### 1. Interruptible movement (`generalFunctions/src/state/state.js`)
|
||||
|
||||
`moveTo`'s `catch` block now detects `Movement aborted` / `Transition aborted` errors and transitions the FSM back to `operational`, unblocking subsequent sequence transitions. A new `movementAborted` event is emitted for observability.
|
||||
|
||||
### 2. Auto-abort on shutdown/emergency-stop (`rotatingMachine/src/specificClass.js`)
|
||||
|
||||
`executeSequence` now:
|
||||
|
||||
- Normalizes the sequence name to lowercase (defensive against parent callers using mixed case).
|
||||
- When `shutdown` or `emergencystop` is requested from `accelerating`/`decelerating`, calls `state.abortCurrentMovement(...)` and waits up to 2 s for the FSM to return to `operational` via the new `_waitForOperational(timeoutMs)` helper that listens on the state emitter.
|
||||
|
||||
### 3. Config schema sync (`generalFunctions/src/configs/rotatingMachine.json`)
|
||||
|
||||
Added to the schema:
|
||||
|
||||
- `functionality.distance`, `.distanceUnit`, `.distanceDescription` (produced by the HTML editor).
|
||||
- Top-level `output.process` / `output.dbase` (produced by `buildConfig`).
|
||||
|
||||
Also reverted an overly broad `buildConfig` addition to only emit `distance` (not `distanceUnit`/`distanceDescription`) so other nodes aren't forced to add these to their schemas.
|
||||
|
||||
### 4. UX polish
|
||||
|
||||
- Fixed typo "acurate" → "accurate" in the single-side pressure warning, plus made the message actionable.
|
||||
- Added unit hints to Reaction Speed / Startup / Warmup / Shutdown / Cooldown fields in the editor.
|
||||
- Expanded the Node-RED help panel with a topic reference, state diagram, prediction rules, and port documentation.
|
||||
|
||||
## Tests added
|
||||
|
||||
`test/integration/interruptible-movement.integration.test.js` — three regression tests for the FSM fix:
|
||||
|
||||
- `shutdown during accelerating aborts the move and reaches idle`
|
||||
- `emergency stop during accelerating reaches off`
|
||||
- `executeSequence accepts mixed-case sequence names`
|
||||
|
||||
`test/integration/curve-prediction.integration.test.js` — 12 parametrized tests across both shipped pump curves (`hidrostal-H05K-S03R` and `hidrostal-C5-D03R-SHN1`):
|
||||
|
||||
- Curve loader returns nq + np with matching pressure slices.
|
||||
- Predicted flow and power at mid-pressure / mid-ctrl are finite and inside the curve envelope.
|
||||
- Flow is monotonically non-decreasing across a ctrl sweep at fixed pressure.
|
||||
- Flow decreases (or stays level) when pressure rises at fixed ctrl — centrifugal-pump physics.
|
||||
- CoG / NCog are computed, finite, and inside [0, 100] controller units.
|
||||
- Reverse predictor (flow → ctrl via reversed nq) round-trips within 10 % of the known controller position.
|
||||
|
||||
`test/e2e/curve-prediction-benchmark.py` + `test/e2e/README.md` — live Dockerized Node-RED benchmark that deploys one rotatingMachine per curve and records a (pressure × ctrl) sweep.
|
||||
|
||||
Full unit suite: **91/91 passing** (was 76/76 on the morning review).
|
||||
|
||||
## E2E verification (Dockerized Node-RED)
|
||||
|
||||
Via `/tmp/rm_e2e_verify.py` — deploys the example flow to `docker compose`-hosted Node-RED, drives it via `POST /inject/:id`, captures port-output via `ws://localhost:1880/comms`.
|
||||
|
||||
| Scenario | Observed state sequence | Pass? |
|
||||
|---|---|---|
|
||||
| Shutdown fired while `accelerating` | starting → warmingup → operational → accelerating → decelerating → stopping → coolingdown → **idle** | ✅ |
|
||||
| Emergency stop fired while `accelerating` | starting → warmingup → operational → accelerating → **off** | ✅ |
|
||||
| Clean startup → shutdown (regression) | starting → warmingup → operational → stopping → coolingdown → idle | ✅ |
|
||||
|
||||
Container log scan over a 3-minute window:
|
||||
|
||||
- `Unknown key` warns: 0 (was 6+ per deploy)
|
||||
- `acurate` typo: 0 (was 2)
|
||||
- `Invalid transition from accelerating/decelerating to ...` errors: 0 (was 3+)
|
||||
- `Sequence '...' not defined`: 0 (was 1)
|
||||
|
||||
### Dual-curve prediction sweep
|
||||
|
||||
Via `nodes/rotatingMachine/test/e2e/curve-prediction-benchmark.py`. Deploys two live rotatingMachines, one per pump curve, and runs a (pressure × ctrl) sweep per pump. Each pump is tested only inside its own curve envelope.
|
||||
|
||||
| Pump | Pressures swept (mbar) | Ctrl setpoints (%) | Samples in envelope | Flow monotonic | Flow observed (m³/h) | Power observed (kW) |
|
||||
|---|---|---|---|---|---|---|
|
||||
| hidrostal-H05K-S03R | 700 / 2300 / 3900 | 20 / 40 / 60 / 80 | 12/12 ✅ | ✅ | 10.3 – 208.3 | 12.3 – 50.3 |
|
||||
| hidrostal-C5-D03R-SHN1 | 400 / 1700 / 2900 | 20 / 40 / 60 / 80 | 12/12 ✅ | ✅ | 8.7 – 45.6 | 0.7 – 13.0 |
|
||||
|
||||
Inverse-pressure monotonicity (centrifugal-pump physics) also verified: for both pumps, flow at the highest pressure slice is strictly lower than flow at the lowest pressure slice for the same ctrl.
|
||||
|
||||
**Known limitation** captured in the memory file: extrapolating pressure *below* the curve's minimum slice produces nonsensical flow values (e.g. H05K at 400 mbar ctrl=20% predicts ~30 000 m³/h vs envelope max 227 m³/h). Upstream `measurement` nodes are expected to clamp sensors to realistic ranges; rotatingMachine itself does not.
|
||||
|
||||
Separately, the C5 curve still exhibits the previously-documented power non-monotonicity at p=1700 mbar (sparse-data spline artefact noted in the 2026-04-07 session); this is compensated by the group-optimization marginal-cost refinement loop.
|
||||
|
||||
## Files changed
|
||||
|
||||
```
|
||||
nodes/generalFunctions/src/state/state.js # abort recovery
|
||||
nodes/generalFunctions/src/configs/index.js # buildConfig trim
|
||||
nodes/generalFunctions/src/configs/rotatingMachine.json # schema sync
|
||||
nodes/rotatingMachine/src/specificClass.js # exec + typo
|
||||
nodes/rotatingMachine/rotatingMachine.html # UX hints + help
|
||||
nodes/rotatingMachine/test/integration/interruptible-movement.integration.test.js # +3 tests (FSM)
|
||||
nodes/rotatingMachine/test/integration/curve-prediction.integration.test.js # +12 tests (dual curve)
|
||||
nodes/rotatingMachine/test/e2e/curve-prediction-benchmark.py # new E2E benchmark
|
||||
nodes/rotatingMachine/test/e2e/README.md # benchmark docs
|
||||
nodes/rotatingMachine/README.md # rewrite
|
||||
```
|
||||
|
||||
## Production readiness
|
||||
|
||||
Status: **trial-ready**. The caveats flagged in the 2026-04-13 memory file (`node_rotatingMachine.md`) are resolved. Remaining items are in the wishlist (interruptible curve validation feedback, domain review of ctrl≈0% + backpressure flow prediction, opt-in full-snapshot port-0 mode, per-machine `/health` endpoint).
|
||||
|
||||
## Verification command
|
||||
|
||||
```bash
|
||||
cd /mnt/d/gitea/EVOLV
|
||||
docker compose up -d nodered influxdb
|
||||
cd nodes/rotatingMachine && npm test
|
||||
python3 /tmp/rm_e2e_verify.py # end-to-end smoke
|
||||
```
|
||||
Reference in New Issue
Block a user