From 48a2e509c2b19b01823a7469d09b4334020f4f19 Mon Sep 17 00:00:00 2001 From: znetsixe Date: Mon, 11 May 2026 21:49:49 +0200 Subject: [PATCH] Refactor master wiki: 7 new master pages + accurate Home + archive cleanup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the prior 9 Architecture-* pre-refactor planning docs and the broken Home with a complete visual-first redesign. All Mermaid diagrams have been verified against the actual configure() declarations in each node's specificClass.js — no fabricated edges. New master pages: - Home — accurate platform overview, full live-nodes table, concept index, refactor status - Architecture — 3-tier code structure, generalFunctions API surface (12-row), output ports, child-registration sequence - Topology-Patterns — 5 verified plant configurations with worked examples (PS+MGC+pumps, reactor/settler train, VGC manifold, composite sampling, dashboard provisioning) - Topic-Conventions — set./cmd./evt./data./child. semantics, unit policy, S88 palette, measurement key shape, status badge, HealthStatus - Telemetry — Port 0/1/2 contracts, InfluxDB line protocol layout, FlowFuse chart wiring, Grafana provisioning - Getting-Started — clone/install/Docker/local Node-RED setup - Glossary — S88, EVOLV runtime, WWTP, pumps, control, project terms - _Sidebar.md — gitea wiki navigation Concept/finding/manual pages: flattened from source wiki/concepts, wiki/findings, wiki/manuals/node-red with proper prefixes. Pre-existing 9 wiki pages (Architecture-*, AI-assisted coding) moved to Archive- prefix with the archive banner stamped on top. 20 source-tree archive pages mirrored as Archive-Source-* for completeness. --- Architecture.md | 170 +++++ ...ng.-.md => Archive-AI-assisted-coding.-.md | 9 + ...ure-Configuration-Model-and-Tagcodering.md | 9 + ...Archive-Architecture-Container-Topology.md | 9 + ...chive-Architecture-Deployment-Blueprint.md | 9 + ...hitecture-Deployment-Controls-Checklist.md | 9 + ... Archive-Architecture-Platform-Overview.md | 9 + ...itecture-Security-and-Access-Boundaries.md | 9 + ...tecture-Security-and-Regulatory-Mapping.md | 9 + ...rchitecture-Telemetry-and-Smart-Storage.md | 9 + Archive-Source-SCHEMA.md | 96 +++ Archive-Source-architecture-3d-pump-curves.md | 64 ++ ...ource-architecture-deployment-blueprint.md | 286 ++++++++ ...-Source-architecture-group-optimization.md | 53 ++ ...e-Source-architecture-node-architecture.md | 434 ++++++++++++ ...e-Source-architecture-platform-overview.md | 166 +++++ Archive-Source-architecture-stack-review.md | 640 ++++++++++++++++++ ...ve-Source-concepts-generalfunctions-api.md | 462 +++++++++++++ Archive-Source-concepts-sources-readme.md | 28 + ...ive-Source-findings-open-issues-2026-03.md | 96 +++ Archive-Source-index.md | 65 ++ Archive-Source-knowledge-graph.yaml | 168 +++++ Archive-Source-log.md | 19 + Archive-Source-manuals-nodes-measurement.md | 211 ++++++ ...ve-Source-manuals-nodes-rotatingMachine.md | 255 +++++++ Archive-Source-metrics.md | 64 ++ Archive-Source-overview.md | 78 +++ ...essions-2026-04-07-production-hardening.md | 54 ++ ...ons-2026-04-13-measurement-digital-mode.md | 117 ++++ ...-2026-04-13-rotatingMachine-trial-ready.md | 142 ++++ Archive.md | 56 +- Concept-ASM-Models.md | 127 ++++ Concept-InfluxDB-Schema-Design.md | 122 ++++ Concept-OT-Security-IEC62443.md | 149 ++++ Concept-PID-Control-Theory.md | 168 +++++ Concept-Pump-Affinity-Laws.md | 154 +++++ Concept-Settling-Models.md | 130 ++++ Concept-Signal-Processing-Sensors.md | 157 +++++ Concept-Wastewater-Compliance-NL.md | 117 ++++ Finding-BEP-Gravitation-Proof.md | 38 ++ Finding-Curve-Non-Convexity.md | 34 + Finding-NCog-Behavior.md | 42 ++ Finding-Pump-Switching-Stability.md | 34 + Getting-Started.md | 172 +++++ Glossary.md | 132 ++++ Home.md | 120 ++-- ...odeRED-Flowfuse-Dashboard-Layout-Manual.md | 47 ++ Manual-NodeRED-Flowfuse-Ui-Button-Manual.md | 62 ++ Manual-NodeRED-Flowfuse-Ui-Chart-Manual.md | 121 ++++ Manual-NodeRED-Flowfuse-Ui-Config-Manual.md | 103 +++ Manual-NodeRED-Flowfuse-Ui-Gauge-Manual.md | 108 +++ Manual-NodeRED-Flowfuse-Ui-Template-Manual.md | 112 +++ Manual-NodeRED-Flowfuse-Ui-Text-Manual.md | 72 ++ Manual-NodeRED-Flowfuse-Widgets-Catalog.md | 49 ++ Manual-NodeRED-Function-Node-Patterns.md | 29 + Manual-NodeRED-INDEX.md | 30 + ...l-NodeRED-Messages-And-Editor-Structure.md | 27 + Manual-NodeRED-Runtime-Node-Js.md | 31 + Telemetry.md | 202 ++++++ Topic-Conventions.md | 185 +++++ Topology-Patterns.md | 248 +++++++ _Sidebar.md | 45 ++ 62 files changed, 6921 insertions(+), 51 deletions(-) create mode 100755 Architecture.md rename AI-assisted coding.-.md => Archive-AI-assisted-coding.-.md (64%) rename Architecture-Configuration-Model-and-Tagcodering.md => Archive-Architecture-Configuration-Model-and-Tagcodering.md (73%) rename Architecture-Container-Topology.md => Archive-Architecture-Container-Topology.md (79%) rename Architecture-Deployment-Blueprint.md => Archive-Architecture-Deployment-Blueprint.md (85%) rename Architecture-Deployment-Controls-Checklist.md => Archive-Architecture-Deployment-Controls-Checklist.md (93%) rename Architecture-Platform-Overview.md => Archive-Architecture-Platform-Overview.md (82%) rename Architecture-Security-and-Access-Boundaries.md => Archive-Architecture-Security-and-Access-Boundaries.md (91%) rename Architecture-Security-and-Regulatory-Mapping.md => Archive-Architecture-Security-and-Regulatory-Mapping.md (91%) rename Architecture-Telemetry-and-Smart-Storage.md => Archive-Architecture-Telemetry-and-Smart-Storage.md (80%) create mode 100755 Archive-Source-SCHEMA.md create mode 100755 Archive-Source-architecture-3d-pump-curves.md create mode 100755 Archive-Source-architecture-deployment-blueprint.md create mode 100755 Archive-Source-architecture-group-optimization.md create mode 100755 Archive-Source-architecture-node-architecture.md create mode 100755 Archive-Source-architecture-platform-overview.md create mode 100755 Archive-Source-architecture-stack-review.md create mode 100755 Archive-Source-concepts-generalfunctions-api.md create mode 100755 Archive-Source-concepts-sources-readme.md create mode 100755 Archive-Source-findings-open-issues-2026-03.md create mode 100755 Archive-Source-index.md create mode 100755 Archive-Source-knowledge-graph.yaml create mode 100755 Archive-Source-log.md create mode 100755 Archive-Source-manuals-nodes-measurement.md create mode 100755 Archive-Source-manuals-nodes-rotatingMachine.md create mode 100755 Archive-Source-metrics.md create mode 100755 Archive-Source-overview.md create mode 100755 Archive-Source-sessions-2026-04-07-production-hardening.md create mode 100755 Archive-Source-sessions-2026-04-13-measurement-digital-mode.md create mode 100755 Archive-Source-sessions-2026-04-13-rotatingMachine-trial-ready.md create mode 100755 Concept-ASM-Models.md create mode 100755 Concept-InfluxDB-Schema-Design.md create mode 100755 Concept-OT-Security-IEC62443.md create mode 100755 Concept-PID-Control-Theory.md create mode 100755 Concept-Pump-Affinity-Laws.md create mode 100755 Concept-Settling-Models.md create mode 100755 Concept-Signal-Processing-Sensors.md create mode 100755 Concept-Wastewater-Compliance-NL.md create mode 100755 Finding-BEP-Gravitation-Proof.md create mode 100755 Finding-Curve-Non-Convexity.md create mode 100755 Finding-NCog-Behavior.md create mode 100755 Finding-Pump-Switching-Stability.md create mode 100755 Getting-Started.md create mode 100755 Glossary.md create mode 100755 Manual-NodeRED-Flowfuse-Dashboard-Layout-Manual.md create mode 100755 Manual-NodeRED-Flowfuse-Ui-Button-Manual.md create mode 100755 Manual-NodeRED-Flowfuse-Ui-Chart-Manual.md create mode 100755 Manual-NodeRED-Flowfuse-Ui-Config-Manual.md create mode 100755 Manual-NodeRED-Flowfuse-Ui-Gauge-Manual.md create mode 100755 Manual-NodeRED-Flowfuse-Ui-Template-Manual.md create mode 100755 Manual-NodeRED-Flowfuse-Ui-Text-Manual.md create mode 100755 Manual-NodeRED-Flowfuse-Widgets-Catalog.md create mode 100755 Manual-NodeRED-Function-Node-Patterns.md create mode 100755 Manual-NodeRED-INDEX.md create mode 100755 Manual-NodeRED-Messages-And-Editor-Structure.md create mode 100755 Manual-NodeRED-Runtime-Node-Js.md create mode 100755 Telemetry.md create mode 100755 Topic-Conventions.md create mode 100755 Topology-Patterns.md create mode 100755 _Sidebar.md diff --git a/Architecture.md b/Architecture.md new file mode 100755 index 0000000..769dfbb --- /dev/null +++ b/Architecture.md @@ -0,0 +1,170 @@ +# Architecture + +> **Reflects code as of `9ab9f6b` · regenerated `2026-05-11`** + +How every EVOLV node is structured, and what the shared `generalFunctions` library provides. + +## The 3-tier node pattern + +```mermaid +flowchart LR + rt["Node-RED runtime"]:::neutral + subgraph node["Custom node (one folder under nodes/)"] + entry[".js
(entry — registers node type with RED)"]:::tier1 + nc["src/NodeClass.js
(nodeClass — Node-RED adapter)"]:::tier2 + sc["src/SpecificClass.js
(specificClass — pure domain logic)"]:::tier3 + end + rt -->|RED.nodes.registerType| entry + entry -->|new| nc + nc -->|new + configure()| sc + + classDef neutral fill:#dddddd,color:#000 + classDef tier1 fill:#a9daee,color:#000 + classDef tier2 fill:#86bbdd,color:#000 + classDef tier3 fill:#50a8d9,color:#000 +``` + +| Tier | Owns | Touches RED.* API? | Tested by | +|---|---|---|---| +| entry (`.js`) | Type registration, HTTP admin endpoints | yes | smoke tests | +| nodeClass (`src/...NodeClass.js`, extends `BaseNodeAdapter`) | msg routing, tick loop, output port wiring, status badge updates | yes | integration tests | +| specificClass (`src/...SpecificClass.js`, extends `BaseDomain`) | All business logic; emits via `this.emitter`; calls `this.measurements` / `this.router` | **no** — must be free of RED imports | unit tests | + +**Rule:** never import Node-RED APIs in the specificClass. The specificClass is unit-testable by `new SpecificClass(config)`. If you find `RED.*` calls outside the entry/nodeClass tiers, that's a bug. + +## generalFunctions — what it provides + +The `nodes/generalFunctions` submodule is a plain-JS library every node depends on. Public exports (top-level `require('generalFunctions')`): + +| Export | Role | +|---|---| +| `BaseDomain` | Base class for every specificClass. Owns `measurements`, `router`, `emitter`, `logger`, `unitPolicy`. | +| `BaseNodeAdapter` | Base class for every nodeClass. Wires `commandRegistry` to `node.on('input')`, owns tick loop. | +| `ChildRouter` | Declarative child-registration matcher. `router.onRegister(softwareType, handler)`, `router.onMeasurement(...)`. | +| `commandRegistry` | Topic → handler descriptor map. Owns alias resolution + unit coercion. | +| `UnitPolicy` | Per-node canonical + output units. Coerces incoming `msg.unit` to canonical. | +| `MeasurementContainer` | Chainable storage: `type(t).variant(v).position(p).value(x, ts, unit)`. Key shape: `...`. | +| `statusBadge` | Composer for `node.status({fill,shape,text})` updates. | +| `HealthStatus` | Standardised `{ level: 0..3, flags: [], message, source }` shape. | +| `LatestWinsGate` | Mutex with supersede semantics — keeps only the freshest in-flight call. | +| `logger` | Structured logger (use this; never `console.log`). | +| `configManager` | Loads JSON schemas from `src/configs/.json`. | +| `MenuManager` | Dynamic editor dropdowns (asset lists). | +| `outputUtils` | Delta-compressed Port-0 + InfluxDB-line-protocol Port-1 formatting. | + +See [generalFunctions Home →](https://gitea.wbd-rd.nl/RnD/generalFunctions/wiki/Home) for the full 34-row API table. + +## Output ports + +Every EVOLV node emits on three ports: + +```mermaid +flowchart LR + sc[specificClass]:::tier3 + p0[(Port 0
process data)]:::p0 + p1[(Port 1
InfluxDB line)]:::p1 + p2[(Port 2
registration / control)]:::p2 + sc --> p0 + sc --> p1 + sc --> p2 + + p0 -.-> dn1[downstream Node-RED nodes
dashboards, function nodes] + p1 -.-> influx[(InfluxDB)] + p2 -.-> parent[parent EVOLV node
via child.register] + + classDef tier3 fill:#50a8d9,color:#000 + classDef p0 fill:#86bbdd + classDef p1 fill:#a9daee + classDef p2 fill:#dddddd +``` + +| Port | Carries | Format | Cardinality | +|---|---|---|---| +| **0** Process | Delta-compressed measurement / state snapshot for downstream Node-RED logic. | `msg.payload` = object of changed keys only. | One msg per tick when something changed. | +| **1** Telemetry | InfluxDB line-protocol strings: `measurement,tag=val field=val ts`. | `msg.payload` = `string` (or array of strings). | One msg per tick when something changed; all numeric outputs. | +| **2** Registration / control | `child.register` upward on adapter init; control replies. | `{topic, payload: nodeRef}` | At init time + on demand. | + +See [Telemetry](Telemetry) for the full Port-1 schema and InfluxDB conventions. + +## Topic conventions + +| Prefix | Direction | Used for | +|---|---|---| +| `set.` | inbound | Set a configurable value (mode, setpoint). Idempotent. | +| `cmd.` | inbound | Trigger an action (startup, shutdown, calibrate). Has side-effects. | +| `data.` | inbound or outbound | Carries measurement data between child ↔ parent. | +| `evt.` | outbound | Signal that something happened (state change, alarm). | +| `child.` | inbound (on parent) | Child node registers itself with this parent. | + +See [Topic-Conventions](Topic-Conventions) for the full list, payload shapes, alias deprecation map. + +## Child registration + +When a node is configured with `parent` = some other node's id, on `init()` the nodeClass emits a `child.register` message on Port 2 toward the parent. The parent's `commandRegistry` routes it into `ChildRouter`, which fires the matching `onRegister(softwareType, handler)` declared in `configure()`. + +```mermaid +sequenceDiagram + participant childNc as Child nodeClass + participant parentReg as Parent commandRegistry + participant parentRouter as Parent ChildRouter + participant parentSc as Parent specificClass + + childNc->>parentReg: msg{topic: child.register, softwareType, ref} + parentReg->>parentRouter: dispatch(child.register, ref) + parentRouter->>parentRouter: match softwareType + parentRouter->>parentSc: invoke registered handler + parentSc->>parentSc: store ref, wire emitter.on(...) +``` + +A child is anything the parent's `configure()` declares via `router.onRegister(, handler)`. Examples: + +| Parent | Accepts children with softwareType | +|---|---| +| pumpingStation | `measurement`, `machine`, `machinegroup`, `pumpingstation` | +| machineGroupControl | `machine`, `measurement` | +| valveGroupControl | `valve`, `machine`, `machinegroup`, `pumpingstation`, `valvegroupcontrol` (last 4 as flow sources) | +| reactor | `measurement`, `reactor` | +| settler | `measurement`, `reactor`, `machine` | +| monster | `measurement` | +| diffuser | `measurement` | +| rotatingMachine | `measurement` | +| valve | `measurement` | +| dashboardAPI | any (used for Grafana provisioning) | + +## Where business logic lives + +```mermaid +flowchart TB + subgraph node["A node's src/ folder"] + sc["specificClass.js
orchestration only"] + subgraph concerns["Concern subdirs (per-node)"] + c1[basin/ or curves/ or kinetics/
physics / math] + c2[state/
FSM transitions] + c3[dispatch/ or safety/
action / guard logic] + c4[commands/
topic → handler descriptors] + c5[io/
output composition] + end + end + sc --> c1 + sc --> c2 + sc --> c3 + sc --> c4 + sc --> c5 +``` + +specificClass should be **stitching only** — instantiate concern modules in `configure()`, call them in `tick()` or in router handlers. Concerns are individually testable. + +## Reading order for newcomers + +1. `.claude/refactor/CONTRACTS.md` — every API shape this wiki abstracts over. +2. `.claude/refactor/CONVENTIONS.md` — code style, file size, naming. +3. `.claude/refactor/MODULE_SPLIT.md` — concern layout per node. +4. One node's `wiki/Home.md` (pumpingStation is the most mature pilot — start there). +5. The corresponding `src/` folder, top-down: specificClass → concern modules. + +## Related pages + +- [Topology-Patterns](Topology-Patterns) — typical plant configurations +- [Topic-Conventions](Topic-Conventions) — naming and units +- [Telemetry](Telemetry) — Port-1 InfluxDB schema +- [Getting-Started](Getting-Started) — hands-on first run diff --git a/AI-assisted coding.-.md b/Archive-AI-assisted-coding.-.md similarity index 64% rename from AI-assisted coding.-.md rename to Archive-AI-assisted-coding.-.md index 702d0ee..27eabb8 100644 --- a/AI-assisted coding.-.md +++ b/Archive-AI-assisted-coding.-.md @@ -1,3 +1,12 @@ +> **⚠️ ARCHIVED — pre-refactor wiki page** +> +> This page describes the architecture before the platform refactor (Tier 1–4, 2026-05). +> The current pages are **[Home](Home)**, **[Architecture](Architecture)**, **[Topology-Patterns](Topology-Patterns)**, **[Topic-Conventions](Topic-Conventions)**, and **[Telemetry](Telemetry)**. +> +> Kept for historical reference only. **Do not update.** + +--- + # Claude API Token Permissions For a safe **read + assist** role (Claude reads code, you approve and push), Claude needs: diff --git a/Architecture-Configuration-Model-and-Tagcodering.md b/Archive-Architecture-Configuration-Model-and-Tagcodering.md similarity index 73% rename from Architecture-Configuration-Model-and-Tagcodering.md rename to Archive-Architecture-Configuration-Model-and-Tagcodering.md index ad2fa47..adacd45 100644 --- a/Architecture-Configuration-Model-and-Tagcodering.md +++ b/Archive-Architecture-Configuration-Model-and-Tagcodering.md @@ -1,3 +1,12 @@ +> **⚠️ ARCHIVED — pre-refactor wiki page** +> +> This page describes the architecture before the platform refactor (Tier 1–4, 2026-05). +> The current pages are **[Home](Home)**, **[Architecture](Architecture)**, **[Topology-Patterns](Topology-Patterns)**, **[Topic-Conventions](Topic-Conventions)**, and **[Telemetry](Telemetry)**. +> +> Kept for historical reference only. **Do not update.** + +--- + # EVOLV Configuration Model and Tagcodering The intended long-term configuration authority for EVOLV is the database-backed `tagcodering` model. diff --git a/Architecture-Container-Topology.md b/Archive-Architecture-Container-Topology.md similarity index 79% rename from Architecture-Container-Topology.md rename to Archive-Architecture-Container-Topology.md index 6c6cbf9..f733e72 100644 --- a/Architecture-Container-Topology.md +++ b/Archive-Architecture-Container-Topology.md @@ -1,3 +1,12 @@ +> **⚠️ ARCHIVED — pre-refactor wiki page** +> +> This page describes the architecture before the platform refactor (Tier 1–4, 2026-05). +> The current pages are **[Home](Home)**, **[Architecture](Architecture)**, **[Topology-Patterns](Topology-Patterns)**, **[Topic-Conventions](Topic-Conventions)**, and **[Telemetry](Telemetry)**. +> +> Kept for historical reference only. **Do not update.** + +--- + # EVOLV Container Topology This page translates the deployment blueprint into a practical container/service split. diff --git a/Architecture-Deployment-Blueprint.md b/Archive-Architecture-Deployment-Blueprint.md similarity index 85% rename from Architecture-Deployment-Blueprint.md rename to Archive-Architecture-Deployment-Blueprint.md index 698f27d..c8ca73b 100644 --- a/Architecture-Deployment-Blueprint.md +++ b/Archive-Architecture-Deployment-Blueprint.md @@ -1,3 +1,12 @@ +> **⚠️ ARCHIVED — pre-refactor wiki page** +> +> This page describes the architecture before the platform refactor (Tier 1–4, 2026-05). +> The current pages are **[Home](Home)**, **[Architecture](Architecture)**, **[Topology-Patterns](Topology-Patterns)**, **[Topic-Conventions](Topic-Conventions)**, and **[Telemetry](Telemetry)**. +> +> Kept for historical reference only. **Do not update.** + +--- + # EVOLV Deployment Blueprint This page turns the architecture into a concrete deployment model. diff --git a/Architecture-Deployment-Controls-Checklist.md b/Archive-Architecture-Deployment-Controls-Checklist.md similarity index 93% rename from Architecture-Deployment-Controls-Checklist.md rename to Archive-Architecture-Deployment-Controls-Checklist.md index f0816cd..37a26fe 100644 --- a/Architecture-Deployment-Controls-Checklist.md +++ b/Archive-Architecture-Deployment-Controls-Checklist.md @@ -1,3 +1,12 @@ +> **⚠️ ARCHIVED — pre-refactor wiki page** +> +> This page describes the architecture before the platform refactor (Tier 1–4, 2026-05). +> The current pages are **[Home](Home)**, **[Architecture](Architecture)**, **[Topology-Patterns](Topology-Patterns)**, **[Topic-Conventions](Topic-Conventions)**, and **[Telemetry](Telemetry)**. +> +> Kept for historical reference only. **Do not update.** + +--- + # EVOLV Deployment Controls Checklist This page translates the EVOLV architecture into deployment controls that can be checked during implementation, review, FAT/SAT, and audit preparation. diff --git a/Architecture-Platform-Overview.md b/Archive-Architecture-Platform-Overview.md similarity index 82% rename from Architecture-Platform-Overview.md rename to Archive-Architecture-Platform-Overview.md index b441558..ecb0cae 100644 --- a/Architecture-Platform-Overview.md +++ b/Archive-Architecture-Platform-Overview.md @@ -1,3 +1,12 @@ +> **⚠️ ARCHIVED — pre-refactor wiki page** +> +> This page describes the architecture before the platform refactor (Tier 1–4, 2026-05). +> The current pages are **[Home](Home)**, **[Architecture](Architecture)**, **[Topology-Patterns](Topology-Patterns)**, **[Topic-Conventions](Topic-Conventions)**, and **[Telemetry](Telemetry)**. +> +> Kept for historical reference only. **Do not update.** + +--- + # EVOLV Platform Overview EVOLV is a layered automation platform: diff --git a/Architecture-Security-and-Access-Boundaries.md b/Archive-Architecture-Security-and-Access-Boundaries.md similarity index 91% rename from Architecture-Security-and-Access-Boundaries.md rename to Archive-Architecture-Security-and-Access-Boundaries.md index 85ff5c4..63d6c18 100644 --- a/Architecture-Security-and-Access-Boundaries.md +++ b/Archive-Architecture-Security-and-Access-Boundaries.md @@ -1,3 +1,12 @@ +> **⚠️ ARCHIVED — pre-refactor wiki page** +> +> This page describes the architecture before the platform refactor (Tier 1–4, 2026-05). +> The current pages are **[Home](Home)**, **[Architecture](Architecture)**, **[Topology-Patterns](Topology-Patterns)**, **[Topic-Conventions](Topic-Conventions)**, and **[Telemetry](Telemetry)**. +> +> Kept for historical reference only. **Do not update.** + +--- + # EVOLV Security and Access Boundaries EVOLV should expose central services, not field-edge systems, to external consumers. diff --git a/Architecture-Security-and-Regulatory-Mapping.md b/Archive-Architecture-Security-and-Regulatory-Mapping.md similarity index 91% rename from Architecture-Security-and-Regulatory-Mapping.md rename to Archive-Architecture-Security-and-Regulatory-Mapping.md index 2b9bc6c..845acfa 100644 --- a/Architecture-Security-and-Regulatory-Mapping.md +++ b/Archive-Architecture-Security-and-Regulatory-Mapping.md @@ -1,3 +1,12 @@ +> **⚠️ ARCHIVED — pre-refactor wiki page** +> +> This page describes the architecture before the platform refactor (Tier 1–4, 2026-05). +> The current pages are **[Home](Home)**, **[Architecture](Architecture)**, **[Topology-Patterns](Topology-Patterns)**, **[Topic-Conventions](Topic-Conventions)**, and **[Telemetry](Telemetry)**. +> +> Kept for historical reference only. **Do not update.** + +--- + # EVOLV Security and Regulatory Mapping This page maps major security and resilience requirement areas to the EVOLV architecture direction. diff --git a/Architecture-Telemetry-and-Smart-Storage.md b/Archive-Architecture-Telemetry-and-Smart-Storage.md similarity index 80% rename from Architecture-Telemetry-and-Smart-Storage.md rename to Archive-Architecture-Telemetry-and-Smart-Storage.md index bec3fbe..54c827b 100644 --- a/Architecture-Telemetry-and-Smart-Storage.md +++ b/Archive-Architecture-Telemetry-and-Smart-Storage.md @@ -1,3 +1,12 @@ +> **⚠️ ARCHIVED — pre-refactor wiki page** +> +> This page describes the architecture before the platform refactor (Tier 1–4, 2026-05). +> The current pages are **[Home](Home)**, **[Architecture](Architecture)**, **[Topology-Patterns](Topology-Patterns)**, **[Topic-Conventions](Topic-Conventions)**, and **[Telemetry](Telemetry)**. +> +> Kept for historical reference only. **Do not update.** + +--- + # EVOLV Telemetry and Smart Storage EVOLV uses InfluxDB on multiple levels: diff --git a/Archive-Source-SCHEMA.md b/Archive-Source-SCHEMA.md new file mode 100755 index 0000000..93b193c --- /dev/null +++ b/Archive-Source-SCHEMA.md @@ -0,0 +1,96 @@ +> **⚠️ ARCHIVED — pre-refactor (Tier 1–4, 2026-05)** +> +> This page describes the architecture before the platform refactor. +> The current page is the per-node wiki on **[gitea.wbd-rd.nl/RnD](https://gitea.wbd-rd.nl/RnD)** or **[Home](../Home)**. +> +> Kept for historical reference only. **Do not update.** + +# Project Wiki Schema + +## Purpose +LLM-maintained knowledge base for this project. The LLM writes and maintains everything. You read it (ideally in Obsidian). Knowledge compounds across sessions instead of being lost in chat history. + +## Directory Structure +``` +wiki/ + SCHEMA.md — this file (how to maintain the wiki) + index.md — catalog of all pages with one-line summaries + log.md — chronological record of updates + overview.md — project overview and current status + metrics.md — all numbers with provenance + knowledge-graph.yaml — structured data, machine-queryable + tools/ — search, lint, query scripts + concepts/ — core ideas and mechanisms + architecture/ — design decisions, system internals + findings/ — honest results (what worked AND what didn't) + sessions/ — per-session summaries +``` + +## Page Conventions + +### Frontmatter +Every page starts with YAML frontmatter: +```yaml +--- +title: Page Title +created: YYYY-MM-DD +updated: YYYY-MM-DD +status: proven | disproven | evolving | speculative +tags: [tag1, tag2] +sources: [path/to/file.py, commit abc1234] +--- +``` + +### Status values +- **proven**: tested and verified with evidence +- **disproven**: tested and honestly shown NOT to work (document WHY) +- **evolving**: partially working, boundary not fully mapped +- **speculative**: proposed but not yet tested + +### Cross-references +Use `[[Page Name]]` Obsidian-style wikilinks. + +### Contradictions +When new evidence contradicts a prior claim, DON'T delete the old claim. Add: +``` +> [!warning] Superseded +> This was shown to be incorrect on YYYY-MM-DD. See [[New Finding]]. +``` + +### Honesty rule +If something doesn't work, say so. If a result was a false positive, document how it was discovered. The wiki must be trustworthy. + +## Operations + +### Ingest (after a session or new source) +1. Read outputs, commits, findings +2. Update relevant pages +3. Create new pages for new concepts +4. Update `index.md`, `log.md`, `knowledge-graph.yaml` +5. Check for contradictions with existing pages + +### Query +1. Use `python3 wiki/tools/query.py` for structured lookup +2. Use `wiki/tools/search.sh` for full-text +3. Read `index.md` to find relevant pages +4. File valuable answers back into the wiki + +### Lint (periodically) +```bash +bash wiki/tools/lint.sh +``` +Checks: orphan pages, broken wikilinks, missing frontmatter, index completeness. + +## Data Layer + +- `knowledge-graph.yaml` — structured YAML with every metric and data point +- `metrics.md` — human-readable dashboard +- When adding new results, update BOTH the wiki page AND the knowledge graph +- The knowledge graph is the single source of truth for numbers + +## Source of Truth Hierarchy +1. **Test results** (actual outputs) — highest authority +2. **Code** (current state) — second authority +3. **Knowledge graph** (knowledge-graph.yaml) — structured metrics +4. **Wiki pages** — synthesis, may lag +5. **Chat/memory** — ephemeral, may be stale diff --git a/Archive-Source-architecture-3d-pump-curves.md b/Archive-Source-architecture-3d-pump-curves.md new file mode 100755 index 0000000..1840dc6 --- /dev/null +++ b/Archive-Source-architecture-3d-pump-curves.md @@ -0,0 +1,64 @@ +--- +title: 3D Pump Curve Architecture +created: 2026-04-07 +updated: 2026-04-07 +status: proven +tags: [predict, curves, interpolation, rotatingMachine] +sources: [nodes/generalFunctions/src/predict/predict_class.js, nodes/rotatingMachine/src/specificClass.js] +--- + +> **⚠️ ARCHIVED — pre-refactor (Tier 1–4, 2026-05)** +> +> This page describes the architecture before the platform refactor. +> The current page is the per-node wiki on **[gitea.wbd-rd.nl/RnD](https://gitea.wbd-rd.nl/RnD)** or **[Home](../Home)**. +> +> Kept for historical reference only. **Do not update.** + + +# 3D Pump Curve Prediction + +## Data Structure +A family of 2D curves indexed by pressure (f-dimension): +- **X-axis**: control position (0-100%) +- **Y-axis**: flow (nq) or power (np) in canonical units +- **F-dimension**: pressure (Pa) — the 3rd dimension + +Raw curves are in curve units (m3/h, kW, mbar). `_normalizeMachineCurve()` converts to canonical (m3/s, W, Pa). + +## Interpolation +Monotonic cubic spline (Fritsch-Carlson) in both dimensions: +- **X-Y splines**: at each discrete pressure level +- **F-splines**: across pressure levels for intermediate pressure interpolation + +## Prediction Flow +``` +predict.y(x): + 1. Clamp x to [currentFxyXMin, currentFxyXMax] + 2. Normalize x to [normMin, normMax] + 3. Evaluate spline at normalized x for current fDimension + 4. Return y in canonical units (m3/s or W) +``` + +## Unit Conversion Chain +``` +Raw curve (m3/h, kW, mbar) + → _normalizeMachineCurve → canonical (m3/s, W, Pa) + → predict class → canonical output + → MeasurementContainer.getCurrentValue(outputUnit) → output units +``` + +No double-conversion. Clean separation: specificClass handles units, predict handles normalization/interpolation. + +## Three Predict Instances per Machine +- `predictFlow`: control % → flow (nq curve) +- `predictPower`: control % → power (np curve) +- `predictCtrl`: flow → control % (reversed nq curve) + +## Boundary Behavior +- Below/above curve X range: flat extrapolation (clamped) +- Below/above f-dimension range: clamped to min/max pressure level + +## Performance +- `y(x)`: O(log n), effectively O(1) for 5-10 data points +- `buildAllFxyCurves`: sub-10ms for typical curves +- Full caching of normalized curves, splines, and calculated curves diff --git a/Archive-Source-architecture-deployment-blueprint.md b/Archive-Source-architecture-deployment-blueprint.md new file mode 100755 index 0000000..f4fb4c7 --- /dev/null +++ b/Archive-Source-architecture-deployment-blueprint.md @@ -0,0 +1,286 @@ +--- +title: EVOLV Deployment Blueprint +created: 2026-03-01 +updated: 2026-04-07 +status: evolving +tags: [deployment, docker, edge, site, central] +--- + +> **⚠️ ARCHIVED — pre-refactor (Tier 1–4, 2026-05)** +> +> This page describes the architecture before the platform refactor. +> The current page is the per-node wiki on **[gitea.wbd-rd.nl/RnD](https://gitea.wbd-rd.nl/RnD)** or **[Home](../Home)**. +> +> Kept for historical reference only. **Do not update.** + + +# EVOLV Deployment Blueprint + +## Purpose + +This document turns the current EVOLV architecture into a concrete deployment model. + +It focuses on: + +- target infrastructure layout +- container/service topology +- environment and secret boundaries +- rollout order from edge to site to central + +It is the local source document behind the wiki deployment pages. + +## 1. Deployment Principles + +- edge-first operation: plant logic must continue when central is unavailable +- site mediation: site services protect field systems and absorb plant-specific complexity +- central governance: external APIs, analytics, IAM, CI/CD, and shared dashboards terminate centrally +- layered telemetry: InfluxDB exists where operationally justified at edge, site, and central +- configuration authority: `tagcodering` should become the source of truth for configuration +- secrets hygiene: tracked manifests contain variables only; secrets live in server-side env or secret stores + +## 2. Layered Deployment Model + +### 2.1 Edge node + +Purpose: + +- interface with PLCs and field assets +- execute local Node-RED logic +- retain local telemetry for resilience and digital-twin use cases + +Recommended services: + +- `evolv-edge-nodered` +- `evolv-edge-influxdb` +- optional `evolv-edge-grafana` +- optional `evolv-edge-broker` + +Should not host: + +- public API ingress +- central IAM +- source control or CI/CD + +### 2.2 Site node + +Purpose: + +- aggregate one or more edge nodes +- host plant-local dashboards and engineering visibility +- mediate traffic between edge and central + +Recommended services: + +- `evolv-site-nodered` or `coresync-site` +- `evolv-site-influxdb` +- `evolv-site-grafana` +- optional `evolv-site-broker` + +### 2.3 Central platform + +Purpose: + +- fleet-wide analytics +- API and integration ingress +- engineering lifecycle and releases +- identity and governance + +Recommended services: + +- reverse proxy / ingress +- API gateway +- IAM +- central InfluxDB +- central Grafana +- Gitea +- CI/CD runner/controller +- optional broker for asynchronous site/central workflows +- configuration services over `tagcodering` + +## 3. Target Container Topology + +### 3.1 Edge host + +Minimum viable edge stack: + +```text +edge-host-01 + - Node-RED + - InfluxDB + - optional Grafana +``` + +Preferred production edge stack: + +```text +edge-host-01 + - Node-RED + - InfluxDB + - local health/export service + - optional local broker + - optional local dashboard service +``` + +### 3.2 Site host + +Minimum viable site stack: + +```text +site-host-01 + - Site Node-RED / CoreSync + - Site InfluxDB + - Site Grafana +``` + +Preferred production site stack: + +```text +site-host-01 + - Site Node-RED / CoreSync + - Site InfluxDB + - Site Grafana + - API relay / sync service + - optional site broker +``` + +### 3.3 Central host group + +Central should not be one giant undifferentiated host forever. It should trend toward at least these responsibility groups: + +```text +central-ingress + - reverse proxy + - API gateway + - IAM + +central-observability + - central InfluxDB + - Grafana + +central-engineering + - Gitea + - CI/CD + - deployment orchestration + +central-config + - tagcodering-backed config services +``` + +For early rollout these may be colocated, but the responsibility split should remain clear. + +## 4. Compose Strategy + +The current repository shows: + +- `docker-compose.yml` as a development stack +- `temp/cloud.yml` as a broad central-stack example + +For production, EVOLV should not rely on one flat compose file for every layer. + +Recommended split: + +- `compose.edge.yml` +- `compose.site.yml` +- `compose.central.yml` +- optional overlay files for site-specific differences + +Benefits: + +- clearer ownership per layer +- smaller blast radius during updates +- easier secret and env separation +- easier rollout per site + +## 5. Environment And Secrets Strategy + +### 5.1 Current baseline + +`temp/cloud.yml` now uses environment variables instead of inline credentials. That is the minimum acceptable baseline. + +### 5.2 Recommended production rule + +- tracked compose files contain `${VARIABLE}` placeholders only +- real secrets live in server-local `.env` files or a managed secret store +- no shared default production passwords in git +- separate env files per layer and per environment + +Suggested structure: + +```text +/opt/evolv/ + compose.edge.yml + compose.site.yml + compose.central.yml + env/ + edge.env + site.env + central.env +``` + +## 6. Recommended Network Flow + +### 6.1 Northbound + +- edge publishes or syncs upward to site +- site aggregates and forwards selected data to central +- central exposes APIs and dashboards to approved consumers + +### 6.2 Southbound + +- central issues advice, approved config, or mediated requests +- site validates and relays to edge where appropriate +- edge remains the execution point near PLCs + +### 6.3 Forbidden direct path + +- enterprise or internet clients should not directly query PLC-connected edge runtimes + +## 7. Rollout Order + +### Phase 1: Edge baseline + +- deploy edge Node-RED +- deploy local InfluxDB +- validate PLC connectivity +- validate local telemetry and resilience + +### Phase 2: Site mediation + +- deploy site Node-RED / CoreSync +- connect one or more edge nodes +- validate site-local dashboards and outage behavior + +### Phase 3: Central services + +- deploy ingress, IAM, API, Grafana, central InfluxDB +- deploy Gitea and CI/CD services +- validate controlled northbound access + +### Phase 4: Configuration backbone + +- connect runtime layers to `tagcodering` +- reduce config duplication in flows +- formalize config promotion and rollback + +### Phase 5: Smart telemetry policy + +- classify signals +- define reconstruction rules +- define authoritative layer per horizon +- validate analytics and auditability + +## 8. Immediate Technical Recommendations + +- treat `docker/settings.js` as development-only and create hardened production settings separately +- split deployment manifests by layer +- define env files per layer and environment +- formalize healthchecks and backup procedures for every persistent service +- define whether broker usage is required at edge, site, central, or only selectively + +## 9. Next Technical Work Items + +1. create draft `compose.edge.yml`, `compose.site.yml`, and `compose.central.yml` +2. define server directory layout and env-file conventions +3. define production Node-RED settings profile +4. define site-to-central sync path +5. define deployment and rollback runbook diff --git a/Archive-Source-architecture-group-optimization.md b/Archive-Source-architecture-group-optimization.md new file mode 100755 index 0000000..8fb8d41 --- /dev/null +++ b/Archive-Source-architecture-group-optimization.md @@ -0,0 +1,53 @@ +--- +title: Group Optimization Architecture +created: 2026-04-07 +updated: 2026-04-07 +status: proven +tags: [machineGroupControl, optimization, BEP-Gravitation] +sources: [nodes/machineGroupControl/src/specificClass.js] +--- + +> **⚠️ ARCHIVED — pre-refactor (Tier 1–4, 2026-05)** +> +> This page describes the architecture before the platform refactor. +> The current page is the per-node wiki on **[gitea.wbd-rd.nl/RnD](https://gitea.wbd-rd.nl/RnD)** or **[Home](../Home)**. +> +> Kept for historical reference only. **Do not update.** + + +# machineGroupControl Optimization + +## Algorithm: BEP-Gravitation + Marginal-Cost Refinement + +### Step 1 — Pressure Equalization +Sets all non-operational pumps to the group's max downstream / min upstream pressure. Ensures fair curve evaluation across combinations. + +### Step 2 — Combination Enumeration +Generates all 2^n pump subsets (n = number of machines). Filters by: +- Machine state (excludes off, cooling, stopping, emergency) +- Mode compatibility (`execsequence` allowed in auto) +- Flow bounds: `sumMinFlow ≤ Qd ≤ sumMaxFlow` +- Optional power cap + +### Step 3 — BEP-Gravitation Distribution (per combination) +1. **BEP seed**: `estimatedBEP = minFlow + span * NCog` per pump +2. **Slope estimation**: samples dP/dQ at BEP ± delta (directional: slopeLeft, slopeRight) +3. **Slope redistribution**: iteratively shifts flow from steep to flat curves (weight = 1/slope) +4. **Marginal-cost refinement**: after slope redistribution, shifts flow from highest actual dP/dQ to lowest using real `inputFlowCalcPower` evaluations. Converges regardless of curve convexity. Max 50 iterations, typically 5-15. + +### Step 4 — Best Selection +Pick combination with lowest total power. Tiebreak by deviation from BEP. + +### Step 5 — Execution +Start/stop pumps as needed, send `flowmovement` commands in output units via `_canonicalToOutputFlow()`. + +## Three Control Modes + +| Mode | Distribution | Combination Selection | +|------|-------------|----------------------| +| optimalControl | BEP-Gravitation + refinement | exhaustive 2^n | +| priorityControl | equal split, priority-ordered | sequential add/remove | +| priorityPercentageControl | percentage-based, normalized | count-based | + +## Key Design Decision +The `flowmovement` command sends flow in the **machine's output units** (m3/h), not canonical (m3/s). The `_canonicalToOutputFlow()` helper converts before sending. Without this conversion, every pump stays at minimum flow (the critical bug fixed on 2026-04-07). diff --git a/Archive-Source-architecture-node-architecture.md b/Archive-Source-architecture-node-architecture.md new file mode 100755 index 0000000..e732141 --- /dev/null +++ b/Archive-Source-architecture-node-architecture.md @@ -0,0 +1,434 @@ +--- +title: EVOLV Architecture +created: 2026-03-01 +updated: 2026-04-07 +status: evolving +tags: [architecture, node-red, three-layer] +--- + +> **⚠️ ARCHIVED — pre-refactor (Tier 1–4, 2026-05)** +> +> This page describes the architecture before the platform refactor. +> The current page is the per-node wiki on **[gitea.wbd-rd.nl/RnD](https://gitea.wbd-rd.nl/RnD)** or **[Home](../Home)**. +> +> Kept for historical reference only. **Do not update.** + + +# EVOLV Architecture + +## 1. System Overview + +High-level view of how EVOLV fits into the wastewater treatment automation stack. + +```mermaid +graph LR + NR[Node-RED Runtime] <-->|msg objects| EVOLV[EVOLV Nodes] + EVOLV -->|InfluxDB line protocol| INFLUX[(InfluxDB)] + INFLUX -->|queries| GRAFANA[Grafana Dashboards] + EVOLV -->|process output| NR + EVOLV -->|parent output| NR + + style NR fill:#b22222,color:#fff + style EVOLV fill:#0f52a5,color:#fff + style INFLUX fill:#0c99d9,color:#fff + style GRAFANA fill:#50a8d9,color:#fff +``` + +Each EVOLV node produces three outputs: +| Port | Name | Purpose | +|------|------|---------| +| 0 | process | Process data forwarded to downstream nodes | +| 1 | dbase | InfluxDB-formatted measurement data | +| 2 | parent | Control messages to parent nodes (e.g. registerChild) | + +--- + +## 2. Node Architecture (Three-Layer Pattern) + +Every node follows a consistent three-layer design that separates Node-RED wiring from domain logic. + +```mermaid +graph TB + subgraph "Node-RED Runtime" + REG["RED.nodes.registerType()"] + end + + subgraph "Layer 1 — Wrapper (valve.js)" + W[wrapper .js] + W -->|"new nodeClass(config, RED, this, name)"| NC + W -->|MenuManager| MENU[HTTP /name/menu.js] + W -->|configManager| CFG[HTTP /name/configData.js] + end + + subgraph "Layer 2 — Node Adapter (src/nodeClass.js)" + NC[nodeClass] + NC -->|_loadConfig| CFGM[configManager] + NC -->|_setupSpecificClass| SC + NC -->|_attachInputHandler| INPUT[onInput routing] + NC -->|_startTickLoop| TICK[1s tick loop] + NC -->|_tick → outputUtils| OUT[formatMsg] + end + + subgraph "Layer 3 — Domain Logic (src/specificClass.js)" + SC[specificClass] + SC -->|measurements| MC[MeasurementContainer] + SC -->|state machine| ST[state] + SC -->|hydraulics / biology| DOMAIN[domain models] + end + + subgraph "generalFunctions" + GF[shared library] + end + + REG --> W + GF -.->|logger, outputUtils, configManager,\nMeasurementContainer, validation, ...| NC + GF -.->|MeasurementContainer, state,\nconvert, predict, ...| SC + + style W fill:#0f52a5,color:#fff + style NC fill:#0c99d9,color:#fff + style SC fill:#50a8d9,color:#fff + style GF fill:#86bbdd,color:#000 +``` + +--- + +## 3. generalFunctions Module Map + +The shared library (`nodes/generalFunctions/`) provides all cross-cutting concerns. + +```mermaid +graph TB + GF[generalFunctions/index.js] + + subgraph "Core Helpers (src/helper/)" + LOGGER[logger] + OUTPUT[outputUtils] + CHILD[childRegistrationUtils] + CFGUTIL[configUtils] + ASSERT[assertionUtils] + VALID[validationUtils] + end + + subgraph "Validators (src/helper/validators/)" + TV[typeValidators] + CV[collectionValidators] + CURV[curveValidator] + end + + subgraph "Domain Modules (src/)" + MC[MeasurementContainer] + CFGMGR[configManager] + MENUMGR[MenuManager] + STATE[state] + CONVERT[convert / Fysics] + PREDICT[predict / interpolation] + NRMSE[nrmse / errorMetrics] + COOLPROP[coolprop] + end + + subgraph "Data (datasets/)" + CURVES[assetData/curves] + ASSETS[assetData/assetData.json] + UNITS[unitData.json] + end + + subgraph "Constants (src/constants/)" + POS[POSITIONS / POSITION_VALUES] + end + + GF --> LOGGER + GF --> OUTPUT + GF --> CHILD + GF --> CFGUTIL + GF --> ASSERT + GF --> VALID + VALID --> TV + VALID --> CV + VALID --> CURV + GF --> MC + GF --> CFGMGR + GF --> MENUMGR + GF --> STATE + GF --> CONVERT + GF --> PREDICT + GF --> NRMSE + GF --> COOLPROP + GF --> CURVES + GF --> POS + + style GF fill:#0f52a5,color:#fff + style LOGGER fill:#86bbdd,color:#000 + style OUTPUT fill:#86bbdd,color:#000 + style VALID fill:#86bbdd,color:#000 + style MC fill:#50a8d9,color:#fff + style CFGMGR fill:#50a8d9,color:#fff + style MENUMGR fill:#50a8d9,color:#fff +``` + +--- + +## 4. Data Flow (Message Lifecycle) + +Sequence diagram showing a typical input message and the periodic tick output cycle. + +```mermaid +sequenceDiagram + participant NR as Node-RED + participant W as wrapper.js + participant NC as nodeClass + participant SC as specificClass + participant OU as outputUtils + + Note over W: Node startup + W->>NC: new nodeClass(config, RED, node, name) + NC->>NC: _loadConfig (configManager.buildConfig) + NC->>SC: new specificClass(config, stateConfig, options) + NC->>NR: send([null, null, {topic: registerChild}]) + + Note over NC: Every 1 second (tick loop) + NC->>SC: getOutput() + SC-->>NC: raw measurement data + NC->>OU: formatMsg(raw, config, 'process') + NC->>OU: formatMsg(raw, config, 'influxdb') + NC->>NR: send([processMsg, influxMsg]) + + Note over NR: Incoming control message + NR->>W: msg {topic: 'execMovement', payload: {...}} + W->>NC: onInput(msg) + NC->>SC: handleInput(source, action, setpoint) + SC->>SC: update state machine & measurements +``` + +--- + +## 5. Node Types + +| Node | S88 Level | Purpose | +|------|-----------|---------| +| **measurement** | Control Module | Generic measurement point — reads, validates, and stores sensor values | +| **valve** | Control Module | Valve simulation with hydraulic model, position control, flow/pressure prediction | +| **rotatingMachine** | Control Module | Pumps, blowers, mixers — rotating equipment with speed control and efficiency curves | +| **diffuser** | Control Module | Aeration diffuser — models oxygen transfer and pressure drop | +| **settler** | Equipment | Sludge settler — models settling behavior and sludge blanket | +| **reactor** | Equipment | Hydraulic tank and biological process simulator (activated sludge, digestion) | +| **monster** | Equipment | MONitoring and STrEam Routing — complex measurement aggregation | +| **pumpingStation** | Unit | Coordinates multiple pumps as a pumping station | +| **valveGroupControl** | Unit | Manages multiple valves as a coordinated group — distributes flow, monitors pressure | +| **machineGroupControl** | Unit | Group control for rotating machines — load balancing and sequencing | +| **dashboardAPI** | Utility | Exposes data and unit conversion endpoints for external dashboards | +# EVOLV Architecture + +## Node Hierarchy (S88) + +EVOLV follows the ISA-88 (S88) batch control standard. Each node maps to an S88 level and uses a consistent color scheme in the Node-RED editor. + +```mermaid +graph TD + classDef area fill:#0f52a5,color:#fff,stroke:#0a3d7a + classDef processCell fill:#0c99d9,color:#fff,stroke:#0977aa + classDef unit fill:#50a8d9,color:#fff,stroke:#3d89b3 + classDef equipment fill:#86bbdd,color:#000,stroke:#6a9bb8 + classDef controlModule fill:#a9daee,color:#000,stroke:#87b8cc + classDef standalone fill:#f0f0f0,color:#000,stroke:#999 + + %% S88 Levels + subgraph "S88: Area" + PS[pumpingStation] + end + + subgraph "S88: Equipment" + MGC[machineGroupControl] + VGC[valveGroupControl] + end + + subgraph "S88: Control Module" + RM[rotatingMachine] + V[valve] + M[measurement] + R[reactor] + S[settler] + end + + subgraph "Standalone" + MON[monster] + DASH[dashboardAPI] + DIFF[diffuser - not implemented] + end + + %% Parent-child registration relationships + PS -->|"accepts: measurement"| M + PS -->|"accepts: machine"| RM + PS -->|"accepts: machineGroup"| MGC + PS -->|"accepts: pumpingStation"| PS2[pumpingStation] + + MGC -->|"accepts: machine"| RM + + RM -->|"accepts: measurement"| M2[measurement] + RM -->|"accepts: reactor"| R + + VGC -->|"accepts: valve"| V + VGC -->|"accepts: machine / rotatingmachine"| RM2[rotatingMachine] + VGC -->|"accepts: machinegroup / machinegroupcontrol"| MGC2[machineGroupControl] + VGC -->|"accepts: pumpingstation / valvegroupcontrol"| PS3["pumpingStation / valveGroupControl"] + + R -->|"accepts: measurement"| M3[measurement] + R -->|"accepts: reactor"| R2[reactor] + + S -->|"accepts: measurement"| M4[measurement] + S -->|"accepts: reactor"| R3[reactor] + S -->|"accepts: machine"| RM3[rotatingMachine] + + %% Styling + class PS,PS2,PS3 area + class MGC,MGC2 equipment + class VGC equipment + class RM,RM2,RM3 controlModule + class V controlModule + class M,M2,M3,M4 controlModule + class R,R2,R3 controlModule + class S controlModule + class MON,DASH,DIFF standalone +``` + +### Registration Summary + +```mermaid +graph LR + classDef parent fill:#0c99d9,color:#fff + classDef child fill:#a9daee,color:#000 + + PS[pumpingStation] -->|measurement| LEAF1((leaf)) + PS -->|machine| RM1[rotatingMachine] + PS -->|machineGroup| MGC1[machineGroupControl] + PS -->|pumpingStation| PS1[pumpingStation] + + MGC[machineGroupControl] -->|machine| RM2[rotatingMachine] + + VGC[valveGroupControl] -->|valve| V1[valve] + VGC -->|source| SRC["machine, machinegroup,
pumpingstation, valvegroupcontrol"] + + RM[rotatingMachine] -->|measurement| LEAF2((leaf)) + RM -->|reactor| R1[reactor] + + R[reactor] -->|measurement| LEAF3((leaf)) + R -->|reactor| R2[reactor] + + S[settler] -->|measurement| LEAF4((leaf)) + S -->|reactor| R3[reactor] + S -->|machine| RM3[rotatingMachine] + + class PS,MGC,VGC,RM,R,S parent + class LEAF1,LEAF2,LEAF3,LEAF4,RM1,RM2,RM3,MGC1,PS1,V1,SRC,R1,R2,R3 child +``` + +## Node Types + +| Node | S88 Level | softwareType | role | Accepts Children | Outputs | +|------|-----------|-------------|------|-----------------|---------| +| **pumpingStation** | Area | `pumpingstation` | StationController | measurement, machine (rotatingMachine), machineGroup, pumpingStation | [process, dbase, parent] | +| **machineGroupControl** | Equipment | `machinegroupcontrol` | GroupController | machine (rotatingMachine) | [process, dbase, parent] | +| **valveGroupControl** | Equipment | `valvegroupcontrol` | ValveGroupController | valve, machine, rotatingmachine, machinegroup, machinegroupcontrol, pumpingstation, valvegroupcontrol | [process, dbase, parent] | +| **rotatingMachine** | Control Module | `rotatingmachine` | RotationalDeviceController | measurement, reactor | [process, dbase, parent] | +| **valve** | Control Module | `valve` | controller | _(leaf node, no children)_ | [process, dbase, parent] | +| **measurement** | Control Module | `measurement` | Sensor | _(leaf node, no children)_ | [process, dbase, parent] | +| **reactor** | Control Module | `reactor` | Biological reactor | measurement, reactor (upstream chaining) | [process, dbase, parent] | +| **settler** | Control Module | `settler` | Secondary settler | measurement, reactor (upstream), machine (return pump) | [process, dbase, parent] | +| **monster** | Standalone | - | - | dual-parent, standalone | - | +| **dashboardAPI** | Standalone | - | - | accepts any child (Grafana integration) | - | +| **diffuser** | Standalone | - | - | _(not implemented)_ | - | + +## Data Flow + +### Measurement Data Flow (upstream to downstream) + +```mermaid +sequenceDiagram + participant Sensor as measurement (sensor) + participant Machine as rotatingMachine + participant Group as machineGroupControl + participant Station as pumpingStation + + Note over Sensor: Sensor reads value
(pressure, flow, level, temp) + + Sensor->>Sensor: measurements.type(t).variant("measured").position(p).value(v) + Sensor->>Sensor: emitter.emit("type.measured.position", eventData) + + Sensor->>Machine: Event: "pressure.measured.upstream" + Machine->>Machine: Store in own MeasurementContainer + Machine->>Machine: getMeasuredPressure() -> calcFlow() -> calcPower() + Machine->>Machine: emitter.emit("flow.predicted.downstream", eventData) + + Machine->>Group: Event: "flow.predicted.downstream" + Group->>Group: handlePressureChange() + Group->>Group: Aggregate flows across all machines + Group->>Group: Calculate group totals and efficiency + + Machine->>Station: Event: "flow.predicted.downstream" + Station->>Station: Store predicted flow in/out + Station->>Station: _updateVolumePrediction() + Station->>Station: _calcNetFlow(), _calcTimeRemaining() +``` + +### Control Command Flow (downstream to upstream) + +```mermaid +sequenceDiagram + participant Station as pumpingStation + participant Group as machineGroupControl + participant Machine as rotatingMachine + participant Machine2 as rotatingMachine (2) + + Station->>Group: handleInput("parent", action, param) + + Group->>Group: Determine scaling strategy + Group->>Group: Calculate setpoints per machine + + Group->>Machine: handleInput("parent", "execMovement", setpoint) + Group->>Machine2: handleInput("parent", "execMovement", setpoint) + + Machine->>Machine: setpoint() -> state.moveTo(pos) + Machine->>Machine: updatePosition() -> calcFlow(), calcPower() + Machine->>Machine: emitter.emit("flow.predicted.downstream") + + Machine2->>Machine2: setpoint() -> state.moveTo(pos) + Machine2->>Machine2: updatePosition() -> calcFlow(), calcPower() + Machine2->>Machine2: emitter.emit("flow.predicted.downstream") +``` + +### Wastewater Treatment Process Flow + +```mermaid +graph LR + classDef process fill:#50a8d9,color:#fff + classDef equipment fill:#86bbdd,color:#000 + + PS_IN[pumpingStation
Influent] -->|flow| R1[reactor
Anoxic] + R1 -->|effluent| R2[reactor
Aerated] + R2 -->|effluent| SET[settler] + SET -->|effluent out| PS_OUT[pumpingStation
Effluent] + SET -->|sludge return| RM_RET[rotatingMachine
Return pump] + RM_RET -->|recirculation| R1 + + PS_IN --- MGC_IN[machineGroupControl] + MGC_IN --- RM_IN[rotatingMachine
Influent pumps] + + class PS_IN,PS_OUT process + class R1,R2,SET process + class MGC_IN,RM_IN,RM_RET equipment +``` + +### Event-Driven Communication Pattern + +All parent-child communication uses Node.js `EventEmitter`: + +1. **Registration**: Parent calls `childRegistrationUtils.registerChild(child, position)` which stores the child and calls the parent's `registerChild(child, softwareType)` method. +2. **Event binding**: The parent's `registerChild()` subscribes to the child's `measurements.emitter` events (e.g., `"flow.predicted.downstream"`). +3. **Data propagation**: When a child updates a measurement, it emits an event. The parent's listener stores the value in its own `MeasurementContainer` and runs its domain logic. +4. **Three outputs**: Every node sends data to three Node-RED outputs: `[process, dbase, parent]` -- process data for downstream nodes, InfluxDB for persistence, and parent aggregation data. + +### Position Convention + +Children register with a position relative to their parent: +- `upstream` -- before the parent in the flow direction +- `downstream` -- after the parent in the flow direction +- `atEquipment` -- physically located at/on the parent equipment diff --git a/Archive-Source-architecture-platform-overview.md b/Archive-Source-architecture-platform-overview.md new file mode 100755 index 0000000..b35bbda --- /dev/null +++ b/Archive-Source-architecture-platform-overview.md @@ -0,0 +1,166 @@ +--- +title: EVOLV Platform Architecture +created: 2026-03-01 +updated: 2026-04-07 +status: evolving +tags: [architecture, platform, edge-first] +--- + +> **⚠️ ARCHIVED — pre-refactor (Tier 1–4, 2026-05)** +> +> This page describes the architecture before the platform refactor. +> The current page is the per-node wiki on **[gitea.wbd-rd.nl/RnD](https://gitea.wbd-rd.nl/RnD)** or **[Home](../Home)**. +> +> Kept for historical reference only. **Do not update.** + + +# EVOLV Platform Architecture + +## At A Glance + +EVOLV is not only a Node-RED package. It is a layered automation platform: + +- edge for plant-side execution +- site for local aggregation and resilience +- central for coordination, analytics, APIs, and governance + +```mermaid +flowchart LR + subgraph EDGE["Edge"] + PLC["PLC / IO"] + ENR["Node-RED"] + EDB["Local InfluxDB"] + EUI["Local Monitoring"] + end + + subgraph SITE["Site"] + SNR["CoreSync / Site Node-RED"] + SDB["Site InfluxDB"] + SUI["Site Dashboards"] + end + + subgraph CENTRAL["Central"] + API["API Gateway"] + CFG["Tagcodering"] + CDB["Central InfluxDB"] + CGR["Grafana"] + INTEL["Overview Intelligence"] + GIT["Gitea + CI/CD"] + end + + PLC --> ENR + ENR --> EDB + ENR --> EUI + ENR <--> SNR + EDB <--> SDB + SNR --> SUI + SNR <--> API + API <--> CFG + API --> INTEL + SDB <--> CDB + CDB --> CGR + GIT --> ENR + GIT --> SNR +``` + +## Core Principles + +### 1. Edge-first operation + +The edge layer must remain useful and safe when central systems are down. + +That means: + +- local logic remains operational +- local telemetry remains queryable +- local dashboards can keep working + +### 2. Multi-level telemetry + +InfluxDB is expected on multiple levels: + +- local for resilience and digital-twin use +- site for plant diagnostics +- central for fleet analytics and advisory logic + +### 3. Smart storage + +Telemetry should not be stored only with naive deadband rules. + +The target model is signal-aware: + +- preserve critical change points +- reduce low-information flat sections +- allow downstream reconstruction where justified + +```mermaid +flowchart LR + SIG["Process Signal"] --> EVAL["Slope / Event Evaluation"] + EVAL --> KEEP["Keep critical points"] + EVAL --> REDUCE["Reduce reconstructable points"] + KEEP --> L0["Local InfluxDB"] + REDUCE --> L0 + L0 --> L1["Site InfluxDB"] + L1 --> L2["Central InfluxDB"] +``` + +### 4. Central is the safe entry point + +External systems should enter through central APIs, not by directly calling field-edge systems. + +```mermaid +flowchart TD + EXT["External Request"] --> API["Central API Gateway"] + API --> AUTH["Auth / Policy"] + AUTH --> SITE["Site Layer"] + SITE --> EDGE["Edge Layer"] + EDGE --> PLC["Field Assets"] + + EXT -. blocked .-> EDGE + EXT -. blocked .-> PLC +``` + +### 5. Configuration belongs in `tagcodering` + +The intended configuration source of truth is the database-backed `tagcodering` model: + +- machine metadata +- asset configuration +- runtime-consumable configuration +- future central/site configuration services + +This already exists partially but still needs more work before it fully serves that role. + +## Layer Roles + +### Edge + +- PLC connectivity +- local logic +- protocol translation +- local telemetry buffering +- local monitoring and digital-twin support + +### Site + +- aggregation of edge systems +- local dashboards and diagnostics +- mediation between OT and central +- protected handoff for central requests + +### Central + +- enterprise/API gateway +- fleet dashboards +- analytics and intelligence +- source control and CI/CD +- configuration governance through `tagcodering` + +## Why This Matters + +This architecture gives EVOLV: + +- better resilience +- safer external integration +- better data quality for analytics +- a path from Node-RED package to platform diff --git a/Archive-Source-architecture-stack-review.md b/Archive-Source-architecture-stack-review.md new file mode 100755 index 0000000..5dad204 --- /dev/null +++ b/Archive-Source-architecture-stack-review.md @@ -0,0 +1,640 @@ +--- +title: EVOLV Architecture Review +created: 2026-03-01 +updated: 2026-04-07 +status: evolving +tags: [architecture, stack, review] +--- + +> **⚠️ ARCHIVED — pre-refactor (Tier 1–4, 2026-05)** +> +> This page describes the architecture before the platform refactor. +> The current page is the per-node wiki on **[gitea.wbd-rd.nl/RnD](https://gitea.wbd-rd.nl/RnD)** or **[Home](../Home)**. +> +> Kept for historical reference only. **Do not update.** + + +# EVOLV Architecture Review + +## Purpose + +This document captures: + +- the architecture implemented in this repository today +- the broader edge/site/central architecture shown in the drawings under `temp/` +- the key strengths and weaknesses of that direction +- the currently preferred target stack based on owner decisions from this review + +It is the local staging document for a later wiki update. + +## Evidence Used + +Implemented stack evidence: + +- `docker-compose.yml` +- `docker/settings.js` +- `docker/grafana/provisioning/datasources/influxdb.yaml` +- `package.json` +- `nodes/*` + +Target-state evidence: + +- `temp/fullStack.pdf` +- `temp/edge.pdf` +- `temp/CoreSync.drawio.pdf` +- `temp/cloud.yml` + +Owner decisions from this review: + +- local InfluxDB is required for operational resilience +- central acts as the advisory/intelligence and API-entry layer, not as a direct field caller +- intended configuration authority is the database-backed `tagcodering` model +- architecture wiki pages should be visual, not text-only + +## 1. What Exists Today + +### 1.1 Product/runtime layer + +The codebase is currently a modular Node-RED package for wastewater/process automation: + +- EVOLV ships custom Node-RED nodes for plant assets and process logic +- nodes emit both process/control messages and telemetry-oriented outputs +- shared helper logic lives in `nodes/generalFunctions/` +- Grafana-facing integration exists through `dashboardAPI` and Influx-oriented outputs + +### 1.2 Implemented development stack + +The concrete development stack in this repository is: + +- Node-RED +- InfluxDB 2.x +- Grafana + +That gives a clear local flow: + +1. EVOLV logic runs in Node-RED. +2. Telemetry is emitted in a time-series-oriented shape. +3. InfluxDB stores the telemetry. +4. Grafana renders operational dashboards. + +### 1.3 Existing runtime pattern in the nodes + +A recurring EVOLV pattern is: + +- output 0: process/control message +- output 1: Influx/telemetry message +- output 2: registration/control plumbing where relevant + +So even in its current implemented form, EVOLV is not only a Node-RED project. It is already a control-plus-observability platform, with Node-RED as orchestration/runtime and InfluxDB/Grafana as telemetry and visualization services. + +## 2. What The Drawings Describe + +Across `temp/fullStack.pdf` and `temp/CoreSync.drawio.pdf`, the intended platform is broader and layered. + +### 2.1 Edge / OT layer + +The drawings consistently place these capabilities at the edge: + +- PLC / OPC UA connectivity +- Node-RED container as protocol translator and logic runtime +- local broker in some variants +- local InfluxDB / Prometheus style storage in some variants +- local Grafana/SCADA in some variants + +This is the plant-side operational layer. + +### 2.2 Site / local server layer + +The CoreSync drawings also show a site aggregation layer: + +- RWZI-local server +- Node-RED / CoreSync services +- site-local broker +- site-local database +- upward API-based synchronization + +This layer decouples field assets from central services and absorbs plant-specific complexity. + +### 2.3 Central / cloud layer + +The broader stack drawings and `temp/cloud.yml` show a central platform layer with: + +- Gitea +- Jenkins +- reverse proxy / ingress +- Grafana +- InfluxDB +- Node-RED +- RabbitMQ / messaging +- VPN / tunnel concepts +- Keycloak in the drawing +- Portainer in the drawing + +This is a platform-services layer, not just an application runtime. + +## 3. Architecture Decisions From This Review + +These decisions now shape the preferred EVOLV target architecture. + +### 3.1 Local telemetry is mandatory for resilience + +Local InfluxDB is not optional. It is required so that: + +- operations continue when central SCADA or central services are down +- local dashboards and advanced digital-twin workflows can still consume recent and relevant process history +- local edge/site layers can make smarter decisions without depending on round-trips to central + +### 3.2 Multi-level InfluxDB is part of the architecture + +InfluxDB should exist on multiple levels where it adds operational value: + +- edge/local for resilience and near-real-time replay +- site for plant-level history, diagnostics, and resilience +- central for fleet-wide analytics, benchmarking, and advisory intelligence + +This is not just copy-paste storage at each level. The design intent is event-driven and selective. + +### 3.3 Storage should be smart, not only deadband-driven + +The target is not simple "store every point" or only a fixed deadband rule such as 1%. + +The desired storage approach is: + +- observe signal slope and change behavior +- preserve points where state is changing materially +- store fewer points where the signal can be reconstructed downstream with sufficient fidelity +- carry enough metadata or conventions so reconstruction quality is auditable + +This implies EVOLV should evolve toward smart storage and signal-aware retention rather than naive event dumping. + +### 3.4 Central is the intelligence and API-entry layer + +Central may advise and coordinate edge/site layers, but external API requests should not hit field-edge systems directly. + +The intended pattern is: + +- external and enterprise integrations terminate centrally +- central evaluates, aggregates, authorizes, and advises +- site/edge layers receive mediated requests, policies, or setpoints +- field-edge remains protected behind an intermediate layer + +This aligns with the stated security direction. + +### 3.5 Configuration source of truth should be database-backed + +The intended configuration authority is the database-backed `tagcodering` model, which already exists but is not yet complete enough to serve as the fully realized source of truth. + +That means the architecture should assume: + +- asset and machine metadata belong in `tagcodering` +- Node-RED flows should consume configuration rather than silently becoming the only configuration store +- more work is still needed before this behaves as the intended central configuration backbone + +## 4. Visual Model + +### 4.1 Platform topology + +```mermaid +flowchart LR + subgraph OT["OT / Field"] + PLC["PLC / IO"] + DEV["Sensors / Machines"] + end + + subgraph EDGE["Edge Layer"] + ENR["Edge Node-RED"] + EDB["Local InfluxDB"] + EUI["Local Grafana / Local Monitoring"] + EBR["Optional Local Broker"] + end + + subgraph SITE["Site Layer"] + SNR["Site Node-RED / CoreSync"] + SDB["Site InfluxDB"] + SUI["Site Grafana / SCADA Support"] + SBR["Site Broker"] + end + + subgraph CENTRAL["Central Layer"] + API["API / Integration Gateway"] + INTEL["Overview Intelligence / Advisory Logic"] + CDB["Central InfluxDB"] + CGR["Central Grafana"] + CFG["Tagcodering Config Model"] + GIT["Gitea"] + CI["CI/CD"] + IAM["IAM / Keycloak"] + end + + DEV --> PLC + PLC --> ENR + ENR --> EDB + ENR --> EUI + ENR --> EBR + ENR <--> SNR + EDB <--> SDB + SNR --> SDB + SNR --> SUI + SNR --> SBR + SNR <--> API + API --> INTEL + API <--> CFG + SDB <--> CDB + INTEL --> SNR + CGR --> CDB + CI --> GIT + IAM --> API + IAM --> CGR +``` + +### 4.2 Command and access boundary + +```mermaid +flowchart TD + EXT["External APIs / Enterprise Requests"] --> API["Central API Gateway"] + API --> AUTH["AuthN/AuthZ / Policy Checks"] + AUTH --> INTEL["Central Advisory / Decision Support"] + INTEL --> SITE["Site Integration Layer"] + SITE --> EDGE["Edge Runtime"] + EDGE --> PLC["PLC / Field Assets"] + + EXT -. no direct access .-> EDGE + EXT -. no direct access .-> PLC +``` + +### 4.3 Smart telemetry flow + +```mermaid +flowchart LR + RAW["Raw Signal"] --> EDGELOGIC["Edge Signal Evaluation"] + EDGELOGIC --> KEEP["Keep Critical Change Points"] + EDGELOGIC --> SKIP["Skip Reconstructable Flat Points"] + EDGELOGIC --> LOCAL["Local InfluxDB"] + LOCAL --> SITE["Site InfluxDB"] + SITE --> CENTRAL["Central InfluxDB"] + KEEP --> LOCAL + SKIP -. reconstruction assumptions / metadata .-> SITE + CENTRAL --> DASH["Fleet Dashboards / Analytics"] +``` + +## 5. Upsides Of This Direction + +### 5.1 Strong separation between control and observability + +Node-RED for runtime/orchestration and InfluxDB/Grafana for telemetry is still the right structural split: + +- control stays close to the process +- telemetry storage/querying stays in time-series-native tooling +- dashboards do not need to overload Node-RED itself + +### 5.2 Edge-first matches operational reality + +For wastewater/process systems, edge-first remains correct: + +- lower latency +- better degraded-mode behavior +- less dependence on WAN or central platform uptime +- clearer OT trust boundary + +### 5.3 Site mediation improves safety and security + +Using central as the enterprise/API entry point and site as the mediator improves posture: + +- field systems are less exposed +- policy decisions can be centralized +- external integrations do not probe the edge directly +- site can continue operating even when upstream is degraded + +### 5.4 Multi-level storage enables better analytics + +Multiple Influx layers can support: + +- local resilience +- site diagnostics +- fleet benchmarking +- smarter retention and reconstruction strategies + +That is substantially more capable than a single central historian model. + +### 5.5 `tagcodering` is the right long-term direction + +A database-backed configuration authority is stronger than embedding configuration only in flows because it supports: + +- machine metadata management +- controlled rollout of configuration changes +- clearer versioning and provenance +- future API-driven configuration services + +## 6. Downsides And Risks + +### 6.1 Smart storage raises algorithmic and governance complexity + +Signal-aware storage and reconstruction is promising, but it creates architectural obligations: + +- reconstruction rules must be explicit +- acceptable reconstruction error must be defined per signal type +- operators must know whether they see raw or reconstructed history +- compliance-relevant data may need stricter retention than operational convenience data + +Without those rules, smart storage can become opaque and hard to trust. + +### 6.2 Multi-level databases can create ownership confusion + +If edge, site, and central all store telemetry, you must define: + +- which layer is authoritative for which time horizon +- when backfill is allowed +- when data is summarized vs copied +- how duplicates or gaps are detected + +Otherwise operations will argue over which trend is "the real one." + +### 6.3 Central intelligence must remain advisory-first + +Central guidance can become valuable, but direct closed-loop dependency on central would be risky. + +The architecture should therefore preserve: + +- local control authority at edge/site +- bounded and explicit central advice +- safe behavior if central recommendations stop arriving + +### 6.4 `tagcodering` is not yet complete enough to lean on blindly + +It is the right target, but its current partial state means there is still architecture debt: + +- incomplete config workflows +- likely mismatch between desired and implemented schema behavior +- temporary duplication between flows, node config, and database-held metadata + +This should be treated as a core platform workstream, not a side issue. + +### 6.5 Broker responsibilities are still not crisp enough + +The materials still reference MQTT/AMQP/RabbitMQ/brokers without one stable responsibility split. That needs to be resolved before large-scale deployment. + +Questions still open: + +- command bus or event bus? +- site-only or cross-site? +- telemetry transport or only synchronization/eventing? +- durability expectations and replay behavior? + +## 7. Security And Regulatory Positioning + +### 7.1 Purdue-style layering is a good fit + +EVOLV's preferred structure aligns well with a Purdue-style OT/IT layering approach: + +- PLCs and field assets stay at the operational edge +- edge runtimes stay close to the process +- site systems mediate between OT and broader enterprise concerns +- central services host APIs, identity, analytics, and engineering workflows + +That is important because it supports segmented trust boundaries instead of direct enterprise-to-field reach-through. + +### 7.2 NIS2 alignment + +Directive (EU) 2022/2555 (NIS2) requires cybersecurity risk-management measures, incident handling, and stronger governance for covered entities. + +This architecture supports that by: + +- limiting direct exposure of field systems +- separating operational layers +- enabling central policy and oversight +- preserving local operation during upstream failure + +### 7.3 CER alignment + +Directive (EU) 2022/2557 (Critical Entities Resilience Directive) focuses on resilience of essential services. + +The edge-plus-site approach supports that direction because: + +- local/site layers can continue during central disruption +- essential service continuity does not depend on one central runtime +- degraded-mode behavior can be explicitly designed per layer + +### 7.4 Cyber Resilience Act alignment + +Regulation (EU) 2024/2847 (Cyber Resilience Act) creates cybersecurity requirements for products with digital elements. + +For EVOLV, that means the platform should keep strengthening: + +- secure configuration handling +- vulnerability and update management +- release traceability +- lifecycle ownership of components and dependencies + +### 7.5 GDPR alignment where personal data is present + +Regulation (EU) 2016/679 (GDPR) applies whenever EVOLV processes personal data. + +The architecture helps by: + +- centralizing ingress +- reducing unnecessary propagation of data to field layers +- making access, retention, and audit boundaries easier to define + +### 7.6 What can and cannot be claimed + +The defensible claim is that EVOLV can be deployed in a way that supports compliance with strict European cybersecurity and resilience expectations. + +The non-defensible claim is that EVOLV is automatically compliant purely because of the architecture diagram. + +Actual compliance still depends on implementation and operations, including: + +- access control +- patch and vulnerability management +- incident response +- logging and audit evidence +- retention policy +- data classification + +## 8. Recommended Ideal Stack + +The ideal EVOLV stack should be layered around operational boundaries, not around tools. + +### 7.1 Layer A: Edge execution + +Purpose: + +- connect to PLCs and field assets +- execute time-sensitive local logic +- preserve operation during WAN/central loss +- provide local telemetry access for resilience and digital-twin use cases + +Recommended components: + +- Node-RED runtime for EVOLV edge flows +- OPC UA and protocol adapters +- local InfluxDB +- optional local Grafana for local engineering/monitoring +- optional local broker only when multiple participants need decoupling + +Principle: + +- edge remains safe and useful when disconnected + +### 7.2 Layer B: Site integration + +Purpose: + +- aggregate multiple edge systems at plant/site level +- host plant-local dashboards and diagnostics +- mediate between raw OT detail and central standardization +- serve as the protected step between field systems and central requests + +Recommended components: + +- site Node-RED / CoreSync services +- site InfluxDB +- site Grafana / SCADA-supporting dashboards +- site broker where asynchronous eventing is justified + +Principle: + +- site absorbs plant complexity and protects field assets + +### 7.3 Layer C: Central platform + +Purpose: + +- fleet-wide analytics +- shared dashboards +- engineering lifecycle +- enterprise/API entry point +- overview intelligence and advisory logic + +Recommended components: + +- Gitea +- CI/CD +- central InfluxDB +- central Grafana +- API/integration gateway +- IAM +- VPN/private connectivity +- `tagcodering`-backed configuration services + +Principle: + +- central coordinates, advises, and governs; it is not the direct field caller + +### 7.4 Cross-cutting platform services + +These should be explicit architecture elements: + +- secrets management +- certificate management +- backup/restore +- audit logging +- monitoring/alerting of the platform itself +- versioned configuration and schema management +- rollout/rollback strategy + +## 9. Recommended Opinionated Choices + +### 8.1 Keep Node-RED as the orchestration layer, not the whole platform + +Node-RED should own: + +- process orchestration +- protocol mediation +- edge/site logic +- KPI production + +It should not become the sole owner of: + +- identity +- long-term configuration authority +- secret management +- compliance/audit authority + +### 8.2 Use InfluxDB by function and horizon + +Recommended split: + +- edge: resilience, local replay, digital-twin input +- site: plant diagnostics and local continuity +- central: fleet analytics, advisory intelligence, benchmarking, and long-term cross-site views + +### 8.3 Prefer smart telemetry retention over naive point dumping + +Recommended rule: + +- keep information-rich points +- reduce information-poor flat spans +- document reconstruction assumptions +- define signal-class-specific fidelity expectations + +This needs design discipline, but it is a real differentiator if executed well. + +### 8.4 Put enterprise/API ingress at central, not at edge + +This should become a hard architectural rule: + +- external requests land centrally +- central authenticates and authorizes +- central or site mediates downward +- edge never becomes the exposed public integration surface + +### 8.5 Make `tagcodering` the target configuration backbone + +The architecture should be designed so that `tagcodering` can mature into: + +- machine and asset registry +- configuration source of truth +- site/central configuration exchange point +- API-served configuration source for runtime layers + +## 10. Suggested Phasing + +### Phase 1: Stabilize contracts + +- define topic and payload contracts +- define telemetry classes and reconstruction policy +- define asset, machine, and site identity model +- define `tagcodering` scope and schema ownership + +### Phase 2: Harden local/site resilience + +- formalize edge and site runtime patterns +- define local telemetry retention and replay behavior +- define central-loss behavior +- define dashboard behavior during isolation + +### Phase 3: Harden central platform + +- IAM +- API gateway +- central observability +- CI/CD +- backup and disaster recovery +- config services over `tagcodering` + +### Phase 4: Introduce selective synchronization and intelligence + +- event-driven telemetry propagation rules +- smart-storage promotion/backfill policies +- advisory services from central +- auditability of downward recommendations and configuration changes + +## 11. Immediate Open Questions Before Wiki Finalization + +1. Which signals are allowed to use reconstruction-aware smart storage, and which must remain raw or near-raw for audit/compliance reasons? +2. How should `tagcodering` be exposed to runtime layers: direct database access, a dedicated API, or both? +3. What exact responsibility split should EVOLV use between API synchronization and broker-based eventing? + +## 12. Recommended Wiki Structure + +The wiki should not be one long page. It should be split into: + +1. platform overview with the main topology diagram +2. edge-site-central runtime model +3. telemetry and smart storage model +4. security and access-boundary model +5. configuration architecture centered on `tagcodering` + +## 13. Next Step + +Use this document as the architecture baseline. The companion markdown page in `architecture/` can then be shaped into a wiki-ready visual overview page with Mermaid diagrams and shorter human-readable sections. diff --git a/Archive-Source-concepts-generalfunctions-api.md b/Archive-Source-concepts-generalfunctions-api.md new file mode 100755 index 0000000..694a940 --- /dev/null +++ b/Archive-Source-concepts-generalfunctions-api.md @@ -0,0 +1,462 @@ +--- +title: generalFunctions API Reference +created: 2026-03-01 +updated: 2026-04-07 +status: evolving +tags: [api, generalFunctions, reference] +--- + +> **⚠️ ARCHIVED — pre-refactor (Tier 1–4, 2026-05)** +> +> This page describes the architecture before the platform refactor. +> The current page is the per-node wiki on **[gitea.wbd-rd.nl/RnD](https://gitea.wbd-rd.nl/RnD)** or **[Home](../Home)**. +> +> Kept for historical reference only. **Do not update.** + + +# generalFunctions API Reference + +Shared library (`nodes/generalFunctions/`) used across all EVOLV Node-RED nodes. + +```js +const { logger, outputUtils, MeasurementContainer, ... } = require('generalFunctions'); +``` + +--- + +## Table of Contents + +1. [Logger](#logger) +2. [OutputUtils](#outpututils) +3. [ValidationUtils](#validationutils) +4. [MeasurementContainer](#measurementcontainer) +5. [ConfigManager](#configmanager) +6. [ChildRegistrationUtils](#childregistrationutils) +7. [MenuUtils](#menuutils) +8. [EndpointUtils](#endpointutils) +9. [Positions](#positions) +10. [AssetLoader / loadCurve](#assetloader--loadcurve) + +--- + +## Logger + +Structured, level-filtered console logger. + +**File:** `src/helper/logger.js` + +### Constructor + +```js +new Logger(logging = true, logLevel = 'debug', nameModule = 'N/A') +``` + +| Param | Type | Default | Description | +|---|---|---|---| +| `logging` | `boolean` | `true` | Enable/disable all output | +| `logLevel` | `string` | `'debug'` | Minimum severity: `'debug'` \| `'info'` \| `'warn'` \| `'error'` | +| `nameModule` | `string` | `'N/A'` | Label prefixed to every message | + +### Methods + +| Method | Signature | Description | +|---|---|---| +| `debug` | `(message: string): void` | Log at DEBUG level | +| `info` | `(message: string): void` | Log at INFO level | +| `warn` | `(message: string): void` | Log at WARN level | +| `error` | `(message: string): void` | Log at ERROR level | +| `setLogLevel` | `(level: string): void` | Change minimum level at runtime | +| `toggleLogging` | `(): void` | Flip logging on/off | + +### Example + +```js +const Logger = require('generalFunctions').logger; +const log = new Logger(true, 'info', 'MyNode'); +log.info('Node started'); // [INFO] -> MyNode: Node started +log.debug('ignored'); // silent (below 'info') +log.setLogLevel('debug'); +log.debug('now visible'); // [DEBUG] -> MyNode: now visible +``` + +--- + +## OutputUtils + +Tracks output state and formats messages for InfluxDB or process outputs. Only emits changed fields. + +**File:** `src/helper/outputUtils.js` + +### Constructor + +```js +new OutputUtils() // no parameters +``` + +### Methods + +| Method | Signature | Returns | Description | +|---|---|---|---| +| `formatMsg` | `(output, config, format)` | `object \| undefined` | Diff against last output; returns formatted msg or `undefined` if nothing changed | +| `checkForChanges` | `(output, format)` | `object` | Returns only the key/value pairs that changed since last call | + +**`format`** must be `'influxdb'` or `'process'`. + +### Example + +```js +const out = new OutputUtils(); +const msg = out.formatMsg( + { temperature: 22.5, pressure: 1013 }, + config, + 'influxdb' +); +// msg = { topic: 'nodeName', payload: { measurement, fields, tags, timestamp } } +``` + +--- + +## ValidationUtils + +Schema-driven config validation with type coercion, range clamping, and nested object support. + +**File:** `src/helper/validationUtils.js` + +### Constructor + +```js +new ValidationUtils(loggerEnabled = true, loggerLevel = 'warn') +``` + +### Methods + +| Method | Signature | Returns | Description | +|---|---|---|---| +| `validateSchema` | `(config, schema, name)` | `object` | Walk the schema, validate every field, return a clean config. Unknown keys are stripped. Missing keys get their schema default. | +| `constrain` | `(value, min, max)` | `number` | Clamp a numeric value to `[min, max]` | +| `removeUnwantedKeys` | `(obj)` | `object` | Strip `rules`/`description` metadata, collapse `default` values | + +**Supported `rules.type` values:** `number`, `integer`, `boolean`, `string`, `enum`, `array`, `set`, `object`, `curve`, `machineCurve`. + +### Example + +```js +const ValidationUtils = require('generalFunctions').validation; +const v = new ValidationUtils(true, 'warn'); + +const schema = { + temperature: { default: 20, rules: { type: 'number', min: -40, max: 100 } }, + unit: { default: 'C', rules: { type: 'enum', values: [{ value: 'C' }, { value: 'F' }] } } +}; + +const validated = v.validateSchema({ temperature: 999 }, schema, 'myNode'); +// validated.temperature === 100 (clamped) +// validated.unit === 'C' (default applied) +``` + +--- + +## MeasurementContainer + +Chainable measurement storage organised by **type / variant / position**. Supports auto unit conversion, windowed statistics, events, and positional difference calculations. + +**File:** `src/measurements/MeasurementContainer.js` + +### Constructor + +```js +new MeasurementContainer(options = {}, logger) +``` + +| Option | Type | Default | Description | +|---|---|---|---| +| `windowSize` | `number` | `10` | Rolling window for statistics | +| `defaultUnits` | `object` | `{ pressure:'mbar', flow:'m3/h', ... }` | Default unit per measurement type | +| `autoConvert` | `boolean` | `true` | Auto-convert values to target unit | +| `preferredUnits` | `object` | `{}` | Per-type unit overrides | + +### Chainable Setters + +All return `this` for chaining. + +```js +container + .type('pressure') + .variant('static') + .position('upstream') + .distance(5) + .unit('bar') + .value(3.2, Date.now(), 'bar'); +``` + +| Method | Signature | Description | +|---|---|---| +| `type` | `(typeName): this` | Set measurement type (e.g. `'pressure'`) | +| `variant` | `(variantName): this` | Set variant (e.g. `'static'`, `'differential'`) | +| `position` | `(positionValue): this` | Set position (e.g. `'upstream'`, `'downstream'`) | +| `distance` | `(distance): this` | Set physical distance from parent | +| `unit` | `(unitName): this` | Set unit on the underlying measurement | +| `value` | `(val, timestamp?, sourceUnit?): this` | Store a value; auto-converts if `sourceUnit` differs from target | + +### Terminal / Query Methods + +| Method | Signature | Returns | Description | +|---|---|---|---| +| `get` | `()` | `Measurement \| null` | Get the raw measurement object | +| `getCurrentValue` | `(requestedUnit?)` | `number \| null` | Latest value, optionally converted | +| `getAverage` | `(requestedUnit?)` | `number \| null` | Windowed average | +| `getMin` | `()` | `number \| null` | Window minimum | +| `getMax` | `()` | `number \| null` | Window maximum | +| `getAllValues` | `()` | `array \| null` | All stored samples | +| `getLaggedValue` | `(lag?, requestedUnit?)` | `number \| null` | Value from `lag` samples ago | +| `getLaggedSample` | `(lag?, requestedUnit?)` | `object \| null` | Full sample `{ value, timestamp, unit }` from `lag` samples ago | +| `exists` | `({ type?, variant?, position?, requireValues? })` | `boolean` | Check if a measurement series exists | +| `difference` | `({ from?, to?, unit? })` | `object \| null` | Compute `{ value, avgDiff, unit }` between two positions | + +### Introspection / Lifecycle + +| Method | Signature | Returns | Description | +|---|---|---|---| +| `getTypes` | `()` | `string[]` | All registered measurement types | +| `getVariants` | `()` | `string[]` | Variants under current type | +| `getPositions` | `()` | `string[]` | Positions under current type+variant | +| `getAvailableUnits` | `(measurementType?)` | `string[]` | Units available for a type | +| `getBestUnit` | `(excludeUnits?)` | `object \| null` | Best human-readable unit for current value | +| `setPreferredUnit` | `(type, unit)` | `this` | Override default unit for a type | +| `setChildId` | `(id)` | `this` | Tag container with a child node ID | +| `setChildName` | `(name)` | `this` | Tag container with a child node name | +| `setParentRef` | `(parent)` | `this` | Store reference to parent node | +| `clear` | `()` | `void` | Reset all measurements and chain state | + +### Events + +The internal `emitter` fires `"type.variant.position"` on every `value()` call with: + +```js +{ value, originalValue, unit, sourceUnit, timestamp, position, distance, variant, type, childId, childName, parentRef } +``` + +### Example + +```js +const { MeasurementContainer } = require('generalFunctions'); +const mc = new MeasurementContainer({ windowSize: 5 }); + +mc.type('pressure').variant('static').position('upstream').value(3.2); +mc.type('pressure').variant('static').position('downstream').value(2.8); + +const diff = mc.type('pressure').variant('static').difference(); +// diff = { value: -0.4, avgDiff: -0.4, unit: 'mbar', from: 'downstream', to: 'upstream' } +``` + +--- + +## ConfigManager + +Loads JSON config files from disk and builds merged runtime configs. + +**File:** `src/configs/index.js` + +### Constructor + +```js +new ConfigManager(relPath = '.') +``` + +`relPath` is resolved relative to the configs directory. + +### Methods + +| Method | Signature | Returns | Description | +|---|---|---|---| +| `getConfig` | `(configName)` | `object` | Load and parse `.json` | +| `getAvailableConfigs` | `()` | `string[]` | List config names (without `.json`) | +| `hasConfig` | `(configName)` | `boolean` | Check existence | +| `getBaseConfig` | `()` | `object` | Shortcut for `getConfig('baseConfig')` | +| `buildConfig` | `(nodeName, uiConfig, nodeId, domainConfig?)` | `object` | Merge base schema + UI overrides into a runtime config | +| `createEndpoint` | `(nodeName)` | `string` | Generate browser JS that injects config into `window.EVOLV.nodes` | + +### Example + +```js +const { configManager } = require('generalFunctions'); +const cfg = configManager.buildConfig('measurement', uiConfig, node.id, { + scaling: { enabled: true, inputMin: 0, inputMax: 100 } +}); +``` + +--- + +## ChildRegistrationUtils + +Manages parent-child node relationships: registration, lookup, and structure storage. + +**File:** `src/helper/childRegistrationUtils.js` + +### Constructor + +```js +new ChildRegistrationUtils(mainClass) +``` + +`mainClass` is the parent node instance (must expose `.logger` and optionally `.registerChild()`). + +### Methods + +| Method | Signature | Returns | Description | +|---|---|---|---| +| `registerChild` | `(child, positionVsParent, distance?)` | `Promise` | Register a child node under the parent. Sets up parent refs, measurement context, and stores by softwareType/category. | +| `getChildrenOfType` | `(softwareType, category?)` | `array` | Get children filtered by software type and optional category | +| `getChildById` | `(childId)` | `object \| null` | Lookup a single child by its ID | +| `getAllChildren` | `()` | `array` | All registered children | +| `logChildStructure` | `()` | `void` | Debug-print the full child tree | + +### Example + +```js +const { childRegistrationUtils: CRU } = require('generalFunctions'); +const cru = new CRU(parentNode); +await cru.registerChild(sensorNode, 'upstream'); +cru.getChildrenOfType('measurement'); // [sensorNode] +``` + +--- + +## MenuUtils + +Browser-side UI helper for Node-RED editor. Methods are mixed in from separate modules: toggles, data fetching, URL utils, dropdown population, and HTML generation. + +**File:** `src/helper/menuUtils.js` + +### Constructor + +```js +new MenuUtils() // no parameters; sets isCloud=false, configData=null +``` + +### Key Methods + +**Toggles** -- control UI element visibility: + +| Method | Signature | Description | +|---|---|---| +| `initBasicToggles` | `(elements)` | Bind log-level row visibility to log checkbox | +| `initMeasurementToggles` | `(elements)` | Bind scaling input rows to scaling checkbox | +| `initTensionToggles` | `(elements, node)` | Show/hide tension row based on interpolation method | + +**Data Fetching:** + +| Method | Signature | Returns | Description | +|---|---|---|---| +| `fetchData` | `(url, fallbackUrl)` | `Promise` | Fetch JSON from primary URL; fall back on failure | +| `fetchProjectData` | `(url)` | `Promise` | Fetch project-level data | +| `apiCall` | `(node)` | `Promise` | POST to asset-register API | + +**URL Construction:** + +| Method | Signature | Returns | Description | +|---|---|---|---| +| `getSpecificConfigUrl` | `(nodeName, cloudAPI)` | `{ cloudConfigURL, localConfigURL }` | Build cloud + local config URLs | +| `constructUrl` | `(base, ...paths)` | `string` | Join URL segments safely | +| `constructCloudURL` | `(base, ...paths)` | `string` | Same as `constructUrl`, for cloud endpoints | + +**Dropdown Population:** + +| Method | Signature | Description | +|---|---|---| +| `fetchAndPopulateDropdowns` | `(configUrls, elements, node)` | Cascading supplier > subType > model > unit dropdowns | +| `populateDropdown` | `(htmlElement, options, node, property, callback?)` | Fill a `