docs: consolidate scattered documentation into wiki
Some checks failed
CI / lint-and-test (push) Has been cancelled
Some checks failed
CI / lint-and-test (push) Has been cancelled
Move architecture/, docs/ content into wiki/ for a single source of truth: - architecture/deployment-blueprint.md → wiki/architecture/ - architecture/stack-architecture-review.md → wiki/architecture/ - architecture/wiki-platform-overview.md → wiki/architecture/ - docs/ARCHITECTURE.md → wiki/architecture/node-architecture.md - docs/API_REFERENCE.md → wiki/concepts/generalfunctions-api.md - docs/ISSUES.md → wiki/findings/open-issues-2026-03.md Remove stale files: - FUNCTIONAL_ISSUES_BACKLOG.md (was just a redirect pointer) - temp/ (stale cloud env examples) Fix README.md gitea URL (centraal.wbd-rd.nl → wbd-rd.nl). Update wiki index with all consolidated pages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
znetsixe
278
wiki/architecture/deployment-blueprint.md
Normal file
278
wiki/architecture/deployment-blueprint.md
Normal file
@@ -0,0 +1,278 @@
|
||||
---
|
||||
title: EVOLV Deployment Blueprint
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [deployment, docker, edge, site, central]
|
||||
---
|
||||
|
||||
# EVOLV Deployment Blueprint
|
||||
|
||||
## Purpose
|
||||
|
||||
This document turns the current EVOLV architecture into a concrete deployment model.
|
||||
|
||||
It focuses on:
|
||||
|
||||
- target infrastructure layout
|
||||
- container/service topology
|
||||
- environment and secret boundaries
|
||||
- rollout order from edge to site to central
|
||||
|
||||
It is the local source document behind the wiki deployment pages.
|
||||
|
||||
## 1. Deployment Principles
|
||||
|
||||
- edge-first operation: plant logic must continue when central is unavailable
|
||||
- site mediation: site services protect field systems and absorb plant-specific complexity
|
||||
- central governance: external APIs, analytics, IAM, CI/CD, and shared dashboards terminate centrally
|
||||
- layered telemetry: InfluxDB exists where operationally justified at edge, site, and central
|
||||
- configuration authority: `tagcodering` should become the source of truth for configuration
|
||||
- secrets hygiene: tracked manifests contain variables only; secrets live in server-side env or secret stores
|
||||
|
||||
## 2. Layered Deployment Model
|
||||
|
||||
### 2.1 Edge node
|
||||
|
||||
Purpose:
|
||||
|
||||
- interface with PLCs and field assets
|
||||
- execute local Node-RED logic
|
||||
- retain local telemetry for resilience and digital-twin use cases
|
||||
|
||||
Recommended services:
|
||||
|
||||
- `evolv-edge-nodered`
|
||||
- `evolv-edge-influxdb`
|
||||
- optional `evolv-edge-grafana`
|
||||
- optional `evolv-edge-broker`
|
||||
|
||||
Should not host:
|
||||
|
||||
- public API ingress
|
||||
- central IAM
|
||||
- source control or CI/CD
|
||||
|
||||
### 2.2 Site node
|
||||
|
||||
Purpose:
|
||||
|
||||
- aggregate one or more edge nodes
|
||||
- host plant-local dashboards and engineering visibility
|
||||
- mediate traffic between edge and central
|
||||
|
||||
Recommended services:
|
||||
|
||||
- `evolv-site-nodered` or `coresync-site`
|
||||
- `evolv-site-influxdb`
|
||||
- `evolv-site-grafana`
|
||||
- optional `evolv-site-broker`
|
||||
|
||||
### 2.3 Central platform
|
||||
|
||||
Purpose:
|
||||
|
||||
- fleet-wide analytics
|
||||
- API and integration ingress
|
||||
- engineering lifecycle and releases
|
||||
- identity and governance
|
||||
|
||||
Recommended services:
|
||||
|
||||
- reverse proxy / ingress
|
||||
- API gateway
|
||||
- IAM
|
||||
- central InfluxDB
|
||||
- central Grafana
|
||||
- Gitea
|
||||
- CI/CD runner/controller
|
||||
- optional broker for asynchronous site/central workflows
|
||||
- configuration services over `tagcodering`
|
||||
|
||||
## 3. Target Container Topology
|
||||
|
||||
### 3.1 Edge host
|
||||
|
||||
Minimum viable edge stack:
|
||||
|
||||
```text
|
||||
edge-host-01
|
||||
- Node-RED
|
||||
- InfluxDB
|
||||
- optional Grafana
|
||||
```
|
||||
|
||||
Preferred production edge stack:
|
||||
|
||||
```text
|
||||
edge-host-01
|
||||
- Node-RED
|
||||
- InfluxDB
|
||||
- local health/export service
|
||||
- optional local broker
|
||||
- optional local dashboard service
|
||||
```
|
||||
|
||||
### 3.2 Site host
|
||||
|
||||
Minimum viable site stack:
|
||||
|
||||
```text
|
||||
site-host-01
|
||||
- Site Node-RED / CoreSync
|
||||
- Site InfluxDB
|
||||
- Site Grafana
|
||||
```
|
||||
|
||||
Preferred production site stack:
|
||||
|
||||
```text
|
||||
site-host-01
|
||||
- Site Node-RED / CoreSync
|
||||
- Site InfluxDB
|
||||
- Site Grafana
|
||||
- API relay / sync service
|
||||
- optional site broker
|
||||
```
|
||||
|
||||
### 3.3 Central host group
|
||||
|
||||
Central should not be one giant undifferentiated host forever. It should trend toward at least these responsibility groups:
|
||||
|
||||
```text
|
||||
central-ingress
|
||||
- reverse proxy
|
||||
- API gateway
|
||||
- IAM
|
||||
|
||||
central-observability
|
||||
- central InfluxDB
|
||||
- Grafana
|
||||
|
||||
central-engineering
|
||||
- Gitea
|
||||
- CI/CD
|
||||
- deployment orchestration
|
||||
|
||||
central-config
|
||||
- tagcodering-backed config services
|
||||
```
|
||||
|
||||
For early rollout these may be colocated, but the responsibility split should remain clear.
|
||||
|
||||
## 4. Compose Strategy
|
||||
|
||||
The current repository shows:
|
||||
|
||||
- `docker-compose.yml` as a development stack
|
||||
- `temp/cloud.yml` as a broad central-stack example
|
||||
|
||||
For production, EVOLV should not rely on one flat compose file for every layer.
|
||||
|
||||
Recommended split:
|
||||
|
||||
- `compose.edge.yml`
|
||||
- `compose.site.yml`
|
||||
- `compose.central.yml`
|
||||
- optional overlay files for site-specific differences
|
||||
|
||||
Benefits:
|
||||
|
||||
- clearer ownership per layer
|
||||
- smaller blast radius during updates
|
||||
- easier secret and env separation
|
||||
- easier rollout per site
|
||||
|
||||
## 5. Environment And Secrets Strategy
|
||||
|
||||
### 5.1 Current baseline
|
||||
|
||||
`temp/cloud.yml` now uses environment variables instead of inline credentials. That is the minimum acceptable baseline.
|
||||
|
||||
### 5.2 Recommended production rule
|
||||
|
||||
- tracked compose files contain `${VARIABLE}` placeholders only
|
||||
- real secrets live in server-local `.env` files or a managed secret store
|
||||
- no shared default production passwords in git
|
||||
- separate env files per layer and per environment
|
||||
|
||||
Suggested structure:
|
||||
|
||||
```text
|
||||
/opt/evolv/
|
||||
compose.edge.yml
|
||||
compose.site.yml
|
||||
compose.central.yml
|
||||
env/
|
||||
edge.env
|
||||
site.env
|
||||
central.env
|
||||
```
|
||||
|
||||
## 6. Recommended Network Flow
|
||||
|
||||
### 6.1 Northbound
|
||||
|
||||
- edge publishes or syncs upward to site
|
||||
- site aggregates and forwards selected data to central
|
||||
- central exposes APIs and dashboards to approved consumers
|
||||
|
||||
### 6.2 Southbound
|
||||
|
||||
- central issues advice, approved config, or mediated requests
|
||||
- site validates and relays to edge where appropriate
|
||||
- edge remains the execution point near PLCs
|
||||
|
||||
### 6.3 Forbidden direct path
|
||||
|
||||
- enterprise or internet clients should not directly query PLC-connected edge runtimes
|
||||
|
||||
## 7. Rollout Order
|
||||
|
||||
### Phase 1: Edge baseline
|
||||
|
||||
- deploy edge Node-RED
|
||||
- deploy local InfluxDB
|
||||
- validate PLC connectivity
|
||||
- validate local telemetry and resilience
|
||||
|
||||
### Phase 2: Site mediation
|
||||
|
||||
- deploy site Node-RED / CoreSync
|
||||
- connect one or more edge nodes
|
||||
- validate site-local dashboards and outage behavior
|
||||
|
||||
### Phase 3: Central services
|
||||
|
||||
- deploy ingress, IAM, API, Grafana, central InfluxDB
|
||||
- deploy Gitea and CI/CD services
|
||||
- validate controlled northbound access
|
||||
|
||||
### Phase 4: Configuration backbone
|
||||
|
||||
- connect runtime layers to `tagcodering`
|
||||
- reduce config duplication in flows
|
||||
- formalize config promotion and rollback
|
||||
|
||||
### Phase 5: Smart telemetry policy
|
||||
|
||||
- classify signals
|
||||
- define reconstruction rules
|
||||
- define authoritative layer per horizon
|
||||
- validate analytics and auditability
|
||||
|
||||
## 8. Immediate Technical Recommendations
|
||||
|
||||
- treat `docker/settings.js` as development-only and create hardened production settings separately
|
||||
- split deployment manifests by layer
|
||||
- define env files per layer and environment
|
||||
- formalize healthchecks and backup procedures for every persistent service
|
||||
- define whether broker usage is required at edge, site, central, or only selectively
|
||||
|
||||
## 9. Next Technical Work Items
|
||||
|
||||
1. create draft `compose.edge.yml`, `compose.site.yml`, and `compose.central.yml`
|
||||
2. define server directory layout and env-file conventions
|
||||
3. define production Node-RED settings profile
|
||||
4. define site-to-central sync path
|
||||
5. define deployment and rollback runbook
|
||||
426
wiki/architecture/node-architecture.md
Normal file
426
wiki/architecture/node-architecture.md
Normal file
@@ -0,0 +1,426 @@
|
||||
---
|
||||
title: EVOLV Architecture
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [architecture, node-red, three-layer]
|
||||
---
|
||||
|
||||
# EVOLV Architecture
|
||||
|
||||
## 1. System Overview
|
||||
|
||||
High-level view of how EVOLV fits into the wastewater treatment automation stack.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
NR[Node-RED Runtime] <-->|msg objects| EVOLV[EVOLV Nodes]
|
||||
EVOLV -->|InfluxDB line protocol| INFLUX[(InfluxDB)]
|
||||
INFLUX -->|queries| GRAFANA[Grafana Dashboards]
|
||||
EVOLV -->|process output| NR
|
||||
EVOLV -->|parent output| NR
|
||||
|
||||
style NR fill:#b22222,color:#fff
|
||||
style EVOLV fill:#0f52a5,color:#fff
|
||||
style INFLUX fill:#0c99d9,color:#fff
|
||||
style GRAFANA fill:#50a8d9,color:#fff
|
||||
```
|
||||
|
||||
Each EVOLV node produces three outputs:
|
||||
| Port | Name | Purpose |
|
||||
|------|------|---------|
|
||||
| 0 | process | Process data forwarded to downstream nodes |
|
||||
| 1 | dbase | InfluxDB-formatted measurement data |
|
||||
| 2 | parent | Control messages to parent nodes (e.g. registerChild) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Node Architecture (Three-Layer Pattern)
|
||||
|
||||
Every node follows a consistent three-layer design that separates Node-RED wiring from domain logic.
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Node-RED Runtime"
|
||||
REG["RED.nodes.registerType()"]
|
||||
end
|
||||
|
||||
subgraph "Layer 1 — Wrapper (valve.js)"
|
||||
W[wrapper .js]
|
||||
W -->|"new nodeClass(config, RED, this, name)"| NC
|
||||
W -->|MenuManager| MENU[HTTP /name/menu.js]
|
||||
W -->|configManager| CFG[HTTP /name/configData.js]
|
||||
end
|
||||
|
||||
subgraph "Layer 2 — Node Adapter (src/nodeClass.js)"
|
||||
NC[nodeClass]
|
||||
NC -->|_loadConfig| CFGM[configManager]
|
||||
NC -->|_setupSpecificClass| SC
|
||||
NC -->|_attachInputHandler| INPUT[onInput routing]
|
||||
NC -->|_startTickLoop| TICK[1s tick loop]
|
||||
NC -->|_tick → outputUtils| OUT[formatMsg]
|
||||
end
|
||||
|
||||
subgraph "Layer 3 — Domain Logic (src/specificClass.js)"
|
||||
SC[specificClass]
|
||||
SC -->|measurements| MC[MeasurementContainer]
|
||||
SC -->|state machine| ST[state]
|
||||
SC -->|hydraulics / biology| DOMAIN[domain models]
|
||||
end
|
||||
|
||||
subgraph "generalFunctions"
|
||||
GF[shared library]
|
||||
end
|
||||
|
||||
REG --> W
|
||||
GF -.->|logger, outputUtils, configManager,\nMeasurementContainer, validation, ...| NC
|
||||
GF -.->|MeasurementContainer, state,\nconvert, predict, ...| SC
|
||||
|
||||
style W fill:#0f52a5,color:#fff
|
||||
style NC fill:#0c99d9,color:#fff
|
||||
style SC fill:#50a8d9,color:#fff
|
||||
style GF fill:#86bbdd,color:#000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. generalFunctions Module Map
|
||||
|
||||
The shared library (`nodes/generalFunctions/`) provides all cross-cutting concerns.
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
GF[generalFunctions/index.js]
|
||||
|
||||
subgraph "Core Helpers (src/helper/)"
|
||||
LOGGER[logger]
|
||||
OUTPUT[outputUtils]
|
||||
CHILD[childRegistrationUtils]
|
||||
CFGUTIL[configUtils]
|
||||
ASSERT[assertionUtils]
|
||||
VALID[validationUtils]
|
||||
end
|
||||
|
||||
subgraph "Validators (src/helper/validators/)"
|
||||
TV[typeValidators]
|
||||
CV[collectionValidators]
|
||||
CURV[curveValidator]
|
||||
end
|
||||
|
||||
subgraph "Domain Modules (src/)"
|
||||
MC[MeasurementContainer]
|
||||
CFGMGR[configManager]
|
||||
MENUMGR[MenuManager]
|
||||
STATE[state]
|
||||
CONVERT[convert / Fysics]
|
||||
PREDICT[predict / interpolation]
|
||||
NRMSE[nrmse / errorMetrics]
|
||||
COOLPROP[coolprop]
|
||||
end
|
||||
|
||||
subgraph "Data (datasets/)"
|
||||
CURVES[assetData/curves]
|
||||
ASSETS[assetData/assetData.json]
|
||||
UNITS[unitData.json]
|
||||
end
|
||||
|
||||
subgraph "Constants (src/constants/)"
|
||||
POS[POSITIONS / POSITION_VALUES]
|
||||
end
|
||||
|
||||
GF --> LOGGER
|
||||
GF --> OUTPUT
|
||||
GF --> CHILD
|
||||
GF --> CFGUTIL
|
||||
GF --> ASSERT
|
||||
GF --> VALID
|
||||
VALID --> TV
|
||||
VALID --> CV
|
||||
VALID --> CURV
|
||||
GF --> MC
|
||||
GF --> CFGMGR
|
||||
GF --> MENUMGR
|
||||
GF --> STATE
|
||||
GF --> CONVERT
|
||||
GF --> PREDICT
|
||||
GF --> NRMSE
|
||||
GF --> COOLPROP
|
||||
GF --> CURVES
|
||||
GF --> POS
|
||||
|
||||
style GF fill:#0f52a5,color:#fff
|
||||
style LOGGER fill:#86bbdd,color:#000
|
||||
style OUTPUT fill:#86bbdd,color:#000
|
||||
style VALID fill:#86bbdd,color:#000
|
||||
style MC fill:#50a8d9,color:#fff
|
||||
style CFGMGR fill:#50a8d9,color:#fff
|
||||
style MENUMGR fill:#50a8d9,color:#fff
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Data Flow (Message Lifecycle)
|
||||
|
||||
Sequence diagram showing a typical input message and the periodic tick output cycle.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant NR as Node-RED
|
||||
participant W as wrapper.js
|
||||
participant NC as nodeClass
|
||||
participant SC as specificClass
|
||||
participant OU as outputUtils
|
||||
|
||||
Note over W: Node startup
|
||||
W->>NC: new nodeClass(config, RED, node, name)
|
||||
NC->>NC: _loadConfig (configManager.buildConfig)
|
||||
NC->>SC: new specificClass(config, stateConfig, options)
|
||||
NC->>NR: send([null, null, {topic: registerChild}])
|
||||
|
||||
Note over NC: Every 1 second (tick loop)
|
||||
NC->>SC: getOutput()
|
||||
SC-->>NC: raw measurement data
|
||||
NC->>OU: formatMsg(raw, config, 'process')
|
||||
NC->>OU: formatMsg(raw, config, 'influxdb')
|
||||
NC->>NR: send([processMsg, influxMsg])
|
||||
|
||||
Note over NR: Incoming control message
|
||||
NR->>W: msg {topic: 'execMovement', payload: {...}}
|
||||
W->>NC: onInput(msg)
|
||||
NC->>SC: handleInput(source, action, setpoint)
|
||||
SC->>SC: update state machine & measurements
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Node Types
|
||||
|
||||
| Node | S88 Level | Purpose |
|
||||
|------|-----------|---------|
|
||||
| **measurement** | Control Module | Generic measurement point — reads, validates, and stores sensor values |
|
||||
| **valve** | Control Module | Valve simulation with hydraulic model, position control, flow/pressure prediction |
|
||||
| **rotatingMachine** | Control Module | Pumps, blowers, mixers — rotating equipment with speed control and efficiency curves |
|
||||
| **diffuser** | Control Module | Aeration diffuser — models oxygen transfer and pressure drop |
|
||||
| **settler** | Equipment | Sludge settler — models settling behavior and sludge blanket |
|
||||
| **reactor** | Equipment | Hydraulic tank and biological process simulator (activated sludge, digestion) |
|
||||
| **monster** | Equipment | MONitoring and STrEam Routing — complex measurement aggregation |
|
||||
| **pumpingStation** | Unit | Coordinates multiple pumps as a pumping station |
|
||||
| **valveGroupControl** | Unit | Manages multiple valves as a coordinated group — distributes flow, monitors pressure |
|
||||
| **machineGroupControl** | Unit | Group control for rotating machines — load balancing and sequencing |
|
||||
| **dashboardAPI** | Utility | Exposes data and unit conversion endpoints for external dashboards |
|
||||
# EVOLV Architecture
|
||||
|
||||
## Node Hierarchy (S88)
|
||||
|
||||
EVOLV follows the ISA-88 (S88) batch control standard. Each node maps to an S88 level and uses a consistent color scheme in the Node-RED editor.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
classDef area fill:#0f52a5,color:#fff,stroke:#0a3d7a
|
||||
classDef processCell fill:#0c99d9,color:#fff,stroke:#0977aa
|
||||
classDef unit fill:#50a8d9,color:#fff,stroke:#3d89b3
|
||||
classDef equipment fill:#86bbdd,color:#000,stroke:#6a9bb8
|
||||
classDef controlModule fill:#a9daee,color:#000,stroke:#87b8cc
|
||||
classDef standalone fill:#f0f0f0,color:#000,stroke:#999
|
||||
|
||||
%% S88 Levels
|
||||
subgraph "S88: Area"
|
||||
PS[pumpingStation]
|
||||
end
|
||||
|
||||
subgraph "S88: Equipment"
|
||||
MGC[machineGroupControl]
|
||||
VGC[valveGroupControl]
|
||||
end
|
||||
|
||||
subgraph "S88: Control Module"
|
||||
RM[rotatingMachine]
|
||||
V[valve]
|
||||
M[measurement]
|
||||
R[reactor]
|
||||
S[settler]
|
||||
end
|
||||
|
||||
subgraph "Standalone"
|
||||
MON[monster]
|
||||
DASH[dashboardAPI]
|
||||
DIFF[diffuser - not implemented]
|
||||
end
|
||||
|
||||
%% Parent-child registration relationships
|
||||
PS -->|"accepts: measurement"| M
|
||||
PS -->|"accepts: machine"| RM
|
||||
PS -->|"accepts: machineGroup"| MGC
|
||||
PS -->|"accepts: pumpingStation"| PS2[pumpingStation]
|
||||
|
||||
MGC -->|"accepts: machine"| RM
|
||||
|
||||
RM -->|"accepts: measurement"| M2[measurement]
|
||||
RM -->|"accepts: reactor"| R
|
||||
|
||||
VGC -->|"accepts: valve"| V
|
||||
VGC -->|"accepts: machine / rotatingmachine"| RM2[rotatingMachine]
|
||||
VGC -->|"accepts: machinegroup / machinegroupcontrol"| MGC2[machineGroupControl]
|
||||
VGC -->|"accepts: pumpingstation / valvegroupcontrol"| PS3["pumpingStation / valveGroupControl"]
|
||||
|
||||
R -->|"accepts: measurement"| M3[measurement]
|
||||
R -->|"accepts: reactor"| R2[reactor]
|
||||
|
||||
S -->|"accepts: measurement"| M4[measurement]
|
||||
S -->|"accepts: reactor"| R3[reactor]
|
||||
S -->|"accepts: machine"| RM3[rotatingMachine]
|
||||
|
||||
%% Styling
|
||||
class PS,PS2,PS3 area
|
||||
class MGC,MGC2 equipment
|
||||
class VGC equipment
|
||||
class RM,RM2,RM3 controlModule
|
||||
class V controlModule
|
||||
class M,M2,M3,M4 controlModule
|
||||
class R,R2,R3 controlModule
|
||||
class S controlModule
|
||||
class MON,DASH,DIFF standalone
|
||||
```
|
||||
|
||||
### Registration Summary
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
classDef parent fill:#0c99d9,color:#fff
|
||||
classDef child fill:#a9daee,color:#000
|
||||
|
||||
PS[pumpingStation] -->|measurement| LEAF1((leaf))
|
||||
PS -->|machine| RM1[rotatingMachine]
|
||||
PS -->|machineGroup| MGC1[machineGroupControl]
|
||||
PS -->|pumpingStation| PS1[pumpingStation]
|
||||
|
||||
MGC[machineGroupControl] -->|machine| RM2[rotatingMachine]
|
||||
|
||||
VGC[valveGroupControl] -->|valve| V1[valve]
|
||||
VGC -->|source| SRC["machine, machinegroup,<br/>pumpingstation, valvegroupcontrol"]
|
||||
|
||||
RM[rotatingMachine] -->|measurement| LEAF2((leaf))
|
||||
RM -->|reactor| R1[reactor]
|
||||
|
||||
R[reactor] -->|measurement| LEAF3((leaf))
|
||||
R -->|reactor| R2[reactor]
|
||||
|
||||
S[settler] -->|measurement| LEAF4((leaf))
|
||||
S -->|reactor| R3[reactor]
|
||||
S -->|machine| RM3[rotatingMachine]
|
||||
|
||||
class PS,MGC,VGC,RM,R,S parent
|
||||
class LEAF1,LEAF2,LEAF3,LEAF4,RM1,RM2,RM3,MGC1,PS1,V1,SRC,R1,R2,R3 child
|
||||
```
|
||||
|
||||
## Node Types
|
||||
|
||||
| Node | S88 Level | softwareType | role | Accepts Children | Outputs |
|
||||
|------|-----------|-------------|------|-----------------|---------|
|
||||
| **pumpingStation** | Area | `pumpingstation` | StationController | measurement, machine (rotatingMachine), machineGroup, pumpingStation | [process, dbase, parent] |
|
||||
| **machineGroupControl** | Equipment | `machinegroupcontrol` | GroupController | machine (rotatingMachine) | [process, dbase, parent] |
|
||||
| **valveGroupControl** | Equipment | `valvegroupcontrol` | ValveGroupController | valve, machine, rotatingmachine, machinegroup, machinegroupcontrol, pumpingstation, valvegroupcontrol | [process, dbase, parent] |
|
||||
| **rotatingMachine** | Control Module | `rotatingmachine` | RotationalDeviceController | measurement, reactor | [process, dbase, parent] |
|
||||
| **valve** | Control Module | `valve` | controller | _(leaf node, no children)_ | [process, dbase, parent] |
|
||||
| **measurement** | Control Module | `measurement` | Sensor | _(leaf node, no children)_ | [process, dbase, parent] |
|
||||
| **reactor** | Control Module | `reactor` | Biological reactor | measurement, reactor (upstream chaining) | [process, dbase, parent] |
|
||||
| **settler** | Control Module | `settler` | Secondary settler | measurement, reactor (upstream), machine (return pump) | [process, dbase, parent] |
|
||||
| **monster** | Standalone | - | - | dual-parent, standalone | - |
|
||||
| **dashboardAPI** | Standalone | - | - | accepts any child (Grafana integration) | - |
|
||||
| **diffuser** | Standalone | - | - | _(not implemented)_ | - |
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Measurement Data Flow (upstream to downstream)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Sensor as measurement (sensor)
|
||||
participant Machine as rotatingMachine
|
||||
participant Group as machineGroupControl
|
||||
participant Station as pumpingStation
|
||||
|
||||
Note over Sensor: Sensor reads value<br/>(pressure, flow, level, temp)
|
||||
|
||||
Sensor->>Sensor: measurements.type(t).variant("measured").position(p).value(v)
|
||||
Sensor->>Sensor: emitter.emit("type.measured.position", eventData)
|
||||
|
||||
Sensor->>Machine: Event: "pressure.measured.upstream"
|
||||
Machine->>Machine: Store in own MeasurementContainer
|
||||
Machine->>Machine: getMeasuredPressure() -> calcFlow() -> calcPower()
|
||||
Machine->>Machine: emitter.emit("flow.predicted.downstream", eventData)
|
||||
|
||||
Machine->>Group: Event: "flow.predicted.downstream"
|
||||
Group->>Group: handlePressureChange()
|
||||
Group->>Group: Aggregate flows across all machines
|
||||
Group->>Group: Calculate group totals and efficiency
|
||||
|
||||
Machine->>Station: Event: "flow.predicted.downstream"
|
||||
Station->>Station: Store predicted flow in/out
|
||||
Station->>Station: _updateVolumePrediction()
|
||||
Station->>Station: _calcNetFlow(), _calcTimeRemaining()
|
||||
```
|
||||
|
||||
### Control Command Flow (downstream to upstream)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Station as pumpingStation
|
||||
participant Group as machineGroupControl
|
||||
participant Machine as rotatingMachine
|
||||
participant Machine2 as rotatingMachine (2)
|
||||
|
||||
Station->>Group: handleInput("parent", action, param)
|
||||
|
||||
Group->>Group: Determine scaling strategy
|
||||
Group->>Group: Calculate setpoints per machine
|
||||
|
||||
Group->>Machine: handleInput("parent", "execMovement", setpoint)
|
||||
Group->>Machine2: handleInput("parent", "execMovement", setpoint)
|
||||
|
||||
Machine->>Machine: setpoint() -> state.moveTo(pos)
|
||||
Machine->>Machine: updatePosition() -> calcFlow(), calcPower()
|
||||
Machine->>Machine: emitter.emit("flow.predicted.downstream")
|
||||
|
||||
Machine2->>Machine2: setpoint() -> state.moveTo(pos)
|
||||
Machine2->>Machine2: updatePosition() -> calcFlow(), calcPower()
|
||||
Machine2->>Machine2: emitter.emit("flow.predicted.downstream")
|
||||
```
|
||||
|
||||
### Wastewater Treatment Process Flow
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
classDef process fill:#50a8d9,color:#fff
|
||||
classDef equipment fill:#86bbdd,color:#000
|
||||
|
||||
PS_IN[pumpingStation<br/>Influent] -->|flow| R1[reactor<br/>Anoxic]
|
||||
R1 -->|effluent| R2[reactor<br/>Aerated]
|
||||
R2 -->|effluent| SET[settler]
|
||||
SET -->|effluent out| PS_OUT[pumpingStation<br/>Effluent]
|
||||
SET -->|sludge return| RM_RET[rotatingMachine<br/>Return pump]
|
||||
RM_RET -->|recirculation| R1
|
||||
|
||||
PS_IN --- MGC_IN[machineGroupControl]
|
||||
MGC_IN --- RM_IN[rotatingMachine<br/>Influent pumps]
|
||||
|
||||
class PS_IN,PS_OUT process
|
||||
class R1,R2,SET process
|
||||
class MGC_IN,RM_IN,RM_RET equipment
|
||||
```
|
||||
|
||||
### Event-Driven Communication Pattern
|
||||
|
||||
All parent-child communication uses Node.js `EventEmitter`:
|
||||
|
||||
1. **Registration**: Parent calls `childRegistrationUtils.registerChild(child, position)` which stores the child and calls the parent's `registerChild(child, softwareType)` method.
|
||||
2. **Event binding**: The parent's `registerChild()` subscribes to the child's `measurements.emitter` events (e.g., `"flow.predicted.downstream"`).
|
||||
3. **Data propagation**: When a child updates a measurement, it emits an event. The parent's listener stores the value in its own `MeasurementContainer` and runs its domain logic.
|
||||
4. **Three outputs**: Every node sends data to three Node-RED outputs: `[process, dbase, parent]` -- process data for downstream nodes, InfluxDB for persistence, and parent aggregation data.
|
||||
|
||||
### Position Convention
|
||||
|
||||
Children register with a position relative to their parent:
|
||||
- `upstream` -- before the parent in the flow direction
|
||||
- `downstream` -- after the parent in the flow direction
|
||||
- `atEquipment` -- physically located at/on the parent equipment
|
||||
158
wiki/architecture/platform-overview.md
Normal file
158
wiki/architecture/platform-overview.md
Normal file
@@ -0,0 +1,158 @@
|
||||
---
|
||||
title: EVOLV Platform Architecture
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [architecture, platform, edge-first]
|
||||
---
|
||||
|
||||
# EVOLV Platform Architecture
|
||||
|
||||
## At A Glance
|
||||
|
||||
EVOLV is not only a Node-RED package. It is a layered automation platform:
|
||||
|
||||
- edge for plant-side execution
|
||||
- site for local aggregation and resilience
|
||||
- central for coordination, analytics, APIs, and governance
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph EDGE["Edge"]
|
||||
PLC["PLC / IO"]
|
||||
ENR["Node-RED"]
|
||||
EDB["Local InfluxDB"]
|
||||
EUI["Local Monitoring"]
|
||||
end
|
||||
|
||||
subgraph SITE["Site"]
|
||||
SNR["CoreSync / Site Node-RED"]
|
||||
SDB["Site InfluxDB"]
|
||||
SUI["Site Dashboards"]
|
||||
end
|
||||
|
||||
subgraph CENTRAL["Central"]
|
||||
API["API Gateway"]
|
||||
CFG["Tagcodering"]
|
||||
CDB["Central InfluxDB"]
|
||||
CGR["Grafana"]
|
||||
INTEL["Overview Intelligence"]
|
||||
GIT["Gitea + CI/CD"]
|
||||
end
|
||||
|
||||
PLC --> ENR
|
||||
ENR --> EDB
|
||||
ENR --> EUI
|
||||
ENR <--> SNR
|
||||
EDB <--> SDB
|
||||
SNR --> SUI
|
||||
SNR <--> API
|
||||
API <--> CFG
|
||||
API --> INTEL
|
||||
SDB <--> CDB
|
||||
CDB --> CGR
|
||||
GIT --> ENR
|
||||
GIT --> SNR
|
||||
```
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. Edge-first operation
|
||||
|
||||
The edge layer must remain useful and safe when central systems are down.
|
||||
|
||||
That means:
|
||||
|
||||
- local logic remains operational
|
||||
- local telemetry remains queryable
|
||||
- local dashboards can keep working
|
||||
|
||||
### 2. Multi-level telemetry
|
||||
|
||||
InfluxDB is expected on multiple levels:
|
||||
|
||||
- local for resilience and digital-twin use
|
||||
- site for plant diagnostics
|
||||
- central for fleet analytics and advisory logic
|
||||
|
||||
### 3. Smart storage
|
||||
|
||||
Telemetry should not be stored only with naive deadband rules.
|
||||
|
||||
The target model is signal-aware:
|
||||
|
||||
- preserve critical change points
|
||||
- reduce low-information flat sections
|
||||
- allow downstream reconstruction where justified
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
SIG["Process Signal"] --> EVAL["Slope / Event Evaluation"]
|
||||
EVAL --> KEEP["Keep critical points"]
|
||||
EVAL --> REDUCE["Reduce reconstructable points"]
|
||||
KEEP --> L0["Local InfluxDB"]
|
||||
REDUCE --> L0
|
||||
L0 --> L1["Site InfluxDB"]
|
||||
L1 --> L2["Central InfluxDB"]
|
||||
```
|
||||
|
||||
### 4. Central is the safe entry point
|
||||
|
||||
External systems should enter through central APIs, not by directly calling field-edge systems.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
EXT["External Request"] --> API["Central API Gateway"]
|
||||
API --> AUTH["Auth / Policy"]
|
||||
AUTH --> SITE["Site Layer"]
|
||||
SITE --> EDGE["Edge Layer"]
|
||||
EDGE --> PLC["Field Assets"]
|
||||
|
||||
EXT -. blocked .-> EDGE
|
||||
EXT -. blocked .-> PLC
|
||||
```
|
||||
|
||||
### 5. Configuration belongs in `tagcodering`
|
||||
|
||||
The intended configuration source of truth is the database-backed `tagcodering` model:
|
||||
|
||||
- machine metadata
|
||||
- asset configuration
|
||||
- runtime-consumable configuration
|
||||
- future central/site configuration services
|
||||
|
||||
This already exists partially but still needs more work before it fully serves that role.
|
||||
|
||||
## Layer Roles
|
||||
|
||||
### Edge
|
||||
|
||||
- PLC connectivity
|
||||
- local logic
|
||||
- protocol translation
|
||||
- local telemetry buffering
|
||||
- local monitoring and digital-twin support
|
||||
|
||||
### Site
|
||||
|
||||
- aggregation of edge systems
|
||||
- local dashboards and diagnostics
|
||||
- mediation between OT and central
|
||||
- protected handoff for central requests
|
||||
|
||||
### Central
|
||||
|
||||
- enterprise/API gateway
|
||||
- fleet dashboards
|
||||
- analytics and intelligence
|
||||
- source control and CI/CD
|
||||
- configuration governance through `tagcodering`
|
||||
|
||||
## Why This Matters
|
||||
|
||||
This architecture gives EVOLV:
|
||||
|
||||
- better resilience
|
||||
- safer external integration
|
||||
- better data quality for analytics
|
||||
- a path from Node-RED package to platform
|
||||
632
wiki/architecture/stack-review.md
Normal file
632
wiki/architecture/stack-review.md
Normal file
@@ -0,0 +1,632 @@
|
||||
---
|
||||
title: EVOLV Architecture Review
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [architecture, stack, review]
|
||||
---
|
||||
|
||||
# EVOLV Architecture Review
|
||||
|
||||
## Purpose
|
||||
|
||||
This document captures:
|
||||
|
||||
- the architecture implemented in this repository today
|
||||
- the broader edge/site/central architecture shown in the drawings under `temp/`
|
||||
- the key strengths and weaknesses of that direction
|
||||
- the currently preferred target stack based on owner decisions from this review
|
||||
|
||||
It is the local staging document for a later wiki update.
|
||||
|
||||
## Evidence Used
|
||||
|
||||
Implemented stack evidence:
|
||||
|
||||
- `docker-compose.yml`
|
||||
- `docker/settings.js`
|
||||
- `docker/grafana/provisioning/datasources/influxdb.yaml`
|
||||
- `package.json`
|
||||
- `nodes/*`
|
||||
|
||||
Target-state evidence:
|
||||
|
||||
- `temp/fullStack.pdf`
|
||||
- `temp/edge.pdf`
|
||||
- `temp/CoreSync.drawio.pdf`
|
||||
- `temp/cloud.yml`
|
||||
|
||||
Owner decisions from this review:
|
||||
|
||||
- local InfluxDB is required for operational resilience
|
||||
- central acts as the advisory/intelligence and API-entry layer, not as a direct field caller
|
||||
- intended configuration authority is the database-backed `tagcodering` model
|
||||
- architecture wiki pages should be visual, not text-only
|
||||
|
||||
## 1. What Exists Today
|
||||
|
||||
### 1.1 Product/runtime layer
|
||||
|
||||
The codebase is currently a modular Node-RED package for wastewater/process automation:
|
||||
|
||||
- EVOLV ships custom Node-RED nodes for plant assets and process logic
|
||||
- nodes emit both process/control messages and telemetry-oriented outputs
|
||||
- shared helper logic lives in `nodes/generalFunctions/`
|
||||
- Grafana-facing integration exists through `dashboardAPI` and Influx-oriented outputs
|
||||
|
||||
### 1.2 Implemented development stack
|
||||
|
||||
The concrete development stack in this repository is:
|
||||
|
||||
- Node-RED
|
||||
- InfluxDB 2.x
|
||||
- Grafana
|
||||
|
||||
That gives a clear local flow:
|
||||
|
||||
1. EVOLV logic runs in Node-RED.
|
||||
2. Telemetry is emitted in a time-series-oriented shape.
|
||||
3. InfluxDB stores the telemetry.
|
||||
4. Grafana renders operational dashboards.
|
||||
|
||||
### 1.3 Existing runtime pattern in the nodes
|
||||
|
||||
A recurring EVOLV pattern is:
|
||||
|
||||
- output 0: process/control message
|
||||
- output 1: Influx/telemetry message
|
||||
- output 2: registration/control plumbing where relevant
|
||||
|
||||
So even in its current implemented form, EVOLV is not only a Node-RED project. It is already a control-plus-observability platform, with Node-RED as orchestration/runtime and InfluxDB/Grafana as telemetry and visualization services.
|
||||
|
||||
## 2. What The Drawings Describe
|
||||
|
||||
Across `temp/fullStack.pdf` and `temp/CoreSync.drawio.pdf`, the intended platform is broader and layered.
|
||||
|
||||
### 2.1 Edge / OT layer
|
||||
|
||||
The drawings consistently place these capabilities at the edge:
|
||||
|
||||
- PLC / OPC UA connectivity
|
||||
- Node-RED container as protocol translator and logic runtime
|
||||
- local broker in some variants
|
||||
- local InfluxDB / Prometheus style storage in some variants
|
||||
- local Grafana/SCADA in some variants
|
||||
|
||||
This is the plant-side operational layer.
|
||||
|
||||
### 2.2 Site / local server layer
|
||||
|
||||
The CoreSync drawings also show a site aggregation layer:
|
||||
|
||||
- RWZI-local server
|
||||
- Node-RED / CoreSync services
|
||||
- site-local broker
|
||||
- site-local database
|
||||
- upward API-based synchronization
|
||||
|
||||
This layer decouples field assets from central services and absorbs plant-specific complexity.
|
||||
|
||||
### 2.3 Central / cloud layer
|
||||
|
||||
The broader stack drawings and `temp/cloud.yml` show a central platform layer with:
|
||||
|
||||
- Gitea
|
||||
- Jenkins
|
||||
- reverse proxy / ingress
|
||||
- Grafana
|
||||
- InfluxDB
|
||||
- Node-RED
|
||||
- RabbitMQ / messaging
|
||||
- VPN / tunnel concepts
|
||||
- Keycloak in the drawing
|
||||
- Portainer in the drawing
|
||||
|
||||
This is a platform-services layer, not just an application runtime.
|
||||
|
||||
## 3. Architecture Decisions From This Review
|
||||
|
||||
These decisions now shape the preferred EVOLV target architecture.
|
||||
|
||||
### 3.1 Local telemetry is mandatory for resilience
|
||||
|
||||
Local InfluxDB is not optional. It is required so that:
|
||||
|
||||
- operations continue when central SCADA or central services are down
|
||||
- local dashboards and advanced digital-twin workflows can still consume recent and relevant process history
|
||||
- local edge/site layers can make smarter decisions without depending on round-trips to central
|
||||
|
||||
### 3.2 Multi-level InfluxDB is part of the architecture
|
||||
|
||||
InfluxDB should exist on multiple levels where it adds operational value:
|
||||
|
||||
- edge/local for resilience and near-real-time replay
|
||||
- site for plant-level history, diagnostics, and resilience
|
||||
- central for fleet-wide analytics, benchmarking, and advisory intelligence
|
||||
|
||||
This is not just copy-paste storage at each level. The design intent is event-driven and selective.
|
||||
|
||||
### 3.3 Storage should be smart, not only deadband-driven
|
||||
|
||||
The target is not simple "store every point" or only a fixed deadband rule such as 1%.
|
||||
|
||||
The desired storage approach is:
|
||||
|
||||
- observe signal slope and change behavior
|
||||
- preserve points where state is changing materially
|
||||
- store fewer points where the signal can be reconstructed downstream with sufficient fidelity
|
||||
- carry enough metadata or conventions so reconstruction quality is auditable
|
||||
|
||||
This implies EVOLV should evolve toward smart storage and signal-aware retention rather than naive event dumping.
|
||||
|
||||
### 3.4 Central is the intelligence and API-entry layer
|
||||
|
||||
Central may advise and coordinate edge/site layers, but external API requests should not hit field-edge systems directly.
|
||||
|
||||
The intended pattern is:
|
||||
|
||||
- external and enterprise integrations terminate centrally
|
||||
- central evaluates, aggregates, authorizes, and advises
|
||||
- site/edge layers receive mediated requests, policies, or setpoints
|
||||
- field-edge remains protected behind an intermediate layer
|
||||
|
||||
This aligns with the stated security direction.
|
||||
|
||||
### 3.5 Configuration source of truth should be database-backed
|
||||
|
||||
The intended configuration authority is the database-backed `tagcodering` model, which already exists but is not yet complete enough to serve as the fully realized source of truth.
|
||||
|
||||
That means the architecture should assume:
|
||||
|
||||
- asset and machine metadata belong in `tagcodering`
|
||||
- Node-RED flows should consume configuration rather than silently becoming the only configuration store
|
||||
- more work is still needed before this behaves as the intended central configuration backbone
|
||||
|
||||
## 4. Visual Model
|
||||
|
||||
### 4.1 Platform topology
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph OT["OT / Field"]
|
||||
PLC["PLC / IO"]
|
||||
DEV["Sensors / Machines"]
|
||||
end
|
||||
|
||||
subgraph EDGE["Edge Layer"]
|
||||
ENR["Edge Node-RED"]
|
||||
EDB["Local InfluxDB"]
|
||||
EUI["Local Grafana / Local Monitoring"]
|
||||
EBR["Optional Local Broker"]
|
||||
end
|
||||
|
||||
subgraph SITE["Site Layer"]
|
||||
SNR["Site Node-RED / CoreSync"]
|
||||
SDB["Site InfluxDB"]
|
||||
SUI["Site Grafana / SCADA Support"]
|
||||
SBR["Site Broker"]
|
||||
end
|
||||
|
||||
subgraph CENTRAL["Central Layer"]
|
||||
API["API / Integration Gateway"]
|
||||
INTEL["Overview Intelligence / Advisory Logic"]
|
||||
CDB["Central InfluxDB"]
|
||||
CGR["Central Grafana"]
|
||||
CFG["Tagcodering Config Model"]
|
||||
GIT["Gitea"]
|
||||
CI["CI/CD"]
|
||||
IAM["IAM / Keycloak"]
|
||||
end
|
||||
|
||||
DEV --> PLC
|
||||
PLC --> ENR
|
||||
ENR --> EDB
|
||||
ENR --> EUI
|
||||
ENR --> EBR
|
||||
ENR <--> SNR
|
||||
EDB <--> SDB
|
||||
SNR --> SDB
|
||||
SNR --> SUI
|
||||
SNR --> SBR
|
||||
SNR <--> API
|
||||
API --> INTEL
|
||||
API <--> CFG
|
||||
SDB <--> CDB
|
||||
INTEL --> SNR
|
||||
CGR --> CDB
|
||||
CI --> GIT
|
||||
IAM --> API
|
||||
IAM --> CGR
|
||||
```
|
||||
|
||||
### 4.2 Command and access boundary
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
EXT["External APIs / Enterprise Requests"] --> API["Central API Gateway"]
|
||||
API --> AUTH["AuthN/AuthZ / Policy Checks"]
|
||||
AUTH --> INTEL["Central Advisory / Decision Support"]
|
||||
INTEL --> SITE["Site Integration Layer"]
|
||||
SITE --> EDGE["Edge Runtime"]
|
||||
EDGE --> PLC["PLC / Field Assets"]
|
||||
|
||||
EXT -. no direct access .-> EDGE
|
||||
EXT -. no direct access .-> PLC
|
||||
```
|
||||
|
||||
### 4.3 Smart telemetry flow
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
RAW["Raw Signal"] --> EDGELOGIC["Edge Signal Evaluation"]
|
||||
EDGELOGIC --> KEEP["Keep Critical Change Points"]
|
||||
EDGELOGIC --> SKIP["Skip Reconstructable Flat Points"]
|
||||
EDGELOGIC --> LOCAL["Local InfluxDB"]
|
||||
LOCAL --> SITE["Site InfluxDB"]
|
||||
SITE --> CENTRAL["Central InfluxDB"]
|
||||
KEEP --> LOCAL
|
||||
SKIP -. reconstruction assumptions / metadata .-> SITE
|
||||
CENTRAL --> DASH["Fleet Dashboards / Analytics"]
|
||||
```
|
||||
|
||||
## 5. Upsides Of This Direction
|
||||
|
||||
### 5.1 Strong separation between control and observability
|
||||
|
||||
Node-RED for runtime/orchestration and InfluxDB/Grafana for telemetry is still the right structural split:
|
||||
|
||||
- control stays close to the process
|
||||
- telemetry storage/querying stays in time-series-native tooling
|
||||
- dashboards do not need to overload Node-RED itself
|
||||
|
||||
### 5.2 Edge-first matches operational reality
|
||||
|
||||
For wastewater/process systems, edge-first remains correct:
|
||||
|
||||
- lower latency
|
||||
- better degraded-mode behavior
|
||||
- less dependence on WAN or central platform uptime
|
||||
- clearer OT trust boundary
|
||||
|
||||
### 5.3 Site mediation improves safety and security
|
||||
|
||||
Using central as the enterprise/API entry point and site as the mediator improves posture:
|
||||
|
||||
- field systems are less exposed
|
||||
- policy decisions can be centralized
|
||||
- external integrations do not probe the edge directly
|
||||
- site can continue operating even when upstream is degraded
|
||||
|
||||
### 5.4 Multi-level storage enables better analytics
|
||||
|
||||
Multiple Influx layers can support:
|
||||
|
||||
- local resilience
|
||||
- site diagnostics
|
||||
- fleet benchmarking
|
||||
- smarter retention and reconstruction strategies
|
||||
|
||||
That is substantially more capable than a single central historian model.
|
||||
|
||||
### 5.5 `tagcodering` is the right long-term direction
|
||||
|
||||
A database-backed configuration authority is stronger than embedding configuration only in flows because it supports:
|
||||
|
||||
- machine metadata management
|
||||
- controlled rollout of configuration changes
|
||||
- clearer versioning and provenance
|
||||
- future API-driven configuration services
|
||||
|
||||
## 6. Downsides And Risks
|
||||
|
||||
### 6.1 Smart storage raises algorithmic and governance complexity
|
||||
|
||||
Signal-aware storage and reconstruction is promising, but it creates architectural obligations:
|
||||
|
||||
- reconstruction rules must be explicit
|
||||
- acceptable reconstruction error must be defined per signal type
|
||||
- operators must know whether they see raw or reconstructed history
|
||||
- compliance-relevant data may need stricter retention than operational convenience data
|
||||
|
||||
Without those rules, smart storage can become opaque and hard to trust.
|
||||
|
||||
### 6.2 Multi-level databases can create ownership confusion
|
||||
|
||||
If edge, site, and central all store telemetry, you must define:
|
||||
|
||||
- which layer is authoritative for which time horizon
|
||||
- when backfill is allowed
|
||||
- when data is summarized vs copied
|
||||
- how duplicates or gaps are detected
|
||||
|
||||
Otherwise operations will argue over which trend is "the real one."
|
||||
|
||||
### 6.3 Central intelligence must remain advisory-first
|
||||
|
||||
Central guidance can become valuable, but direct closed-loop dependency on central would be risky.
|
||||
|
||||
The architecture should therefore preserve:
|
||||
|
||||
- local control authority at edge/site
|
||||
- bounded and explicit central advice
|
||||
- safe behavior if central recommendations stop arriving
|
||||
|
||||
### 6.4 `tagcodering` is not yet complete enough to lean on blindly
|
||||
|
||||
It is the right target, but its current partial state means there is still architecture debt:
|
||||
|
||||
- incomplete config workflows
|
||||
- likely mismatch between desired and implemented schema behavior
|
||||
- temporary duplication between flows, node config, and database-held metadata
|
||||
|
||||
This should be treated as a core platform workstream, not a side issue.
|
||||
|
||||
### 6.5 Broker responsibilities are still not crisp enough
|
||||
|
||||
The materials still reference MQTT/AMQP/RabbitMQ/brokers without one stable responsibility split. That needs to be resolved before large-scale deployment.
|
||||
|
||||
Questions still open:
|
||||
|
||||
- command bus or event bus?
|
||||
- site-only or cross-site?
|
||||
- telemetry transport or only synchronization/eventing?
|
||||
- durability expectations and replay behavior?
|
||||
|
||||
## 7. Security And Regulatory Positioning
|
||||
|
||||
### 7.1 Purdue-style layering is a good fit
|
||||
|
||||
EVOLV's preferred structure aligns well with a Purdue-style OT/IT layering approach:
|
||||
|
||||
- PLCs and field assets stay at the operational edge
|
||||
- edge runtimes stay close to the process
|
||||
- site systems mediate between OT and broader enterprise concerns
|
||||
- central services host APIs, identity, analytics, and engineering workflows
|
||||
|
||||
That is important because it supports segmented trust boundaries instead of direct enterprise-to-field reach-through.
|
||||
|
||||
### 7.2 NIS2 alignment
|
||||
|
||||
Directive (EU) 2022/2555 (NIS2) requires cybersecurity risk-management measures, incident handling, and stronger governance for covered entities.
|
||||
|
||||
This architecture supports that by:
|
||||
|
||||
- limiting direct exposure of field systems
|
||||
- separating operational layers
|
||||
- enabling central policy and oversight
|
||||
- preserving local operation during upstream failure
|
||||
|
||||
### 7.3 CER alignment
|
||||
|
||||
Directive (EU) 2022/2557 (Critical Entities Resilience Directive) focuses on resilience of essential services.
|
||||
|
||||
The edge-plus-site approach supports that direction because:
|
||||
|
||||
- local/site layers can continue during central disruption
|
||||
- essential service continuity does not depend on one central runtime
|
||||
- degraded-mode behavior can be explicitly designed per layer
|
||||
|
||||
### 7.4 Cyber Resilience Act alignment
|
||||
|
||||
Regulation (EU) 2024/2847 (Cyber Resilience Act) creates cybersecurity requirements for products with digital elements.
|
||||
|
||||
For EVOLV, that means the platform should keep strengthening:
|
||||
|
||||
- secure configuration handling
|
||||
- vulnerability and update management
|
||||
- release traceability
|
||||
- lifecycle ownership of components and dependencies
|
||||
|
||||
### 7.5 GDPR alignment where personal data is present
|
||||
|
||||
Regulation (EU) 2016/679 (GDPR) applies whenever EVOLV processes personal data.
|
||||
|
||||
The architecture helps by:
|
||||
|
||||
- centralizing ingress
|
||||
- reducing unnecessary propagation of data to field layers
|
||||
- making access, retention, and audit boundaries easier to define
|
||||
|
||||
### 7.6 What can and cannot be claimed
|
||||
|
||||
The defensible claim is that EVOLV can be deployed in a way that supports compliance with strict European cybersecurity and resilience expectations.
|
||||
|
||||
The non-defensible claim is that EVOLV is automatically compliant purely because of the architecture diagram.
|
||||
|
||||
Actual compliance still depends on implementation and operations, including:
|
||||
|
||||
- access control
|
||||
- patch and vulnerability management
|
||||
- incident response
|
||||
- logging and audit evidence
|
||||
- retention policy
|
||||
- data classification
|
||||
|
||||
## 8. Recommended Ideal Stack
|
||||
|
||||
The ideal EVOLV stack should be layered around operational boundaries, not around tools.
|
||||
|
||||
### 7.1 Layer A: Edge execution
|
||||
|
||||
Purpose:
|
||||
|
||||
- connect to PLCs and field assets
|
||||
- execute time-sensitive local logic
|
||||
- preserve operation during WAN/central loss
|
||||
- provide local telemetry access for resilience and digital-twin use cases
|
||||
|
||||
Recommended components:
|
||||
|
||||
- Node-RED runtime for EVOLV edge flows
|
||||
- OPC UA and protocol adapters
|
||||
- local InfluxDB
|
||||
- optional local Grafana for local engineering/monitoring
|
||||
- optional local broker only when multiple participants need decoupling
|
||||
|
||||
Principle:
|
||||
|
||||
- edge remains safe and useful when disconnected
|
||||
|
||||
### 7.2 Layer B: Site integration
|
||||
|
||||
Purpose:
|
||||
|
||||
- aggregate multiple edge systems at plant/site level
|
||||
- host plant-local dashboards and diagnostics
|
||||
- mediate between raw OT detail and central standardization
|
||||
- serve as the protected step between field systems and central requests
|
||||
|
||||
Recommended components:
|
||||
|
||||
- site Node-RED / CoreSync services
|
||||
- site InfluxDB
|
||||
- site Grafana / SCADA-supporting dashboards
|
||||
- site broker where asynchronous eventing is justified
|
||||
|
||||
Principle:
|
||||
|
||||
- site absorbs plant complexity and protects field assets
|
||||
|
||||
### 7.3 Layer C: Central platform
|
||||
|
||||
Purpose:
|
||||
|
||||
- fleet-wide analytics
|
||||
- shared dashboards
|
||||
- engineering lifecycle
|
||||
- enterprise/API entry point
|
||||
- overview intelligence and advisory logic
|
||||
|
||||
Recommended components:
|
||||
|
||||
- Gitea
|
||||
- CI/CD
|
||||
- central InfluxDB
|
||||
- central Grafana
|
||||
- API/integration gateway
|
||||
- IAM
|
||||
- VPN/private connectivity
|
||||
- `tagcodering`-backed configuration services
|
||||
|
||||
Principle:
|
||||
|
||||
- central coordinates, advises, and governs; it is not the direct field caller
|
||||
|
||||
### 7.4 Cross-cutting platform services
|
||||
|
||||
These should be explicit architecture elements:
|
||||
|
||||
- secrets management
|
||||
- certificate management
|
||||
- backup/restore
|
||||
- audit logging
|
||||
- monitoring/alerting of the platform itself
|
||||
- versioned configuration and schema management
|
||||
- rollout/rollback strategy
|
||||
|
||||
## 9. Recommended Opinionated Choices
|
||||
|
||||
### 8.1 Keep Node-RED as the orchestration layer, not the whole platform
|
||||
|
||||
Node-RED should own:
|
||||
|
||||
- process orchestration
|
||||
- protocol mediation
|
||||
- edge/site logic
|
||||
- KPI production
|
||||
|
||||
It should not become the sole owner of:
|
||||
|
||||
- identity
|
||||
- long-term configuration authority
|
||||
- secret management
|
||||
- compliance/audit authority
|
||||
|
||||
### 8.2 Use InfluxDB by function and horizon
|
||||
|
||||
Recommended split:
|
||||
|
||||
- edge: resilience, local replay, digital-twin input
|
||||
- site: plant diagnostics and local continuity
|
||||
- central: fleet analytics, advisory intelligence, benchmarking, and long-term cross-site views
|
||||
|
||||
### 8.3 Prefer smart telemetry retention over naive point dumping
|
||||
|
||||
Recommended rule:
|
||||
|
||||
- keep information-rich points
|
||||
- reduce information-poor flat spans
|
||||
- document reconstruction assumptions
|
||||
- define signal-class-specific fidelity expectations
|
||||
|
||||
This needs design discipline, but it is a real differentiator if executed well.
|
||||
|
||||
### 8.4 Put enterprise/API ingress at central, not at edge
|
||||
|
||||
This should become a hard architectural rule:
|
||||
|
||||
- external requests land centrally
|
||||
- central authenticates and authorizes
|
||||
- central or site mediates downward
|
||||
- edge never becomes the exposed public integration surface
|
||||
|
||||
### 8.5 Make `tagcodering` the target configuration backbone
|
||||
|
||||
The architecture should be designed so that `tagcodering` can mature into:
|
||||
|
||||
- machine and asset registry
|
||||
- configuration source of truth
|
||||
- site/central configuration exchange point
|
||||
- API-served configuration source for runtime layers
|
||||
|
||||
## 10. Suggested Phasing
|
||||
|
||||
### Phase 1: Stabilize contracts
|
||||
|
||||
- define topic and payload contracts
|
||||
- define telemetry classes and reconstruction policy
|
||||
- define asset, machine, and site identity model
|
||||
- define `tagcodering` scope and schema ownership
|
||||
|
||||
### Phase 2: Harden local/site resilience
|
||||
|
||||
- formalize edge and site runtime patterns
|
||||
- define local telemetry retention and replay behavior
|
||||
- define central-loss behavior
|
||||
- define dashboard behavior during isolation
|
||||
|
||||
### Phase 3: Harden central platform
|
||||
|
||||
- IAM
|
||||
- API gateway
|
||||
- central observability
|
||||
- CI/CD
|
||||
- backup and disaster recovery
|
||||
- config services over `tagcodering`
|
||||
|
||||
### Phase 4: Introduce selective synchronization and intelligence
|
||||
|
||||
- event-driven telemetry propagation rules
|
||||
- smart-storage promotion/backfill policies
|
||||
- advisory services from central
|
||||
- auditability of downward recommendations and configuration changes
|
||||
|
||||
## 11. Immediate Open Questions Before Wiki Finalization
|
||||
|
||||
1. Which signals are allowed to use reconstruction-aware smart storage, and which must remain raw or near-raw for audit/compliance reasons?
|
||||
2. How should `tagcodering` be exposed to runtime layers: direct database access, a dedicated API, or both?
|
||||
3. What exact responsibility split should EVOLV use between API synchronization and broker-based eventing?
|
||||
|
||||
## 12. Recommended Wiki Structure
|
||||
|
||||
The wiki should not be one long page. It should be split into:
|
||||
|
||||
1. platform overview with the main topology diagram
|
||||
2. edge-site-central runtime model
|
||||
3. telemetry and smart storage model
|
||||
4. security and access-boundary model
|
||||
5. configuration architecture centered on `tagcodering`
|
||||
|
||||
## 13. Next Step
|
||||
|
||||
Use this document as the architecture baseline. The companion markdown page in `architecture/` can then be shaped into a wiki-ready visual overview page with Mermaid diagrams and shorter human-readable sections.
|
||||
454
wiki/concepts/generalfunctions-api.md
Normal file
454
wiki/concepts/generalfunctions-api.md
Normal file
@@ -0,0 +1,454 @@
|
||||
---
|
||||
title: generalFunctions API Reference
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [api, generalFunctions, reference]
|
||||
---
|
||||
|
||||
# generalFunctions API Reference
|
||||
|
||||
Shared library (`nodes/generalFunctions/`) used across all EVOLV Node-RED nodes.
|
||||
|
||||
```js
|
||||
const { logger, outputUtils, MeasurementContainer, ... } = require('generalFunctions');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Logger](#logger)
|
||||
2. [OutputUtils](#outpututils)
|
||||
3. [ValidationUtils](#validationutils)
|
||||
4. [MeasurementContainer](#measurementcontainer)
|
||||
5. [ConfigManager](#configmanager)
|
||||
6. [ChildRegistrationUtils](#childregistrationutils)
|
||||
7. [MenuUtils](#menuutils)
|
||||
8. [EndpointUtils](#endpointutils)
|
||||
9. [Positions](#positions)
|
||||
10. [AssetLoader / loadCurve](#assetloader--loadcurve)
|
||||
|
||||
---
|
||||
|
||||
## Logger
|
||||
|
||||
Structured, level-filtered console logger.
|
||||
|
||||
**File:** `src/helper/logger.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new Logger(logging = true, logLevel = 'debug', nameModule = 'N/A')
|
||||
```
|
||||
|
||||
| Param | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `logging` | `boolean` | `true` | Enable/disable all output |
|
||||
| `logLevel` | `string` | `'debug'` | Minimum severity: `'debug'` \| `'info'` \| `'warn'` \| `'error'` |
|
||||
| `nameModule` | `string` | `'N/A'` | Label prefixed to every message |
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Description |
|
||||
|---|---|---|
|
||||
| `debug` | `(message: string): void` | Log at DEBUG level |
|
||||
| `info` | `(message: string): void` | Log at INFO level |
|
||||
| `warn` | `(message: string): void` | Log at WARN level |
|
||||
| `error` | `(message: string): void` | Log at ERROR level |
|
||||
| `setLogLevel` | `(level: string): void` | Change minimum level at runtime |
|
||||
| `toggleLogging` | `(): void` | Flip logging on/off |
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const Logger = require('generalFunctions').logger;
|
||||
const log = new Logger(true, 'info', 'MyNode');
|
||||
log.info('Node started'); // [INFO] -> MyNode: Node started
|
||||
log.debug('ignored'); // silent (below 'info')
|
||||
log.setLogLevel('debug');
|
||||
log.debug('now visible'); // [DEBUG] -> MyNode: now visible
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## OutputUtils
|
||||
|
||||
Tracks output state and formats messages for InfluxDB or process outputs. Only emits changed fields.
|
||||
|
||||
**File:** `src/helper/outputUtils.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new OutputUtils() // no parameters
|
||||
```
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `formatMsg` | `(output, config, format)` | `object \| undefined` | Diff against last output; returns formatted msg or `undefined` if nothing changed |
|
||||
| `checkForChanges` | `(output, format)` | `object` | Returns only the key/value pairs that changed since last call |
|
||||
|
||||
**`format`** must be `'influxdb'` or `'process'`.
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const out = new OutputUtils();
|
||||
const msg = out.formatMsg(
|
||||
{ temperature: 22.5, pressure: 1013 },
|
||||
config,
|
||||
'influxdb'
|
||||
);
|
||||
// msg = { topic: 'nodeName', payload: { measurement, fields, tags, timestamp } }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ValidationUtils
|
||||
|
||||
Schema-driven config validation with type coercion, range clamping, and nested object support.
|
||||
|
||||
**File:** `src/helper/validationUtils.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new ValidationUtils(loggerEnabled = true, loggerLevel = 'warn')
|
||||
```
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `validateSchema` | `(config, schema, name)` | `object` | Walk the schema, validate every field, return a clean config. Unknown keys are stripped. Missing keys get their schema default. |
|
||||
| `constrain` | `(value, min, max)` | `number` | Clamp a numeric value to `[min, max]` |
|
||||
| `removeUnwantedKeys` | `(obj)` | `object` | Strip `rules`/`description` metadata, collapse `default` values |
|
||||
|
||||
**Supported `rules.type` values:** `number`, `integer`, `boolean`, `string`, `enum`, `array`, `set`, `object`, `curve`, `machineCurve`.
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const ValidationUtils = require('generalFunctions').validation;
|
||||
const v = new ValidationUtils(true, 'warn');
|
||||
|
||||
const schema = {
|
||||
temperature: { default: 20, rules: { type: 'number', min: -40, max: 100 } },
|
||||
unit: { default: 'C', rules: { type: 'enum', values: [{ value: 'C' }, { value: 'F' }] } }
|
||||
};
|
||||
|
||||
const validated = v.validateSchema({ temperature: 999 }, schema, 'myNode');
|
||||
// validated.temperature === 100 (clamped)
|
||||
// validated.unit === 'C' (default applied)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MeasurementContainer
|
||||
|
||||
Chainable measurement storage organised by **type / variant / position**. Supports auto unit conversion, windowed statistics, events, and positional difference calculations.
|
||||
|
||||
**File:** `src/measurements/MeasurementContainer.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new MeasurementContainer(options = {}, logger)
|
||||
```
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `windowSize` | `number` | `10` | Rolling window for statistics |
|
||||
| `defaultUnits` | `object` | `{ pressure:'mbar', flow:'m3/h', ... }` | Default unit per measurement type |
|
||||
| `autoConvert` | `boolean` | `true` | Auto-convert values to target unit |
|
||||
| `preferredUnits` | `object` | `{}` | Per-type unit overrides |
|
||||
|
||||
### Chainable Setters
|
||||
|
||||
All return `this` for chaining.
|
||||
|
||||
```js
|
||||
container
|
||||
.type('pressure')
|
||||
.variant('static')
|
||||
.position('upstream')
|
||||
.distance(5)
|
||||
.unit('bar')
|
||||
.value(3.2, Date.now(), 'bar');
|
||||
```
|
||||
|
||||
| Method | Signature | Description |
|
||||
|---|---|---|
|
||||
| `type` | `(typeName): this` | Set measurement type (e.g. `'pressure'`) |
|
||||
| `variant` | `(variantName): this` | Set variant (e.g. `'static'`, `'differential'`) |
|
||||
| `position` | `(positionValue): this` | Set position (e.g. `'upstream'`, `'downstream'`) |
|
||||
| `distance` | `(distance): this` | Set physical distance from parent |
|
||||
| `unit` | `(unitName): this` | Set unit on the underlying measurement |
|
||||
| `value` | `(val, timestamp?, sourceUnit?): this` | Store a value; auto-converts if `sourceUnit` differs from target |
|
||||
|
||||
### Terminal / Query Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `get` | `()` | `Measurement \| null` | Get the raw measurement object |
|
||||
| `getCurrentValue` | `(requestedUnit?)` | `number \| null` | Latest value, optionally converted |
|
||||
| `getAverage` | `(requestedUnit?)` | `number \| null` | Windowed average |
|
||||
| `getMin` | `()` | `number \| null` | Window minimum |
|
||||
| `getMax` | `()` | `number \| null` | Window maximum |
|
||||
| `getAllValues` | `()` | `array \| null` | All stored samples |
|
||||
| `getLaggedValue` | `(lag?, requestedUnit?)` | `number \| null` | Value from `lag` samples ago |
|
||||
| `getLaggedSample` | `(lag?, requestedUnit?)` | `object \| null` | Full sample `{ value, timestamp, unit }` from `lag` samples ago |
|
||||
| `exists` | `({ type?, variant?, position?, requireValues? })` | `boolean` | Check if a measurement series exists |
|
||||
| `difference` | `({ from?, to?, unit? })` | `object \| null` | Compute `{ value, avgDiff, unit }` between two positions |
|
||||
|
||||
### Introspection / Lifecycle
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `getTypes` | `()` | `string[]` | All registered measurement types |
|
||||
| `getVariants` | `()` | `string[]` | Variants under current type |
|
||||
| `getPositions` | `()` | `string[]` | Positions under current type+variant |
|
||||
| `getAvailableUnits` | `(measurementType?)` | `string[]` | Units available for a type |
|
||||
| `getBestUnit` | `(excludeUnits?)` | `object \| null` | Best human-readable unit for current value |
|
||||
| `setPreferredUnit` | `(type, unit)` | `this` | Override default unit for a type |
|
||||
| `setChildId` | `(id)` | `this` | Tag container with a child node ID |
|
||||
| `setChildName` | `(name)` | `this` | Tag container with a child node name |
|
||||
| `setParentRef` | `(parent)` | `this` | Store reference to parent node |
|
||||
| `clear` | `()` | `void` | Reset all measurements and chain state |
|
||||
|
||||
### Events
|
||||
|
||||
The internal `emitter` fires `"type.variant.position"` on every `value()` call with:
|
||||
|
||||
```js
|
||||
{ value, originalValue, unit, sourceUnit, timestamp, position, distance, variant, type, childId, childName, parentRef }
|
||||
```
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const { MeasurementContainer } = require('generalFunctions');
|
||||
const mc = new MeasurementContainer({ windowSize: 5 });
|
||||
|
||||
mc.type('pressure').variant('static').position('upstream').value(3.2);
|
||||
mc.type('pressure').variant('static').position('downstream').value(2.8);
|
||||
|
||||
const diff = mc.type('pressure').variant('static').difference();
|
||||
// diff = { value: -0.4, avgDiff: -0.4, unit: 'mbar', from: 'downstream', to: 'upstream' }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ConfigManager
|
||||
|
||||
Loads JSON config files from disk and builds merged runtime configs.
|
||||
|
||||
**File:** `src/configs/index.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new ConfigManager(relPath = '.')
|
||||
```
|
||||
|
||||
`relPath` is resolved relative to the configs directory.
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `getConfig` | `(configName)` | `object` | Load and parse `<configName>.json` |
|
||||
| `getAvailableConfigs` | `()` | `string[]` | List config names (without `.json`) |
|
||||
| `hasConfig` | `(configName)` | `boolean` | Check existence |
|
||||
| `getBaseConfig` | `()` | `object` | Shortcut for `getConfig('baseConfig')` |
|
||||
| `buildConfig` | `(nodeName, uiConfig, nodeId, domainConfig?)` | `object` | Merge base schema + UI overrides into a runtime config |
|
||||
| `createEndpoint` | `(nodeName)` | `string` | Generate browser JS that injects config into `window.EVOLV.nodes` |
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const { configManager } = require('generalFunctions');
|
||||
const cfg = configManager.buildConfig('measurement', uiConfig, node.id, {
|
||||
scaling: { enabled: true, inputMin: 0, inputMax: 100 }
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ChildRegistrationUtils
|
||||
|
||||
Manages parent-child node relationships: registration, lookup, and structure storage.
|
||||
|
||||
**File:** `src/helper/childRegistrationUtils.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new ChildRegistrationUtils(mainClass)
|
||||
```
|
||||
|
||||
`mainClass` is the parent node instance (must expose `.logger` and optionally `.registerChild()`).
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `registerChild` | `(child, positionVsParent, distance?)` | `Promise<any>` | Register a child node under the parent. Sets up parent refs, measurement context, and stores by softwareType/category. |
|
||||
| `getChildrenOfType` | `(softwareType, category?)` | `array` | Get children filtered by software type and optional category |
|
||||
| `getChildById` | `(childId)` | `object \| null` | Lookup a single child by its ID |
|
||||
| `getAllChildren` | `()` | `array` | All registered children |
|
||||
| `logChildStructure` | `()` | `void` | Debug-print the full child tree |
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const { childRegistrationUtils: CRU } = require('generalFunctions');
|
||||
const cru = new CRU(parentNode);
|
||||
await cru.registerChild(sensorNode, 'upstream');
|
||||
cru.getChildrenOfType('measurement'); // [sensorNode]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MenuUtils
|
||||
|
||||
Browser-side UI helper for Node-RED editor. Methods are mixed in from separate modules: toggles, data fetching, URL utils, dropdown population, and HTML generation.
|
||||
|
||||
**File:** `src/helper/menuUtils.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new MenuUtils() // no parameters; sets isCloud=false, configData=null
|
||||
```
|
||||
|
||||
### Key Methods
|
||||
|
||||
**Toggles** -- control UI element visibility:
|
||||
|
||||
| Method | Signature | Description |
|
||||
|---|---|---|
|
||||
| `initBasicToggles` | `(elements)` | Bind log-level row visibility to log checkbox |
|
||||
| `initMeasurementToggles` | `(elements)` | Bind scaling input rows to scaling checkbox |
|
||||
| `initTensionToggles` | `(elements, node)` | Show/hide tension row based on interpolation method |
|
||||
|
||||
**Data Fetching:**
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `fetchData` | `(url, fallbackUrl)` | `Promise<array>` | Fetch JSON from primary URL; fall back on failure |
|
||||
| `fetchProjectData` | `(url)` | `Promise<object>` | Fetch project-level data |
|
||||
| `apiCall` | `(node)` | `Promise<object>` | POST to asset-register API |
|
||||
|
||||
**URL Construction:**
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `getSpecificConfigUrl` | `(nodeName, cloudAPI)` | `{ cloudConfigURL, localConfigURL }` | Build cloud + local config URLs |
|
||||
| `constructUrl` | `(base, ...paths)` | `string` | Join URL segments safely |
|
||||
| `constructCloudURL` | `(base, ...paths)` | `string` | Same as `constructUrl`, for cloud endpoints |
|
||||
|
||||
**Dropdown Population:**
|
||||
|
||||
| Method | Signature | Description |
|
||||
|---|---|---|
|
||||
| `fetchAndPopulateDropdowns` | `(configUrls, elements, node)` | Cascading supplier > subType > model > unit dropdowns |
|
||||
| `populateDropdown` | `(htmlElement, options, node, property, callback?)` | Fill a `<select>` with options and wire change events |
|
||||
| `populateLogLevelOptions` | `(logLevelSelect, configData, node)` | Populate log-level dropdown from config |
|
||||
| `populateSmoothingMethods` | `(configUrls, elements, node)` | Populate smoothing method dropdown |
|
||||
| `populateInterpolationMethods` | `(configUrls, elements, node)` | Populate interpolation method dropdown |
|
||||
| `generateHtml` | `(htmlElement, options, savedValue)` | Write `<option>` HTML into an element |
|
||||
|
||||
---
|
||||
|
||||
## EndpointUtils
|
||||
|
||||
Server-side helper that serves `MenuUtils` as browser JavaScript via Node-RED HTTP endpoints.
|
||||
|
||||
**File:** `src/helper/endpointUtils.js`
|
||||
|
||||
### Constructor
|
||||
|
||||
```js
|
||||
new EndpointUtils({ MenuUtilsClass? })
|
||||
```
|
||||
|
||||
| Param | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `MenuUtilsClass` | `class` | `MenuUtils` | The MenuUtils constructor to introspect |
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `createMenuUtilsEndpoint` | `(RED, nodeName, customHelpers?)` | `void` | Register `GET /<nodeName>/resources/menuUtils.js` |
|
||||
| `generateMenuUtilsCode` | `(nodeName, customHelpers?)` | `string` | Produce the browser JS string (introspects `MenuUtils.prototype`) |
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const EndpointUtils = require('generalFunctions/src/helper/endpointUtils');
|
||||
const ep = new EndpointUtils();
|
||||
ep.createMenuUtilsEndpoint(RED, 'valve');
|
||||
// Browser can now load: GET /valve/resources/menuUtils.js
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Positions
|
||||
|
||||
Canonical constants for parent-child spatial relationships.
|
||||
|
||||
**File:** `src/constants/positions.js`
|
||||
|
||||
### Exports
|
||||
|
||||
```js
|
||||
const { POSITIONS, POSITION_VALUES, isValidPosition } = require('generalFunctions');
|
||||
```
|
||||
|
||||
| Export | Type | Value |
|
||||
|---|---|---|
|
||||
| `POSITIONS` | `object` | `{ UPSTREAM: 'upstream', DOWNSTREAM: 'downstream', AT_EQUIPMENT: 'atEquipment', DELTA: 'delta' }` |
|
||||
| `POSITION_VALUES` | `string[]` | `['upstream', 'downstream', 'atEquipment', 'delta']` |
|
||||
| `isValidPosition` | `(pos: string): boolean` | Returns `true` if `pos` is one of the four values |
|
||||
|
||||
---
|
||||
|
||||
## AssetLoader / loadCurve
|
||||
|
||||
Loads JSON asset files (machine curves, etc.) from the datasets directory with LRU caching.
|
||||
|
||||
**File:** `datasets/assetData/curves/index.js`
|
||||
|
||||
### Singleton convenience functions
|
||||
|
||||
```js
|
||||
const { loadCurve } = require('generalFunctions');
|
||||
```
|
||||
|
||||
| Function | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `loadCurve` | `(curveType: string)` | `object \| null` | Load `<curveType>.json` from the curves directory |
|
||||
| `loadAsset` | `(datasetType, assetId)` | `object \| null` | Load any JSON asset by dataset folder and ID |
|
||||
| `getAvailableAssets` | `(datasetType)` | `string[]` | List asset IDs in a dataset folder |
|
||||
|
||||
### AssetLoader class
|
||||
|
||||
```js
|
||||
new AssetLoader(maxCacheSize = 100)
|
||||
```
|
||||
|
||||
Same methods as above (`loadCurve`, `loadAsset`, `getAvailableAssets`), plus `clearCache()`.
|
||||
|
||||
### Example
|
||||
|
||||
```js
|
||||
const { loadCurve } = require('generalFunctions');
|
||||
const curve = loadCurve('hidrostal-H05K-S03R');
|
||||
// curve = { flow: [...], head: [...], ... } or null
|
||||
```
|
||||
88
wiki/findings/open-issues-2026-03.md
Normal file
88
wiki/findings/open-issues-2026-03.md
Normal file
@@ -0,0 +1,88 @@
|
||||
---
|
||||
title: Open Issues — EVOLV Codebase
|
||||
created: 2026-03-01
|
||||
updated: 2026-04-07
|
||||
status: evolving
|
||||
tags: [issues, backlog]
|
||||
---
|
||||
|
||||
# Open Issues — EVOLV Codebase
|
||||
|
||||
Issues identified during codebase scan (2026-03-12). Create these on Gitea when ready.
|
||||
|
||||
---
|
||||
|
||||
## Issue 1: Restore diffuser node implementation
|
||||
|
||||
**Labels:** `enhancement`, `node`
|
||||
**Priority:** Medium
|
||||
|
||||
The `nodes/diffuser/` directory contains only `.git`, `LICENSE`, and `README.md` — no implementation. There was a previous experimental version. Needs:
|
||||
|
||||
- Retrieve original diffuser logic from user/backup
|
||||
- Rebuild to current three-layer architecture (wrapper `.js` + `src/nodeClass.js` + `src/specificClass.js`)
|
||||
- Use `require('generalFunctions')` barrel imports
|
||||
- Add config JSON in `generalFunctions/src/configs/diffuser.json`
|
||||
- Register under category `'EVOLV'` with appropriate S88 color
|
||||
- Add tests
|
||||
|
||||
**Blocked on:** User providing original diffuser logic/requirements.
|
||||
|
||||
---
|
||||
|
||||
## Issue 2: Relocate prediction/ML modules to external service
|
||||
|
||||
**Labels:** `enhancement`, `architecture`
|
||||
**Priority:** Medium
|
||||
|
||||
TensorFlow-based influent prediction code was removed from monster node (was broken/incomplete). The prediction functionality needs a new home:
|
||||
|
||||
- LSTM model for 24-hour flow prediction based on precipitation data
|
||||
- Standardization constants: hours `(mean=11.504, std=6.922)`, precipitation `(mean=0.090, std=0.439)`, response `(mean=1188.01, std=1024.19)`
|
||||
- Model was served from `http://127.0.0.1:1880/generalFunctions/datasets/lstmData/tfjs_model/`
|
||||
- Consider: separate microservice, Python-based inference, or ONNX runtime
|
||||
- Monster node should accept predictions via `model_prediction` message topic from external service
|
||||
|
||||
**Related files removed:** `monster_class.js` methods `get_model_prediction()`, `model_loader()`
|
||||
|
||||
---
|
||||
|
||||
## Issue 3: Modernize monster node to three-layer architecture
|
||||
|
||||
**Labels:** `refactor`, `node`
|
||||
**Priority:** Low
|
||||
|
||||
Monster node uses old-style structure (`dependencies/monster/` instead of `src/`). Should be refactored:
|
||||
|
||||
- Move `dependencies/monster/monster_class.js` → `src/specificClass.js`
|
||||
- Create `src/nodeClass.js` adapter (extract from `monster.js`)
|
||||
- Slim down `monster.js` to standard wrapper pattern
|
||||
- Move `monsterConfig.json` → `generalFunctions/src/configs/monster.json`
|
||||
- Remove `modelLoader.js` (TF dependency removed)
|
||||
- Add unit tests
|
||||
|
||||
**Note:** monster_class.js is ~500 lines of domain logic. Keep sampling_program(), aggregation, AQUON integration intact.
|
||||
|
||||
---
|
||||
|
||||
## Issue 4: Clean up inline test/demo code in specificClass files
|
||||
|
||||
**Labels:** `cleanup`
|
||||
**Priority:** Low
|
||||
|
||||
Several specificClass files have test/demo code after `module.exports`:
|
||||
|
||||
- `pumpingStation/src/specificClass.js` (lines 478-697): Demo code guarded with `require.main === module` — acceptable but could move to `test/` or `examples/`
|
||||
- `machineGroupControl/src/specificClass.js` (lines 969-1158): Block-commented test code with `makeMachines()` — dead code, could be removed or moved to test file
|
||||
|
||||
---
|
||||
|
||||
## Issue 5: DashboardAPI node improvements
|
||||
|
||||
**Labels:** `enhancement`, `security`
|
||||
**Priority:** Low
|
||||
|
||||
- Bearer token now relies on `GRAFANA_TOKEN` env var (hardcoded token was removed for security)
|
||||
- Ensure deployment docs mention setting `GRAFANA_TOKEN`
|
||||
- `dashboardapi_class.js` still has `console.log` calls (lines 154, 178) — should use logger
|
||||
- Node doesn't follow three-layer architecture (older style)
|
||||
@@ -11,23 +11,38 @@ updated: 2026-04-07
|
||||
- [Knowledge Graph](knowledge-graph.yaml) — structured data, machine-queryable
|
||||
|
||||
## Architecture
|
||||
- [Node Architecture](architecture/node-architecture.md) — three-layer pattern, ports, mermaid diagrams
|
||||
- [3D Pump Curves](architecture/3d-pump-curves.md) — predict class, spline interpolation, unit chain
|
||||
- [Group Optimization](architecture/group-optimization.md) — BEP-Gravitation, combination selection, marginal-cost refinement
|
||||
- [Platform Overview](architecture/platform-overview.md) — edge/site/central layering, telemetry model
|
||||
- [Deployment Blueprint](architecture/deployment-blueprint.md) — Docker topology, rollout order
|
||||
- [Stack Review](architecture/stack-review.md) — full stack architecture assessment
|
||||
|
||||
## Core Concepts
|
||||
- [generalFunctions API](concepts/generalfunctions-api.md) — logger, MeasurementContainer, configManager, etc.
|
||||
|
||||
## Findings
|
||||
- [BEP-Gravitation Proof](findings/bep-gravitation-proof.md) — within 0.1% of brute-force optimum (proven)
|
||||
- [NCog Behavior](findings/ncog-behavior.md) — when NCog works, when it's zero, how it's used (evolving)
|
||||
- [Curve Non-Convexity](findings/curve-non-convexity.md) — C5 sparse data artifacts (proven)
|
||||
- [Pump Switching Stability](findings/pump-switching-stability.md) — 1-2 transitions, no hysteresis (proven)
|
||||
- [Open Issues (2026-03)](findings/open-issues-2026-03.md) — diffuser, monster refactor, ML relocation, etc.
|
||||
|
||||
## Sessions
|
||||
- [2026-04-07: Production Hardening](sessions/2026-04-07-production-hardening.md) — rotatingMachine + machineGroupControl
|
||||
|
||||
## Other Documentation (outside wiki)
|
||||
- `CLAUDE.md` — Claude Code project guide (root)
|
||||
- `AGENTS.md` — agent routing table, orchestrator policy (root, used by `.claude/agents/`)
|
||||
- `.agents/` — skills, decisions, function-anchors, improvements
|
||||
- `.claude/` — Claude Code agents and rules
|
||||
- `manuals/node-red/` — FlowFuse dashboard and Node-RED reference docs
|
||||
|
||||
## Not Yet Documented
|
||||
- Parent-child registration protocol (Port 2 handshake)
|
||||
- Prediction health scoring algorithm (confidence 0-1)
|
||||
- MeasurementContainer internals (chainable API, delta compression)
|
||||
- PID controller implementation
|
||||
- reactor / settler / monster / measurement nodes
|
||||
- reactor / settler / monster / measurement / valve nodes
|
||||
- pumpingStation node (uses rotatingMachine children)
|
||||
- InfluxDB telemetry format (Port 1)
|
||||
|
||||
Reference in New Issue
Block a user