Compare commits

25 Commits

Author SHA1 Message Date
znetsixe
4336002b77 fix: update submodule refs with bug fixes for validateSchema recursion and nodeClass syntax
Some checks failed
CI / lint-and-test (push) Has been cancelled
- generalFunctions: fix infinite recursion in validateSchema when version string is in schema
- rotatingMachine: fix missing closing brace in emergencystop case block

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 08:46:34 +02:00
znetsixe
f57343f5e3 Update submodule refs after merge with main
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 18:29:31 +02:00
znetsixe
65ceb696ab Merge remote-tracking branch 'origin/main' into dev-rene
# Conflicts:
#	.dockerignore
#	.gitmodules
#	Dockerfile
#	docker-compose.yml
#	nodes/generalFunctions
#	nodes/machineGroupControl
#	nodes/measurement
#	nodes/monster
#	nodes/pumpingStation
#	nodes/reactor
#	nodes/rotatingMachine
#	nodes/settler
#	package-lock.json
#	package.json
2026-03-31 18:29:03 +02:00
root
91a298960c Prepare reactor, diffuser, and settler updates for mainline merge 2026-03-31 14:26:33 +02:00
Rene De Ren
35221fc5dd Sync pushed submodule refs
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-03-12 16:47:08 +01:00
Rene De Ren
93a5b6a90e Expose output format controls across node editors 2026-03-12 16:39:54 +01:00
Rene De Ren
1d98670706 Validate diffuser through the full stack 2026-03-12 16:32:25 +01:00
Rene De Ren
a432eea7fe Track config-driven output formatting support 2026-03-12 16:13:47 +01:00
Rene De Ren
9cb3657bae Track dashboardapi buildConfig adoption 2026-03-12 16:11:10 +01:00
Rene De Ren
bd9432eebb Validate dashboardapi round-trip through Node-RED 2026-03-12 11:40:37 +01:00
Rene De Ren
c9bacb64c8 Add monster coverage and stack validation 2026-03-12 10:32:09 +01:00
Rene De Ren
e580c93c84 docs: add open issues from codebase scan
Tracked issues for diffuser restoration, ML module relocation,
monster architecture modernization, test code cleanup, and dashboardAPI improvements.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 09:34:51 +01:00
Rene De Ren
b02306c42f fix: codebase scan — bug fixes, security, logging consistency, monster modernization
- generalFunctions: add missing migrateConfig(), config versioning, formatters module
- rotatingMachine: fix eneableLog typo, correct child registration ID
- machineGroupControl: console.log → structured logger
- settler/reactor: console.log → logger, throw on unknown reactor type
- monster: modernize imports to require('generalFunctions'), remove broken
  TensorFlow code, add childRegistrationUtils, consolidate input handlers
- dashboardAPI: remove hardcoded Grafana bearer token, use logger

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 09:33:52 +01:00
Rene De Ren
2c76430394 feat: working E2E container stack with Node-RED + InfluxDB + Grafana
- Fix Dockerfile.e2e to install EVOLV properly in Node-RED /data/
- Add measurement node E2E test flow with scaling (4-20mA to 0-5m)
- Add Grafana health check to run-e2e.sh
- Guard pumpingStation demo IIFE with require.main check
- All 10 EVOLV nodes load successfully in containerized Node-RED

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 16:38:14 +01:00
Rene De Ren
49ebd833db feat: add node tests, integration tests, API reference, fix pumpingStation bug
- Add 127 unit tests for measurement, pumpingStation, reactor, settler specificClass
- Add 32 integration tests for parent-child registration flows
- Fix pumpingStation tick() calling non-existent _calcTimeRemaining (was _calcRemainingTime)
- Add API reference documentation for all generalFunctions modules

Total tests: 536 (389 Jest + 23 node:test + 124 legacy), all passing

Closes #17, #19, #20

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 16:32:04 +01:00
Rene De Ren
905a061590 feat: architecture refactor — validators, positions, menuUtils, ESLint, tests, CI
Major improvements across the codebase:

- Extract validationUtils.js (548→217 lines) into strategy pattern validators
- Extract menuUtils.js (543→35 lines) into 6 focused menu modules
- Adopt POSITIONS constants across 23 files (183 replacements)
- Eliminate all 71 ESLint warnings (0 errors, 0 warnings)
- Add 158 unit tests for ConfigManager, MeasurementContainer, ValidationUtils
- Add architecture documentation with Mermaid diagrams
- Add CI pipeline (Docker, ESLint, Jest, Makefile)
- Add E2E infrastructure (docker-compose.e2e.yml)

Test results: 377 total (230 Jest + 23 node:test + 124 legacy), all passing
Lint: 0 errors, 0 warnings

Closes #2, #3, #9, #13, #14, #18

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 15:37:20 +01:00
p.vanderwilt
80de324b32 Update reactor submodule to latest commit 2025-11-28 11:53:57 +01:00
p.vanderwilt
c8d5ea0fce Update submodule commits for reactor and rotatingMachine 2025-11-21 14:49:50 +01:00
p.vanderwilt
b871b23c24 Remove TensorFlow dependencies from package.json 2025-11-21 11:56:59 +01:00
p.vanderwilt
91b681a74d Add additional sensors 2025-11-12 10:48:38 +01:00
p.vanderwilt
76d2008e52 Update submodule commits and package-lock.json dependencies 2025-11-12 10:30:51 +01:00
p.vanderwilt
3c304f14e5 Update submodules for recirculation implementation 2025-11-06 15:03:43 +01:00
p.vanderwilt
24c443840b Add settler to package.json 2025-10-31 12:14:58 +01:00
p.vanderwilt
c4c8629c01 add settler submodule 2025-10-31 12:11:50 +01:00
609c72cedc Merge pull request 'dev-Rene' (#5) from dev-Rene into main
Reviewed-on: https://gitea.centraal.wbd-rd.nl/RnD/EVOLV/pulls/5
2025-10-24 19:25:04 +00:00
28 changed files with 7387 additions and 3115 deletions

View File

@@ -0,0 +1,43 @@
## Context
The single demo bioreactor did not reflect the intended EVOLV biological treatment concept. The owner requested:
- four reactor zones in series
- staged aeration based on effluent NH4
- local visualization per zone for NH4, NO3, O2, and other relevant state variables
- improved PFR numerical stability by increasing reactor resolution
The localhost deployment also needed to remain usable for E2E debugging with Node-RED, InfluxDB, and Grafana.
## Options Considered
1. Keep one large PFR and add more internal profile visualization only.
2. Split the biology into four explicit reactor zones in the flow and control aeration at zone level.
3. Replace the PFR demo with a simpler CSTR train for faster visual response.
## Decision
Choose option 2.
The demo flow now uses four explicit PFR zones in series with:
- equal-zone sizing (`4 x 500 m3`, total `2000 m3`)
- explicit `Fluent` forwarding between zones
- common clocking for all zones
- external `OTR` control instead of fixed `kla`
- staged NH4-based aeration escalation with 30-minute hold logic
- per-zone telemetry to InfluxDB and Node-RED dashboard charts
For runtime stability on localhost, the demo uses a higher spatial resolution with moderate compute load rather than the earlier single-reactor setup.
## Consequences
- The flow is easier to reason about operationally because each aeration zone is explicit.
- Zone-level telemetry is available for dashboarding and debugging.
- PFR outlet response remains residence-time dependent, so zone outlet composition will not change instantly after startup or inflow changes.
- Grafana datasource query round-trip remains valid, but dashboard auto-generation still needs separate follow-up if strict dashboard creation is required in E2E checks.
## Rollback / Migration Notes
- Rolling back to the earlier demo means restoring the single `demo_reactor` topology in `docker/demo-flow.json`.
- Existing E2E checks and dashboards should prefer the explicit zone measurements (`reactor_demo_reactor_z1` ... `reactor_demo_reactor_z4`) going forward.

41
.gitea/workflows/ci.yml Normal file
View File

@@ -0,0 +1,41 @@
name: CI
on:
push:
branches: [main, develop, dev-Rene]
pull_request:
branches: [main]
jobs:
lint-and-test:
runs-on: ubuntu-latest
container:
image: node:20-slim
steps:
- name: Install git
run: apt-get update -qq && apt-get install -y -qq git
- name: Checkout with submodules
uses: actions/checkout@v4
with:
submodules: recursive
- name: Rewrite generalFunctions to local path
run: |
sed -i 's|"generalFunctions": "git+https://[^"]*"|"generalFunctions": "file:./nodes/generalFunctions"|' package.json
- name: Install dependencies
run: npm install --ignore-scripts
- name: Lint
run: npm run lint
- name: Test (Jest)
run: npm test
- name: Test (node:test)
run: npm run test:node
- name: Test (legacy)
run: npm run test:legacy

32
CLAUDE.md Normal file
View File

@@ -0,0 +1,32 @@
# EVOLV - Claude Code Project Guide
## What This Is
Node-RED custom nodes package for wastewater treatment plant automation. Developed by Waterschap Brabantse Delta R&D team. Follows ISA-88 (S88) batch control standard.
## Architecture
Each node follows a three-layer pattern:
1. **Node-RED wrapper** (`<name>.js`) - registers the node type, sets up HTTP endpoints
2. **Node adapter** (`src/nodeClass.js`) - bridges Node-RED API with domain logic, handles config loading, tick loops, events
3. **Domain logic** (`src/specificClass.js`) - pure business logic, no Node-RED dependencies
## Key Shared Library: `nodes/generalFunctions/`
- `logger` - structured logging (use this, NOT console.log)
- `MeasurementContainer` - chainable measurement storage (type/variant/position)
- `configManager` - loads JSON configs from `src/configs/`
- `MenuManager` - dynamic UI dropdowns
- `outputUtils` - formats messages for InfluxDB and process outputs
- `childRegistrationUtils` - parent-child node relationships
- `coolprop` - thermodynamic property calculations
## Conventions
- Nodes register under category `'EVOLV'` in Node-RED
- S88 color scheme: Area=#0f52a5, ProcessCell=#0c99d9, Unit=#50a8d9, Equipment=#86bbdd, ControlModule=#a9daee
- Config JSON files in `generalFunctions/src/configs/` define defaults, types, enums per node
- Tick loop runs at 1000ms intervals for time-based updates
- Three outputs per node: [process, dbase, parent]
## Development Notes
- No build step required - pure Node.js
- Install: `npm install` in root
- Submodule URLs were rewritten from `gitea.centraal.wbd-rd.nl` to `gitea.wbd-rd.nl` for external access
- Dependencies: mathjs, generalFunctions (git submodule)

29
Dockerfile.e2e Normal file
View File

@@ -0,0 +1,29 @@
FROM nodered/node-red:latest
# Switch to root for setup
USER root
# Copy EVOLV directly into where Node-RED looks for custom nodes
COPY package.json /data/node_modules/EVOLV/package.json
COPY nodes/ /data/node_modules/EVOLV/nodes/
# Rewrite generalFunctions dependency to local file path (no-op if already local)
RUN sed -i 's|"generalFunctions": "git+https://[^"]*"|"generalFunctions": "file:./nodes/generalFunctions"|' \
/data/node_modules/EVOLV/package.json
# Fix ownership for node-red user
RUN chown -R node-red:root /data
USER node-red
# Install EVOLV's own dependencies inside the EVOLV package directory
WORKDIR /data/node_modules/EVOLV
RUN npm install --ignore-scripts --production
# Copy test flows into Node-RED data directory
COPY --chown=node-red:root test/e2e/flows.json /data/flows.json
# Reset workdir to Node-RED default
WORKDIR /usr/src/node-red
EXPOSE 1880

50
Makefile Normal file
View File

@@ -0,0 +1,50 @@
.PHONY: install lint lint-fix test test-jest test-node test-legacy ci docker-ci docker-test docker-lint e2e e2e-up e2e-down
install:
@sed -i 's|"generalFunctions": "git+https://[^"]*"|"generalFunctions": "file:./nodes/generalFunctions"|' package.json
npm install
@git checkout -- package.json 2>/dev/null || true
lint:
npx eslint nodes/
lint-fix:
npx eslint nodes/ --fix
test-jest:
npx jest --forceExit
test-node:
node --test \
nodes/valve/test/basic/*.test.js \
nodes/valve/test/edge/*.test.js \
nodes/valve/test/integration/*.test.js \
nodes/valveGroupControl/test/basic/*.test.js \
nodes/valveGroupControl/test/edge/*.test.js \
nodes/valveGroupControl/test/integration/*.test.js
test-legacy:
node nodes/machineGroupControl/src/groupcontrol.test.js
node nodes/generalFunctions/src/nrmse/errorMetric.test.js
test: test-jest test-node test-legacy
ci: lint test
docker-ci:
docker compose run --rm ci
docker-test:
docker compose run --rm test
docker-lint:
docker compose run --rm lint
e2e:
bash test/e2e/run-e2e.sh
e2e-up:
docker compose -f docker-compose.e2e.yml up -d --build
e2e-down:
docker compose -f docker-compose.e2e.yml down

49
docker-compose.e2e.yml Normal file
View File

@@ -0,0 +1,49 @@
services:
influxdb:
image: influxdb:2.7
environment:
- DOCKER_INFLUXDB_INIT_MODE=setup
- DOCKER_INFLUXDB_INIT_USERNAME=admin
- DOCKER_INFLUXDB_INIT_PASSWORD=adminpassword
- DOCKER_INFLUXDB_INIT_ORG=evolv
- DOCKER_INFLUXDB_INIT_BUCKET=evolv
- DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=evolv-e2e-token
ports:
- "8086:8086"
healthcheck:
test: ["CMD", "influx", "ping"]
interval: 5s
timeout: 5s
retries: 5
nodered:
build:
context: .
dockerfile: Dockerfile.e2e
ports:
- "1880:1880"
depends_on:
influxdb:
condition: service_healthy
environment:
- INFLUXDB_URL=http://influxdb:8086
- INFLUXDB_TOKEN=evolv-e2e-token
- INFLUXDB_ORG=evolv
- INFLUXDB_BUCKET=evolv
volumes:
- ./test/e2e/flows.json:/data/flows.json
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:1880/"]
interval: 5s
timeout: 5s
retries: 10
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_AUTH_ANONYMOUS_ENABLED=true
depends_on:
- influxdb

File diff suppressed because it is too large Load Diff

446
docs/API_REFERENCE.md Normal file
View File

@@ -0,0 +1,446 @@
# generalFunctions API Reference
Shared library (`nodes/generalFunctions/`) used across all EVOLV Node-RED nodes.
```js
const { logger, outputUtils, MeasurementContainer, ... } = require('generalFunctions');
```
---
## Table of Contents
1. [Logger](#logger)
2. [OutputUtils](#outpututils)
3. [ValidationUtils](#validationutils)
4. [MeasurementContainer](#measurementcontainer)
5. [ConfigManager](#configmanager)
6. [ChildRegistrationUtils](#childregistrationutils)
7. [MenuUtils](#menuutils)
8. [EndpointUtils](#endpointutils)
9. [Positions](#positions)
10. [AssetLoader / loadCurve](#assetloader--loadcurve)
---
## Logger
Structured, level-filtered console logger.
**File:** `src/helper/logger.js`
### Constructor
```js
new Logger(logging = true, logLevel = 'debug', nameModule = 'N/A')
```
| Param | Type | Default | Description |
|---|---|---|---|
| `logging` | `boolean` | `true` | Enable/disable all output |
| `logLevel` | `string` | `'debug'` | Minimum severity: `'debug'` \| `'info'` \| `'warn'` \| `'error'` |
| `nameModule` | `string` | `'N/A'` | Label prefixed to every message |
### Methods
| Method | Signature | Description |
|---|---|---|
| `debug` | `(message: string): void` | Log at DEBUG level |
| `info` | `(message: string): void` | Log at INFO level |
| `warn` | `(message: string): void` | Log at WARN level |
| `error` | `(message: string): void` | Log at ERROR level |
| `setLogLevel` | `(level: string): void` | Change minimum level at runtime |
| `toggleLogging` | `(): void` | Flip logging on/off |
### Example
```js
const Logger = require('generalFunctions').logger;
const log = new Logger(true, 'info', 'MyNode');
log.info('Node started'); // [INFO] -> MyNode: Node started
log.debug('ignored'); // silent (below 'info')
log.setLogLevel('debug');
log.debug('now visible'); // [DEBUG] -> MyNode: now visible
```
---
## OutputUtils
Tracks output state and formats messages for InfluxDB or process outputs. Only emits changed fields.
**File:** `src/helper/outputUtils.js`
### Constructor
```js
new OutputUtils() // no parameters
```
### Methods
| Method | Signature | Returns | Description |
|---|---|---|---|
| `formatMsg` | `(output, config, format)` | `object \| undefined` | Diff against last output; returns formatted msg or `undefined` if nothing changed |
| `checkForChanges` | `(output, format)` | `object` | Returns only the key/value pairs that changed since last call |
**`format`** must be `'influxdb'` or `'process'`.
### Example
```js
const out = new OutputUtils();
const msg = out.formatMsg(
{ temperature: 22.5, pressure: 1013 },
config,
'influxdb'
);
// msg = { topic: 'nodeName', payload: { measurement, fields, tags, timestamp } }
```
---
## ValidationUtils
Schema-driven config validation with type coercion, range clamping, and nested object support.
**File:** `src/helper/validationUtils.js`
### Constructor
```js
new ValidationUtils(loggerEnabled = true, loggerLevel = 'warn')
```
### Methods
| Method | Signature | Returns | Description |
|---|---|---|---|
| `validateSchema` | `(config, schema, name)` | `object` | Walk the schema, validate every field, return a clean config. Unknown keys are stripped. Missing keys get their schema default. |
| `constrain` | `(value, min, max)` | `number` | Clamp a numeric value to `[min, max]` |
| `removeUnwantedKeys` | `(obj)` | `object` | Strip `rules`/`description` metadata, collapse `default` values |
**Supported `rules.type` values:** `number`, `integer`, `boolean`, `string`, `enum`, `array`, `set`, `object`, `curve`, `machineCurve`.
### Example
```js
const ValidationUtils = require('generalFunctions').validation;
const v = new ValidationUtils(true, 'warn');
const schema = {
temperature: { default: 20, rules: { type: 'number', min: -40, max: 100 } },
unit: { default: 'C', rules: { type: 'enum', values: [{ value: 'C' }, { value: 'F' }] } }
};
const validated = v.validateSchema({ temperature: 999 }, schema, 'myNode');
// validated.temperature === 100 (clamped)
// validated.unit === 'C' (default applied)
```
---
## MeasurementContainer
Chainable measurement storage organised by **type / variant / position**. Supports auto unit conversion, windowed statistics, events, and positional difference calculations.
**File:** `src/measurements/MeasurementContainer.js`
### Constructor
```js
new MeasurementContainer(options = {}, logger)
```
| Option | Type | Default | Description |
|---|---|---|---|
| `windowSize` | `number` | `10` | Rolling window for statistics |
| `defaultUnits` | `object` | `{ pressure:'mbar', flow:'m3/h', ... }` | Default unit per measurement type |
| `autoConvert` | `boolean` | `true` | Auto-convert values to target unit |
| `preferredUnits` | `object` | `{}` | Per-type unit overrides |
### Chainable Setters
All return `this` for chaining.
```js
container
.type('pressure')
.variant('static')
.position('upstream')
.distance(5)
.unit('bar')
.value(3.2, Date.now(), 'bar');
```
| Method | Signature | Description |
|---|---|---|
| `type` | `(typeName): this` | Set measurement type (e.g. `'pressure'`) |
| `variant` | `(variantName): this` | Set variant (e.g. `'static'`, `'differential'`) |
| `position` | `(positionValue): this` | Set position (e.g. `'upstream'`, `'downstream'`) |
| `distance` | `(distance): this` | Set physical distance from parent |
| `unit` | `(unitName): this` | Set unit on the underlying measurement |
| `value` | `(val, timestamp?, sourceUnit?): this` | Store a value; auto-converts if `sourceUnit` differs from target |
### Terminal / Query Methods
| Method | Signature | Returns | Description |
|---|---|---|---|
| `get` | `()` | `Measurement \| null` | Get the raw measurement object |
| `getCurrentValue` | `(requestedUnit?)` | `number \| null` | Latest value, optionally converted |
| `getAverage` | `(requestedUnit?)` | `number \| null` | Windowed average |
| `getMin` | `()` | `number \| null` | Window minimum |
| `getMax` | `()` | `number \| null` | Window maximum |
| `getAllValues` | `()` | `array \| null` | All stored samples |
| `getLaggedValue` | `(lag?, requestedUnit?)` | `number \| null` | Value from `lag` samples ago |
| `getLaggedSample` | `(lag?, requestedUnit?)` | `object \| null` | Full sample `{ value, timestamp, unit }` from `lag` samples ago |
| `exists` | `({ type?, variant?, position?, requireValues? })` | `boolean` | Check if a measurement series exists |
| `difference` | `({ from?, to?, unit? })` | `object \| null` | Compute `{ value, avgDiff, unit }` between two positions |
### Introspection / Lifecycle
| Method | Signature | Returns | Description |
|---|---|---|---|
| `getTypes` | `()` | `string[]` | All registered measurement types |
| `getVariants` | `()` | `string[]` | Variants under current type |
| `getPositions` | `()` | `string[]` | Positions under current type+variant |
| `getAvailableUnits` | `(measurementType?)` | `string[]` | Units available for a type |
| `getBestUnit` | `(excludeUnits?)` | `object \| null` | Best human-readable unit for current value |
| `setPreferredUnit` | `(type, unit)` | `this` | Override default unit for a type |
| `setChildId` | `(id)` | `this` | Tag container with a child node ID |
| `setChildName` | `(name)` | `this` | Tag container with a child node name |
| `setParentRef` | `(parent)` | `this` | Store reference to parent node |
| `clear` | `()` | `void` | Reset all measurements and chain state |
### Events
The internal `emitter` fires `"type.variant.position"` on every `value()` call with:
```js
{ value, originalValue, unit, sourceUnit, timestamp, position, distance, variant, type, childId, childName, parentRef }
```
### Example
```js
const { MeasurementContainer } = require('generalFunctions');
const mc = new MeasurementContainer({ windowSize: 5 });
mc.type('pressure').variant('static').position('upstream').value(3.2);
mc.type('pressure').variant('static').position('downstream').value(2.8);
const diff = mc.type('pressure').variant('static').difference();
// diff = { value: -0.4, avgDiff: -0.4, unit: 'mbar', from: 'downstream', to: 'upstream' }
```
---
## ConfigManager
Loads JSON config files from disk and builds merged runtime configs.
**File:** `src/configs/index.js`
### Constructor
```js
new ConfigManager(relPath = '.')
```
`relPath` is resolved relative to the configs directory.
### Methods
| Method | Signature | Returns | Description |
|---|---|---|---|
| `getConfig` | `(configName)` | `object` | Load and parse `<configName>.json` |
| `getAvailableConfigs` | `()` | `string[]` | List config names (without `.json`) |
| `hasConfig` | `(configName)` | `boolean` | Check existence |
| `getBaseConfig` | `()` | `object` | Shortcut for `getConfig('baseConfig')` |
| `buildConfig` | `(nodeName, uiConfig, nodeId, domainConfig?)` | `object` | Merge base schema + UI overrides into a runtime config |
| `createEndpoint` | `(nodeName)` | `string` | Generate browser JS that injects config into `window.EVOLV.nodes` |
### Example
```js
const { configManager } = require('generalFunctions');
const cfg = configManager.buildConfig('measurement', uiConfig, node.id, {
scaling: { enabled: true, inputMin: 0, inputMax: 100 }
});
```
---
## ChildRegistrationUtils
Manages parent-child node relationships: registration, lookup, and structure storage.
**File:** `src/helper/childRegistrationUtils.js`
### Constructor
```js
new ChildRegistrationUtils(mainClass)
```
`mainClass` is the parent node instance (must expose `.logger` and optionally `.registerChild()`).
### Methods
| Method | Signature | Returns | Description |
|---|---|---|---|
| `registerChild` | `(child, positionVsParent, distance?)` | `Promise<any>` | Register a child node under the parent. Sets up parent refs, measurement context, and stores by softwareType/category. |
| `getChildrenOfType` | `(softwareType, category?)` | `array` | Get children filtered by software type and optional category |
| `getChildById` | `(childId)` | `object \| null` | Lookup a single child by its ID |
| `getAllChildren` | `()` | `array` | All registered children |
| `logChildStructure` | `()` | `void` | Debug-print the full child tree |
### Example
```js
const { childRegistrationUtils: CRU } = require('generalFunctions');
const cru = new CRU(parentNode);
await cru.registerChild(sensorNode, 'upstream');
cru.getChildrenOfType('measurement'); // [sensorNode]
```
---
## MenuUtils
Browser-side UI helper for Node-RED editor. Methods are mixed in from separate modules: toggles, data fetching, URL utils, dropdown population, and HTML generation.
**File:** `src/helper/menuUtils.js`
### Constructor
```js
new MenuUtils() // no parameters; sets isCloud=false, configData=null
```
### Key Methods
**Toggles** -- control UI element visibility:
| Method | Signature | Description |
|---|---|---|
| `initBasicToggles` | `(elements)` | Bind log-level row visibility to log checkbox |
| `initMeasurementToggles` | `(elements)` | Bind scaling input rows to scaling checkbox |
| `initTensionToggles` | `(elements, node)` | Show/hide tension row based on interpolation method |
**Data Fetching:**
| Method | Signature | Returns | Description |
|---|---|---|---|
| `fetchData` | `(url, fallbackUrl)` | `Promise<array>` | Fetch JSON from primary URL; fall back on failure |
| `fetchProjectData` | `(url)` | `Promise<object>` | Fetch project-level data |
| `apiCall` | `(node)` | `Promise<object>` | POST to asset-register API |
**URL Construction:**
| Method | Signature | Returns | Description |
|---|---|---|---|
| `getSpecificConfigUrl` | `(nodeName, cloudAPI)` | `{ cloudConfigURL, localConfigURL }` | Build cloud + local config URLs |
| `constructUrl` | `(base, ...paths)` | `string` | Join URL segments safely |
| `constructCloudURL` | `(base, ...paths)` | `string` | Same as `constructUrl`, for cloud endpoints |
**Dropdown Population:**
| Method | Signature | Description |
|---|---|---|
| `fetchAndPopulateDropdowns` | `(configUrls, elements, node)` | Cascading supplier > subType > model > unit dropdowns |
| `populateDropdown` | `(htmlElement, options, node, property, callback?)` | Fill a `<select>` with options and wire change events |
| `populateLogLevelOptions` | `(logLevelSelect, configData, node)` | Populate log-level dropdown from config |
| `populateSmoothingMethods` | `(configUrls, elements, node)` | Populate smoothing method dropdown |
| `populateInterpolationMethods` | `(configUrls, elements, node)` | Populate interpolation method dropdown |
| `generateHtml` | `(htmlElement, options, savedValue)` | Write `<option>` HTML into an element |
---
## EndpointUtils
Server-side helper that serves `MenuUtils` as browser JavaScript via Node-RED HTTP endpoints.
**File:** `src/helper/endpointUtils.js`
### Constructor
```js
new EndpointUtils({ MenuUtilsClass? })
```
| Param | Type | Default | Description |
|---|---|---|---|
| `MenuUtilsClass` | `class` | `MenuUtils` | The MenuUtils constructor to introspect |
### Methods
| Method | Signature | Returns | Description |
|---|---|---|---|
| `createMenuUtilsEndpoint` | `(RED, nodeName, customHelpers?)` | `void` | Register `GET /<nodeName>/resources/menuUtils.js` |
| `generateMenuUtilsCode` | `(nodeName, customHelpers?)` | `string` | Produce the browser JS string (introspects `MenuUtils.prototype`) |
### Example
```js
const EndpointUtils = require('generalFunctions/src/helper/endpointUtils');
const ep = new EndpointUtils();
ep.createMenuUtilsEndpoint(RED, 'valve');
// Browser can now load: GET /valve/resources/menuUtils.js
```
---
## Positions
Canonical constants for parent-child spatial relationships.
**File:** `src/constants/positions.js`
### Exports
```js
const { POSITIONS, POSITION_VALUES, isValidPosition } = require('generalFunctions');
```
| Export | Type | Value |
|---|---|---|
| `POSITIONS` | `object` | `{ UPSTREAM: 'upstream', DOWNSTREAM: 'downstream', AT_EQUIPMENT: 'atEquipment', DELTA: 'delta' }` |
| `POSITION_VALUES` | `string[]` | `['upstream', 'downstream', 'atEquipment', 'delta']` |
| `isValidPosition` | `(pos: string): boolean` | Returns `true` if `pos` is one of the four values |
---
## AssetLoader / loadCurve
Loads JSON asset files (machine curves, etc.) from the datasets directory with LRU caching.
**File:** `datasets/assetData/curves/index.js`
### Singleton convenience functions
```js
const { loadCurve } = require('generalFunctions');
```
| Function | Signature | Returns | Description |
|---|---|---|---|
| `loadCurve` | `(curveType: string)` | `object \| null` | Load `<curveType>.json` from the curves directory |
| `loadAsset` | `(datasetType, assetId)` | `object \| null` | Load any JSON asset by dataset folder and ID |
| `getAvailableAssets` | `(datasetType)` | `string[]` | List asset IDs in a dataset folder |
### AssetLoader class
```js
new AssetLoader(maxCacheSize = 100)
```
Same methods as above (`loadCurve`, `loadAsset`, `getAvailableAssets`), plus `clearCache()`.
### Example
```js
const { loadCurve } = require('generalFunctions');
const curve = loadCurve('hidrostal-H05K-S03R');
// curve = { flow: [...], head: [...], ... } or null
```

418
docs/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,418 @@
# EVOLV Architecture
## 1. System Overview
High-level view of how EVOLV fits into the wastewater treatment automation stack.
```mermaid
graph LR
NR[Node-RED Runtime] <-->|msg objects| EVOLV[EVOLV Nodes]
EVOLV -->|InfluxDB line protocol| INFLUX[(InfluxDB)]
INFLUX -->|queries| GRAFANA[Grafana Dashboards]
EVOLV -->|process output| NR
EVOLV -->|parent output| NR
style NR fill:#b22222,color:#fff
style EVOLV fill:#0f52a5,color:#fff
style INFLUX fill:#0c99d9,color:#fff
style GRAFANA fill:#50a8d9,color:#fff
```
Each EVOLV node produces three outputs:
| Port | Name | Purpose |
|------|------|---------|
| 0 | process | Process data forwarded to downstream nodes |
| 1 | dbase | InfluxDB-formatted measurement data |
| 2 | parent | Control messages to parent nodes (e.g. registerChild) |
---
## 2. Node Architecture (Three-Layer Pattern)
Every node follows a consistent three-layer design that separates Node-RED wiring from domain logic.
```mermaid
graph TB
subgraph "Node-RED Runtime"
REG["RED.nodes.registerType()"]
end
subgraph "Layer 1 — Wrapper (valve.js)"
W[wrapper .js]
W -->|"new nodeClass(config, RED, this, name)"| NC
W -->|MenuManager| MENU[HTTP /name/menu.js]
W -->|configManager| CFG[HTTP /name/configData.js]
end
subgraph "Layer 2 — Node Adapter (src/nodeClass.js)"
NC[nodeClass]
NC -->|_loadConfig| CFGM[configManager]
NC -->|_setupSpecificClass| SC
NC -->|_attachInputHandler| INPUT[onInput routing]
NC -->|_startTickLoop| TICK[1s tick loop]
NC -->|_tick → outputUtils| OUT[formatMsg]
end
subgraph "Layer 3 — Domain Logic (src/specificClass.js)"
SC[specificClass]
SC -->|measurements| MC[MeasurementContainer]
SC -->|state machine| ST[state]
SC -->|hydraulics / biology| DOMAIN[domain models]
end
subgraph "generalFunctions"
GF[shared library]
end
REG --> W
GF -.->|logger, outputUtils, configManager,\nMeasurementContainer, validation, ...| NC
GF -.->|MeasurementContainer, state,\nconvert, predict, ...| SC
style W fill:#0f52a5,color:#fff
style NC fill:#0c99d9,color:#fff
style SC fill:#50a8d9,color:#fff
style GF fill:#86bbdd,color:#000
```
---
## 3. generalFunctions Module Map
The shared library (`nodes/generalFunctions/`) provides all cross-cutting concerns.
```mermaid
graph TB
GF[generalFunctions/index.js]
subgraph "Core Helpers (src/helper/)"
LOGGER[logger]
OUTPUT[outputUtils]
CHILD[childRegistrationUtils]
CFGUTIL[configUtils]
ASSERT[assertionUtils]
VALID[validationUtils]
end
subgraph "Validators (src/helper/validators/)"
TV[typeValidators]
CV[collectionValidators]
CURV[curveValidator]
end
subgraph "Domain Modules (src/)"
MC[MeasurementContainer]
CFGMGR[configManager]
MENUMGR[MenuManager]
STATE[state]
CONVERT[convert / Fysics]
PREDICT[predict / interpolation]
NRMSE[nrmse / errorMetrics]
COOLPROP[coolprop]
end
subgraph "Data (datasets/)"
CURVES[assetData/curves]
ASSETS[assetData/assetData.json]
UNITS[unitData.json]
end
subgraph "Constants (src/constants/)"
POS[POSITIONS / POSITION_VALUES]
end
GF --> LOGGER
GF --> OUTPUT
GF --> CHILD
GF --> CFGUTIL
GF --> ASSERT
GF --> VALID
VALID --> TV
VALID --> CV
VALID --> CURV
GF --> MC
GF --> CFGMGR
GF --> MENUMGR
GF --> STATE
GF --> CONVERT
GF --> PREDICT
GF --> NRMSE
GF --> COOLPROP
GF --> CURVES
GF --> POS
style GF fill:#0f52a5,color:#fff
style LOGGER fill:#86bbdd,color:#000
style OUTPUT fill:#86bbdd,color:#000
style VALID fill:#86bbdd,color:#000
style MC fill:#50a8d9,color:#fff
style CFGMGR fill:#50a8d9,color:#fff
style MENUMGR fill:#50a8d9,color:#fff
```
---
## 4. Data Flow (Message Lifecycle)
Sequence diagram showing a typical input message and the periodic tick output cycle.
```mermaid
sequenceDiagram
participant NR as Node-RED
participant W as wrapper.js
participant NC as nodeClass
participant SC as specificClass
participant OU as outputUtils
Note over W: Node startup
W->>NC: new nodeClass(config, RED, node, name)
NC->>NC: _loadConfig (configManager.buildConfig)
NC->>SC: new specificClass(config, stateConfig, options)
NC->>NR: send([null, null, {topic: registerChild}])
Note over NC: Every 1 second (tick loop)
NC->>SC: getOutput()
SC-->>NC: raw measurement data
NC->>OU: formatMsg(raw, config, 'process')
NC->>OU: formatMsg(raw, config, 'influxdb')
NC->>NR: send([processMsg, influxMsg])
Note over NR: Incoming control message
NR->>W: msg {topic: 'execMovement', payload: {...}}
W->>NC: onInput(msg)
NC->>SC: handleInput(source, action, setpoint)
SC->>SC: update state machine & measurements
```
---
## 5. Node Types
| Node | S88 Level | Purpose |
|------|-----------|---------|
| **measurement** | Control Module | Generic measurement point — reads, validates, and stores sensor values |
| **valve** | Control Module | Valve simulation with hydraulic model, position control, flow/pressure prediction |
| **rotatingMachine** | Control Module | Pumps, blowers, mixers — rotating equipment with speed control and efficiency curves |
| **diffuser** | Control Module | Aeration diffuser — models oxygen transfer and pressure drop |
| **settler** | Equipment | Sludge settler — models settling behavior and sludge blanket |
| **reactor** | Equipment | Hydraulic tank and biological process simulator (activated sludge, digestion) |
| **monster** | Equipment | MONitoring and STrEam Routing — complex measurement aggregation |
| **pumpingStation** | Unit | Coordinates multiple pumps as a pumping station |
| **valveGroupControl** | Unit | Manages multiple valves as a coordinated group — distributes flow, monitors pressure |
| **machineGroupControl** | Unit | Group control for rotating machines — load balancing and sequencing |
| **dashboardAPI** | Utility | Exposes data and unit conversion endpoints for external dashboards |
# EVOLV Architecture
## Node Hierarchy (S88)
EVOLV follows the ISA-88 (S88) batch control standard. Each node maps to an S88 level and uses a consistent color scheme in the Node-RED editor.
```mermaid
graph TD
classDef area fill:#0f52a5,color:#fff,stroke:#0a3d7a
classDef processCell fill:#0c99d9,color:#fff,stroke:#0977aa
classDef unit fill:#50a8d9,color:#fff,stroke:#3d89b3
classDef equipment fill:#86bbdd,color:#000,stroke:#6a9bb8
classDef controlModule fill:#a9daee,color:#000,stroke:#87b8cc
classDef standalone fill:#f0f0f0,color:#000,stroke:#999
%% S88 Levels
subgraph "S88: Area"
PS[pumpingStation]
end
subgraph "S88: Equipment"
MGC[machineGroupControl]
VGC[valveGroupControl]
end
subgraph "S88: Control Module"
RM[rotatingMachine]
V[valve]
M[measurement]
R[reactor]
S[settler]
end
subgraph "Standalone"
MON[monster]
DASH[dashboardAPI]
DIFF[diffuser - not implemented]
end
%% Parent-child registration relationships
PS -->|"accepts: measurement"| M
PS -->|"accepts: machine"| RM
PS -->|"accepts: machineGroup"| MGC
PS -->|"accepts: pumpingStation"| PS2[pumpingStation]
MGC -->|"accepts: machine"| RM
RM -->|"accepts: measurement"| M2[measurement]
RM -->|"accepts: reactor"| R
VGC -->|"accepts: valve"| V
VGC -->|"accepts: machine / rotatingmachine"| RM2[rotatingMachine]
VGC -->|"accepts: machinegroup / machinegroupcontrol"| MGC2[machineGroupControl]
VGC -->|"accepts: pumpingstation / valvegroupcontrol"| PS3["pumpingStation / valveGroupControl"]
R -->|"accepts: measurement"| M3[measurement]
R -->|"accepts: reactor"| R2[reactor]
S -->|"accepts: measurement"| M4[measurement]
S -->|"accepts: reactor"| R3[reactor]
S -->|"accepts: machine"| RM3[rotatingMachine]
%% Styling
class PS,PS2,PS3 area
class MGC,MGC2 equipment
class VGC equipment
class RM,RM2,RM3 controlModule
class V controlModule
class M,M2,M3,M4 controlModule
class R,R2,R3 controlModule
class S controlModule
class MON,DASH,DIFF standalone
```
### Registration Summary
```mermaid
graph LR
classDef parent fill:#0c99d9,color:#fff
classDef child fill:#a9daee,color:#000
PS[pumpingStation] -->|measurement| LEAF1((leaf))
PS -->|machine| RM1[rotatingMachine]
PS -->|machineGroup| MGC1[machineGroupControl]
PS -->|pumpingStation| PS1[pumpingStation]
MGC[machineGroupControl] -->|machine| RM2[rotatingMachine]
VGC[valveGroupControl] -->|valve| V1[valve]
VGC -->|source| SRC["machine, machinegroup,<br/>pumpingstation, valvegroupcontrol"]
RM[rotatingMachine] -->|measurement| LEAF2((leaf))
RM -->|reactor| R1[reactor]
R[reactor] -->|measurement| LEAF3((leaf))
R -->|reactor| R2[reactor]
S[settler] -->|measurement| LEAF4((leaf))
S -->|reactor| R3[reactor]
S -->|machine| RM3[rotatingMachine]
class PS,MGC,VGC,RM,R,S parent
class LEAF1,LEAF2,LEAF3,LEAF4,RM1,RM2,RM3,MGC1,PS1,V1,SRC,R1,R2,R3 child
```
## Node Types
| Node | S88 Level | softwareType | role | Accepts Children | Outputs |
|------|-----------|-------------|------|-----------------|---------|
| **pumpingStation** | Area | `pumpingstation` | StationController | measurement, machine (rotatingMachine), machineGroup, pumpingStation | [process, dbase, parent] |
| **machineGroupControl** | Equipment | `machinegroupcontrol` | GroupController | machine (rotatingMachine) | [process, dbase, parent] |
| **valveGroupControl** | Equipment | `valvegroupcontrol` | ValveGroupController | valve, machine, rotatingmachine, machinegroup, machinegroupcontrol, pumpingstation, valvegroupcontrol | [process, dbase, parent] |
| **rotatingMachine** | Control Module | `rotatingmachine` | RotationalDeviceController | measurement, reactor | [process, dbase, parent] |
| **valve** | Control Module | `valve` | controller | _(leaf node, no children)_ | [process, dbase, parent] |
| **measurement** | Control Module | `measurement` | Sensor | _(leaf node, no children)_ | [process, dbase, parent] |
| **reactor** | Control Module | `reactor` | Biological reactor | measurement, reactor (upstream chaining) | [process, dbase, parent] |
| **settler** | Control Module | `settler` | Secondary settler | measurement, reactor (upstream), machine (return pump) | [process, dbase, parent] |
| **monster** | Standalone | - | - | dual-parent, standalone | - |
| **dashboardAPI** | Standalone | - | - | accepts any child (Grafana integration) | - |
| **diffuser** | Standalone | - | - | _(not implemented)_ | - |
## Data Flow
### Measurement Data Flow (upstream to downstream)
```mermaid
sequenceDiagram
participant Sensor as measurement (sensor)
participant Machine as rotatingMachine
participant Group as machineGroupControl
participant Station as pumpingStation
Note over Sensor: Sensor reads value<br/>(pressure, flow, level, temp)
Sensor->>Sensor: measurements.type(t).variant("measured").position(p).value(v)
Sensor->>Sensor: emitter.emit("type.measured.position", eventData)
Sensor->>Machine: Event: "pressure.measured.upstream"
Machine->>Machine: Store in own MeasurementContainer
Machine->>Machine: getMeasuredPressure() -> calcFlow() -> calcPower()
Machine->>Machine: emitter.emit("flow.predicted.downstream", eventData)
Machine->>Group: Event: "flow.predicted.downstream"
Group->>Group: handlePressureChange()
Group->>Group: Aggregate flows across all machines
Group->>Group: Calculate group totals and efficiency
Machine->>Station: Event: "flow.predicted.downstream"
Station->>Station: Store predicted flow in/out
Station->>Station: _updateVolumePrediction()
Station->>Station: _calcNetFlow(), _calcTimeRemaining()
```
### Control Command Flow (downstream to upstream)
```mermaid
sequenceDiagram
participant Station as pumpingStation
participant Group as machineGroupControl
participant Machine as rotatingMachine
participant Machine2 as rotatingMachine (2)
Station->>Group: handleInput("parent", action, param)
Group->>Group: Determine scaling strategy
Group->>Group: Calculate setpoints per machine
Group->>Machine: handleInput("parent", "execMovement", setpoint)
Group->>Machine2: handleInput("parent", "execMovement", setpoint)
Machine->>Machine: setpoint() -> state.moveTo(pos)
Machine->>Machine: updatePosition() -> calcFlow(), calcPower()
Machine->>Machine: emitter.emit("flow.predicted.downstream")
Machine2->>Machine2: setpoint() -> state.moveTo(pos)
Machine2->>Machine2: updatePosition() -> calcFlow(), calcPower()
Machine2->>Machine2: emitter.emit("flow.predicted.downstream")
```
### Wastewater Treatment Process Flow
```mermaid
graph LR
classDef process fill:#50a8d9,color:#fff
classDef equipment fill:#86bbdd,color:#000
PS_IN[pumpingStation<br/>Influent] -->|flow| R1[reactor<br/>Anoxic]
R1 -->|effluent| R2[reactor<br/>Aerated]
R2 -->|effluent| SET[settler]
SET -->|effluent out| PS_OUT[pumpingStation<br/>Effluent]
SET -->|sludge return| RM_RET[rotatingMachine<br/>Return pump]
RM_RET -->|recirculation| R1
PS_IN --- MGC_IN[machineGroupControl]
MGC_IN --- RM_IN[rotatingMachine<br/>Influent pumps]
class PS_IN,PS_OUT process
class R1,R2,SET process
class MGC_IN,RM_IN,RM_RET equipment
```
### Event-Driven Communication Pattern
All parent-child communication uses Node.js `EventEmitter`:
1. **Registration**: Parent calls `childRegistrationUtils.registerChild(child, position)` which stores the child and calls the parent's `registerChild(child, softwareType)` method.
2. **Event binding**: The parent's `registerChild()` subscribes to the child's `measurements.emitter` events (e.g., `"flow.predicted.downstream"`).
3. **Data propagation**: When a child updates a measurement, it emits an event. The parent's listener stores the value in its own `MeasurementContainer` and runs its domain logic.
4. **Three outputs**: Every node sends data to three Node-RED outputs: `[process, dbase, parent]` -- process data for downstream nodes, InfluxDB for persistence, and parent aggregation data.
### Position Convention
Children register with a position relative to their parent:
- `upstream` -- before the parent in the flow direction
- `downstream` -- after the parent in the flow direction
- `atEquipment` -- physically located at/on the parent equipment

80
docs/ISSUES.md Normal file
View File

@@ -0,0 +1,80 @@
# Open Issues — EVOLV Codebase
Issues identified during codebase scan (2026-03-12). Create these on Gitea when ready.
---
## Issue 1: Restore diffuser node implementation
**Labels:** `enhancement`, `node`
**Priority:** Medium
The `nodes/diffuser/` directory contains only `.git`, `LICENSE`, and `README.md` — no implementation. There was a previous experimental version. Needs:
- Retrieve original diffuser logic from user/backup
- Rebuild to current three-layer architecture (wrapper `.js` + `src/nodeClass.js` + `src/specificClass.js`)
- Use `require('generalFunctions')` barrel imports
- Add config JSON in `generalFunctions/src/configs/diffuser.json`
- Register under category `'EVOLV'` with appropriate S88 color
- Add tests
**Blocked on:** User providing original diffuser logic/requirements.
---
## Issue 2: Relocate prediction/ML modules to external service
**Labels:** `enhancement`, `architecture`
**Priority:** Medium
TensorFlow-based influent prediction code was removed from monster node (was broken/incomplete). The prediction functionality needs a new home:
- LSTM model for 24-hour flow prediction based on precipitation data
- Standardization constants: hours `(mean=11.504, std=6.922)`, precipitation `(mean=0.090, std=0.439)`, response `(mean=1188.01, std=1024.19)`
- Model was served from `http://127.0.0.1:1880/generalFunctions/datasets/lstmData/tfjs_model/`
- Consider: separate microservice, Python-based inference, or ONNX runtime
- Monster node should accept predictions via `model_prediction` message topic from external service
**Related files removed:** `monster_class.js` methods `get_model_prediction()`, `model_loader()`
---
## Issue 3: Modernize monster node to three-layer architecture
**Labels:** `refactor`, `node`
**Priority:** Low
Monster node uses old-style structure (`dependencies/monster/` instead of `src/`). Should be refactored:
- Move `dependencies/monster/monster_class.js``src/specificClass.js`
- Create `src/nodeClass.js` adapter (extract from `monster.js`)
- Slim down `monster.js` to standard wrapper pattern
- Move `monsterConfig.json``generalFunctions/src/configs/monster.json`
- Remove `modelLoader.js` (TF dependency removed)
- Add unit tests
**Note:** monster_class.js is ~500 lines of domain logic. Keep sampling_program(), aggregation, AQUON integration intact.
---
## Issue 4: Clean up inline test/demo code in specificClass files
**Labels:** `cleanup`
**Priority:** Low
Several specificClass files have test/demo code after `module.exports`:
- `pumpingStation/src/specificClass.js` (lines 478-697): Demo code guarded with `require.main === module` — acceptable but could move to `test/` or `examples/`
- `machineGroupControl/src/specificClass.js` (lines 969-1158): Block-commented test code with `makeMachines()` — dead code, could be removed or moved to test file
---
## Issue 5: DashboardAPI node improvements
**Labels:** `enhancement`, `security`
**Priority:** Low
- Bearer token now relies on `GRAFANA_TOKEN` env var (hardcoded token was removed for security)
- Ensure deployment docs mention setting `GRAFANA_TOKEN`
- `dashboardapi_class.js` still has `console.log` calls (lines 154, 178) — should use logger
- Node doesn't follow three-layer architecture (older style)

29
eslint.config.js Normal file
View File

@@ -0,0 +1,29 @@
const js = require('@eslint/js');
const globals = require('globals');
module.exports = [
js.configs.recommended,
{
languageOptions: {
ecmaVersion: 2022,
sourceType: 'commonjs',
globals: {
...globals.node,
...globals.jest,
RED: 'readonly',
},
},
rules: {
'no-unused-vars': ['warn', { argsIgnorePattern: '^_', varsIgnorePattern: '^_' }],
'no-console': 'off',
'no-prototype-builtins': 'warn',
'no-constant-condition': 'warn',
},
},
{
ignores: [
'node_modules/**',
'nodes/generalFunctions/src/coolprop-node/coolprop/**',
],
},
];

19
jest.config.js Normal file
View File

@@ -0,0 +1,19 @@
module.exports = {
testEnvironment: 'node',
verbose: true,
testMatch: [
'<rootDir>/nodes/generalFunctions/src/coolprop-node/test/**/*.test.js',
'<rootDir>/nodes/generalFunctions/test/**/*.test.js',
'<rootDir>/nodes/dashboardAPI/test/**/*.test.js',
'<rootDir>/nodes/diffuser/test/specificClass.test.js',
'<rootDir>/nodes/monster/test/**/*.test.js',
'<rootDir>/nodes/pumpingStation/test/**/*.test.js',
'<rootDir>/nodes/reactor/test/**/*.test.js',
'<rootDir>/nodes/settler/test/**/*.test.js',
'<rootDir>/nodes/measurement/test/**/*.test.js',
],
testPathIgnorePatterns: [
'/node_modules/',
],
testTimeout: 15000,
};

5891
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -12,18 +12,21 @@
"node-red": {
"nodes": {
"dashboardapi": "nodes/dashboardAPI/dashboardapi.js",
"diffuser": "nodes/diffuser/diffuser.js",
"machineGroupControl": "nodes/machineGroupControl/mgc.js",
"measurement": "nodes/measurement/measurement.js",
"monster": "nodes/monster/monster.js",
"pumpingstation": "nodes/pumpingStation/pumpingStation.js",
"reactor": "nodes/reactor/reactor.js",
"rotatingMachine": "nodes/rotatingMachine/rotatingMachine.js",
"settler": "nodes/settler/settler.js",
"valve": "nodes/valve/valve.js",
"valveGroupControl": "nodes/valveGroupControl/vgc.js",
"pumpingstation": "nodes/pumpingStation/pumpingStation.js",
"settler": "nodes/settler/settler.js"
"valveGroupControl": "nodes/valveGroupControl/vgc.js"
}
},
"scripts": {
"preinstall": "node scripts/patch-deps.js",
"postinstall": "git checkout -- package.json 2>/dev/null || true",
"docker:build": "docker compose build",
"docker:up": "docker compose up -d",
"docker:down": "docker compose down",
@@ -36,7 +39,16 @@
"docker:test:gf": "docker compose exec nodered sh /data/evolv/scripts/test-all.sh gf",
"docker:validate": "docker compose exec nodered sh /data/evolv/scripts/validate-nodes.sh",
"docker:deploy": "docker compose exec nodered sh /data/evolv/scripts/deploy-flow.sh",
"docker:reset": "docker compose down -v && docker compose up -d --build"
"docker:reset": "docker compose down -v && docker compose up -d --build",
"test": "jest --forceExit",
"test:node": "node --test nodes/valve/test/basic/*.test.js nodes/valve/test/edge/*.test.js nodes/valve/test/integration/*.test.js nodes/valveGroupControl/test/basic/*.test.js nodes/valveGroupControl/test/edge/*.test.js nodes/valveGroupControl/test/integration/*.test.js",
"test:legacy": "node nodes/machineGroupControl/src/groupcontrol.test.js && node nodes/generalFunctions/src/nrmse/errorMetric.test.js",
"test:all": "npm test && npm run test:node && npm run test:legacy",
"test:e2e:reactor": "node scripts/e2e-reactor-roundtrip.js",
"lint": "eslint nodes/",
"lint:fix": "eslint nodes/ --fix",
"ci": "npm run lint && npm run test:all",
"test:e2e": "bash test/e2e/run-e2e.sh"
},
"author": "Rene De Ren, Pim Moerman, Janneke Tack, Sjoerd Fijnje, Dieke Gabriels, pieter van der wilt",
"license": "SEE LICENSE",
@@ -46,5 +58,11 @@
"@tensorflow/tfjs-node": "^4.22.0",
"generalFunctions": "file:nodes/generalFunctions",
"mathjs": "^13.2.0"
},
"devDependencies": {
"@eslint/js": "^8.57.0",
"eslint": "^8.57.0",
"globals": "^15.0.0",
"jest": "^29.7.0"
}
}

View File

@@ -0,0 +1,269 @@
#!/usr/bin/env node
/**
* E2E reactor round-trip test:
* Node-RED -> InfluxDB -> Grafana proxy query
*/
const fs = require('node:fs');
const path = require('node:path');
const NR_URL = process.env.NR_URL || 'http://localhost:1880';
const INFLUX_URL = process.env.INFLUX_URL || 'http://localhost:8086';
const GRAFANA_URL = process.env.GRAFANA_URL || 'http://localhost:3000';
const GRAFANA_USER = process.env.GRAFANA_USER || 'admin';
const GRAFANA_PASSWORD = process.env.GRAFANA_PASSWORD || 'evolv';
const INFLUX_ORG = process.env.INFLUX_ORG || 'evolv';
const INFLUX_BUCKET = process.env.INFLUX_BUCKET || 'telemetry';
const INFLUX_TOKEN = process.env.INFLUX_TOKEN || 'evolv-dev-token';
const GRAFANA_DS_UID = process.env.GRAFANA_DS_UID || 'cdzg44tv250jkd';
const FLOW_FILE = path.join(__dirname, '..', 'docker', 'demo-flow.json');
const REQUIRE_GRAFANA_DASHBOARDS = process.env.REQUIRE_GRAFANA_DASHBOARDS === '1';
const REACTOR_MEASUREMENTS = [
'reactor_demo_reactor_z1',
'reactor_demo_reactor_z2',
'reactor_demo_reactor_z3',
'reactor_demo_reactor_z4',
];
const REACTOR_MEASUREMENT = REACTOR_MEASUREMENTS[3];
const QUERY_TIMEOUT_MS = 90000;
const POLL_INTERVAL_MS = 3000;
const REQUIRED_DASHBOARD_TITLES = ['Bioreactor Z1', 'Bioreactor Z2', 'Bioreactor Z3', 'Bioreactor Z4', 'Settler S1'];
async function wait(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function fetchJson(url, options = {}) {
const response = await fetch(url, options);
const text = await response.text();
let body = null;
if (text) {
try {
body = JSON.parse(text);
} catch {
body = text;
}
}
return { response, body, text };
}
async function assertReachable() {
const checks = [
[`${NR_URL}/settings`, 'Node-RED'],
[`${INFLUX_URL}/health`, 'InfluxDB'],
[`${GRAFANA_URL}/api/health`, 'Grafana'],
];
for (const [url, label] of checks) {
const { response, text } = await fetchJson(url, {
headers: label === 'Grafana'
? { Authorization: `Basic ${Buffer.from(`${GRAFANA_USER}:${GRAFANA_PASSWORD}`).toString('base64')}` }
: undefined,
});
if (!response.ok) {
throw new Error(`${label} not reachable at ${url} (${response.status}): ${text}`);
}
console.log(`PASS: ${label} reachable`);
}
}
async function deployDemoFlow() {
const flow = JSON.parse(fs.readFileSync(FLOW_FILE, 'utf8'));
const { response, text } = await fetchJson(`${NR_URL}/flows`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Node-RED-Deployment-Type': 'full',
},
body: JSON.stringify(flow),
});
if (!(response.status === 200 || response.status === 204)) {
throw new Error(`Flow deploy failed (${response.status}): ${text}`);
}
console.log(`PASS: Demo flow deployed (${response.status})`);
}
async function queryInfluxCsv(query) {
const response = await fetch(`${INFLUX_URL}/api/v2/query?org=${encodeURIComponent(INFLUX_ORG)}`, {
method: 'POST',
headers: {
Authorization: `Token ${INFLUX_TOKEN}`,
'Content-Type': 'application/json',
Accept: 'application/csv',
},
body: JSON.stringify({ query }),
});
const text = await response.text();
if (!response.ok) {
throw new Error(`Influx query failed (${response.status}): ${text}`);
}
return text;
}
function countCsvDataRows(csvText) {
return csvText
.split('\n')
.map((line) => line.trim())
.filter((line) => line && !line.startsWith('#') && line.includes(','))
.length;
}
async function waitForReactorTelemetry() {
const deadline = Date.now() + QUERY_TIMEOUT_MS;
while (Date.now() < deadline) {
const counts = {};
for (const measurement of REACTOR_MEASUREMENTS) {
const query = `
from(bucket: "${INFLUX_BUCKET}")
|> range(start: -15m)
|> filter(fn: (r) => r._measurement == "${measurement}")
|> limit(n: 20)
`.trim();
counts[measurement] = countCsvDataRows(await queryInfluxCsv(query));
}
const missing = Object.entries(counts)
.filter(([, rows]) => rows === 0)
.map(([measurement]) => measurement);
if (missing.length === 0) {
const summary = Object.entries(counts)
.map(([measurement, rows]) => `${measurement}=${rows}`)
.join(', ');
console.log(`PASS: Reactor telemetry reached InfluxDB (${summary})`);
return;
}
console.log(`WAIT: reactor telemetry not yet present in InfluxDB for ${missing.join(', ')}`);
await wait(POLL_INTERVAL_MS);
}
throw new Error(`Timed out waiting for reactor telemetry measurements ${REACTOR_MEASUREMENTS.join(', ')}`);
}
async function assertGrafanaDatasource() {
const auth = `Basic ${Buffer.from(`${GRAFANA_USER}:${GRAFANA_PASSWORD}`).toString('base64')}`;
const { response, body, text } = await fetchJson(`${GRAFANA_URL}/api/datasources/uid/${GRAFANA_DS_UID}`, {
headers: { Authorization: auth },
});
if (!response.ok) {
throw new Error(`Grafana datasource lookup failed (${response.status}): ${text}`);
}
if (body?.uid !== GRAFANA_DS_UID) {
throw new Error(`Grafana datasource UID mismatch: expected ${GRAFANA_DS_UID}, got ${body?.uid}`);
}
console.log(`PASS: Grafana datasource ${GRAFANA_DS_UID} is present`);
}
async function queryGrafanaDatasource() {
const auth = `Basic ${Buffer.from(`${GRAFANA_USER}:${GRAFANA_PASSWORD}`).toString('base64')}`;
const response = await fetch(`${GRAFANA_URL}/api/ds/query`, {
method: 'POST',
headers: {
Authorization: auth,
'Content-Type': 'application/json',
},
body: JSON.stringify({
from: 'now-15m',
to: 'now',
queries: [
{
refId: 'A',
datasource: { uid: GRAFANA_DS_UID, type: 'influxdb' },
query: `
from(bucket: "${INFLUX_BUCKET}")
|> range(start: -15m)
|> filter(fn: (r) => r._measurement == "${REACTOR_MEASUREMENT}" and r._field == "S_O")
|> last()
`.trim(),
rawQuery: true,
intervalMs: 1000,
maxDataPoints: 100,
}
],
}),
});
const text = await response.text();
if (!response.ok) {
throw new Error(`Grafana datasource query failed (${response.status}): ${text}`);
}
const body = JSON.parse(text);
const frames = body?.results?.A?.frames || [];
if (frames.length === 0) {
throw new Error('Grafana datasource query returned no reactor frames');
}
console.log(`PASS: Grafana can query reactor telemetry through datasource (${frames.length} frame(s))`);
}
async function waitForGrafanaDashboards(timeoutMs = QUERY_TIMEOUT_MS) {
const deadline = Date.now() + timeoutMs;
const auth = `Basic ${Buffer.from(`${GRAFANA_USER}:${GRAFANA_PASSWORD}`).toString('base64')}`;
while (Date.now() < deadline) {
const response = await fetch(`${GRAFANA_URL}/api/search?query=`, {
headers: { Authorization: auth },
});
const text = await response.text();
if (!response.ok) {
throw new Error(`Grafana dashboard search failed (${response.status}): ${text}`);
}
const results = JSON.parse(text);
const titles = new Set(results.map((item) => item.title));
const missing = REQUIRED_DASHBOARD_TITLES.filter((title) => !titles.has(title));
const pumpingStationCount = results.filter((item) => item.title === 'pumpingStation').length;
if (missing.length === 0 && pumpingStationCount >= 3) {
console.log(`PASS: Grafana dashboards created (${REQUIRED_DASHBOARD_TITLES.join(', ')} + ${pumpingStationCount} pumpingStation dashboards)`);
return;
}
const missingParts = [];
if (missing.length > 0) {
missingParts.push(`missing titled dashboards: ${missing.join(', ')}`);
}
if (pumpingStationCount < 3) {
missingParts.push(`pumpingStation dashboards=${pumpingStationCount}`);
}
console.log(`WAIT: Grafana dashboards not ready: ${missingParts.join(' | ')}`);
await wait(POLL_INTERVAL_MS);
}
throw new Error(`Timed out waiting for Grafana dashboards: ${REQUIRED_DASHBOARD_TITLES.join(', ')} and >=3 pumpingStation dashboards`);
}
async function main() {
console.log('=== EVOLV Reactor E2E Round Trip ===');
await assertReachable();
await deployDemoFlow();
console.log('WAIT: allowing Node-RED inject/tick loops to populate telemetry');
await wait(12000);
await waitForReactorTelemetry();
await assertGrafanaDatasource();
await queryGrafanaDatasource();
if (REQUIRE_GRAFANA_DASHBOARDS) {
await waitForGrafanaDashboards();
console.log('PASS: Node-RED -> InfluxDB -> Grafana round trip is working for reactor telemetry and dashboard generation');
return;
}
try {
await waitForGrafanaDashboards(15000);
console.log('PASS: Node-RED -> InfluxDB -> Grafana round trip is working for reactor telemetry and dashboard generation');
} catch (error) {
console.warn(`WARN: Grafana dashboard auto-generation is not ready yet: ${error.message}`);
console.log('PASS: Node-RED -> InfluxDB -> Grafana round trip is working for live reactor telemetry');
}
}
main().catch((error) => {
console.error(`FAIL: ${error.message}`);
process.exit(1);
});

20
scripts/patch-deps.js Normal file
View File

@@ -0,0 +1,20 @@
/**
* Preinstall script: rewrites the generalFunctions dependency
* from git+https to a local file path when the submodule exists.
* This avoids needing Gitea credentials during npm install.
*/
const fs = require('fs');
const path = require('path');
const pkgPath = path.join(__dirname, '..', 'package.json');
const localGF = path.join(__dirname, '..', 'nodes', 'generalFunctions');
if (fs.existsSync(localGF) && fs.existsSync(path.join(localGF, 'index.js'))) {
const pkg = JSON.parse(fs.readFileSync(pkgPath, 'utf8'));
if (pkg.dependencies && pkg.dependencies.generalFunctions &&
pkg.dependencies.generalFunctions.startsWith('git+')) {
pkg.dependencies.generalFunctions = 'file:./nodes/generalFunctions';
fs.writeFileSync(pkgPath, JSON.stringify(pkg, null, 2) + '\n');
console.log('[patch-deps] Rewrote generalFunctions to local path');
}
}

440
test/e2e/flows.json Normal file
View File

@@ -0,0 +1,440 @@
[
{
"id": "e2e-flow-tab",
"type": "tab",
"label": "E2E Test Flow",
"disabled": false,
"info": "End-to-end test flow that verifies EVOLV nodes load, accept input, and produce output."
},
{
"id": "inject-trigger",
"type": "inject",
"z": "e2e-flow-tab",
"name": "Trigger once on start",
"props": [
{ "p": "payload" },
{ "p": "topic", "vt": "str" }
],
"repeat": "",
"crontab": "",
"once": true,
"onceDelay": "3",
"topic": "e2e-test",
"payload": "",
"payloadType": "date",
"x": 160,
"y": 80,
"wires": [["build-measurement-msg"]]
},
{
"id": "build-measurement-msg",
"type": "function",
"z": "e2e-flow-tab",
"name": "Build measurement input",
"func": "// Simulate an analog sensor reading sent to the measurement node.\n// The measurement node expects a numeric payload on topic 'analogInput'.\nmsg.payload = 4.2 + Math.random() * 15.8; // 4-20 mA range\nmsg.topic = 'analogInput';\nnode.status({ fill: 'green', shape: 'dot', text: 'sent ' + msg.payload.toFixed(2) });\nreturn msg;",
"outputs": 1,
"timeout": "",
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 380,
"y": 80,
"wires": [["measurement-e2e-node"]]
},
{
"id": "measurement-e2e-node",
"type": "measurement",
"z": "e2e-flow-tab",
"name": "E2E-Level-Sensor",
"scaling": true,
"i_min": 4,
"i_max": 20,
"i_offset": 0,
"o_min": 0,
"o_max": 5,
"simulator": false,
"smooth_method": "",
"count": "10",
"uuid": "",
"supplier": "e2e-test",
"category": "level",
"assetType": "sensor",
"model": "e2e-virtual",
"unit": "m",
"enableLog": false,
"logLevel": "error",
"positionVsParent": "upstream",
"positionIcon": "",
"hasDistance": false,
"distance": 0,
"distanceUnit": "m",
"distanceDescription": "",
"x": 600,
"y": 80,
"wires": [
["debug-process"],
["debug-dbase"],
["debug-parent"]
]
},
{
"id": "debug-process",
"type": "debug",
"z": "e2e-flow-tab",
"name": "Process Output",
"active": true,
"tosidebar": true,
"console": true,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 830,
"y": 40,
"wires": []
},
{
"id": "debug-dbase",
"type": "debug",
"z": "e2e-flow-tab",
"name": "Database Output",
"active": true,
"tosidebar": true,
"console": true,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 840,
"y": 80,
"wires": []
},
{
"id": "debug-parent",
"type": "debug",
"z": "e2e-flow-tab",
"name": "Parent Output",
"active": true,
"tosidebar": true,
"console": true,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 830,
"y": 120,
"wires": []
},
{
"id": "inject-periodic",
"type": "inject",
"z": "e2e-flow-tab",
"name": "Periodic (5s)",
"props": [
{ "p": "payload" },
{ "p": "topic", "vt": "str" }
],
"repeat": "5",
"crontab": "",
"once": true,
"onceDelay": "6",
"topic": "e2e-heartbeat",
"payload": "",
"payloadType": "date",
"x": 160,
"y": 200,
"wires": [["heartbeat-func"]]
},
{
"id": "heartbeat-func",
"type": "function",
"z": "e2e-flow-tab",
"name": "Heartbeat check",
"func": "// Verify the EVOLV measurement node is running by querying its presence\nmsg.payload = {\n check: 'heartbeat',\n timestamp: Date.now(),\n nodeCount: global.get('_e2e_msg_count') || 0\n};\n// Increment message counter\nlet count = global.get('_e2e_msg_count') || 0;\nglobal.set('_e2e_msg_count', count + 1);\nnode.status({ fill: 'blue', shape: 'ring', text: 'beat #' + (count+1) });\nreturn msg;",
"outputs": 1,
"timeout": "",
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 380,
"y": 200,
"wires": [["debug-heartbeat"]]
},
{
"id": "debug-heartbeat",
"type": "debug",
"z": "e2e-flow-tab",
"name": "Heartbeat Debug",
"active": true,
"tosidebar": true,
"console": true,
"tostatus": false,
"complete": "payload",
"targetType": "msg",
"statusVal": "",
"statusType": "auto",
"x": 600,
"y": 200,
"wires": []
},
{
"id": "inject-monster-prediction",
"type": "inject",
"z": "e2e-flow-tab",
"name": "Monster prediction",
"props": [
{ "p": "payload" },
{ "p": "topic", "vt": "str" }
],
"repeat": "",
"crontab": "",
"once": true,
"onceDelay": "4",
"topic": "model_prediction",
"payload": "120",
"payloadType": "num",
"x": 150,
"y": 320,
"wires": [["evolv-monster"]]
},
{
"id": "inject-monster-flow",
"type": "inject",
"z": "e2e-flow-tab",
"name": "Monster flow",
"props": [
{ "p": "payload" },
{ "p": "topic", "vt": "str" }
],
"repeat": "3",
"crontab": "",
"once": true,
"onceDelay": "5",
"topic": "i_flow",
"payload": "3600",
"payloadType": "num",
"x": 140,
"y": 360,
"wires": [["evolv-monster"]]
},
{
"id": "inject-monster-start",
"type": "inject",
"z": "e2e-flow-tab",
"name": "Monster start",
"props": [
{ "p": "payload" },
{ "p": "topic", "vt": "str" }
],
"repeat": "",
"crontab": "",
"once": true,
"onceDelay": "6",
"topic": "start",
"payload": "",
"payloadType": "date",
"x": 140,
"y": 400,
"wires": [["evolv-monster"]]
},
{
"id": "evolv-monster",
"type": "monster",
"z": "e2e-flow-tab",
"name": "E2E-Monster",
"samplingtime": 1,
"minvolume": 5,
"maxweight": 23,
"emptyWeightBucket": 3,
"aquon_sample_name": "112100",
"supplier": "e2e-test",
"subType": "samplingCabinet",
"model": "e2e-virtual",
"unit": "m3/h",
"enableLog": false,
"logLevel": "error",
"x": 390,
"y": 360,
"wires": [
["debug-monster-process"],
["debug-monster-dbase"],
[],
[]
]
},
{
"id": "debug-monster-process",
"type": "debug",
"z": "e2e-flow-tab",
"name": "Monster Process Output",
"active": true,
"tosidebar": true,
"console": true,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 660,
"y": 340,
"wires": []
},
{
"id": "debug-monster-dbase",
"type": "debug",
"z": "e2e-flow-tab",
"name": "Monster Database Output",
"active": true,
"tosidebar": true,
"console": true,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 670,
"y": 380,
"wires": []
},
{
"id": "inject-dashboardapi-register",
"type": "inject",
"z": "e2e-flow-tab",
"name": "DashboardAPI register child",
"props": [
{ "p": "payload" },
{ "p": "topic", "vt": "str" }
],
"repeat": "",
"crontab": "",
"once": true,
"onceDelay": "12",
"payload": "",
"payloadType": "date",
"x": 160,
"y": 500,
"wires": [["build-dashboardapi-msg"]]
},
{
"id": "build-dashboardapi-msg",
"type": "function",
"z": "e2e-flow-tab",
"name": "Build dashboardapi input",
"func": "msg.topic = 'registerChild';\nmsg.payload = {\n config: {\n general: {\n name: 'E2E-Level-Sensor'\n },\n functionality: {\n softwareType: 'measurement'\n }\n }\n};\nreturn msg;",
"outputs": 1,
"timeout": "",
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 400,
"y": 500,
"wires": [["dashboardapi-e2e"]]
},
{
"id": "dashboardapi-e2e",
"type": "dashboardapi",
"z": "e2e-flow-tab",
"name": "E2E-DashboardAPI",
"logLevel": "error",
"enableLog": false,
"host": "grafana",
"port": "3000",
"bearerToken": "",
"x": 660,
"y": 500,
"wires": [["debug-dashboardapi-output"]]
},
{
"id": "debug-dashboardapi-output",
"type": "debug",
"z": "e2e-flow-tab",
"name": "DashboardAPI Output",
"active": true,
"tosidebar": true,
"console": true,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 920,
"y": 500,
"wires": []
},
{
"id": "inject-diffuser-flow",
"type": "inject",
"z": "e2e-flow-tab",
"name": "Diffuser airflow",
"props": [
{ "p": "payload" },
{ "p": "topic", "vt": "str" }
],
"repeat": "",
"crontab": "",
"once": true,
"onceDelay": "9",
"topic": "air_flow",
"payload": "24",
"payloadType": "num",
"x": 150,
"y": 620,
"wires": [["diffuser-e2e"]]
},
{
"id": "diffuser-e2e",
"type": "diffuser",
"z": "e2e-flow-tab",
"name": "E2E-Diffuser",
"number": 1,
"i_elements": 4,
"i_diff_density": 2.4,
"i_m_water": 4.5,
"alfaf": 0.7,
"enableLog": false,
"logLevel": "error",
"x": 390,
"y": 620,
"wires": [["debug-diffuser-process"], ["debug-diffuser-dbase"], []]
},
{
"id": "debug-diffuser-process",
"type": "debug",
"z": "e2e-flow-tab",
"name": "Diffuser Process Output",
"active": true,
"tosidebar": true,
"console": true,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 670,
"y": 600,
"wires": []
},
{
"id": "debug-diffuser-dbase",
"type": "debug",
"z": "e2e-flow-tab",
"name": "Diffuser Database Output",
"active": true,
"tosidebar": true,
"console": true,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 680,
"y": 640,
"wires": []
}
]

213
test/e2e/run-e2e.sh Executable file
View File

@@ -0,0 +1,213 @@
#!/usr/bin/env bash
#
# End-to-end test runner for EVOLV Node-RED stack.
# Starts Node-RED + InfluxDB + Grafana via Docker Compose,
# verifies that EVOLV nodes are registered in the palette,
# and tears down the stack on exit.
#
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
COMPOSE_FILE="$PROJECT_ROOT/docker-compose.e2e.yml"
NODERED_URL="http://localhost:1880"
MAX_WAIT=120 # seconds to wait for Node-RED to become healthy
GRAFANA_URL="http://localhost:3000/api/health"
MAX_GRAFANA_WAIT=60
LOG_WAIT=20
# EVOLV node types that must appear in the palette (from package.json node-red.nodes)
EXPECTED_NODES=(
"dashboardapi"
"diffuser"
"machineGroupControl"
"measurement"
"monster"
"pumpingstation"
"reactor"
"rotatingMachine"
"settler"
"valve"
"valveGroupControl"
)
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
log_info() { echo -e "${GREEN}[INFO]${NC} $*"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
log_error() { echo -e "${RED}[ERROR]${NC} $*"; }
wait_for_log_pattern() {
local pattern="$1"
local description="$2"
local required="${3:-false}"
local elapsed=0
local logs=""
while [ $elapsed -lt $LOG_WAIT ]; do
logs=$(run_compose logs nodered 2>&1)
if echo "$logs" | grep -q "$pattern"; then
log_info " [PASS] $description"
return 0
fi
sleep 2
elapsed=$((elapsed + 2))
done
if [ "$required" = true ]; then
log_error " [FAIL] $description not detected in logs"
FAILURES=$((FAILURES + 1))
else
log_warn " [WARN] $description not detected in logs"
fi
return 1
}
# Determine docker compose command (handle permission via sg docker if needed)
USE_SG_DOCKER=false
if ! docker info >/dev/null 2>&1; then
if sg docker -c "docker info" >/dev/null 2>&1; then
USE_SG_DOCKER=true
log_info "Using sg docker for Docker access"
else
log_error "Docker is not accessible. Please ensure Docker is running and you have permissions."
exit 1
fi
fi
run_compose() {
if [ "$USE_SG_DOCKER" = true ]; then
local cmd="docker compose -f $(printf '%q' "$COMPOSE_FILE")"
local arg
for arg in "$@"; do
cmd+=" $(printf '%q' "$arg")"
done
sg docker -c "$cmd"
else
docker compose -f "$COMPOSE_FILE" "$@"
fi
}
cleanup() {
log_info "Tearing down E2E stack..."
run_compose down --volumes --remove-orphans 2>/dev/null || true
}
# Always clean up on exit
trap cleanup EXIT
# --- Step 1: Build and start the stack ---
log_info "Building and starting E2E stack..."
run_compose up -d --build
# --- Step 2: Wait for Node-RED to be healthy ---
log_info "Waiting for Node-RED to become healthy (max ${MAX_WAIT}s)..."
elapsed=0
while [ $elapsed -lt $MAX_WAIT ]; do
if curl -sf "$NODERED_URL/" >/dev/null 2>&1; then
log_info "Node-RED is up after ${elapsed}s"
break
fi
sleep 2
elapsed=$((elapsed + 2))
done
if [ $elapsed -ge $MAX_WAIT ]; then
log_error "Node-RED did not become healthy within ${MAX_WAIT}s"
log_error "Container logs:"
run_compose logs nodered
exit 1
fi
# Give Node-RED a few extra seconds to finish loading all nodes and editor metadata
sleep 8
# --- Step 3: Verify EVOLV nodes are registered in the palette ---
log_info "Querying Node-RED for registered nodes..."
NODES_RESPONSE=$(curl -sf "$NODERED_URL/nodes" 2>&1) || {
log_error "Failed to query Node-RED /nodes endpoint"
exit 1
}
FAILURES=0
PALETTE_MISSES=0
for node_type in "${EXPECTED_NODES[@]}"; do
if echo "$NODES_RESPONSE" | grep -qi "$node_type"; then
log_info " [PASS] Node type '$node_type' found in palette"
else
log_warn " [WARN] Node type '$node_type' not found in /nodes response"
PALETTE_MISSES=$((PALETTE_MISSES + 1))
fi
done
# --- Step 4: Verify flows are deployed ---
log_info "Checking deployed flows..."
FLOWS_RESPONSE=$(curl -sf "$NODERED_URL/flows" 2>&1) || {
log_error "Failed to query Node-RED /flows endpoint"
exit 1
}
if echo "$FLOWS_RESPONSE" | grep -q "e2e-flow-tab"; then
log_info " [PASS] E2E test flow is deployed"
else
log_warn " [WARN] E2E test flow not found in deployed flows (may need manual deploy)"
fi
# --- Step 5: Verify InfluxDB is reachable ---
log_info "Checking InfluxDB health..."
INFLUX_HEALTH=$(curl -sf "http://localhost:8086/health" 2>&1) || {
log_error "Failed to reach InfluxDB health endpoint"
FAILURES=$((FAILURES + 1))
}
if echo "$INFLUX_HEALTH" | grep -q '"status":"pass"'; then
log_info " [PASS] InfluxDB is healthy"
else
log_error " [FAIL] InfluxDB health check failed"
FAILURES=$((FAILURES + 1))
fi
# --- Step 5b: Verify Grafana is reachable ---
log_info "Checking Grafana health..."
GRAFANA_HEALTH=""
elapsed=0
while [ $elapsed -lt $MAX_GRAFANA_WAIT ]; do
GRAFANA_HEALTH=$(curl -sf "$GRAFANA_URL" 2>&1) && break
sleep 2
elapsed=$((elapsed + 2))
done
if echo "$GRAFANA_HEALTH" | grep -Eq '"database"[[:space:]]*:[[:space:]]*"ok"'; then
log_info " [PASS] Grafana is healthy"
else
log_error " [FAIL] Grafana health check failed"
FAILURES=$((FAILURES + 1))
fi
# --- Step 5c: Verify EVOLV measurement node produced output ---
log_info "Checking EVOLV measurement node output in container logs..."
wait_for_log_pattern "Database Output" "EVOLV measurement node produced database output" true || true
wait_for_log_pattern "Process Output" "EVOLV measurement node produced process output" true || true
wait_for_log_pattern "Monster Process Output" "EVOLV monster node produced process output" true || true
wait_for_log_pattern "Monster Database Output" "EVOLV monster node produced database output" true || true
wait_for_log_pattern "Diffuser Process Output" "EVOLV diffuser node produced process output" true || true
wait_for_log_pattern "Diffuser Database Output" "EVOLV diffuser node produced database output" true || true
wait_for_log_pattern "DashboardAPI Output" "EVOLV dashboardapi node produced create output" true || true
# --- Step 6: Summary ---
echo ""
if [ $FAILURES -eq 0 ]; then
log_info "========================================="
log_info " E2E tests PASSED - all checks green"
log_info "========================================="
exit 0
else
log_error "========================================="
log_error " E2E tests FAILED - $FAILURES check(s) failed"
log_error "========================================="
exit 1
fi