release: palette redesign + CoreSync scaffolding + dashboardAPI MODULE_NOT_FOUND fix
PALETTE REDESIGN (2026-05-21)
Sidebar swatches switched from S88 level (all blue) to domain-hue per node.
Family hue = function (rotating=orange, valves=teal, biology=green/olive,
sampling=violet, sensor=amber, aeration=sky-blue, infrastructure=slate);
within a family, darker = higher S88 / "more controller-ish."
Editor-group rectangles in flow.json still follow S88 — only the
registerType colour changed.
Submodule bumps for palette: rotatingMachine, machineGroupControl,
pumpingStation, valve, valveGroupControl, reactor, settler, monster,
measurement, diffuser, dashboardAPI.
Docs touched:
- CLAUDE.md: palette swatch vs. editor-group bullets split out.
- .claude/rules/node-red-flow-layout.md: new §10.0 introduces the two
color systems, full 12-row palette table, and explicit warning not to
mix the two hexes.
- .claude/refactor/MODULE_SPLIT.md: per-node headers annotated with
both `group #XXX` and `palette #XXX`.
- .claude/refactor/WIKI_HOME_TEMPLATE.md + WIKI_TEMPLATE.md: clarify
Mermaid classDefs visualize hierarchy, not palette swatches.
- .claude/refactor/OPEN_QUESTIONS.md: dated decision entry with
rationale, file list, and follow-ups.
CORESYNC SUBMODULE (new)
nodes/coresync added pointing at https://gitea.wbd-rd.nl/RnD/coresync.
FROST/SensorThings handoff path — first version forwards FROST-ready HTTP
request messages on the dbase output; a downstream http-request node
performs the POST and feeds responses back on msg.topic = "frost.response".
Lazy stream resolver, latest-wins queue (keep first + latest, drop middle),
knot-emit on slope change, provenance preserved in Observation parameters.
- .gitmodules: add nodes/coresync entry.
- package.json: register coresync as a Node-RED node.
- generalFunctions bump: new frostFormatter + 4 node config schemas
expose the dbase format option.
- measurement bump: "frost" option added to dbaseOutputFormat dropdown
(plus the in-flight data.measurement unit-handling work).
- machineGroupControl bump: small editor compact-fields tweak alongside
the palette change.
- CORESYNC_FROST_INTERVIEW_HANDOFF.md added at root with interview state
(Q20 open: slope angle vs. relative delta comparison).
DASHBOARDAPI MODULE_NOT_FOUND FIX
package.json: dashboardapi entry path corrected to
nodes/dashboardAPI/dashboardAPI.js. Commit e04c4a1 renamed the files to
camelCase but missed package.json; on case-sensitive filesystems
(Linux/Docker, where the tarball lands) the require resolved to nothing
and the node showed MODULE_NOT_FOUND in the Node-RED palette.
MISC CLEANUP
- examples/README.md + examples/pumpingstation-complete-example/ removal
(build_flow.py, flow.json, README.md superseded by per-node examples).
- jest.config.js: in-progress tweak.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
znetsixe
414
CORESYNC_FROST_INTERVIEW_HANDOFF.md
Normal file
414
CORESYNC_FROST_INTERVIEW_HANDOFF.md
Normal file
@@ -0,0 +1,414 @@
|
||||
# CoreSync FROST Interview Handoff
|
||||
|
||||
Date: 2026-05-19
|
||||
|
||||
## Continue Here First
|
||||
|
||||
Resume the interview at **Question 20**. The last open design topic was the reducer comparison method:
|
||||
|
||||
**Q20. Should slope change be compared by angle in degrees or by relative slope delta?**
|
||||
|
||||
Recommended direction before pausing:
|
||||
|
||||
- Support both eventually.
|
||||
- Default to angle comparison with normalized time/value axes.
|
||||
- Compute `dx = deltaTimeMs / timeScaleMs`.
|
||||
- Compute `dy = deltaValue / valueScale`.
|
||||
- Compare `atan2(dy, dx)` direction changes against `angleToleranceDeg`.
|
||||
|
||||
## Agreed Decisions
|
||||
|
||||
- Use FROST/SensorThings instead of direct InfluxDB for the new CoreSync path.
|
||||
- Keep EVOLV standard outputs:
|
||||
- `process`
|
||||
- `dbase`
|
||||
- `parent`
|
||||
- Add a `dbase` output format option for `frost`.
|
||||
- `dbase = frost` emits FROST-ready HTTP request messages.
|
||||
- The CoreSync node does not post directly to FROST in the first version.
|
||||
- A normal Node-RED HTTP request node sends the FROST messages.
|
||||
- HTTP responses feed back into the same CoreSync input with `msg.topic = "frost.response"`.
|
||||
- All FROST metadata lookup/create/patch requests leave on `dbase`, not `process`.
|
||||
- `process` is reserved for functional process data and optional functional state.
|
||||
- The resolver is lazy: streams are resolved only when telemetry arrives.
|
||||
- Pending queue policy for unresolved/FROST-down streams is keep first + latest, drop middle.
|
||||
- Observation writes use nested Datastream endpoints:
|
||||
- `POST /v1.1/Datastreams({datastreamId})/Observations`
|
||||
- Preserve provenance in Observation `parameters`.
|
||||
- On angle/slope change, emit the previous point as the knot.
|
||||
- Do not forward-fill delta-compressed fields.
|
||||
- Latest values are queried per Datastream:
|
||||
- `/Datastreams(id)/Observations?$orderby=phenomenonTime desc&$top=1`
|
||||
|
||||
## SensorThings Mapping
|
||||
|
||||
- EVOLV asset/apparatus/node -> FROST `Thing`
|
||||
- EVOLV field `type` -> FROST `ObservedProperty`
|
||||
- EVOLV `variant` (`measured`, `predicted`, `setpoint`) -> FROST `Sensor`
|
||||
- EVOLV `position` -> stable FROST `FeatureOfInterest`
|
||||
- EVOLV numeric field -> one FROST `Datastream`
|
||||
- One reducer-kept knot -> one FROST `Observation`
|
||||
|
||||
Stable FOI convention:
|
||||
|
||||
```text
|
||||
{thingId}:upstream
|
||||
{thingId}:atEquipment
|
||||
{thingId}:downstream
|
||||
```
|
||||
|
||||
Also copy position into `Datastream.properties.position` for filtering.
|
||||
|
||||
## Units
|
||||
|
||||
Use EVOLV canonical ingest units. UI conversion happens client-side.
|
||||
|
||||
- pressure: `Pa`
|
||||
- flow: `m3/s`
|
||||
- power: `W`
|
||||
- temperature: `K`
|
||||
- density: `kg/m3`
|
||||
- level: `m`
|
||||
- volume: `m3`
|
||||
- control / percentage / efficiency: normalized ratio `1`
|
||||
|
||||
No leading zeros in engineering tags:
|
||||
|
||||
```text
|
||||
P-1
|
||||
PT-1
|
||||
FT-9999999
|
||||
```
|
||||
|
||||
Never:
|
||||
|
||||
```text
|
||||
P-001
|
||||
PT-0001
|
||||
```
|
||||
|
||||
## Identity And Registry
|
||||
|
||||
- Node-RED is not the source of truth for asset identity.
|
||||
- Future central asset registry owns tag allocation and duplicate detection.
|
||||
- Use one central counter per tag prefix:
|
||||
- `P`
|
||||
- `PT`
|
||||
- `FT`
|
||||
- `TT`
|
||||
- etc.
|
||||
- The central registry, not local Node-RED, performs atomic `+1`.
|
||||
- For now, assume the central registry is future work.
|
||||
- First implementation derives identity when possible and allows overrides.
|
||||
- Keep a boundary like `resolveIdentity(input)` so future registry integration is straightforward.
|
||||
|
||||
First-version identity behavior:
|
||||
|
||||
```text
|
||||
Thing tag: configured/derived, e.g. P-1
|
||||
Sensor tag: configured/derived, e.g. PT-1, MODEL-P-1, CTRL-P-1
|
||||
Stream key: thingTag:type:variant:position:sensorTag
|
||||
```
|
||||
|
||||
## Shared Collector Model
|
||||
|
||||
Use one shared CoreSync per FROST target/stack level.
|
||||
|
||||
Many EVOLV nodes can connect their `dbase` output to the CoreSync input, assuming payloads are structured as:
|
||||
|
||||
```js
|
||||
{
|
||||
measurement: "P-1",
|
||||
fields: {
|
||||
"pressure.measured.upstream.PT-1": 12345
|
||||
},
|
||||
tags: {
|
||||
tagcode: "P-1"
|
||||
},
|
||||
timestamp: Date
|
||||
}
|
||||
```
|
||||
|
||||
Also accept arrays of such payloads.
|
||||
|
||||
Internal stream key:
|
||||
|
||||
```text
|
||||
thingTag:type:variant:position:sensorTag
|
||||
```
|
||||
|
||||
Per-stream state:
|
||||
|
||||
- FROST id cache
|
||||
- latest FROST `phenomenonTime`
|
||||
- reducer anchor point
|
||||
- reducer previous point
|
||||
- pending latest point
|
||||
- bounded pending queue
|
||||
|
||||
## FROST Request Message Shape
|
||||
|
||||
Outgoing request messages should preserve correlation metadata:
|
||||
|
||||
```js
|
||||
{
|
||||
topic: "frost.metadata.lookup",
|
||||
requestId: "thing:P-1:lookup",
|
||||
_coreSync: {
|
||||
kind: "thing",
|
||||
action: "lookup",
|
||||
externalKey: "thing:P-1",
|
||||
streamKey: "P-1:pressure:measured:upstream:PT-1"
|
||||
},
|
||||
method: "GET",
|
||||
url: "...",
|
||||
payload: null
|
||||
}
|
||||
```
|
||||
|
||||
FROST response feedback:
|
||||
|
||||
```js
|
||||
{
|
||||
topic: "frost.response",
|
||||
requestId: "...",
|
||||
statusCode: 200,
|
||||
payload: {},
|
||||
_coreSync: {}
|
||||
}
|
||||
```
|
||||
|
||||
Observation write target:
|
||||
|
||||
```http
|
||||
POST /v1.1/Datastreams({datastreamId})/Observations
|
||||
```
|
||||
|
||||
Observation payload:
|
||||
|
||||
```json
|
||||
{
|
||||
"phenomenonTime": "2026-05-19T10:15:30.000Z",
|
||||
"result": 123.4,
|
||||
"FeatureOfInterest": {
|
||||
"@iot.id": 7
|
||||
},
|
||||
"parameters": {
|
||||
"reduction": "knot",
|
||||
"reductionReason": "first|angle-change|max-gap|flush",
|
||||
"evolvFieldKey": "pressure.measured.upstream.PT-1",
|
||||
"evolvStreamKey": "P-1:pressure:measured:upstream:PT-1",
|
||||
"sourceMeasurement": "Pump A"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Reducer Decisions So Far
|
||||
|
||||
- Reducer runs independently per Datastream.
|
||||
- 2D vector means time on X and numeric field value on Y.
|
||||
- On direction change, emit the previous point.
|
||||
- First point of a stream is kept.
|
||||
- Previous point is kept on angle change.
|
||||
- Pending latest point is emitted on explicit flush, max gap, or close.
|
||||
- No forward-fill.
|
||||
|
||||
Pending queue during unresolved metadata/FROST downtime:
|
||||
|
||||
```text
|
||||
queue empty -> store observation
|
||||
queue has 1 -> keep first, append latest
|
||||
queue has 2 -> keep first, replace second with latest
|
||||
```
|
||||
|
||||
## Open Interview Questions
|
||||
|
||||
## Implementation Progress 2026-05-21
|
||||
|
||||
First coding pass added:
|
||||
|
||||
- New Node-RED node: `nodes/coresync/coresync.js` / `coresync.html`.
|
||||
- New `frost` dbase formatter in `generalFunctions`.
|
||||
- Root Node-RED registration: `package.json` -> `coresync`.
|
||||
- Focused tests: `nodes/coresync/test/basic/coresync.basic.test.js`.
|
||||
- FROST request builder for lazy lookup/create and nested Observation writes.
|
||||
- Per-stream normalized-angle reducer, defaulting to:
|
||||
- `angleToleranceDeg = 5`
|
||||
- `timeScaleMs = 60000`
|
||||
- `maxGapMs = 300000`
|
||||
- keep first + latest pending queue
|
||||
- Minimal response state machine:
|
||||
- `GET lookup`
|
||||
- `POST create if missing`
|
||||
- cache returned `@iot.id`
|
||||
- drain pending Observations once Datastream and FOI ids are known
|
||||
|
||||
Validation run:
|
||||
|
||||
```text
|
||||
npx jest nodes/coresync/test/basic/coresync.basic.test.js --runInBand
|
||||
PASS, 4 tests
|
||||
|
||||
npx eslint nodes/coresync/**/*.js nodes/generalFunctions/src/helper/formatters/frostFormatter.js nodes/generalFunctions/src/helper/formatters/index.js
|
||||
PASS
|
||||
```
|
||||
|
||||
Full `npm run lint` still fails on pre-existing unrelated repo issues, mostly browser globals in editor scripts and older lint findings.
|
||||
|
||||
Q20 decision implemented: default to normalized angle comparison. Relative slope mode is present as an advanced option in the reducer and editor config.
|
||||
|
||||
Q21 decision implemented: first defaults are the candidate defaults from this document, with per-type value-scale defaults in the CoreSync domain and fallback scale `1`.
|
||||
|
||||
Q22 decision implemented: explicit `msg.topic = "coresync.flush"`, `maxGapMs`, and close flush are supported. No periodic flush timer was added.
|
||||
|
||||
Q23 decision implemented: lazy resolver order is:
|
||||
|
||||
```text
|
||||
Thing
|
||||
ObservedProperty
|
||||
Sensor
|
||||
FeatureOfInterest
|
||||
Datastream
|
||||
Observation
|
||||
```
|
||||
|
||||
Each metadata entity uses lookup/create only. PATCH drift correction is not in this pass.
|
||||
|
||||
Q24 decision implemented in editor defaults:
|
||||
|
||||
```text
|
||||
frostBaseUrl
|
||||
serviceVersion
|
||||
assetTagOverride
|
||||
sensorTagOverride
|
||||
comparisonMode
|
||||
angleToleranceDeg
|
||||
timeScaleMs
|
||||
maxGapMs
|
||||
minDeltaTimeMs
|
||||
minDeltaValue
|
||||
maxQueuedObservationsPerStream
|
||||
diagnosticsEnabled
|
||||
```
|
||||
|
||||
Q25 decision implemented: emitted request messages are plain Node-RED HTTP-compatible messages and preserve `requestId` / `_coreSync` correlation fields.
|
||||
|
||||
Q26 decision implemented: id cache is runtime-only.
|
||||
|
||||
Q27 partial: failed metadata responses emit a process diagnostic and clear in-flight metadata. Backoff timing is not implemented yet.
|
||||
|
||||
Q28 implemented scope: skeleton, normalizer, reducer, FROST request builder, and minimal response state machine.
|
||||
|
||||
### Q20. Reducer comparison method
|
||||
|
||||
Should slope change be compared by:
|
||||
|
||||
- angle in degrees, using normalized axes, or
|
||||
- relative slope delta?
|
||||
|
||||
Recommended: default to normalized angle comparison and keep relative slope as an optional advanced mode.
|
||||
|
||||
### Q21. Reducer defaults
|
||||
|
||||
What should the first defaults be?
|
||||
|
||||
Candidate defaults:
|
||||
|
||||
```text
|
||||
angleToleranceDeg = 5
|
||||
timeScaleMs = 60000
|
||||
valueScaleMode = auto
|
||||
minDeltaTimeMs = 0
|
||||
minDeltaValue = 0
|
||||
maxGapMs = 300000
|
||||
```
|
||||
|
||||
Need decide whether `valueScale` is:
|
||||
|
||||
- configured per observed property/unit,
|
||||
- auto-learned per stream,
|
||||
- fixed to `1`.
|
||||
|
||||
Recommended: configured defaults by type, with auto fallback.
|
||||
|
||||
### Q22. Flush behavior
|
||||
|
||||
When should pending latest points flush?
|
||||
|
||||
Options:
|
||||
|
||||
- on node close only,
|
||||
- on explicit `msg.topic = "coresync.flush"`,
|
||||
- on `maxGapMs`,
|
||||
- on periodic flush timer.
|
||||
|
||||
Recommended: support explicit flush and `maxGapMs`; avoid periodic flush unless needed.
|
||||
|
||||
### Q23. Metadata bootstrap order
|
||||
|
||||
What exact lazy resolver chain should the first implementation use?
|
||||
|
||||
Candidate:
|
||||
|
||||
```text
|
||||
Thing
|
||||
ObservedProperty
|
||||
Sensor
|
||||
FeatureOfInterest
|
||||
Datastream
|
||||
Observation
|
||||
```
|
||||
|
||||
Need decide whether each entity is `GET lookup -> POST create if missing`, and whether PATCH metadata drift is included in v1.
|
||||
|
||||
Recommended v1: lookup/create only; no PATCH drift correction yet.
|
||||
|
||||
### Q24. FROST base URL config
|
||||
|
||||
What config fields belong on the CoreSync node?
|
||||
|
||||
Candidate:
|
||||
|
||||
```text
|
||||
frostBaseUrl
|
||||
serviceVersion = v1.1
|
||||
dbaseFormat = frost
|
||||
assetTagOverride
|
||||
sensorTagOverride
|
||||
angleToleranceDeg
|
||||
timeScaleMs
|
||||
maxGapMs
|
||||
maxQueuedObservationsPerStream
|
||||
diagnosticsEnabled
|
||||
```
|
||||
|
||||
Need decide which are required for v1 editor UI.
|
||||
|
||||
### Q25. HTTP node compatibility
|
||||
|
||||
Do we require a wrapper function around Node-RED HTTP request to preserve `_coreSync` and `requestId`, or should CoreSync emit messages exactly in the shape the HTTP node preserves by default?
|
||||
|
||||
Recommended: design emitted messages to survive the standard HTTP node, then add a helper/example flow if needed.
|
||||
|
||||
### Q26. Local id cache persistence
|
||||
|
||||
Should resolved FROST ids be runtime-only, or persisted in Node-RED context?
|
||||
|
||||
Recommended v1: runtime-only cache, because metadata lookup is lazy and deterministic. Add persistent context later if lookups become expensive.
|
||||
|
||||
### Q27. Error handling policy
|
||||
|
||||
For failed FROST responses, should the stream:
|
||||
|
||||
- retry immediately,
|
||||
- back off,
|
||||
- mark unresolved and keep first/latest pending,
|
||||
- drop until manual reset?
|
||||
|
||||
Recommended: exponential-ish backoff per stream plus keep first/latest pending.
|
||||
|
||||
### Q28. First implementation scope
|
||||
|
||||
Should the first coding pass create only the node skeleton plus reducer tests, or include the lazy FROST resolver end-to-end?
|
||||
|
||||
Recommended: implement skeleton, normalizer, reducer, and FROST request builder together; keep HTTP response state machine minimal but functional.
|
||||
Reference in New Issue
Block a user