EVOLV/CORESYNC_FROST_INTERVIEW_HANDOFF.md

# CoreSync FROST Interview Handoff

Date: 2026-05-19

## Continue Here First

Resume the interview at **Question 20**. The last open design topic was the reducer comparison method:

**Q20. Should slope change be compared by angle in degrees or by relative slope delta?**

Recommended direction before pausing:

- Support both eventually.
- Default to angle comparison with normalized time/value axes.
- Compute `dx = deltaTimeMs / timeScaleMs`.
- Compute `dy = deltaValue / valueScale`.
- Compare `atan2(dy, dx)` direction changes against `angleToleranceDeg`.

## Agreed Decisions

- Use FROST/SensorThings instead of direct InfluxDB for the new CoreSync path.
- Keep EVOLV standard outputs:
  - `process`
  - `dbase`
  - `parent`
- Add a `dbase` output format option for `frost`.
- `dbase = frost` emits FROST-ready HTTP request messages.
- The CoreSync node does not post directly to FROST in the first version.
- A normal Node-RED HTTP request node sends the FROST messages.
- HTTP responses feed back into the same CoreSync input with `msg.topic = "frost.response"`.
- All FROST metadata lookup/create/patch requests leave on `dbase`, not `process`.
- `process` is reserved for functional process data and optional functional state.
- The resolver is lazy: streams are resolved only when telemetry arrives.
- Pending queue policy for unresolved/FROST-down streams is keep first + latest, drop middle.
- Observation writes use nested Datastream endpoints:
  - `POST /v1.1/Datastreams({datastreamId})/Observations`
- Preserve provenance in Observation `parameters`.
- On angle/slope change, emit the previous point as the knot.
- Do not forward-fill delta-compressed fields.
- Latest values are queried per Datastream:
  - `/Datastreams(id)/Observations?$orderby=phenomenonTime desc&$top=1`

## SensorThings Mapping

- EVOLV asset/apparatus/node -> FROST `Thing`
- EVOLV field `type` -> FROST `ObservedProperty`
- EVOLV `variant` (`measured`, `predicted`, `setpoint`) -> FROST `Sensor`
- EVOLV `position` -> stable FROST `FeatureOfInterest`
- EVOLV numeric field -> one FROST `Datastream`
- One reducer-kept knot -> one FROST `Observation`

Stable FOI convention:

```text
{thingId}:upstream
{thingId}:atEquipment
{thingId}:downstream
```

Also copy position into `Datastream.properties.position` for filtering.

## Units

Use EVOLV canonical ingest units. UI conversion happens client-side.

- pressure: `Pa`
- flow: `m3/s`
- power: `W`
- temperature: `K`
- density: `kg/m3`
- level: `m`
- volume: `m3`
- control / percentage / efficiency: normalized ratio `1`

No leading zeros in engineering tags:

```text
P-1
PT-1
FT-9999999
```

Never:

```text
P-001
PT-0001
```

## Identity And Registry

- Node-RED is not the source of truth for asset identity.
- Future central asset registry owns tag allocation and duplicate detection.
- Use one central counter per tag prefix:
  - `P`
  - `PT`
  - `FT`
  - `TT`
  - etc.
- The central registry, not local Node-RED, performs atomic `+1`.
- For now, assume the central registry is future work.
- First implementation derives identity when possible and allows overrides.
- Keep a boundary like `resolveIdentity(input)` so future registry integration is straightforward.

First-version identity behavior:

```text
Thing tag: configured/derived, e.g. P-1
Sensor tag: configured/derived, e.g. PT-1, MODEL-P-1, CTRL-P-1
Stream key: thingTag:type:variant:position:sensorTag
```

## Shared Collector Model

Use one shared CoreSync per FROST target/stack level.

Many EVOLV nodes can connect their `dbase` output to the CoreSync input, assuming payloads are structured as:

```js
{
  measurement: "P-1",
  fields: {
    "pressure.measured.upstream.PT-1": 12345
  },
  tags: {
    tagcode: "P-1"
  },
  timestamp: Date
}
```

Also accept arrays of such payloads.

Internal stream key:

```text
thingTag:type:variant:position:sensorTag
```

Per-stream state:

- FROST id cache
- latest FROST `phenomenonTime`
- reducer anchor point
- reducer previous point
- pending latest point
- bounded pending queue

## FROST Request Message Shape

Outgoing request messages should preserve correlation metadata:

```js
{
  topic: "frost.metadata.lookup",
  requestId: "thing:P-1:lookup",
  _coreSync: {
    kind: "thing",
    action: "lookup",
    externalKey: "thing:P-1",
    streamKey: "P-1:pressure:measured:upstream:PT-1"
  },
  method: "GET",
  url: "...",
  payload: null
}
```

FROST response feedback:

```js
{
  topic: "frost.response",
  requestId: "...",
  statusCode: 200,
  payload: {},
  _coreSync: {}
}
```

Observation write target:

```http
POST /v1.1/Datastreams({datastreamId})/Observations
```

Observation payload:

```json
{
  "phenomenonTime": "2026-05-19T10:15:30.000Z",
  "result": 123.4,
  "FeatureOfInterest": {
    "@iot.id": 7
  },
  "parameters": {
    "reduction": "knot",
    "reductionReason": "first|angle-change|max-gap|flush",
    "evolvFieldKey": "pressure.measured.upstream.PT-1",
    "evolvStreamKey": "P-1:pressure:measured:upstream:PT-1",
    "sourceMeasurement": "Pump A"
  }
}
```

## Reducer Decisions So Far

- Reducer runs independently per Datastream.
- 2D vector means time on X and numeric field value on Y.
- On direction change, emit the previous point.
- First point of a stream is kept.
- Previous point is kept on angle change.
- Pending latest point is emitted on explicit flush, max gap, or close.
- No forward-fill.

Pending queue during unresolved metadata/FROST downtime:

```text
queue empty -> store observation
queue has 1 -> keep first, append latest
queue has 2 -> keep first, replace second with latest
```

## Open Interview Questions

## Implementation Progress 2026-05-21

First coding pass added:

- New Node-RED node: `nodes/coresync/coresync.js` / `coresync.html`.
- New `frost` dbase formatter in `generalFunctions`.
- Root Node-RED registration: `package.json` -> `coresync`.
- Focused tests: `nodes/coresync/test/basic/coresync.basic.test.js`.
- FROST request builder for lazy lookup/create and nested Observation writes.
- Per-stream normalized-angle reducer, defaulting to:
  - `angleToleranceDeg = 5`
  - `timeScaleMs = 60000`
  - `maxGapMs = 300000`
  - keep first + latest pending queue
- Minimal response state machine:
  - `GET lookup`
  - `POST create if missing`
  - cache returned `@iot.id`
  - drain pending Observations once Datastream and FOI ids are known

Validation run:

```text
npx jest nodes/coresync/test/basic/coresync.basic.test.js --runInBand
PASS, 4 tests

npx eslint nodes/coresync/**/*.js nodes/generalFunctions/src/helper/formatters/frostFormatter.js nodes/generalFunctions/src/helper/formatters/index.js
PASS
```

Full `npm run lint` still fails on pre-existing unrelated repo issues, mostly browser globals in editor scripts and older lint findings.

Q20 decision implemented: default to normalized angle comparison. Relative slope mode is present as an advanced option in the reducer and editor config.

Q21 decision implemented: first defaults are the candidate defaults from this document, with per-type value-scale defaults in the CoreSync domain and fallback scale `1`.

Q22 decision implemented: explicit `msg.topic = "coresync.flush"`, `maxGapMs`, and close flush are supported. No periodic flush timer was added.

Q23 decision implemented: lazy resolver order is:

```text
Thing
ObservedProperty
Sensor
FeatureOfInterest
Datastream
Observation
```

Each metadata entity uses lookup/create only. PATCH drift correction is not in this pass.

Q24 decision implemented in editor defaults:

```text
frostBaseUrl
serviceVersion
assetTagOverride
sensorTagOverride
comparisonMode
angleToleranceDeg
timeScaleMs
maxGapMs
minDeltaTimeMs
minDeltaValue
maxQueuedObservationsPerStream
diagnosticsEnabled
```

Q25 decision implemented: emitted request messages are plain Node-RED HTTP-compatible messages and preserve `requestId` / `_coreSync` correlation fields.

Q26 decision implemented: id cache is runtime-only.

Q27 partial: failed metadata responses emit a process diagnostic and clear in-flight metadata. Backoff timing is not implemented yet.

Q28 implemented scope: skeleton, normalizer, reducer, FROST request builder, and minimal response state machine.

### Q20. Reducer comparison method

Should slope change be compared by:

- angle in degrees, using normalized axes, or
- relative slope delta?

Recommended: default to normalized angle comparison and keep relative slope as an optional advanced mode.

### Q21. Reducer defaults

What should the first defaults be?

Candidate defaults:

```text
angleToleranceDeg = 5
timeScaleMs = 60000
valueScaleMode = auto
minDeltaTimeMs = 0
minDeltaValue = 0
maxGapMs = 300000
```

Need decide whether `valueScale` is:

- configured per observed property/unit,
- auto-learned per stream,
- fixed to `1`.

Recommended: configured defaults by type, with auto fallback.

### Q22. Flush behavior

When should pending latest points flush?

Options:

- on node close only,
- on explicit `msg.topic = "coresync.flush"`,
- on `maxGapMs`,
- on periodic flush timer.

Recommended: support explicit flush and `maxGapMs`; avoid periodic flush unless needed.

### Q23. Metadata bootstrap order

What exact lazy resolver chain should the first implementation use?

Candidate:

```text
Thing
ObservedProperty
Sensor
FeatureOfInterest
Datastream
Observation
```

Need decide whether each entity is `GET lookup -> POST create if missing`, and whether PATCH metadata drift is included in v1.

Recommended v1: lookup/create only; no PATCH drift correction yet.

### Q24. FROST base URL config

What config fields belong on the CoreSync node?

Candidate:

```text
frostBaseUrl
serviceVersion = v1.1
dbaseFormat = frost
assetTagOverride
sensorTagOverride
angleToleranceDeg
timeScaleMs
maxGapMs
maxQueuedObservationsPerStream
diagnosticsEnabled
```

Need decide which are required for v1 editor UI.

### Q25. HTTP node compatibility

Do we require a wrapper function around Node-RED HTTP request to preserve `_coreSync` and `requestId`, or should CoreSync emit messages exactly in the shape the HTTP node preserves by default?

Recommended: design emitted messages to survive the standard HTTP node, then add a helper/example flow if needed.

### Q26. Local id cache persistence

Should resolved FROST ids be runtime-only, or persisted in Node-RED context?

Recommended v1: runtime-only cache, because metadata lookup is lazy and deterministic. Add persistent context later if lookups become expensive.

### Q27. Error handling policy

For failed FROST responses, should the stream:

- retry immediately,
- back off,
- mark unresolved and keep first/latest pending,
- drop until manual reset?

Recommended: exponential-ish backoff per stream plus keep first/latest pending.

### Q28. First implementation scope

Should the first coding pass create only the node skeleton plus reducer tests, or include the lazy FROST resolver end-to-end?

Recommended: implement skeleton, normalizer, reducer, and FROST request builder together; keep HTTP response state machine minimal but functional.