EVOLV/wiki/concepts/signal-processing-sensors.md

# Sensor Signal Conditioning & Data Quality

> **Used by**: `instrumentation-measurement` agent, `measurement` node
> **Validation**: Verified against IEC 61298, sensor manufacturer literature, and signal processing references

## Signal Conditioning Pipeline

```
Raw Signal → Scaling → Filtering → Outlier Rejection → Quality Flagging → Output
```

## Scaling: Engineering Unit Conversion

### 4-20 mA Standard
```
value = range_min + (I - 4) / (20 - 4) · (range_max - range_min)
```

Where I is the measured current in mA.

### Key Rules
- 0 mA = wire break (fault condition)
- < 4 mA = under-range or fault
- 4 mA = range minimum (0%)
- 20 mA = range maximum (100%)
- > 20 mA = over-range or fault
- NAMUR NE43 recommends 3.8 mA and 20.5 mA as fault thresholds

## Filtering Methods

### Moving Average
```
y[k] = (1/N) · Σ x[k-i]   for i = 0 to N-1
```
- Simple, effective for white noise
- Introduces phase lag proportional to (N-1)/2 samples
- Good for steady-state signals, poor for fast transients

### Exponential Moving Average (EMA)
```
y[k] = α · x[k] + (1-α) · y[k-1]
```
Where α = 2/(N+1) or α = Δt/(τ + Δt) for time-constant-based tuning.
- Less memory than moving average
- Equivalent to first-order low-pass filter
- τ (time constant) sets the cutoff frequency: f_c = 1/(2π·τ)

### Savitzky-Golay Filter
- Fits a polynomial to a window of data points, uses the polynomial value as the filtered output
- Preserves higher-order moments (peaks, edges) better than moving average
- Configurable by window size and polynomial order
- Typical: window = 5-11 points, order = 2-3

## Outlier Detection

### Z-Score Method
```
z = |x - μ| / σ
outlier if z > threshold (typically 3.0)
```
- Assumes normal distribution
- Sensitive to the outliers themselves (they inflate σ)

### Modified Z-Score (MAD-based)
```
MAD = median(|x_i - median(x)|)
modified_z = 0.6745 · (x - median(x)) / MAD
outlier if |modified_z| > threshold (typically 3.5)
```
- Robust to outliers (uses median instead of mean)
- **Recommended for process measurements** where occasional spikes are common
- 0.6745 is the 75th percentile of the standard normal distribution

### IQR Method
```
Q1 = 25th percentile, Q3 = 75th percentile
IQR = Q3 - Q1
outlier if x < Q1 - 1.5·IQR or x > Q3 + 1.5·IQR
```
- Non-parametric, no distribution assumption
- Common in exploratory data analysis
- Less suitable for real-time streaming (needs window of data)

## NRMSE for Drift Detection

Normalized Root Mean Square Error compares a recent measurement window against a reference window to detect sensor drift.

### Calculation
```
RMSE = √(Σ(x_i - x_ref_i)² / N)
NRMSE = RMSE / (x_max - x_min)   or   RMSE / x_mean
```

### Thresholds (typical for process sensors)
| NRMSE Range | Quality | Action |
|-------------|---------|--------|
| 0 – 0.05 | Good | Normal operation |
| 0.05 – 0.15 | Uncertain | Flag for review, increase monitoring |
| 0.15 – 0.30 | Poor | Alarm, reduce weight in control loops |
| > 0.30 | Bad | Remove from control, maintenance required |

### Reference Window Selection
- Calibration data (gold standard)
- Post-maintenance baseline
- Rolling reference from a known-good period
- Multi-sensor cross-validation

## Sensor Accuracy Classes

### IEC 61298 Framework
IEC 61298 "Process measurement and control devices — General methods and procedures for evaluating performance" defines standardized test methods for evaluating sensor accuracy under reference and influence conditions.

### Key Performance Metrics
- **Accuracy**: Closeness of measured value to true value (includes systematic and random errors)
- **Repeatability**: Closeness of successive measurements under identical conditions
- **Hysteresis**: Maximum difference between upscale and downscale readings
- **Linearity**: Maximum deviation from a straight line between zero and span
- **Deadband**: Smallest change in input that produces a detectable output change

### Common Accuracy Specifications
| Sensor Type | Typical Accuracy | Response Time |
|-------------|-----------------|---------------|
| Pressure transmitter | ±0.04 – 0.1% FS | < 100 ms |
| Flow meter (electromagnetic) | ±0.2 – 0.5% of reading | 1-3 s |
| Temperature (RTD/Pt100) | ±0.1 – 0.3°C | 5-30 s (depends on housing) |
| Level (ultrasonic) | ±0.25% FS | 1-5 s |
| pH | ±0.02 – 0.1 pH | 10-60 s |
| Dissolved oxygen | ±1-2% of reading | 30-90 s (membrane) |
| Turbidity (nephelometric) | ±2% of reading | 5-15 s |
| Ammonia (ion-selective) | ±5-10% of reading | 60-180 s |

## Sensor States and Warmup

### State Machine
```
Maintenance → Warmup → Active → Cooldown → Maintenance
```

### Warmup Behavior
- **Duration**: Varies by sensor type (seconds for pressure, minutes for pH, hours for dissolved oxygen)
- **During warmup**: Measurements flagged as "uncertain" quality
- **Completion criterion**: Readings stabilize within defined tolerance for a minimum duration
- **EVOLV convention**: Warmup state prevents measurements from propagating to control loops

### Stabilization Detection
```
stable if std_dev(last_N_readings) < threshold for T_stable seconds
```

## Authoritative References

1. IEC 61298 series (2008). "Process measurement and control devices — General methods and procedures for evaluating performance"
2. IEC 61326-2-3. "Electrical equipment for measurement, control and laboratory use — EMC requirements — Part 2-3: Particular requirements — Transducers with integrated or remote signal conditioning"
3. NAMUR NE43 (2003). "Standardization of the Signal Level for the Failure Information of Digital Transmitters"
4. Bently Nevada / Baker Hughes. "Fundamentals of Rotating Machinery Diagnostics"
5. Oppenheim, A.V. & Willsky, A.S. (1997). "Signals & Systems" 2nd ed., Prentice Hall
6. Press, W.H. et al. (2007). "Numerical Recipes" 3rd ed., Chapter 14 (Savitzky-Golay filters)