Skip to content

Commit 97ea578

Browse files
Ingvordclaude
Ingvord
and
claude
committed
Step 7+: retire AllRecords, add service metrics, fix double polling, move observability
- Remove AllRecords/SingleRecords and their tests (snapshot-only model) - Remove DataStorage layer (Engine now wires directly to InMemoryWriter) - Fix Engine.start(): remove duplicate CompletableFuture.runAsync(task) — polling was firing twice per attribute on startup - MetricsServer: add status_server_monitored/up/failed_attributes summary gauges - Merge prom-grafana-status-server into observability/ subdirectory; update docker-compose.yml with prometheus + grafana services - Refresh both Grafana dashboards: Health (stats + state/status + failed table) and Live Values (snapshot table join + timeseries drilldown) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent f434776 commit 97ea578

15 files changed

Lines changed: 946 additions & 644 deletions

File tree

‎README.md‎

Lines changed: 135 additions & 152 deletions
Original file line numberDiff line numberDiff line change
@@ -1,216 +1,199 @@
11
# StatusServer
22

3+
A non-disturbing data collector for the [X-Environment](http://www.github.com/waltz-controls/xenv) (Integrated Control System for High-throughput Tomography). It aggregates control system data from Tango and/or TINE servers, tracks attribute availability, persists downtime intervals to MariaDB, and exposes live metrics via HTTP/Prometheus.
34

4-
[![codecov](https://codecov.io/gh/hzg-wpi/status-server/branch/master/graph/badge.svg)](https://codecov.io/gh/hzg-wpi/status-server)
5-
[![codebeat badge](https://codebeat.co/badges/a646871c-9772-48bd-91b3-0edee8134aa2)](https://codebeat.co/projects/github-com-xenvhzg-status-server-master)
5+
## Requirements
66

7-
[![Download](https://img.shields.io/github/release/hzg-wpi/status-server.svg?style=flat)](https://github.com/hzg-wpi/status-server/releases/latest)
7+
- Java 21
8+
- Maven 3.8+
9+
- Tango 9+ (for Tango attributes) / TINE (for TINE attributes)
10+
- MariaDB 11 (optional — required for downtime persistence)
811

9-
This project is a part of [X-Environment](http://www.github.com/waltz-controls/xenv) (Integrated Control System for High-throughput Tomography experiments). X-Environment is a bunch of components that server two main goals:
12+
## Quick Start
1013

11-
* Collect data during the High throughput Tomography experiment in a non-disturbing way (does not disturb experiment)
12-
* Provide high level abstraction for beamline scientist to control the experiment
14+
```bash
15+
# Build fat JAR
16+
mvn clean package -Dmaven.test.skip=true
1317

14-
This Tango server corresponds to the first goal. It aggregates data from upstream [Tango](http://www.tango-controls.org) and/or [TINE](http://adweb.desy.de/mcs/tine/) servers it acts as an intermediate buffer for the collected data.
18+
# Start MariaDB (and optional Tango stack)
19+
docker compose up -d status-server-db
1520

16-
# Requirements
17-
18-
* Java 11 server environment
19-
* Tango 9+
21+
# Run
22+
java -jar target/status-server-*.jar path/to/config.xml [http-port]
23+
# http-port defaults to 9190
24+
```
2025

21-
# User guide
26+
## Configuration
2227

23-
[![Documentation Status](https://readthedocs.org/projects/status-server/badge/?version=latest)](http://status-server.readthedocs.io/en/latest/?badge=latest)
28+
Configuration is an XML file. Minimal example:
2429

25-
# How To ...
30+
```xml
31+
<status-server stale-after="3" down-after="6">
32+
<devices>
33+
<device name="my/device/1" server="tango://host:10000">
34+
<attributes>
35+
<attribute name="Temperature" poll-delay="1000" interpolation="LINEAR"/>
36+
</attributes>
37+
</device>
38+
</devices>
39+
</status-server>
40+
```
2641

27-
## ... resolve dependencies (ver > 1.0.0)
28-
goto {SS_ROOT}/devkit and execute install.bat(.sh)
42+
Optional MariaDB section (omit to disable persistence):
2943

30-
for older versions manually download and install tine and TangoAPI jars.
44+
```xml
45+
<mariadb>
46+
<jdbc-url>jdbc:mariadb://localhost:3306/statusserver</jdbc-url>
47+
<user>ss</user>
48+
<password>ss</password>
49+
</mariadb>
50+
```
3151

32-
## ... build from source
52+
| Parameter | Description | Default |
53+
|---|---|---|
54+
| `stale-after` | Consecutive failures before UP→STALE | 3 |
55+
| `down-after` | Consecutive failures before STALE→DOWN | 6 |
3356

34-
There are still some tests that are strictly environment depended (e.g. TangoClientTest) to build the project simply turn off tests in maven: ```mvn package -Dmaven.test.skip=true```
57+
## Architecture
3558

36-
## ... run StatusServer without Tango environment (ver < 1.0.0)
59+
```
60+
DeviceSource (XML)
61+
│
62+
▼
63+
EngineFactory ──► Engine
64+
│
65+
┌─────────┴──────────┐
66+
▼ ▼
67+
PollTask EventTask
68+
│ │
69+
▼ ▼
70+
EventSink<SingleRecord<?>> EventSink<TechnicalEvent>
71+
(telemetry) (AvailabilityAnalyzer)
72+
│ │
73+
▼ ▼
74+
InMemoryWriter EventSink<DomainEvent>
75+
(Snapshot only) (EventDispatcher fan-out)
76+
│ │
77+
▼ ┌────┴────┐
78+
MetricsServer logger MariaDbSink
79+
(HTTP /metrics)
80+
```
3781

38-
As the for version 1.8 there is no way to run StatusServer without Tango installed. At least programmatically. But it is possible to adjust code so it will start:
82+
### Key components
3983

40-
1. change the code of the Launcher.java (see below)
41-
2. change the code of the StatusServerClass.java (see below)
42-
3. execute {{{mvn clean package}}}
43-
4. deploy
44-
5. run with additional JVM parameter: -DOAPort=56234 (see below full command)
84+
**`EventSink<T>`** — unified observer interface replacing the old `RecordWriter` and `TechnicalEventListener`. Everything that consumes events implements this single generic interface.
4585

46-
In Launcher.java change main method:
86+
**`EventDispatcher<T>`** — fan-out dispatcher. Calls all registered sinks, isolates failures per sink.
4787

48-
```
49-
#!java
50-
Launcher.java
51-
public static void main(String[] args) {
52-
...
53-
log.info("Initializing Tango framework...");
54-
Util util = Util.init(new String[]{configuration.getInstanceName(),"-nodb","-dlist","development/local/0"}, configuration.getServerName());
55-
util.add_class(configuration.getServerName());
56-
Util.set_serial_model(TangoConst.NO_SYNC);
57-
log.info("Done.");
58-
...
59-
}
60-
```
88+
**`AvailabilityAnalyzer`** — consumes `TechnicalEvent`s, maintains per-attribute state machines (UP/STALE/DOWN), emits `DomainEvent`s.
6189

62-
In StatusServerClass.java comment ```write_class_property``` and ```get_class_property``` methods call in the constructor (```StatusServerClass(String s)```)
90+
**`AttributeAvailability`** — per-attribute state machine:
6391

6492
```
65-
#!java
66-
StatusServerClass.java
67-
...
68-
protected StatusServerClass(String s) throws DevFailed {
69-
super(s);
70-
71-
// write_class_property();
72-
// get_class_property();
73-
}
74-
...
93+
consecutive failures ≥ stale-after → UP → STALE
94+
consecutive failures ≥ down-after → STALE → DOWN (+ DowntimeOpened)
95+
any success → any → UP (+ DowntimeClosed if from DOWN)
7596
```
7697

77-
To run StatusServer use the following command:
78-
```
79-
java -DXmx1G -Dss.home={SS_HOME} -DOAPort=56234 wpn.hdri.ss.Launcher --config {PATH_TO_CONFIG}
80-
```
98+
**`InMemoryWriter`** — snapshot-only in-memory store backing the `/metrics` endpoint.
8199

82-
# Benchmark tests
100+
**`MariaDbSink`** — persists domain events to MariaDB in ERPNext-compatible tables. Reconnects automatically on failure.
83101

84-
## Server and client are running on the same machine.
102+
## HTTP Endpoints
85103

86-
Machine configuration:
87-
* Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz
88-
* RAM 8Gb
89-
* RAM&CPU clock //TODO
90-
* java version "1.6.0_29"
91-
* Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
92-
* Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
104+
| Endpoint | Description |
105+
|---|---|
106+
| `GET /metrics` | Prometheus gauge format. Each attribute emits a value gauge and `_up` (1=healthy, 0=failing). |
107+
| `GET /health` | Liveness — always 200. |
108+
| `GET /ready` | Readiness — 503 until engine has started. |
93109

94-
The application started in -server mode
110+
## Availability Tracking & Downtime Persistence
95111

96-
100K getLatestSnapshot were performed to warm up the JVM
112+
StatusServer classifies each read outcome as a technical event:
97113

98-
Results measured on 10K invocations of getLatestSnapshot:
114+
| Event | Trigger |
115+
|---|---|
116+
| `ReadSuccess` | Successful attribute read |
117+
| `ReadFailure` | Client exception during read |
118+
| `Timeout` | Read timed out |
119+
| `Disconnect` / `Reconnect` | Connection lost / restored |
99120

100-
```
101-
Delta time in getLatestValues (nano) = 11746809173
102-
Delta time in getLatestValues (millis) = 11746
103-
Average time in getLatestValues (nano) = 1174680
104-
Average time in getLatestValues (millis) = 1
105-
Average time in getLatestValues (seconds) = 0
106-
```
121+
The `AvailabilityAnalyzer` aggregates these per attribute and emits domain events when thresholds are crossed:
107122

108-
## Server and client are running on different machines connected with 1Gbit network.
123+
| Domain Event | Meaning |
124+
|---|---|
125+
| `AvailabilityTransitioned` | State changed (UP↔STALE↔DOWN) |
126+
| `DowntimeOpened` | Attribute entered DOWN state |
127+
| `DowntimeClosed` | Attribute recovered from DOWN |
109128

110-
Server machine configuration:
129+
### MariaDB Schema
111130

112-
* Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
113-
* RAM 48Gb
114-
* RAM&CPU clock //TODO
115-
* java version "1.6.0_33"
116-
* Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
117-
* Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
131+
Three ERPNext-compatible tables (standard `tab{DocType}` naming, standard audit columns):
118132

119-
The application started in -server mode.
133+
```
134+
tabState Transition — full history of every state change
135+
tabCurrent State — one row per attribute, UPSERT on every transition
136+
tabDowntime Interval — one open row per active downtime, closed on recovery
137+
```
120138

121-
100K getLatestSnapshot were performed to warm up the JVM
139+
Apply schema to a fresh database:
122140

123-
Results measured on 10K invocations of getLatestSnapshot:
141+
```bash
142+
# Fresh container (wipes existing data)
143+
docker compose down -v
144+
docker compose up -d status-server-db
124145

125-
```
126-
Delta time in getLatestValues (nano) = 71063988508
127-
Delta time in getLatestValues (millis) = 71063
128-
Average time in getLatestValues (nano) = 7106398
129-
Average time in getLatestValues (millis) = 7
130-
Average time in getLatestValues (seconds) = 0
146+
# Or apply manually to a running container
147+
docker exec -i status-server-db mariadb -u ss -pss statusserver < db/schema.sql
131148
```
132149

133-
Ping:
134-
```
135-
Pinging 131.169.65.240 with 32 bytes of data:
136-
Reply from 131.169.65.240: bytes=32 time<1ms TTL=64
137-
Reply from 131.169.65.240: bytes=32 time<1ms TTL=64
138-
Reply from 131.169.65.240: bytes=32 time<1ms TTL=64
139-
Reply from 131.169.65.240: bytes=32 time<1ms TTL=64
140-
141-
Ping statistics for 131.169.65.240:
142-
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
143-
Approximate round trip times in milli-seconds:
144-
Minimum = 0ms, Maximum = 0ms, Average = 0ms
145-
```
150+
On startup, if MariaDB is configured, StatusServer reads `tabCurrent State` and seeds the in-memory state machines — attributes that were DOWN when the server last stopped resume tracking correctly.
146151

152+
## Development
147153

148-
## 1.0.10 Benchmark test:
154+
```bash
155+
# Build (skip environment-dependent tests)
156+
mvn clean package -Dmaven.test.skip=true
149157

150-
The test was performed on a single host.
158+
# Run unit tests (no Tango/TINE required)
159+
mvn test -Dtest="AvailabilityAnalyzerTest,StatusServerStatusServerConfigurationTest"
151160

161+
# Run all tests (requires live Tango)
162+
mvn test
152163
```
153-
Result "test_getLatestSnapshot":
154-
1042.600 ±(99.9%) 2.616 ops/s [Average]
155-
(min, avg, max) = (575.350, 1042.600, 1724.124), stdev = 74.460
156-
CI (99.9%): [1039.984, 1045.216] (assumes normal distribution)
157164

165+
Safe unit tests (no external dependencies): `data2/`, `configuration/`, `engine2/AvailabilityAnalyzerTest`.
158166

159-
# Run complete. Total time: 02:36:29
167+
## Architecture Decision Records
160168

161-
Benchmark Mode Cnt Score Error Units
162-
tango.ss.benchmark.SimpleBenchmark.test_getLatestSnapshot thrpt 8780 1042.600 ± 2.616 ops/s
163-
```
169+
### ADR-1: XML-only device source
164170

165-
## 1.3.2 Benchmark test:
171+
Frappe/ERPNext is used as a configuration front-end. It exports device/attribute lists as XML files. StatusServer reads that XML directly. There is no runtime Frappe dependency — this removes the HTTP round-trip on startup and makes the server runnable without an ERPNext instance.
166172

173+
### ADR-2: Unified `EventSink<T>` interface
167174

168-
One attribute 100K records
175+
`RecordWriter` (telemetry) and `TechnicalEventListener` (availability signals) were merged into a single `EventSink<T>` functional interface. Rationale: a writer *is* a listener — it reacts to events. The generic parameter carries the event type, keeping the type system honest while eliminating the duplicate observer hierarchies.
169176

170-
```
171-
# Run complete. Total time: 00:13:55
177+
### ADR-3: `EventDispatcher<T>` replaces `WriterDispatcher`
172178

173-
Benchmark Mode Cnt Score Error Units
174-
SearchBenchmark.benchmarkLatest thrpt 200 8751053,333 ? 32440,027 ops/s
175-
SearchBenchmark.benchmarkSnapshot thrpt 200 3256768,953 ? 14576,755 ops/s
176-
SearchBenchmark.benchmarkRange thrpt 200 1290584,274 ? 19983,747 ops/s
179+
A single generic fan-out dispatcher replaces the telemetry-specific `WriterDispatcher`. Both the telemetry pipeline (`SingleRecord<?>`) and the domain event pipeline (`DomainEvent`) use the same implementation. Failures in one sink are logged and isolated; other sinks always receive the event.
177180

178-
```
181+
### ADR-4: Snapshot-only in-memory store
179182

180-
100 attributes with 100K randomly distributed records
183+
`AllRecords` (full time-series history) was retired. The in-memory store now keeps only the latest value per attribute (`Snapshot`). Rationale: historical queries are served by MariaDB; keeping a second growing in-memory copy adds memory pressure with no remaining consumer.
181184

182-
```
183-
# Run complete. Total time: 00:27:19
185+
### ADR-5: Global availability thresholds
184186

185-
Benchmark Mode Cnt Score Error Units
186-
SearchBenchmark.benchmarkLatest thrpt 200 120810,181 ? 884,829 ops/s
187-
SearchBenchmark.benchmarkRange thrpt 200 54811,091 ? 875,401 ops/s
188-
SearchBenchmark.benchmarkSnapshot thrpt 200 36707,332 ? 228,565 ops/s
189-
SearchBenchmark.benchmarkUpdates thrpt 200 36955,929 ? 196,897 ops/s
190-
```
191-
192-
## 2.0.0 Benchmark test:
187+
`stale-after` and `down-after` are global values configured at the server level, not per attribute. Rationale: in this deployment all attributes are polled at similar rates; per-attribute thresholds add configuration complexity without practical benefit.
193188

194-
One attribute 100K records
189+
### ADR-6: ERPNext-compatible MariaDB schema
195190

196-
```
197-
# Run complete. Total time: 00:21:54
191+
Tables follow ERPNext naming (`tab{DocType}`) and include the standard Frappe audit columns (`name`, `creation`, `modified`, `modified_by`, `owner`, `docstatus`, `idx`). Rationale: rows can be imported into or consumed by a Frappe/ERPNext instance without transformation, enabling ERP-level downtime reporting and billing workflows.
198192

199-
Benchmark Mode Cnt Score Error Units
200-
SearchBenchmark2.benchmarkLatest thrpt 200 376546078,535 ? 1343521,015 ops/s
201-
SearchBenchmark2.benchmarkRange thrpt 200 18960847,908 ? 60166,410 ops/s
202-
SearchBenchmark2.benchmarkSnapshot thrpt 200 6079969,349 ? 71723,323 ops/s
203-
SearchBenchmark2.benchmarkUpdates thrpt 200 36036591,098 ? 164357,240 ops/s
204-
```
193+
### ADR-7: No connection pool (plain JDBC with auto-reconnect)
205194

206-
100 attributes with 100K randomly distributed records
195+
`MariaDbSink` uses a single JDBC connection with `isValid()` check before each use and silent reconnect on failure. Rationale: domain events are low-frequency (state changes, not every poll); a full connection pool (HikariCP etc.) adds a dependency and warm-up complexity for negligible benefit at this throughput.
207196

208-
```
209-
# Run complete. Total time: 00:27:07
197+
### ADR-8: Java 21 virtual threads
210198

211-
Benchmark Mode Cnt Score Error Units
212-
SearchBenchmark3.benchmarkLatest thrpt 200 368993808,387 ? 1728088,058 ops/s
213-
SearchBenchmark3.benchmarkRange thrpt 200 18638307,115 ? 67312,940 ops/s
214-
SearchBenchmark3.benchmarkSnapshot thrpt 200 1785893,066 ? 23765,739 ops/s
215-
SearchBenchmark3.benchmarkUpdates thrpt 200 36079956,416 ? 133265,984 ops/s
216-
```
199+
The HTTP server and engine polling tasks use `Thread.ofVirtual()`. This allows a large number of concurrent blocking I/O operations (Tango/TINE reads) without the overhead of a large platform thread pool.

‎docker-compose.yml‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ networks:
55

66
volumes:
77
mariadb_data:
8+
prometheus_data:
9+
grafana_data:
810

911
services:
1012
status-server-db:
@@ -109,4 +111,34 @@ services:
109111
timeout: 1s
110112
retries: 3
111113
command:
112-
- "rest"
114+
- "rest"
115+
116+
prometheus:
117+
image: prom/prometheus:latest
118+
container_name: local-prometheus
119+
restart: unless-stopped
120+
ports:
121+
- "9090:9090"
122+
extra_hosts:
123+
- "host.docker.internal:host-gateway"
124+
volumes:
125+
- ./observability/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
126+
- prometheus_data:/prometheus
127+
command:
128+
- --config.file=/etc/prometheus/prometheus.yml
129+
- --storage.tsdb.path=/prometheus
130+
131+
grafana:
132+
image: grafana/grafana:latest
133+
container_name: local-grafana
134+
restart: unless-stopped
135+
depends_on:
136+
- prometheus
137+
ports:
138+
- "3000:3000"
139+
volumes:
140+
- grafana_data:/var/lib/grafana
141+
- ./observability/grafana/provisioning:/etc/grafana/provisioning:ro
142+
environment:
143+
GF_SECURITY_ADMIN_USER: admin
144+
GF_SECURITY_ADMIN_PASSWORD: admin

0 commit comments

Comments
 (0)