Skip to content

feat: Add SBE 911+ HEX reader support - #139

Merged
ysorge merged 7 commits into
mainfrom
feat/sbe911-hex-reader
Aug 10, 2026
Merged

feat: Add SBE 911+ HEX reader support#139
ysorge merged 7 commits into
mainfrom
feat/sbe911-hex-reader

Conversation

@eleanorfrajka

@eleanorfrajka eleanorfrajka commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Extends SbeHexReader to handle SBE 911plus/917plus .hex files alongside the existing SBE37 family, using a single-class family-dispatch architecture.
  • Adds detect_sbe_hex_family(), a new * key = value header parser, and an index-keyed xmlcon channel-map reader that correctly handles dual T/C and all 8 auxiliary voltage channels.
  • Implements a table-driven dataset builder using seabirdscientific's frequency converters (convert_temperature_frequency, convert_pressure_digiquartz, convert_conductivity, convert_sbe43_oxygen), fixing ctdproc's hardcoded channel wiring.
  • Constructs the time coordinate from System UpLoad Time + scan index × sample_interval (derived from Number of Scans Averaged / 24 Hz), since 911+ files carry no per-scan "date time" column.
  • Fixes 7 silent failure traps in the existing SBE37 path (zero-substitution fallbacks, swallowed calibration parse errors, first-index-wins sensor deduplication).
  • Corrects mislabelling: FluoroWetlabECO_AFL_FL_Sensor output is fluorescence, not chlorophyll.
  • Commits a trimmed real-cruise fixture (MIXSED2_000.hex, 500 scans, + .xmlcon) and adds an end-to-end integration test against it.

Variables produced for a full 911+ configuration (dual T/C, 8 aux channels)

temp, cond, press, temp2, cond2, oxygen, oxygen_ml_l, fluorescence, turbidity, altimeter, transmissometer, volt0_raw (user polynomial / UVP6), volt3_raw (pH raw voltage), nmea_latitude, nmea_longitude

Sensors declared NotInUse are skipped with a logged explanation. Sensors present in the xmlcon but without a conversion implementation store raw voltage and log a warning rather than silently disappearing.

Breaking changes

  • SbeHexReader.format_name() changes from "SeaBird SBE37 HEX" to "SeaBird SBE HEX". Any downstream code that hardcodes the old string to identify this reader will need updating.
  • Calibration-coefficient fallbacks that previously substituted zeros (temperature compensation, T/P in conductivity conversion) now raise ValueError instead. Files that previously loaded with wrong silent substitutions will now error; fix: supply the correct xmlcon.
  • SBE37SM is no longer the silent fallback for an unrecognised SBE37 device type — it now raises ValueError. The 911+ path never uses this code.

Test plan

  • pytest tests/readers/test_sbe_hex_reader.py -v — all tests pass
  • Added multiple versions of *hex/*xmlcon files to tests/readers/fixtures/ for SBE911+

Test fixtures added

All .hex files are trimmed to ~500 scans (header preserved verbatim). MIXSED2_000.hex is the primary end-to-end fixture; the others exercise specific xmlcon configurations.

File Family CTD SN Seasave Voltage words Notes
m84_3_287_short.hex 911+ 1294 7.20c 5 Old XMLCON (pre-7.22, no Coefficients[@equation] wrapper)
64PE426_000_1_short.hex 911+ 1294 7.21d 2 2 volt words suppressed
msm_021_1_168_short.hex 911+ 1294 7.21f 4
MSM72_002_2_short.hex 911+ 1717 7.22 3 1 volt word suppressed
M104_154_01_short.hex 911+ 5283 7.22.4 5 PAR + SPAR sensors
PS129_014_01_short.hex 911+ 1374 7.23.1 4 Southern Ocean (55°S)
P503_358_short.hex 911+ 1294 7.25.0.151 4
M166_025_1_short.hex 911+ 1294 7.26.7.107 2 2 volt words suppressed
m212_005_short.hex 911+ 4145 7.26.7.121 4
msm_142_1_056_short.hex 911+ 4823 7.26.7.121 4
MIXSED2_000.hex 911+ 4798 7.26.7.121 4 Primary end-to-end fixture; full cruise file

@eleanorfrajka
eleanorfrajka marked this pull request as draft August 8, 2026 21:53
@eleanorfrajka
eleanorfrajka marked this pull request as ready for review August 8, 2026 22:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Sea-Bird SBE 911plus/917plus .hex support to the existing SBE HEX reader path, expands parameter/mapping metadata for newly produced variables, and introduces multiple real-world fixtures plus integration tests to validate end-to-end decoding and calibration.

Changes:

  • Extend SbeHexReader/sbe_hex_reader to detect and parse SBE911+ HEX + XMLCON configurations and build calibrated datasets.
  • Add/adjust parameter metadata and mappings (notably par, plus updated oxygen/salinity metadata).
  • Add new SBE911+ fixtures and integration tests; adjust reader completeness tests to ignore helper dataclasses.

Reviewed changes

Copilot reviewed 30 out of 31 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
seasenselib/readers/sbe_hex_reader.py Implements/extends SBE911+ parsing, XMLCON mapping, and dataset construction.
tests/readers/test_sbe_hex_reader.py Adds extensive SBE911+ unit/integration tests and updates existing expectations.
tests/test_readers_completeness.py Excludes helper dataclasses from “reader class” enforcement.
seasenselib/parameters.py Adds par and updates parameter metadata (incl. oxygen, fluorescence, salinity).
seasenselib/knowledge/pipeline/metadata_enrichment/parameters_metadata.json Mirrors metadata updates for enrichment pipeline.
seasenselib/knowledge/pipeline/mapping/mappings_default.json Adds default name mappings for par.
seasenselib/knowledge/pipeline/mapping/allowed_parameters.json Adds par to allowed parameters.
tests/readers/fixtures/*.hex, tests/readers/fixtures/*.XMLCON, tests/readers/fixtures/*.xmlcon Adds real-world SBE911+ fixtures for test coverage.
.gitignore Ignores .coverage file.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/readers/test_sbe_hex_reader.py Outdated
Comment thread seasenselib/parameters.py
Comment on lines +90 to +96
FLUORESCENCE: {
'long_name': "Fluorescence",
'units': 'mg m-3',
'coverage_content_type': 'physicalMeasurement',
'standard_name': 'volume_fraction_of_oxygen_in_sea_water'
'standard_name': 'mass_concentration_of_chlorophyll_in_sea_water',
'measurement_type': 'Measured',
},
Comment on lines +78 to +84
"fluorescence": {
"coverage_content_type": "physicalMeasurement",
"long_name": "Fluorescence",
"measurement_type": "Measured",
"standard_name": "mass_concentration_of_chlorophyll_in_sea_water",
"units": "mg m-3"
},
Comment on lines +1564 to +1586
# ----- Primary conductivity (freq 1) -----
c1_info = xmlcon_map.sensors.get(("frequency", 1))
if c1_info is None or c1_info.sensor_type != "conductivity":
raise ValueError(
"No primary conductivity sensor at frequency channel 1 in channel map."
)
# convert_conductivity with scalar=1.0 returns mS/cm for 911+ frequency input.
cond_primary = conv.convert_conductivity(
conductivity_count=raw["conductivity"].values,
temperature=temp_primary,
pressure=pressure,
coefs=c1_info.coefficients,
)
_median_cond = float(np.median(cond_primary[np.isfinite(cond_primary)]))
if not (10.0 < _median_cond < 1000.0):
warnings.warn(
f"SBE 911+ primary conductivity has median {_median_cond:.4g} mS/cm, "
"outside the expected range (10–1000). "
"Check instrument calibration coefficients.",
UserWarning,
stacklevel=2,
)
data_vars["cond"] = (params.TIME, cond_primary)
Comment on lines +2428 to 2429
ds["oxygen"].attrs["units"] = "umol l-1"
ds["oxygen"].attrs["long_name"] = "Dissolved Oxygen"
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@ysorge
ysorge merged commit 395554c into main Aug 10, 2026
5 checks passed
@ysorge ysorge changed the title [FEAT] Add SBE 911+ HEX reader support feat: Add SBE 911+ HEX reader support Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants