Add CLI and environment passthrough for RocksDB and ScyllaDB tuning options#6477
Draft
ma2bd wants to merge 1 commit into
Draft
Add CLI and environment passthrough for RocksDB and ScyllaDB tuning options#6477ma2bd wants to merge 1 commit into
ma2bd wants to merge 1 commit into
Conversation
Instruction Count Benchmark Results
Deterministic metrics — reproducible across runs (34 benchmarks)
Cache-dependent metrics — expect fluctuations between runs (34 benchmarks)
|
pull Bot
pushed a commit
to Mu-L/linera-protocol
that referenced
this pull request
Jun 29, 2026
…-io#6511) ## Motivation RocksDB backs the Linera client's local store, and the PM workers *are* the client — so it's now production-critical infrastructure. Yet the binary exports nothing about RocksDB's internal engine state (block-cache effectiveness, compaction / write amplification, write stalls, LSM structure). We have the metering-wrapper operation latencies but not the engine internals needed to understand or tune RocksDB — e.g. the write-stall / compaction behaviour behind recent worker incidents. Validators get rich ScyllaDB dashboards; the client's RocksDB deserves the same. ## Proposal Add opt-in export of RocksDB's internal statistics as Prometheus metrics, off by default: - Two flags on `CommonStorageOptions`: `--rocksdb-enable-statistics` (default off) and `--rocksdb-statistics-level` (default `except-histogram-or-timers`; one of the six RocksDB `StatsLevel`s). Off by default so clients that don't scrape metrics pay nothing; enabled explicitly on the workers. - `RocksDbStoreInternalConfig` gains `enable_statistics` + `statistics_level` (`#[serde(default)]`), wired in `build()` to `enable_statistics()` / `set_statistics_level()`. - A lazy Prometheus `Collector` (behind `with_metrics`) that reads cumulative tickers (`get_ticker_count`) and instantaneous LSM state (`GetIntProperty`) at scrape time — no background task — exporting `linera_rocksdb_*` metrics: block-cache hit/miss, compaction read/write bytes, write-stall micros, flush bytes, bloom/memtable hits, plus L0 file count, pending-compaction bytes, write-stopped, SST sizes, memtable size, etc. The default level (tickers only) is cheap — the op-latency histograms (the expensive part) are deliberately skipped, since we already export them from the metering wrapper; bumping the level turns on the engine histograms for latency investigation. Cumulative tickers are exported as gauges (the `prometheus` crate's `IntCounter` can't be set to an external absolute without stateful delta-tracking); they're monotonic so `rate()` works across restarts. No format-defining option is touched — this only toggles statistics collection. ## Test Plan - Unit tests: the level `ValueEnum` parses and maps 1:1 to the views enum; statistics are off by default; the collector's metric definitions build unique, valid gauges. - `cargo clippy --all-targets -- -D warnings` clean on `linera-views` and `linera-storage-runtime`; `cargo fmt`; `cargo doc` strict; `cargo machete`; `CLI.md` regenerated. - The default-off path leaves `build()` behaviour identical (the new blocks are no-ops when `enable_statistics` is false). ## Release Plan - These changes should be backported to the latest `testnet` branch (the workers run on `testnet_conway`). No SDK or validator hotfix needed — this is client/worker observability only. ## Links - Complements linera-io#6477 (RocksDB/ScyllaDB tuning-option passthrough): these metrics provide the observability to drive those tuning knobs. No logical dependency; the overlapping files will be rebased on whichever lands first. - Follow-up: a RocksDB Grafana dashboard once the metrics are live; optionally expose `PerfContext` for deep latency profiling. - [reviewer checklist](https://github.com/linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Many RocksDB and ScyllaDB tuning knobs are currently hardcoded in the backend
build()/session-construction code, so operators cannot adapt them to theirhardware or workload without recompiling. We want a way to pass a much larger
set of database options via CLI and environment, with the hard constraint that
changing an option must always be safe — it must never alter the on-disk
storage format or table schema.
Proposal
Add a generic key=value passthrough per backend, wired through the existing
CommonStorageOptions→add_common_storage_optionsplumbing:--rocksdb-opt KEY=VALUE(repeatable) /LINERA_ROCKSDB_OPTS(comma-separated).Supports the safe runtime/new-data knobs:
write_buffer_size,max_write_buffer_number, thelevel_zero_*triggers,parallelism,max_background_jobs,max_subcompactions,target_file_size_base,block_cache_size,write_buffer_manager_size,max_open_files,block_size,bloom_filter_bits_per_key,compression_type.--scylladb-opt KEY=VALUE(repeatable) /LINERA_SCYLLADB_OPTS.Supports driver/session options:
consistency,request_timeout_ms,connection_pool_size_per_host,compression,token_aware.Each backend parses its pairs into a typed
*TuningOptionsstruct; unset fieldskeep the current hardcoded default, so default behavior is unchanged. Unknown
keys, malformed pairs, and unparseable values are rejected with descriptive
errors. For RocksDB, format-defining keys (
prefix_extractor,format_version,compaction_style,whole_key_filtering, …) are explicitly rejected — this isthe safety guarantee. ScyllaDB create-time keyspace/table properties
(replication, compaction, table compression,
gc_grace_seconds) areintentionally out of scope here since they only apply at namespace creation.
Test Plan
input, and every rejection path (unknown key, malformed pair, unparseable
value, unknown enum token, format-sensitive keys, zero pool size).
conditionally-set
max_open_filesand cache-size paths throughDB::open).cargo clippyclean (with the repo's-D warnings) onlinera-views,linera-storage-runtime,linera-indexerforrocksdb,scylladb, and thecombined dual-feature build.
CLI.mdregenerated; nightlyrustfmtclean.Release Plan
Links