Skip to content

Centralize Datadog config inventory with build-time metadata support. #1788

@webern

Description

@webern

Problem

Updating a single Datadog config key in ADP currently requires touching several files. There are two
sources of truth with no sync mechanism between them.

Three additional problems compound this:

  • The config registry conflates production code with test support, requiring allow(dead_code),
    cfg(test), and feature = "config-test-support" guards throughout (see Separate prod from test/build concerns in the config registry. #1623).
  • All config metadata (which keys are supported, which pipelines they affect, what they're used for)
    is encoded in hand-written Rust structs and is only visible at runtime.
  • The ledger and documentation exist as separate artifacts with no generation step.

Proposed Solution

Replace all existing per-key annotation sources with a single hand-maintained schema_overlay.yaml,
joined at build time with the vendored Datadog schema. The overlay will be the single source of
truth for Datadog config coverage, documentation, runtime classification, and smoke-test support.

lib/datadog-agent-commons/build.rs will read both files and:

  • Validate that every key in the vendored schema appears exactly once in the overlay.
  • Generate config registry annotation constants, replacing the hand-written datadog/*.rs files.
  • Generate the documentation markdown.
  • Generate a DatadogConfiguration struct (via typify or a custom generator, depending on the
    schema language).
  • Generate a "witness trait" (DatadogConfigConsumer) trait with one method per supported key, plus
    a drive function — forcing a human to explicitly handle every supported key at compile time.

A hand-written Translator will implement the trait, converting DatadogConfiguration into the
internal topology and dataplane config structs. Adding a key to the overlay without updating the
translator will result in a compilation error. Removing a key leaves an orphan impl method — also a
compilation error.

The audit ledger will be deleted. Documentation will be generated. The registry annotation files
will be deleted at the end of the process when the smoke tests switch over to using the new code.

Layout

lib/datadog-agent-commons/config/schema.yaml          # vendored, untouched
lib/datadog-agent-commons/config/schema_overlay.yaml  # hand-written, single source of truth
lib/datadog-agent-schema-model/                       # types for deserializing the overlay

Anticipated PRs

If everything goes according to plan, then the following sequence makes sense to me. If it goes off
the rails, all bets are off!

  • (small): New datadog-agent-schema-model crate: Rust types for deserializing
    schema_overlay.yaml
  • (small): Infra scaffold: move schema to datadog-agent-commons/config/, add build.rs that
    validates overlay structure (no coverage enforcement yet)
  • (medium): Generation pipeline with seed data: build.rs generates config registry constants
    and documentation markdown, proved with a small set of synthetic overlay entries
  • (xtra-large): Migrate all keys to schema_overlay.yaml; enable full coverage enforcement;
    replace the hand-written config_registry/datadog/*.rs with generated files (TBD exactly what,
    but preserve the smoke tests for now). Delete the audit ledger. Generate the documentation file.
  • (small): Generate DatadogConfiguration struct from the vendored schema
  • (medium): Generate trait witness: DatadogConfigConsumer trait and drive function
  • (medium): Implement Translator; replace the config-reading entry point; smoke tests pass
  • (unknown): replace GenericConfiguration access with this???
  • (probably small): make sure Separate prod from test/build concerns in the config registry. #1623 is fixed

Metadata

Metadata

Assignees

Labels

area/configConfiguration.area/docsReference documentation.area/testAll things testing: unit/integration, correctness, SMP regression, etc.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions