wip: ddtest support#18584
Draft
gnufede wants to merge 13 commits into
Draft
Conversation
Add DD_CI_TEST_DISCOVERY_MODE_ENABLED support to the ddtrace/testing pytest plugin, matching the format produced by datadog-ci-rb. When enabled, pytest collects tests and writes them as JSON Lines to DD_CI_TEST_DISCOVERY_OUTPUT_PATH (default: ddtest/test_discovery/tests.json), then exits without running any test. Each line contains name, suite, module, parameters, and suiteSourceFile fields. Tests decorated with pytest.mark.skip or pytest.mark.skipif with a truthy non-string condition are excluded from the output. CI Visibility initialisation is suppressed when discovery mode is active. Also moves _get_test_parameters_json and _encode_test_parameter from plugin.py to utils.py so they can be imported by _discovery.py without a circular dependency. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ery module Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED and DD_TEST_OPTIMIZATION_DISCOVERY_FILE (ddtest's canonical names) instead of the Ruby-aligned DD_CI_TEST_DISCOVERY_MODE_ENABLED / DD_CI_TEST_DISCOVERY_OUTPUT_PATH. ddtest owns the shared env var contract; individual language plugins should conform to it. Also update the default output path to match ddtest's .testoptimization/tests-discovery/tests.json. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…output The module field must match what the plugin reports at run time so the backend can correlate discovery results with test runs via FQN (module.suite.name). Using the static string "pytest" as module would collapse every test across all repos into the same namespace. nodeid_to_names already extracts the correct value (e.g. "tests.unit" for tests/unit/test_foo.py); we just needed to stop discarding it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…y mode Register TestOptHooks specs during discovery so item_to_test_ref can call the same custom hook chain used at run time. This means name/suite/ module resolution stays in sync with ITR matching if the logic ever changes, rather than being a separate parallel implementation. pytest-bdd is intentionally not wired up: BddTestOptPlugin is not registered in discovery mode, so BDD tests fall back to nodeid-based names (same as before). A TODO marks where to add that support. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED and DD_TEST_OPTIMIZATION_DISCOVERY_FILE to supported-configurations.json and regenerate _supported_configurations.py so the CI check passes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…est plugin - Sort discovery imports alphabetically into the ddtrace.testing block - Drop the pytest_collection_finish module-level re-export (caused F811 clash with BddTestOptPlugin.pytest_collection_finish); register the _discovery module directly as a plugin in pytest_configure instead - Fix import order and remove spurious f-string prefix in test_pytest_discovery.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pytester.monkeypatch was added in pytest 6.2; older envs (py3.9 CI) don't have it. Accept monkeypatch as a separate fixture parameter. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In manifest mode (activated by DD_TEST_OPTIMIZATION_MANIFEST_FILE), CachedFileDataProvider.get_skippable_tests was a hard no-op, and session_manager unconditionally forced itr_enabled and skipping_enabled to False. This made sense for Bazel where the build system handles test sharding, but prevents ddtest (Datadog's test runner) from using the skippable tests it fetched and cached during its plan phase. With this change: - get_skippable_tests reads and parses cache/http/skippable_tests.json, mirroring the same parsing logic used by the HTTP API client - The forced itr_enabled = False / skipping_enabled = False override is removed; settings come from the cached settings.json as for other features (EFD, ATR, known tests) This is safe for Bazel: if skippable_tests.json is absent the function returns an empty set, so no tests are skipped. Bazel builds that do not write the cache file are unaffected. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Version 1 (Bazel): original behaviour preserved — test skipping is disabled because the build system handles selection. Version 2 (ddtest): skippable_tests.json is read and applied, and the forced itr_enabled/skipping_enabled override is skipped. Changes: - constants.py: replace SUPPORTED_MANIFEST_VERSION scalar with SUPPORTED_MANIFEST_VERSIONS frozenset and add MANIFEST_VERSION_DDTEST = 2 - offline_mode.py: _validate_manifest → _read_manifest_version (returns int|None); OfflineMode gains apply_cached_skipping = (version >= 2) - cached_file_provider.py: new apply_cached_skipping constructor arg; get_skippable_tests returns early when False - session_manager.py: CachedFileDataProvider receives apply_cached_skipping; forced-False override is conditioned on not apply_cached_skipping Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…zel payload-files Previously all manifest-mode sessions had itr_enabled and skipping_enabled forced to False, and get_skippable_tests was a hard no-op. This prevented ddtest from using the skippable tests it cached during its plan phase. The correct Bazel guard is DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES, not manifest mode itself: that env var is set in Bazel sandboxes where the build system owns test selection. When it is absent (ddtest workers), the cached skippable_tests.json should be read and applied. Changes: - cached_file_provider.py: get_skippable_tests returns early only when DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES is set; otherwise reads and parses cache/http/skippable_tests.json mirroring the API client logic - session_manager.py: remove the manifest-mode forced itr_enabled=False override; CachedFileDataProvider.get_skippable_tests now owns the Bazel no-op decision - constants.py / offline_mode.py: revert manifest version scaffolding added in the previous attempt; SUPPORTED_MANIFEST_VERSION stays 1 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tests' into gnufede/ddtest-support
Contributor
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Testing
Risks
Additional Notes