Add better self-testing of the local coverage plugin and meta-coverage reporting#2367
Draft
sirosen wants to merge 11 commits into
Draft
Add better self-testing of the local coverage plugin and meta-coverage reporting#2367sirosen wants to merge 11 commits into
sirosen wants to merge 11 commits into
Conversation
These are either chronically uncovered up to a certain line or have flaky coverage. Either way, having them in is rather harmful for measurements.
These are version-dependent but are sometimes applied to conditionals checking the deps versions. They won't work perfectly but it's a start.
A new local plugin in `plugins/coverage/` as a standalone module, which
injects coverage excludes of the form
pragma: pip{comparator}{version} no cover
In order for this plugin to be picked up, `.coveragerc` is updated to
list the plugin by name and `PYTHONPATH` is set via `setenv` in order to
make the module importable.
All conditional checks on the pip version are now marked with the
relevant matching pragmas.
When looking at lost coverage reporting in codecov, it appears that it is counting the new plugin code as "uncovered". Though true -- it isn't being tested rigorously -- trying to measure it seems incorrect at present.
In order to support these, the logic of the plugin had to become a bit more organized -- it now uses a list of supported operators to build out the full suite of pragmas. With these new pragmas available, update the various usage sites to use the affirmative 'cover' pragmas on comparisons, with the corresponding 'no cover' pragmaes on else branches. To improve branch coverage, all of the `else` branches are explicitly enumerated and captured with the appropriate pragmas, even though many are `else: pass`.
This is added to the python path via pytest options, and is excluded from pytest discovery (norecursedirs). Also, add more pip-version pragmas to test code which only runs on specific pip versions.
In order to determine the lowest supported version of `pip`, read the `project.dependencies` list from `pyproject.toml`, find the only listed `pip` dependency, and pull its specifier. If any of the data does not match expectations, a ValueError will be thrown (crashing any run of coverage/pytest-cov) with a message stating that the plugin needs to be updated. This lets us get a working solution without trying to solve a more general case than what we need.
Unit tests of the local coverage plugin exercise its various helpers, and the module-level docstring explains that we can test the parts even if we can't easily measure code coverage on the plugin itself.
These tests need `coverage` to be installed, and it is missing 1. In CI builds on pypy where`coverage` is not installed 2. When local tests run without the `coverage` factor
Local coverage runs won't report on this file, as configured, but codecov is still flagging it as uncovered code.
In order to enable coverage reporting for the plugin itself, define a separate tox environment and CI job which test the plugin, all using a separate, dedicated coverage config.
sirosen
force-pushed
the
testing/self-test-coverage-local-plugin
branch
from
816de2f to
7ae2040
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| steps: | ||
| - uses: actions/checkout@v5 | ||
| - name: Set up Python ${{ matrix.python-version }} | ||
| uses: actions/setup-python@v6 |
| - name: Set up Python ${{ matrix.python-version }} | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} |
| - name: Pip cache | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ${{ steps.pip-cache.outputs.dir }} |
| - name: Pip cache | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ${{ steps.pip-cache.outputs.dir }} |
| if: >- | ||
| !cancelled() | ||
| && !inputs.cpython-pip-version | ||
| uses: codecov/codecov-action@v5 |
Comment on lines
+433
to
+517
| test-local-coverage-plugin: | ||
| name: test-coverage-plugin / ${{ matrix.runner-vm }} / ${{ matrix.python-version }} | ||
| runs-on: ${{ matrix.runner-vm }} | ||
| timeout-minutes: 9 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| runner-vm: | ||
| - ubuntu-24.04 | ||
| - macos-15-intel | ||
| - windows-2025 | ||
| python-version: | ||
| - "3.9" | ||
| - "3.14" | ||
| env: | ||
| TOXENV: test-coverage-plugin | ||
| steps: | ||
| - uses: actions/checkout@v5 | ||
| - name: Set up Python ${{ matrix.python-version }} | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| - name: Get pip cache dir | ||
| id: pip-cache | ||
| shell: bash | ||
| run: | | ||
| echo "dir=$(pip cache dir)" >> "${GITHUB_OUTPUT}" | ||
| - name: Pip cache | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ${{ steps.pip-cache.outputs.dir }} | ||
| key: >- | ||
| ${{ runner.os }}-pip-${{ hashFiles('setup.cfg') }}-${{ | ||
| hashFiles('pyproject.toml') }}-${{ hashFiles('tox.ini') }}-${{ | ||
| hashFiles('.pre-commit-config.yaml') }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-pip- | ||
| ${{ runner.os }}- | ||
| - name: Install tox | ||
| run: pip install tox | ||
| - name: Prepare test environment | ||
| run: tox --notest -p auto | ||
| - name: Run Tests | ||
| run: tox --skip-pkg-install | ||
| - name: Re-run the failing tests with maximum verbosity | ||
| if: >- | ||
| !cancelled() | ||
| && failure() | ||
| run: >- # `exit 1` makes sure that the job remains red with flaky runs | ||
| python -Xutf8 -Im | ||
| tox | ||
| --parallel=auto | ||
| --parallel-live | ||
| --skip-missing-interpreters=false | ||
| --skip-pkg-install | ||
| -vvvvv | ||
| -- | ||
| --continue-on-collection-errors | ||
| --full-trace | ||
| --last-failed | ||
| --numprocesses=0 | ||
| --showlocals | ||
| --trace-config | ||
| -rA | ||
| -vvvvv | ||
| && exit 1 | ||
| shell: bash | ||
| - name: Upload coverage to Codecov | ||
| if: >- | ||
| !cancelled() | ||
| && !inputs.cpython-pip-version | ||
| uses: codecov/codecov-action@v5 | ||
| with: | ||
| files: ./coverage.xml | ||
| flags: >- | ||
| CI-GHA, | ||
| OS-${{ runner.os }}, | ||
| VM-${{ matrix.runner-vm }}, | ||
| Py-${{ matrix.python-version }}, | ||
| LocalCoveragePlugin | ||
| name: >- | ||
| OS-${{ runner.os }}, | ||
| VM-${{ matrix.runner-vm }}, | ||
| Py-${{ matrix.python-version }}, | ||
| LocalCoveragePlugin |
| steps: | ||
| - uses: actions/checkout@v5 | ||
| - name: Set up Python ${{ matrix.python-version }} | ||
| uses: actions/setup-python@v6 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a draft on top of #2347; see that PR for more context.
Add a separate tox environment and CI job for doing the testing of the coverage
plugin itself.
Because the plugin uses a separate coverage config for this test, both it and
normal coverage can be maintained at 100%, and codecov can aggregate the two.
Because this is a draft right now, I'm not clear on whether or not it should get
a separate changelog? I feel like it should fold into the one for #2347.
Contributor checklist
changelog.d/(seechangelog.d/README.mdfor instructions) or the PR text says "no changelog needed".
Maintainer checklist
bot:chronographer:skiplabel.(following Semantic Versioning).