Skip to content

Commit 1873c0b

Browse files
committed
consistency: report adds Code & Data Availability + 4 cross-artefact fixes
Four small repo-wide consistency fixes plus a new "Code and Data Availability" section in the report so reviewers can locate every artefact described in the paper. - cleantest_agent/__init__.py: replace the hard-coded __version__ = '0.1.0' (drifted past 0.1.1 on the last release) with importlib.metadata.version('cleantest-agent'). pyproject.toml is now the single source of truth for the version, and python-side __version__ tracks pip's view of the installed distribution. - docs/skill-distribution-guide.md: replace the version literal in the import-and-print walkthrough with X.Y.Z + a note that the string is resolved at import time, not embedded. - report/references.bib: add a self-cite @misc{yang2026cleantestagent} pointing at github.com and pypi.org so the BibTeX corpus matches the citation stanza already published in README.md. - report/main.tex: * \authornote on the title page links the GitHub + PyPI URLs and forwards the reader to the new section. (\thanks on \title blew up acmart's metadata stack; \authornote is the template-canonical equivalent.) * Introduction outline now ends with one sentence pointing at the availability section, then a paragraph that names every artefact (package, skills, tests, CI matrix, CD pipeline, Filter 3 metrics, LaTeX sources) and cites the self-reference. * New Section 10 "Code and Data Availability" (§10 in the new numbering; Conclusion is §9) carries a 14-row table mapping every empirical claim to its reproducible source, plus three paragraphs on reproduction, supply-chain integrity (sigstore), and contribution channels. - report/main.pdf: rebuilt locally with TeX Live 2026; 67 pp. -> 68 pp., bibliography now has 40 entries (was 39), all citations resolved, no missing references.
1 parent c2d6d68 commit 1873c0b

5 files changed

Lines changed: 119 additions & 3 deletions

File tree

cleantest_agent/__init__.py

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,21 @@
99
)
1010
"""
1111

12-
__version__ = "0.1.0"
12+
# Single source of truth for the version is `pyproject.toml`;
13+
# `__version__` is resolved at import time from the installed
14+
# distribution metadata so the two cannot drift out of sync.
15+
# Falls back to "0.0.0+unknown" only when the package is imported
16+
# directly from a source tree that has never been `pip install`-ed
17+
# (e.g. by running tests from a freshly cloned repo without pip);
18+
# all CI / CD / PyPI / sigstore paths always go through an install.
19+
try:
20+
from importlib.metadata import PackageNotFoundError, version as _pkg_version
21+
try:
22+
__version__ = _pkg_version("cleantest-agent")
23+
except PackageNotFoundError:
24+
__version__ = "0.0.0+unknown"
25+
except ImportError: # pragma: no cover - importlib.metadata is stdlib on 3.8+
26+
__version__ = "0.0.0+unknown"
1327

1428
from cleantest_agent.data_loader import load_csv, save_csv
1529
from cleantest_agent.report_generator import NoiseReport

docs/skill-distribution-guide.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,9 @@ Verify:
5454

5555
```bash
5656
python -c "import cleantest_agent; print(cleantest_agent.__version__)"
57-
# 0.1.0
57+
# X.Y.Z (e.g. 0.1.1; the version is resolved at import time from
58+
# the installed distribution metadata, so it always tracks
59+
# whatever pip installed)
5860

5961
cleantest --help
6062
# usage: cleantest [-h] --input_csv INPUT_CSV --output_dir OUTPUT_DIR ...

report/main.pdf

9.3 KB
Binary file not shown.

report/main.tex

Lines changed: 89 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,14 @@
106106
\title[CleanTest-Agent]{CleanTest-Agent: A Multi-Agent Skill-Orchestrated System for Unit Test Training Data Quality Assurance}
107107

108108
\author{Yong Yang}
109+
\authornote{Source code, datasets, the trained Filter 3 checkpoint
110+
metadata, and a reproducible end-to-end notebook are released
111+
under the MIT license at
112+
\url{https://github.com/jimmy0717/cleantest-agent}; a
113+
pip-installable distribution is at
114+
\url{https://pypi.org/project/cleantest-agent/}. See
115+
Section~\ref{sec:availability} for the full availability
116+
statement.}
109117
\affiliation{%
110118
\institution{Beihang University}
111119
\department{School of Software}
@@ -189,7 +197,15 @@ \section{Introduction}
189197
\item \textbf{Empirical evaluation}: We compare four approaches (rule-based, zero-shot LLM, few-shot LLM, hybrid) and demonstrate that our hybrid approach achieves the best F1 (0.965) while the pure LLM approach achieves only 0.387, validated with real DeepSeek-V4-Flash API calls.
190198
\end{enumerate}
191199

192-
The remainder of this paper is organized as follows. Section~\ref{sec:background} reviews related work on unit test generation, data quality, LLM-based code analysis, and agent skill architectures. Section~\ref{sec:requirements} presents a requirements analysis including stakeholder analysis, use cases, functional and non-functional requirements, and a traceability matrix. Section~\ref{sec:model-driven} defines the model-driven approach and contrasts it with the pure-LLM approach. Section~\ref{sec:design} details the system design including architecture, filter specifications, and the pipeline algorithm. Section~\ref{sec:implementation} covers the implementation with technology choices, design decisions, and testing strategy. Section~\ref{sec:evaluation} presents the experimental evaluation through four research questions, including a case study on real samples and an ablation study. Section~\ref{sec:discussion} discusses findings, lessons learned, connections to course topics, and threats to validity. Section~\ref{sec:conclusion} concludes the paper and outlines future work directions. Appendices provide LLM prompt templates, CI/CD configuration, the full noise report, noise type examples with real Java code, a complete API reference, and skill directory structure documentation.
200+
The remainder of this paper is organized as follows. Section~\ref{sec:background} reviews related work on unit test generation, data quality, LLM-based code analysis, and agent skill architectures. Section~\ref{sec:requirements} presents a requirements analysis including stakeholder analysis, use cases, functional and non-functional requirements, and a traceability matrix. Section~\ref{sec:model-driven} defines the model-driven approach and contrasts it with the pure-LLM approach. Section~\ref{sec:design} details the system design including architecture, filter specifications, and the pipeline algorithm. Section~\ref{sec:implementation} covers the implementation with technology choices, design decisions, and testing strategy. Section~\ref{sec:evaluation} presents the experimental evaluation through four research questions, including a case study on real samples and an ablation study. Section~\ref{sec:discussion} discusses findings, lessons learned, connections to course topics, and threats to validity. Section~\ref{sec:conclusion} concludes the paper and outlines future work directions. Section~\ref{sec:availability} states code, data, and reproducibility links. Appendices provide LLM prompt templates, CI/CD configuration, the full noise report, noise type examples with real Java code, a complete API reference, and skill directory structure documentation.
201+
202+
All artefacts described in this paper --- including the
203+
Python package \texttt{cleantest-agent}, the four SKILL.md skill
204+
bundles, the 36-case pytest suite, the GitHub Actions CI matrix, the
205+
tag-driven PyPI publish workflow with sigstore signing, the curated
206+
Filter 3 model-mode metrics, and the LaTeX sources of this paper
207+
--- are released under the MIT license and available at
208+
\url{https://github.com/jimmy0717/cleantest-agent}~\cite{yang2026cleantestagent}.
193209

194210
\subsection{Scope and Context}
195211

@@ -3077,6 +3093,78 @@ \subsection{Final Reflections}
30773093

30783094
The skill-based architecture also illustrates how modern software design patterns adapt to new paradigms: the \texttt{SKILL.md} protocol is effectively an interface contract between the skill author and the AI coding assistant runtime --- a natural evolution of API design for the age of natural-language-driven software interaction.
30793095

3096+
% ============================================================================
3097+
\section{Code and Data Availability}
3098+
\label{sec:availability}
3099+
% ============================================================================
3100+
3101+
In line with the ACM Artifact Review and Badging policy, all
3102+
artefacts described in this paper are publicly available under the
3103+
MIT license. Table~\ref{tab:availability} maps every empirical claim
3104+
in the paper to its reproducible source.
3105+
3106+
\begin{table}[htbp]
3107+
\centering
3108+
\small
3109+
\caption{Code and data availability map.}
3110+
\label{tab:availability}
3111+
\begin{tabularx}{\linewidth}{@{}lY@{}}
3112+
\toprule
3113+
\textbf{Artefact} & \textbf{Location} \\
3114+
\midrule
3115+
Source repository (MIT) & \url{https://github.com/jimmy0717/cleantest-agent} \\
3116+
Python package (pip-installable) & \url{https://pypi.org/project/cleantest-agent/} \\
3117+
Tagged release (this paper) & \url{https://github.com/jimmy0717/cleantest-agent/releases/tag/v0.1.1} \\
3118+
This paper (PDF, signed asset) & \url{https://github.com/jimmy0717/cleantest-agent/releases/download/v0.1.0/main.pdf} \\
3119+
SKILL.md skill bundles (4) & \texttt{skills/} in the repository \\
3120+
Filter 3 training notebook & \texttt{experiments/main-final.ipynb} \\
3121+
Filter 3 metrics (held-out test) & \texttt{experiments/results/coverage\_run/test\_metrics.json} \\
3122+
Bundled 5{,}000-row sample & \texttt{data/sample\_5000.csv} (Methods2Test, MIT) \\
3123+
Annotation dictionary (21{,}954 patterns) & \texttt{cleantest\_agent/data/noise\_modifier\_fm.txt} \\
3124+
500-sample evaluation labels & \texttt{experiments/results/labeled\_samples.csv} \\
3125+
DeepSeek baseline runner & \texttt{experiments/run\_baselines.py} \\
3126+
CI workflow (Python 3.10/3.11/3.12) & \texttt{.github/workflows/ci.yml} \\
3127+
CD workflow (PyPI + sigstore) & \texttt{.github/workflows/publish.yml} \\
3128+
Upstream LessIsMore-FSE2025 data & \url{https://doi.org/10.5281/zenodo.15347368} \\
3129+
\bottomrule
3130+
\end{tabularx}
3131+
\end{table}
3132+
3133+
\textbf{Reproducing the headline results.}
3134+
The 593{,}953-sample full-pipeline run reported in
3135+
Section~\ref{sec:evaluation} (RQ1) is reproducible by cloning the
3136+
repository, exporting Methods2Test via the LessIsMore-FSE2025
3137+
replication package, and invoking
3138+
\texttt{cleantest --input\_csv all\_train.csv --output\_dir out/}.
3139+
The 500-sample stratified subset evaluation (RQ2--RQ4) is
3140+
reproducible via
3141+
\texttt{python experiments/run\_baselines.py}, which issues real
3142+
DeepSeek-V4-Flash API calls; the per-sample predictions used for
3143+
the F1\,=\,0.965 hybrid result and the LLM zero-shot / few-shot
3144+
baselines are checked into
3145+
\texttt{experiments/results/labeled\_samples.csv}. The Filter 3
3146+
model-mode held-out MAE\,=\,0.0309 result is reproducible from
3147+
\texttt{experiments/main-final.ipynb} on a single A800 80\,GB; the
3148+
metrics archived under
3149+
\texttt{experiments/results/coverage\_run/} are the verbatim
3150+
artefacts that produced the numbers in
3151+
Section~\ref{sec:evaluation} (RQ3 model mode).
3152+
3153+
\textbf{Supply-chain integrity.}
3154+
PyPI distributions for v0.1.1 onward are uploaded via OIDC Trusted
3155+
Publisher (no long-lived API token) and signed with sigstore;
3156+
the signed wheel and source bundles are attached as release assets
3157+
on GitHub. End users can verify provenance with the
3158+
\texttt{sigstore verify identity} command documented in
3159+
\texttt{docs/PYPI-PUBLISHING.md}.
3160+
3161+
\textbf{Issue tracking and contribution.}
3162+
The repository ships with structured GitHub issue templates (bug,
3163+
feature, question), a pull-request template, the Contributor
3164+
Covenant 2.1 code of conduct, and a development-workflow
3165+
\texttt{CONTRIBUTING.md}. Bug reports, feature requests, and pull
3166+
requests are welcomed at the URLs above.
3167+
30803168
% ============================================================================
30813169
% Acknowledgements (ACM-recommended `acks` environment, not `\section{...}`).
30823170
% ============================================================================

report/references.bib

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -314,3 +314,15 @@ @book{meszaros2007xunit
314314
modalities of automated test design (Chapters 11--12); referenced
315315
by Filter 2's reflection rule checklist (R2, R3) in this paper.}
316316
}
317+
@misc{yang2026cleantestagent,
318+
title = {{CleanTest-Agent}: A Multi-Agent Skill-Orchestrated System
319+
for Unit Test Training Data Quality Assurance},
320+
author = {Yang, Yong},
321+
year = {2026},
322+
howpublished = {\url{https://github.com/jimmy0717/cleantest-agent}},
323+
note = {Source code, datasets, and a reproducible Filter 3
324+
training notebook are released under the MIT license at
325+
\url{https://github.com/jimmy0717/cleantest-agent}; the
326+
pip-installable package is at
327+
\url{https://pypi.org/project/cleantest-agent/}.}
328+
}

0 commit comments

Comments
 (0)