|
106 | 106 | \title[CleanTest-Agent]{CleanTest-Agent: A Multi-Agent Skill-Orchestrated System for Unit Test Training Data Quality Assurance} |
107 | 107 |
|
108 | 108 | \author{Yong Yang} |
| 109 | +\authornote{Source code, datasets, the trained Filter 3 checkpoint |
| 110 | + metadata, and a reproducible end-to-end notebook are released |
| 111 | + under the MIT license at |
| 112 | + \url{https://github.com/jimmy0717/cleantest-agent}; a |
| 113 | + pip-installable distribution is at |
| 114 | + \url{https://pypi.org/project/cleantest-agent/}. See |
| 115 | + Section~\ref{sec:availability} for the full availability |
| 116 | + statement.} |
109 | 117 | \affiliation{% |
110 | 118 | \institution{Beihang University} |
111 | 119 | \department{School of Software} |
@@ -189,7 +197,15 @@ \section{Introduction} |
189 | 197 | \item \textbf{Empirical evaluation}: We compare four approaches (rule-based, zero-shot LLM, few-shot LLM, hybrid) and demonstrate that our hybrid approach achieves the best F1 (0.965) while the pure LLM approach achieves only 0.387, validated with real DeepSeek-V4-Flash API calls. |
190 | 198 | \end{enumerate} |
191 | 199 |
|
192 | | -The remainder of this paper is organized as follows. Section~\ref{sec:background} reviews related work on unit test generation, data quality, LLM-based code analysis, and agent skill architectures. Section~\ref{sec:requirements} presents a requirements analysis including stakeholder analysis, use cases, functional and non-functional requirements, and a traceability matrix. Section~\ref{sec:model-driven} defines the model-driven approach and contrasts it with the pure-LLM approach. Section~\ref{sec:design} details the system design including architecture, filter specifications, and the pipeline algorithm. Section~\ref{sec:implementation} covers the implementation with technology choices, design decisions, and testing strategy. Section~\ref{sec:evaluation} presents the experimental evaluation through four research questions, including a case study on real samples and an ablation study. Section~\ref{sec:discussion} discusses findings, lessons learned, connections to course topics, and threats to validity. Section~\ref{sec:conclusion} concludes the paper and outlines future work directions. Appendices provide LLM prompt templates, CI/CD configuration, the full noise report, noise type examples with real Java code, a complete API reference, and skill directory structure documentation. |
| 200 | +The remainder of this paper is organized as follows. Section~\ref{sec:background} reviews related work on unit test generation, data quality, LLM-based code analysis, and agent skill architectures. Section~\ref{sec:requirements} presents a requirements analysis including stakeholder analysis, use cases, functional and non-functional requirements, and a traceability matrix. Section~\ref{sec:model-driven} defines the model-driven approach and contrasts it with the pure-LLM approach. Section~\ref{sec:design} details the system design including architecture, filter specifications, and the pipeline algorithm. Section~\ref{sec:implementation} covers the implementation with technology choices, design decisions, and testing strategy. Section~\ref{sec:evaluation} presents the experimental evaluation through four research questions, including a case study on real samples and an ablation study. Section~\ref{sec:discussion} discusses findings, lessons learned, connections to course topics, and threats to validity. Section~\ref{sec:conclusion} concludes the paper and outlines future work directions. Section~\ref{sec:availability} states code, data, and reproducibility links. Appendices provide LLM prompt templates, CI/CD configuration, the full noise report, noise type examples with real Java code, a complete API reference, and skill directory structure documentation. |
| 201 | + |
| 202 | +All artefacts described in this paper --- including the |
| 203 | +Python package \texttt{cleantest-agent}, the four SKILL.md skill |
| 204 | +bundles, the 36-case pytest suite, the GitHub Actions CI matrix, the |
| 205 | +tag-driven PyPI publish workflow with sigstore signing, the curated |
| 206 | +Filter 3 model-mode metrics, and the LaTeX sources of this paper |
| 207 | +--- are released under the MIT license and available at |
| 208 | +\url{https://github.com/jimmy0717/cleantest-agent}~\cite{yang2026cleantestagent}. |
193 | 209 |
|
194 | 210 | \subsection{Scope and Context} |
195 | 211 |
|
@@ -3077,6 +3093,78 @@ \subsection{Final Reflections} |
3077 | 3093 |
|
3078 | 3094 | The skill-based architecture also illustrates how modern software design patterns adapt to new paradigms: the \texttt{SKILL.md} protocol is effectively an interface contract between the skill author and the AI coding assistant runtime --- a natural evolution of API design for the age of natural-language-driven software interaction. |
3079 | 3095 |
|
| 3096 | +% ============================================================================ |
| 3097 | +\section{Code and Data Availability} |
| 3098 | +\label{sec:availability} |
| 3099 | +% ============================================================================ |
| 3100 | + |
| 3101 | +In line with the ACM Artifact Review and Badging policy, all |
| 3102 | +artefacts described in this paper are publicly available under the |
| 3103 | +MIT license. Table~\ref{tab:availability} maps every empirical claim |
| 3104 | +in the paper to its reproducible source. |
| 3105 | + |
| 3106 | +\begin{table}[htbp] |
| 3107 | +\centering |
| 3108 | +\small |
| 3109 | +\caption{Code and data availability map.} |
| 3110 | +\label{tab:availability} |
| 3111 | +\begin{tabularx}{\linewidth}{@{}lY@{}} |
| 3112 | +\toprule |
| 3113 | +\textbf{Artefact} & \textbf{Location} \\ |
| 3114 | +\midrule |
| 3115 | +Source repository (MIT) & \url{https://github.com/jimmy0717/cleantest-agent} \\ |
| 3116 | +Python package (pip-installable) & \url{https://pypi.org/project/cleantest-agent/} \\ |
| 3117 | +Tagged release (this paper) & \url{https://github.com/jimmy0717/cleantest-agent/releases/tag/v0.1.1} \\ |
| 3118 | +This paper (PDF, signed asset) & \url{https://github.com/jimmy0717/cleantest-agent/releases/download/v0.1.0/main.pdf} \\ |
| 3119 | +SKILL.md skill bundles (4) & \texttt{skills/} in the repository \\ |
| 3120 | +Filter 3 training notebook & \texttt{experiments/main-final.ipynb} \\ |
| 3121 | +Filter 3 metrics (held-out test) & \texttt{experiments/results/coverage\_run/test\_metrics.json} \\ |
| 3122 | +Bundled 5{,}000-row sample & \texttt{data/sample\_5000.csv} (Methods2Test, MIT) \\ |
| 3123 | +Annotation dictionary (21{,}954 patterns) & \texttt{cleantest\_agent/data/noise\_modifier\_fm.txt} \\ |
| 3124 | +500-sample evaluation labels & \texttt{experiments/results/labeled\_samples.csv} \\ |
| 3125 | +DeepSeek baseline runner & \texttt{experiments/run\_baselines.py} \\ |
| 3126 | +CI workflow (Python 3.10/3.11/3.12) & \texttt{.github/workflows/ci.yml} \\ |
| 3127 | +CD workflow (PyPI + sigstore) & \texttt{.github/workflows/publish.yml} \\ |
| 3128 | +Upstream LessIsMore-FSE2025 data & \url{https://doi.org/10.5281/zenodo.15347368} \\ |
| 3129 | +\bottomrule |
| 3130 | +\end{tabularx} |
| 3131 | +\end{table} |
| 3132 | + |
| 3133 | +\textbf{Reproducing the headline results.} |
| 3134 | +The 593{,}953-sample full-pipeline run reported in |
| 3135 | +Section~\ref{sec:evaluation} (RQ1) is reproducible by cloning the |
| 3136 | +repository, exporting Methods2Test via the LessIsMore-FSE2025 |
| 3137 | +replication package, and invoking |
| 3138 | +\texttt{cleantest --input\_csv all\_train.csv --output\_dir out/}. |
| 3139 | +The 500-sample stratified subset evaluation (RQ2--RQ4) is |
| 3140 | +reproducible via |
| 3141 | +\texttt{python experiments/run\_baselines.py}, which issues real |
| 3142 | +DeepSeek-V4-Flash API calls; the per-sample predictions used for |
| 3143 | +the F1\,=\,0.965 hybrid result and the LLM zero-shot / few-shot |
| 3144 | +baselines are checked into |
| 3145 | +\texttt{experiments/results/labeled\_samples.csv}. The Filter 3 |
| 3146 | +model-mode held-out MAE\,=\,0.0309 result is reproducible from |
| 3147 | +\texttt{experiments/main-final.ipynb} on a single A800 80\,GB; the |
| 3148 | +metrics archived under |
| 3149 | +\texttt{experiments/results/coverage\_run/} are the verbatim |
| 3150 | +artefacts that produced the numbers in |
| 3151 | +Section~\ref{sec:evaluation} (RQ3 model mode). |
| 3152 | + |
| 3153 | +\textbf{Supply-chain integrity.} |
| 3154 | +PyPI distributions for v0.1.1 onward are uploaded via OIDC Trusted |
| 3155 | +Publisher (no long-lived API token) and signed with sigstore; |
| 3156 | +the signed wheel and source bundles are attached as release assets |
| 3157 | +on GitHub. End users can verify provenance with the |
| 3158 | +\texttt{sigstore verify identity} command documented in |
| 3159 | +\texttt{docs/PYPI-PUBLISHING.md}. |
| 3160 | + |
| 3161 | +\textbf{Issue tracking and contribution.} |
| 3162 | +The repository ships with structured GitHub issue templates (bug, |
| 3163 | +feature, question), a pull-request template, the Contributor |
| 3164 | +Covenant 2.1 code of conduct, and a development-workflow |
| 3165 | +\texttt{CONTRIBUTING.md}. Bug reports, feature requests, and pull |
| 3166 | +requests are welcomed at the URLs above. |
| 3167 | + |
3080 | 3168 | % ============================================================================ |
3081 | 3169 | % Acknowledgements (ACM-recommended `acks` environment, not `\section{...}`). |
3082 | 3170 | % ============================================================================ |
|
0 commit comments