Skip to content

Commit 33e01eb

Browse files
committed
feat(release): ship 5.2.0 operational reliability upgrades
- Add markdown scanner config support (.a11y-markdown-config.json) - Add markdown SARIF export and configurable gate modes (none/error/warning) - Add workflow_dispatch controls and SARIF artifact upload in a11y-check - Add orchestrator-specialist dispatch contract validator + CI workflow - Upgrade web-severity-scoring guidance with v2 calibration/drift metadata - Add metadata and markup conventions guide + AGENTS/README/docs updates - Add templates/markdown-config-moderate.json starter profile - Bump release version to 5.2.0 across plugin.yaml, gemini-extension.json, mcp-server/package.json - Update CHANGELOG and add RELEASE-5.2.0.md Notes: - Intentionally excludes signing/key-management changes per release scope
1 parent f321216 commit 33e01eb

16 files changed

Lines changed: 857 additions & 114 deletions

.github/scripts/markdown-a11y-lint.mjs

Lines changed: 257 additions & 109 deletions
Large diffs are not rendered by default.

.github/skills/web-severity-scoring/SKILL.md

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,18 @@ Weights:
2727
Floor: 0 (minimum score)
2828
```
2929

30+
### Scoring Profiles
31+
32+
Use a profile to tune strictness by context while keeping comparable grade bands:
33+
34+
| Profile | Intended Use | Multiplier |
35+
|---------|--------------|------------|
36+
| balanced (default) | Standard product delivery | 1.0 |
37+
| strict | Regulated/public-sector releases | 1.15 |
38+
| advisory | Early design and prototyping | 0.8 |
39+
40+
Apply the profile multiplier to each final deduction after confidence handling.
41+
3042
### Formula
3143

3244
```pseudocode
@@ -43,6 +55,26 @@ The values in the lookup table above are **base deductions** (pre-multiplier).
4355

4456
**Example:** One Critical finding at confirmed confidence = 18 (base) × 1.2 = **21.6 points** deducted → page score 78.
4557

58+
### Calibration Layer (v2)
59+
60+
To reduce false-positive inflation and stabilize trends, apply a calibration coefficient by rule family:
61+
62+
```text
63+
calibrated_deduction = deduction × calibration_coefficient(rule_family)
64+
```
65+
66+
Recommended initial coefficients:
67+
68+
| Rule Family | Coefficient | Rationale |
69+
|-------------|-------------|-----------|
70+
| Keyboard/focus | 1.1 | High functional impact at runtime |
71+
| Forms/labels/errors | 1.05 | High completion risk for core tasks |
72+
| Semantics/structure | 1.0 | Baseline scoring |
73+
| Link text/context | 0.9 | Higher context variance |
74+
| Content quality (alt/link clarity) | 0.85 | Needs human review more often |
75+
76+
Update coefficients quarterly from confirmed outcomes. Avoid changing coefficients more than +/-0.1 per cycle.
77+
4678
## Score Grades
4779

4880
| Score | Grade | Meaning |
@@ -74,6 +106,20 @@ Issues found by all three sources (axe-core + agent review + Playwright behavior
74106

75107
When Playwright is not available, the maximum achievable confidence remains **High (100%)**. The confirmed tier is additive — it never downgrades findings.
76108

109+
### Confidence Drift Guard
110+
111+
Track predicted confidence versus post-triage outcome and compute drift:
112+
113+
```text
114+
drift = abs(predicted_confidence_score - observed_confirmation_rate)
115+
```
116+
117+
Operational guideline:
118+
119+
- drift <= 0.10: stable
120+
- drift 0.11-0.20: tune coefficients and source mapping
121+
- drift > 0.20: freeze profile changes and run rule-level review
122+
77123
## Scorecard Format
78124

79125
### Single Page
@@ -131,6 +177,37 @@ When Playwright is not available, the maximum achievable confidence remains **Hi
131177
- **Pages improved:** count of pages with higher scores than previous audit
132178
- **Trend:** improving (score up 5+), stable (within 5), declining (score down 5+)
133179

180+
### Normalized Trend Metric (Cross-Audit)
181+
182+
When audit scope changes between runs, use normalized change:
183+
184+
```text
185+
normalized_score = raw_score - (scope_variance_penalty)
186+
scope_variance_penalty = min(10, abs(previous_pages - current_pages) * 0.8)
187+
```
188+
189+
Use normalized score for trend charts and use raw score for release gates.
190+
191+
## Output Metadata (Recommended)
192+
193+
Include these fields in generated score artifacts for reproducibility:
194+
195+
```yaml
196+
scoring:
197+
model: web-severity-scoring-v2
198+
profile: balanced
199+
calibrationVersion: 2026-q2
200+
confidenceSources:
201+
- axe-core
202+
- agent-review
203+
- playwright
204+
failThresholds:
205+
critical: 1
206+
score: 75
207+
```
208+
209+
This metadata allows deterministic re-runs and audit-to-audit comparisons.
210+
134211
## Issue Severity Categories
135212
136213
### Critical

.github/workflows/a11y-check.yml

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,16 @@ on:
1313
- "**/*.css"
1414
- "**/*.scss"
1515
- "**/*.md"
16+
workflow_dispatch:
17+
inputs:
18+
enforcement_mode:
19+
description: "Markdown lint gate mode (none|error|warning)"
20+
required: false
21+
default: "error"
22+
output_format:
23+
description: "Markdown lint output format (text|sarif|both)"
24+
required: false
25+
default: "both"
1626

1727
jobs:
1828
a11y-lint:
@@ -92,6 +102,9 @@ jobs:
92102
markdown-lint:
93103
name: Markdown Accessibility Lint
94104
runs-on: ubuntu-latest
105+
env:
106+
A11Y_ENFORCE_MODE: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.enforcement_mode || vars.A11Y_MARKDOWN_FAIL_ON || 'error' }}
107+
A11Y_OUTPUT_FORMAT: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.output_format || vars.A11Y_MARKDOWN_FORMAT || 'both' }}
95108
steps:
96109
- uses: actions/checkout@v6
97110

@@ -101,7 +114,19 @@ jobs:
101114
node-version: "20"
102115

103116
- name: Run markdown accessibility lint
104-
run: node .github/scripts/markdown-a11y-lint.mjs .
117+
run: |
118+
node .github/scripts/markdown-a11y-lint.mjs . \
119+
--fail-on "$A11Y_ENFORCE_MODE" \
120+
--format "$A11Y_OUTPUT_FORMAT" \
121+
--output artifacts/markdown-a11y.sarif
122+
123+
- name: Upload markdown a11y SARIF artifact
124+
if: always() && (env.A11Y_OUTPUT_FORMAT == 'sarif' || env.A11Y_OUTPUT_FORMAT == 'both')
125+
uses: actions/upload-artifact@v4
126+
with:
127+
name: markdown-a11y-sarif
128+
path: artifacts/markdown-a11y.sarif
129+
if-no-files-found: ignore
105130

106131
# Optional: GitHub Accessibility Scanner for deployed pages
107132
# Uncomment and configure URLs to enable automated scanning on PR.
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
name: Validate Orchestrator Contracts
2+
3+
on:
4+
push:
5+
branches: [main]
6+
paths:
7+
- '.claude/agents/**'
8+
- '.claude/specialists/**'
9+
- 'scripts/validate-orchestrator-dispatch.js'
10+
- '.github/workflows/validate-orchestrator-contracts.yml'
11+
pull_request:
12+
paths:
13+
- '.claude/agents/**'
14+
- '.claude/specialists/**'
15+
- 'scripts/validate-orchestrator-dispatch.js'
16+
- '.github/workflows/validate-orchestrator-contracts.yml'
17+
18+
jobs:
19+
validate-dispatch-contracts:
20+
runs-on: ubuntu-latest
21+
22+
steps:
23+
- name: Checkout repository
24+
uses: actions/checkout@v6
25+
26+
- name: Setup Node.js
27+
uses: actions/setup-node@v6
28+
with:
29+
node-version: '20'
30+
31+
- name: Validate orchestrator-specialist dispatch contracts
32+
run: node scripts/validate-orchestrator-dispatch.js

AGENTS.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,48 @@ description: What this skill provides
156156
Skill content in markdown...
157157
```
158158

159+
### Recommended Metadata (2026)
160+
161+
Use `metadata` for stable machine-readable attributes that improve routing, validation, and release reporting.
162+
163+
Agent frontmatter example:
164+
165+
```yaml
166+
---
167+
name: agent-name
168+
description: What this agent does
169+
tools: ['read', 'edit', 'search']
170+
metadata:
171+
owner: accessibility-team
172+
domain: web|document|github|developer
173+
maturity: stable|beta|experimental
174+
release-phase: ga|preview
175+
capability-tags: ["wcag-2.2", "keyboard", "forms"]
176+
dispatch-contract: required|optional
177+
---
178+
```
179+
180+
Skill frontmatter example:
181+
182+
```yaml
183+
---
184+
name: skill-name
185+
description: What this skill provides
186+
metadata:
187+
spec-version: "2026-05"
188+
model-compatibility: ["copilot", "claude", "gemini"]
189+
scoring-model: "v2"
190+
compliance-profiles: ["wcag-2.2-aa", "en-301-549"]
191+
---
192+
```
193+
194+
Instruction markup guidance:
195+
196+
- Keep one H1 per file.
197+
- Use explicit `## Decision Matrix` and `## Non-Negotiable Standards` headings for scanners.
198+
- Prefer checklist-style acceptance criteria sections for automation (`## Acceptance Criteria`).
199+
- Keep line-based examples in fenced code blocks with explicit language tags.
200+
159201
## Build & Test
160202

161203
No build step required - agents are markdown files.

CHANGELOG.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,30 @@ All notable changes to the Accessibility Agents project will be documented in th
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [5.2.0] - 2026-05-06
9+
10+
### Added
11+
12+
- **Markdown scanner config support** via `.a11y-markdown-config.json` in `.github/scripts/markdown-a11y-lint.mjs` with per-rule enable/disable, severity overrides, ignore directory control, and configurable per-rule output limits.
13+
- **SARIF export support** for markdown accessibility findings (`--format sarif|both`, `--output`) to enable machine-readable CI artifacts.
14+
- **New template**: `templates/markdown-config-moderate.json` for fast adoption of markdown scanning defaults.
15+
- **Orchestrator dispatch contract validator**: `scripts/validate-orchestrator-dispatch.js` to enforce required `Specialist Dispatch` sections and verify specialist file references exist.
16+
- **New CI workflow**: `.github/workflows/validate-orchestrator-contracts.yml` to run dispatch contract validation on PRs and pushes.
17+
- **New documentation guide**: `docs/guides/metadata-markup-conventions.md` defining recommended metadata and markup conventions for agents, skills, and instruction files.
18+
19+
### Changed
20+
21+
- **`a11y-check.yml` markdown lint job** now supports gate modes (`none|error|warning`) and output modes (`text|sarif|both`) through workflow dispatch inputs and repo variables (`A11Y_MARKDOWN_FAIL_ON`, `A11Y_MARKDOWN_FORMAT`).
22+
- **`web-severity-scoring` skill** upgraded with v2 guidance: scoring profiles, calibration coefficients, confidence-drift guardrails, normalized trend metric, and recommended scoring metadata fields.
23+
- **`docs/getting-started.md`** updated with markdown scanner config examples, CI gate mode behavior, and SARIF usage patterns.
24+
- **`AGENTS.md`** expanded with recommended metadata frontmatter conventions for agents and skills, plus instruction markup conventions.
25+
- **`README.md`** documentation index now includes the metadata and markup conventions guide.
26+
27+
### Fixed
28+
29+
- **Markdown lint CI resilience** improved by making scanner behavior configurable without code changes.
30+
- **Orchestrator-specialist drift risk** reduced by adding CI-enforced dispatch contract validation.
31+
832
## [5.1.0] - 2026-05-03
933

1034
### Added

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -482,6 +482,7 @@ The following guides cover advanced configuration, cross-platform handoff, and d
482482
|-------|---------------|
483483
| [Cross-Platform Handoff](docs/advanced/cross-platform-handoff.md) | Seamless handoff between Claude Code and Copilot |
484484
| [Advanced Scanning Patterns](docs/advanced/advanced-scanning-patterns.md) | Background scanning, worktree isolation, large libraries |
485+
| [Metadata and Markup Conventions](docs/guides/metadata-markup-conventions.md) | Recommended metadata fields and markdown structure conventions for agents, skills, and instructions |
485486
| [Plugin Packaging](docs/advanced/plugin-packaging.md) | Packaging and distributing agents for different environments |
486487
| [Platform References](docs/advanced/platform-references.md) | External documentation sources with feature-to-source mapping |
487488
| [Experimental Codex Multi-Agent Roles](docs/guides/codex-experimental-multi-agent.md) | Optional TOML-based Codex roles for focused accessibility passes |

RELEASE-5.2.0.md

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Accessibility Agents 5.2: Calibrated Scoring, Configurable Gates, Stronger Contracts
2+
3+
## Overview
4+
5+
Accessibility Agents 5.2 focuses on operational quality and reliability: configurable markdown accessibility gates, SARIF output for machine-readable CI, calibrated web severity scoring guidance, and contract validation between orchestrators and specialists.
6+
7+
This release intentionally excludes signing and key-management changes.
8+
9+
---
10+
11+
## Highlights
12+
13+
### Configurable Markdown Accessibility Scanning
14+
15+
`markdown-a11y-lint.mjs` now supports:
16+
17+
- repository config file: `.a11y-markdown-config.json`
18+
- per-rule `enabled` and `severity` overrides
19+
- custom ignored directories
20+
- configurable per-rule output limits
21+
- runtime gate mode controls (`none`, `error`, `warning`)
22+
23+
### SARIF Output for CI Pipelines
24+
25+
Markdown accessibility findings can now be exported as SARIF:
26+
27+
```bash
28+
node .github/scripts/markdown-a11y-lint.mjs . \
29+
--format both \
30+
--output artifacts/markdown-a11y.sarif
31+
```
32+
33+
This enables downstream code-scanning and artifact workflows without custom parsers.
34+
35+
### CI Gate Maturity Controls
36+
37+
`a11y-check.yml` now supports dispatch-time and variable-driven gate controls:
38+
39+
- workflow inputs: `enforcement_mode`, `output_format`
40+
- repo variables: `A11Y_MARKDOWN_FAIL_ON`, `A11Y_MARKDOWN_FORMAT`
41+
42+
Teams can adopt advisory mode first and tighten enforcement later without editing workflow logic.
43+
44+
### Orchestrator-Specialist Contract Validation
45+
46+
New validator:
47+
48+
- `scripts/validate-orchestrator-dispatch.js`
49+
50+
New workflow:
51+
52+
- `.github/workflows/validate-orchestrator-contracts.yml`
53+
54+
The validator enforces required dispatch sections, verifies `Read(".claude/specialists/*.md")` references, checks `Task(...)` usage, and confirms referenced specialists exist.
55+
56+
### Web Severity Scoring v2 Guidance
57+
58+
`web-severity-scoring` now documents:
59+
60+
- profile-based scoring (`balanced`, `strict`, `advisory`)
61+
- calibration coefficients by rule family
62+
- confidence drift guardrails
63+
- normalized trend scoring for cross-audit comparability
64+
- recommended output metadata for reproducibility
65+
66+
### New Metadata and Markup Conventions
67+
68+
A new guide standardizes metadata and markdown structure:
69+
70+
- `docs/guides/metadata-markup-conventions.md`
71+
72+
It includes recommended frontmatter metadata patterns for agents and skills and instruction markup conventions for automation-friendly docs.
73+
74+
---
75+
76+
## Additional Docs and Templates
77+
78+
- Added template: `templates/markdown-config-moderate.json`
79+
- Updated: `docs/getting-started.md` with markdown scanner and CI gate usage
80+
- Updated: `AGENTS.md` with metadata conventions
81+
- Updated: `README.md` docs index to include metadata and markup conventions guide
82+
83+
---
84+
85+
## Why This Matters
86+
87+
5.2 improves how teams operate these agents at scale:
88+
89+
- less CI fragility through explicit configuration
90+
- better interop through SARIF
91+
- reduced orchestration drift via contract validation
92+
- more reliable trend analysis through calibrated scoring guidance
93+
- clearer machine-readable metadata standards for future automation
94+
95+
---
96+
97+
## Full Changelog
98+
99+
See `CHANGELOG.md` for complete details.

0 commit comments

Comments
 (0)