[ab-advisor] A/B experiment: prefetch_strategy for daily-safe-output-optimizer by Copilot · Pull Request #38096 · github/gh-aw

Copilot · 2026-06-09T11:45:04Z

Adds the log_fetch_strategy A/B experiment to daily-safe-output-optimizer, comparing eager pre-fetching (setup step downloads logs before agent starts) against lazy on-demand fetching (agent calls MCP logs tool at runtime). Also introduces the infrastructure required to support if:-conditioned imports.

New infrastructure: conditional import support

ImportSpec and PromptImportEntry gain an If string field. The condition propagates through the full import pipeline and produces two effects at compile time:

Steps — each step from the conditional import gets if: needs.activation.outputs.<name> == '<value>' injected (via applyIfConditionToStepsYAML + experimentConditionRegex).

Prompt — the {{#runtime-import}} in the lock file is wrapped in {{#if experiments.<name> == '<value>'}}…{{/if}}, which interpolate_prompt.cjs evaluates at runtime (runtime-imports are expanded before {{#if}} blocks).

Usage in workflow frontmatter:

imports:
  - uses: shared/aw-logs-24h-fetch.md
    if: "experiments.log_fetch_strategy == 'eager'"

JSON schema updated to allow if on all four import object variants (top-level path/uses and aw-subfield path/uses).

Workflow changes (`daily-safe-output-optimizer.md`)

shared/aw-logs-24h-fetch.md import is now conditional on eager variant; the download step in the lock file gets if: needs.activation.outputs.log_fetch_strategy == 'eager'
Added agentic-workflows: tool and cache-memory: true directly to the main workflow so the lazy variant retains full MCP access
Added experiments: log_fetch_strategy block (variants: eager/lazy, 50/50, Mann-Whitney U, min_samples: 20, guardrail: issue_creation_success_rate >= 0.80)
Phase 1 prompt section replaced with {{#if experiments.log_fetch_strategy == 'eager'}}…{{else}}…{{/if}}: eager instructs the agent to read from /tmp/gh-aw/aw-mcp/logs/ directly; lazy retains the existing MCP logs tool call instruction

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

…sages Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Pull request overview

Adds a new A/B experiment (log_fetch_strategy) to the daily-safe-output-optimizer workflow and introduces conditional import support so imports can be guarded by runtime experiment conditions, affecting both generated workflow steps and prompt content.

Changes:

Add if: support to workflow import specs and propagate it into prompt assembly and step generation.
Inject step-level if: guards for steps originating from conditional imports by rewriting experiments.<name> → needs.activation.outputs.<name>.
Update daily-safe-output-optimizer to run an eager-vs-lazy log fetching experiment and persist experiment state.

Show a summary per file

File	Description
pkg/workflow/compiler_yaml.go	Wraps prompt imports/runtime-import macros in `{{#if ...}}` blocks when an import has an `if:` condition.
pkg/parser/schemas/main_workflow_schema.json	Extends schema to allow `if` on import object variants (`path`/`uses` and `aw` subfield).
pkg/parser/import_remote.go	Adds `ifCondition` field to queued import items for conditional import propagation.
pkg/parser/import_processor.go	Extends `ImportSpec` and `PromptImportEntry` with an `If` condition field.
pkg/parser/import_field_extractor.go	Injects conditional `if:` guards into imported YAML step lists and adds helpers for condition rewriting.
pkg/parser/import_field_extractor_test.go	Updates tests for the new `extractConfigFields` signature.
pkg/parser/import_bfs.go	Parses top-level import `if:` field and threads it into queue items.
.github/workflows/daily-safe-output-optimizer.md	Adds experiment config, conditional import of log-fetcher, and prompt branching for eager vs lazy behavior.
.github/workflows/daily-safe-output-optimizer.lock.yml	Regenerates the compiled workflow to include experiment selection/state steps and conditional log-download step.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 9/9 changed files
Comments generated: 6

+	for i := range steps {
+		if _, hasIf := steps[i]["if"]; !hasIf {
+			steps[i]["if"] = stepCondition
+		}
+	}


+func enqueueImportPath(state *importBFSState, importPath, fullPath, sectionName, baseDir string, inputs map[string]any, ifCondition string, origin *remoteImportOrigin) error {
 	if !state.visited[fullPath] {
 		state.visited[fullPath] = true
 		state.visitedInputs[fullPath] = inputs
 		state.queue = append(state.queue, importQueueItem{
-			importPath: importPath, fullPath: fullPath, sectionName: sectionName, baseDir: baseDir, inputs: inputs, remoteOrigin: origin,
+			importPath: importPath, fullPath: fullPath, sectionName: sectionName, baseDir: baseDir, inputs: inputs, ifCondition: ifCondition, remoteOrigin: origin,
 		})


+				if entry.If != "" {
+					userPromptChunks = append(userPromptChunks, fmt.Sprintf("{{#if %s}}", entry.If))
+				}
 				chunks, exprMaps := extractPromptChunksFromMarkdown(cleaned)
 				userPromptChunks = append(userPromptChunks, chunks...)
 				expressionMappings = append(expressionMappings, exprMaps...)
+				if entry.If != "" {
+					userPromptChunks = append(userPromptChunks, "{{/if}}")
+				}


+{{#if experiments.log_fetch_strategy == 'eager'}}
+Logs have been pre-downloaded to `/tmp/gh-aw/aw-mcp/logs/` by the setup step. Use this pre-fetched data directly — do **not** call the `logs` MCP tool.
+


+func applyIfConditionToStepsYAML(stepsYAML string, ifCondition string) string {
+	if ifCondition == "" {
+		return stepsYAML
+	}
+	stepCondition := transformImportConditionForStep(ifCondition)
+	var steps []map[string]any
+	if err := yaml.UnmarshalWithOptions([]byte(stepsYAML), &steps, yaml.UseOrderedMap()); err != nil {
+		parserLog.Printf("Warning: failed to parse steps YAML for if-condition injection (condition=%q), using original: %v", ifCondition, err)
+		return stepsYAML
+	}
+	for i := range steps {
+		if _, hasIf := steps[i]["if"]; !hasIf {
+			steps[i]["if"] = stepCondition
+		}
+	}
+	out, err := yaml.Marshal(steps)
+	if err != nil {
+		parserLog.Printf("Warning: failed to marshal steps YAML after if-condition injection (condition=%q), using original: %v", ifCondition, err)
+		return stepsYAML
+	}
+	return strings.TrimSpace(string(out))
+}


pelikhan · 2026-06-09T13:00:24Z

@copilot run pr-finisher skill

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-06-09T13:37:43Z

@copilot run pr-finisher skill

Ran pr-finisher pass and pushed fixes in 14b7577. Addressed conditional import gating/propagation, strict imports[].if validation, prompt condition quote normalization, workflow prompt quote fix, and added focused tests. Local make fmt, make lint, make test-unit, make test, and make recompile all pass.

Initial plan

3129681

Copilot AI assigned Copilot and pelikhan Jun 9, 2026

Copilot started work on behalf of pelikhan June 9, 2026 11:45 View session

Copilot AI linked an issue Jun 9, 2026 that may be closed by this pull request

[ab-advisor] Experiment campaign for daily-safe-output-optimizer: A/B test prefetch_strategy #38094

Closed

8 tasks

Copilot AI and others added 2 commits June 9, 2026 12:24

Fix schema validation for conditional imports and experiments config

b2ee3f5

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Address code review: rename regex var, add ifCondition to warning mes…

4f43448

…sages Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Experiment with A/B test prefetch_strategy for daily-safe-output-optimizer~~ [ab-advisor] A/B experiment: prefetch_strategy for daily-safe-output-optimizer Jun 9, 2026

Copilot AI requested a review from pelikhan June 9, 2026 12:29

Copilot finished work on behalf of pelikhan June 9, 2026 12:29

pelikhan marked this pull request as ready for review June 9, 2026 12:35

Copilot AI review requested due to automatic review settings June 9, 2026 12:35

Copilot started reviewing on behalf of pelikhan June 9, 2026 12:36 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

Copilot started work on behalf of pelikhan June 9, 2026 13:09 View session

Fix conditional import gating and prompt condition handling

14b7577

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan June 9, 2026 13:38

pelikhan merged commit 7094d2b into main Jun 9, 2026
29 checks passed

pelikhan deleted the copilot/ab-advisor-experiment-campaign-daily-safe-output-o branch June 9, 2026 13:47

github-actions Bot mentioned this pull request Jun 10, 2026

[aw] No-Op Runs #37963

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ab-advisor] A/B experiment: prefetch_strategy for daily-safe-output-optimizer#38096

[ab-advisor] A/B experiment: prefetch_strategy for daily-safe-output-optimizer#38096
pelikhan merged 4 commits into
mainfrom
copilot/ab-advisor-experiment-campaign-daily-safe-output-o

Copilot AI commented Jun 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

pelikhan commented Jun 9, 2026

Uh oh!

Copilot AI commented Jun 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		{{#if experiments.log_fetch_strategy == 'eager'}}
		Logs have been pre-downloaded to `/tmp/gh-aw/aw-mcp/logs/` by the setup step. Use this pre-fetched data directly — do not call the `logs` MCP tool.

Conversation

Copilot AI commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New infrastructure: conditional import support

Workflow changes (daily-safe-output-optimizer.md)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

pelikhan commented Jun 9, 2026

Uh oh!

Copilot AI commented Jun 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jun 9, 2026 •

edited

Loading

Workflow changes (`daily-safe-output-optimizer.md`)