Skip to content
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 149 additions & 0 deletions .codex/skills/ci-perf-analysis/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
---
name: ci-perf-analysis
description: 用于从 GitHub Actions 的 gem5 性能 CI 中定位 summary、score.txt 和归档目录,并结合本地 gem5_data_proc 对 spec06/spec17 结果做 weighted score、benchmark 子项对比和通用 stats 归因。适用于用户给出 run URL/run id、commit、workflow run,或要求分析 CI 跑分变化来源时。
---

# CI 性能分析

## 概览

这个 skill 只做一条固定链路:

1. 用 `gh` 从 CI run 找到 summary 对应的 `score.txt` 和归档目录。
2. 用本地 `gem5_data_proc/run.py` 把 `spec_all/` 处理成 `csv`、`weighted.csv`、`score.csv`。
3. 对比 benchmark 级收益,并在需要时继续下钻到 `stats.txt` 或其他归档结果。

## 快速开始

### 1. 先拿 summary 和归档目录

优先使用 bundled script:

```bash
python3 .codex/skills/ci-perf-analysis/scripts/ci_perf_info.py \
https://github.com/OpenXiangShan/GEM5/actions/runs/<run_id>
```

输出会包含:

- `archive_path`
- `spec_all` 目录
- `score.txt` 的最后 42 行

这和 workflow 在 GitHub summary 里展示的内容一致,因为 CI 本身就是把 `score.txt` 的最后 42 行写进 step summary。

### 2. 优先使用本地 `gem5_data_proc`

默认优先使用:

```bash
/nfs/home/yanyue/workspace/gem5_data_proc
```

如果用户本地没有,再执行:

```bash
git clone https://github.com/jensen-yan/gem5_data_proc
# 设置环境变量
export $GEM5_DATA_PROC_HOME=xxx
```
Comment thread
jensen-yan marked this conversation as resolved.


### 3. 用 `gem5_data_proc` 处理整个归档

```bash
cd $GEM5_DATA_PROC_HOME
python3 run.py /nfs/home/share/gem5_ci/performance_data/spec06-0.3c/<archive_dir> \
--out-dir /tmp/gem5_proc_runA \
--tag runA
```

关键输出:

- `<tag>.csv`:point 级或 benchmark 聚合后的原始统计
- `<tag>-weighted.csv`:按权重聚合后的 benchmark 统计
- `<tag>-score.csv`:最终 score/time/coverage

### 4. 对比两个 run

最常见的是比较两次 CI:

```bash
python3 .codex/skills/ci-perf-analysis/scripts/ci_perf_info.py <runA>
python3 .codex/skills/ci-perf-analysis/scripts/ci_perf_info.py <runB>

cd $GEM5_DATA_PROC_HOME
python3 run.py <archiveA> --out-dir /tmp/gem5_proc_A --tag A
python3 run.py <archiveB> --out-dir /tmp/gem5_proc_B --tag B
```

然后用短 Python 片段读取两个 `*-score.csv` / `*-weighted.csv` 做对比。优先关注:

- 总 score 变化
- benchmark 级 `time` / `score` 变化
- 用户关心的 stats 指标变化

## 下钻分析

### 1. 看 benchmark 级收益

- 对两个归档分别运行 `run.py`
- 比较 `*-score.csv` 里的 `time` 和 `score`
- 按 `score_delta_pct` 或 `time_delta_pct` 排序

### 2. 看 stats 指标变化

- 先看 `*-weighted.csv` 里的通用统计
- 如果用户已经给出重点指标,直接围绕这些指标对比
- 如果用户没有指定,优先从 `time`、`cpi`、前端、后端、内存、分支等大类里挑变化最明显的项
- 归因时优先描述“哪些 stats 在变”,再解释这些变化更像支持哪类根因

### 3. 需要时再读归档里的其他文件

- `stats.txt`:看原始统计,位置在
/nfs/home/share/gem5_ci/performance_data/spec06-0.3c/<archive_dir>/<spec_bmk>/m5out/stats.txt
- `score.txt`:对照 summary
- 其他 CSV 或日志:按用户问题决定是否下钻

## 常用命令

### 已知 run URL,直接拿 archive path

```bash
python3 .codex/skills/ci-perf-analysis/scripts/ci_perf_info.py <run_url_or_id>
```

### 已知 archive path,直接处理

```bash
cd $GEM5_DATA_PROC_HOME
python3 run.py <archive_dir> --out-dir /tmp/gem5_proc --tag run
```

## 输出组织建议

回答这类问题时,优先按下面的顺序组织:

1. commit / run / workflow / 配置差异
2. summary 里的总分变化
3. benchmark 级主要收益和回退项
4. 相关 stats 指标变化
5. 对根因的判断

结论要尽量区分:

- “哪个 benchmark 涨了”
- “哪些 stats 在变”
- “这些 stats 更像支持哪类根因”

## 资源

### scripts/

- `ci_perf_info.py`
- 输入 run URL 或 run id
- 输出 archive path、spec_all 路径和 `score.txt` tail

### references/

当前不需要额外参考文件。后续如果这套流程扩展到更多 workflow 或更多统计口径,再新增参考文档。
92 changes: 92 additions & 0 deletions .codex/skills/ci-perf-analysis/scripts/ci_perf_info.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
#!/usr/bin/env python3
"""Locate archived CI performance data and print the score summary."""

import argparse
import json
import os
import re
import subprocess
import sys
from pathlib import Path


REPO = "OpenXiangShan/GEM5"
ARCHIVE_RE = re.compile(
r"Archiving performance data to "
r"(/nfs/home/share/gem5_ci/performance_data/\S+)"
)


def run_cmd(cmd: list[str]) -> str:
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
raise SystemExit(result.stderr.strip() or f"command failed: {' '.join(cmd)}")
return result.stdout


def parse_run_id(text: str) -> str:
match = re.search(r"/actions/runs/(\d+)", text)
if match:
return match.group(1)
if text.isdigit():
return text
raise SystemExit(f"cannot parse run id from: {text}")


def get_job_id(run_id: str) -> str:
output = run_cmd(["gh", "api", f"repos/{REPO}/actions/runs/{run_id}/jobs"])
data = json.loads(output)
jobs = data.get("jobs", [])
if not jobs:
raise SystemExit(f"no jobs found for run {run_id}")
return str(jobs[0]["id"])


def get_archive_path(job_id: str) -> str:
log_text = run_cmd(["gh", "api", f"repos/{REPO}/actions/jobs/{job_id}/logs"])
match = ARCHIVE_RE.search(log_text)
if not match:
raise SystemExit(f"cannot find archive path in logs for job {job_id}")
return match.group(1)


def print_score(score_path: Path, tail_lines: int) -> None:
if not score_path.is_file():
print(f"score.txt not found: {score_path}", file=sys.stderr)
return

lines = score_path.read_text().splitlines()
print(f"score.txt: {score_path}")
print(f"--- tail -n {tail_lines} ---")
for line in lines[-tail_lines:]:
print(line)


def main() -> None:
parser = argparse.ArgumentParser(
description="Print CI archive path and score summary for a GEM5 perf run."
)
parser.add_argument("run", help="GitHub Actions run URL or run id")
parser.add_argument(
"--tail-lines",
type=int,
default=42,
help="how many trailing lines of score.txt to print",
)
args = parser.parse_args()

run_id = parse_run_id(args.run)
job_id = get_job_id(run_id)
archive_path = Path(get_archive_path(job_id))

print(f"run_id: {run_id}")
print(f"job_id: {job_id}")
print(f"archive_path: {archive_path}")
print(f"spec_all: {archive_path / 'spec_all'}")

score_path = archive_path / "score.txt"
print_score(score_path, args.tail_lines)


if __name__ == "__main__":
main()
18 changes: 18 additions & 0 deletions configs/common/xiangshan.py
Original file line number Diff line number Diff line change
Expand Up @@ -705,6 +705,24 @@ def xiangshan_system_init():
Options.addCommonOptions(parser, configure_xiangshan=True)
Options.addXiangshanFSOptions(parser)
Options.addXiangshanTraceOptions(parser)
parser.set_defaults(btb_tage_upper_bound=True)
parser.add_argument(
"--btb-tage-upper-bound",
dest="btb_tage_upper_bound",
action="store_true",
help="Use BTBTAGEUpperBound in kmhv3 (enabled by default)",
)
parser.add_argument(
"--disable-btb-tage-upper-bound",
dest="btb_tage_upper_bound",
action="store_false",
help="Use regular BTBTAGE instead of BTBTAGEUpperBound in kmhv3",
)
parser.add_argument(
"--btb-tage-upper-bound-path-hash",
action="store_true",
help="Use path-hash history for BTBTAGEUpperBound instead of outcome history",
)

# Add the ruby specific and protocol specific args
if '--ruby' in sys.argv:
Expand Down
12 changes: 11 additions & 1 deletion configs/example/kmhv3.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,11 @@ def setKmhV3Params(args, system):
cpu.branchPred.ftq_size = 64
cpu.branchPred.fsq_size = 64

if args.btb_tage_upper_bound:
# Default to UB-S(outcome); optionally switch to UB-P(path-hash).
cpu.branchPred.tage = BTBTAGEUpperBound(
usePathHashHistory=args.btb_tage_upper_bound_path_hash)

cpu.branchPred.mbtb.resolvedUpdate = True
cpu.branchPred.tage.resolvedUpdate = True
cpu.branchPred.ittage.resolvedUpdate = True
Expand All @@ -104,8 +109,13 @@ def setKmhV3Params(args, system):
cpu.branchPred.microtage.enabled = False
cpu.branchPred.mbtb.enabled = True
cpu.branchPred.tage.enabled = True
cpu.branchPred.tage.useBranchPcForIndex = False
cpu.branchPred.tage.usePositionForIndexMix = True
cpu.branchPred.tage.indexMixTables = 4
cpu.branchPred.tage.numWays = 2
cpu.branchPred.tage.tableSizes = [2048] * 8
cpu.branchPred.ittage.enabled = True
cpu.branchPred.mgsc.enabled = True
cpu.branchPred.mgsc.enabled = False
Comment thread
jensen-yan marked this conversation as resolved.
Outdated
cpu.branchPred.ras.enabled = True

# RTL alignment: only enable bias + path + IMLI tables, disable PC threshold
Expand Down
Loading
Loading