Skip to content

PERF.md

Latest commit

 

History

History
180 lines (145 loc) · 6.06 KB

PERF.md

File metadata and controls

180 lines (145 loc) · 6.06 KB
Raw

Performance Harness

This document defines the reproducible performance harness used for Phase 5.

Workload profile (pinned)

  • CPU governor: performance
  • Target host class: dedicated self-hosted perf runner
  • Kernel: captured in CI artifact metadata (uname -r)
  • Filesystem target: /etc/hosts for open/read microbenches
  • Network target: loopback (127.0.0.1) with UDP connect() for connect microbench
  • Agent mode: --audit for overhead measurement (no deny action side effects)
  • Policy state: deny maps cleared (no deny rules). In audit-only mode with an empty deny policy, the daemon intentionally skips attaching file/network LSM hooks to measure idle overhead rather than enforcement cost.

Noise controls

  • Open benchmark default iterations: 200000
  • Connect benchmark default iterations: 50000
  • Open regression compare uses median of repeated runs (REPEATS=3 default)
  • Perf gate compares baseline vs with-agent on the same host and run
  • Hosted benchmark.yml is advisory only; strict gating runs in .github/workflows/perf.yml on deterministic runners
  • Hosted benchmark runs pin execution to one CPU (taskset -c 0) when available
  • Hosted benchmark comparison filters out unstable rows:
    • only mean aggregates are kept
    • rows below 50ns are excluded from alerting
    • only high-signal families listed in config/benchmark_focus_patterns.txt are compared
  • Hosted benchmark alert threshold is intentionally looser (20%) than strict perf SLO gating to account for shared-runner variance on shared runners
  • PR benchmark comments are disabled to prevent noisy review threads; strict pass/fail performance evidence is reported by .github/workflows/perf.yml
  • Use stable thresholds in scripts/perf_workload_suite.sh
  • Strict KPI gate: p95_with_agent / p95_baseline <= 1.05 for both open and connect profiles (validated by scripts/validate_perf_artifacts.py)
  • Canonical SLO gate table is generated by scripts/perf_slo_check.sh and stored as perf-slo-report.md + perf-slo-summary.json

Hosted benchmark alert interpretation

Treat Benchmark workflow alerts as early signals, not release blockers:

  • If hosted alert is present but perf.yml passes on self-hosted perf runners, treat the result as noise or non-critical drift.
  • If both hosted alert and perf.yml gate fail, treat it as a real regression and require mitigation before promotion.

Benchmarks

Open/close latency (includes percentiles)

ITERATIONS=200000 FILE=/etc/hosts scripts/perf_open_bench.sh
WITH_AGENT=1 BIN=./build/aegisbpf ITERATIONS=200000 FILE=/etc/hosts scripts/perf_open_bench.sh

Output includes:

  • us_per_op
  • p50_us
  • p95_us
  • p99_us

Loopback connect latency (includes percentiles)

ITERATIONS=50000 scripts/perf_connect_bench.sh
WITH_AGENT=1 BIN=./build/aegisbpf ITERATIONS=50000 scripts/perf_connect_bench.sh

The connect microbench uses UDP sockets to isolate syscall overhead and avoid accept-thread scheduling noise from TCP listener setups.

Output includes:

  • us_per_op
  • p50_us
  • p95_us
  • p99_us

Workload suite gate

sudo BIN=./build/aegisbpf \
  FILE=/etc/hosts \
  OPEN_ITERATIONS=200000 \
  CONNECT_ITERATIONS=50000 \
  READ_ITERATIONS=50000 \
  STAT_SAMPLE=400 \
  STAT_ITERATIONS=50 \
  MAX_OPEN_PCT=15 \
  MAX_CONNECT_PCT=15 \
  MAX_READ_PCT=15 \
  MAX_STAT_PCT=15 \
  scripts/perf_workload_suite.sh

The suite fails if any workload exceeds threshold. The strict open_close <= 10% gate remains enforced by scripts/perf_compare.sh in .github/workflows/perf.yml.

Event-rate and drop behavior

  • Soak reliability and drop-ratio evidence are collected via:
    • scripts/soak_reliability.sh
    • .github/workflows/soak.yml
  • Product reliability target remains <0.1% event-drop ratio under sustained benchmarked load (see CI perf artifacts).

JSON output

Both open and connect benches support FORMAT=json and OUT=/path:

FORMAT=json OUT=/tmp/open.json ITERATIONS=200000 FILE=/etc/hosts scripts/perf_open_bench.sh
FORMAT=json OUT=/tmp/connect.json ITERATIONS=50000 scripts/perf_connect_bench.sh

CI stores these JSON profiles as perf-profiles artifacts in .github/workflows/perf.yml.

Artifact schema validation

Perf CI validates artifact schema before upload:

python3 scripts/validate_perf_artifacts.py \
  --open-baseline artifacts/perf/open_baseline.json \
  --open-with-agent artifacts/perf/open_with_agent.json \
  --connect-baseline artifacts/perf/connect_baseline.json \
  --connect-with-agent artifacts/perf/connect_with_agent.json \
  --workload artifacts/perf/workload_suite.json \
  --report artifacts/perf/perf-evidence-report.md

Validation guarantees:

  • required JSON keys exist for each profile
  • percentile ordering is sane (p50 <= p95 <= p99)
  • workload suite includes all required rows (open_close, connect_loopback, full_read, stat_walk)
  • with-agent profiles are labeled consistently
  • KPI ratios stay within gates:
    • open_p95_ratio <= 1.05
    • connect_p95_ratio <= 1.05

Canonical SLO gate

Use the deterministic SLO gate script to produce one canonical table:

OPEN_JSON=artifacts/perf/open_compare.json \
WORKLOAD_JSON=artifacts/perf/workload_suite.json \
OPEN_BASELINE_JSON=artifacts/perf/open_baseline.json \
OPEN_WITH_AGENT_JSON=artifacts/perf/open_with_agent.json \
CONNECT_BASELINE_JSON=artifacts/perf/connect_baseline.json \
CONNECT_WITH_AGENT_JSON=artifacts/perf/connect_with_agent.json \
REPORT_OUT=artifacts/perf/perf-slo-report.md \
SUMMARY_OUT=artifacts/perf/perf-slo-summary.json \
scripts/perf_slo_check.sh

The table rows are scenario-based (audit_mode_open, audit_mode_connect, workload rows, optional soak rows) and each row carries a concrete budget and pass/fail status.

Perf artifact contract

Required artifacts for each strict perf run:

  • open_compare.json
  • open_baseline.json
  • open_with_agent.json
  • connect_baseline.json
  • connect_with_agent.json
  • workload_suite.json
  • perf-slo-report.md
  • perf-slo-summary.json
  • perf-evidence-report.md
  • kernel.txt
  • os-release.txt
  • cpu.txt
  • fs.txt