Skip to content

Commit e9a1e46

Browse files
samcmsamcm
authored
ci: strengthen CI with race detection, govulncheck, and more (#759)
* ci: strengthen CI with race detection, module verification, and protobuf validation Add race detection and go mod tidy checks as parallel jobs in the test workflow. Add buf lint and breaking change detection for protobuf schemas. Add concurrency groups to test, lint, and build workflows to cancel stale runs. Add path filtering to smoke tests so they skip on non-code changes. Switch smoke test Docker builds to use GHA layer caching for faster rebuilds. Enable exhaustive, protogetter, and sloglint linters. Fix a flaky test in mempool watcher exposed by the race detector where the background processor could delete a transaction before the test asserted on it. * ci: consolidate test workflow into single sequential job Merge race detection, mod-tidy, and test jobs into one runner. The race detector already runs all tests, so a separate non-race run was redundant. * ci: use archive beacon node for cannon smoke test The non-archive node returns 404 for historical committee/proposer duty queries that cannon needs. * ci: add health checks and improve smoke test diagnostics - Add xatu-server health check step before starting sentry/cannon (catches DNS/startup failures early with clear error messages) - Bump sentry smoke test timeout from 10 to 15 minutes (builds can take 9+ minutes, leaving no time for verification) - Change failure artifact collection to run on failure OR cancelled (job timeouts previously killed diagnostic steps) - Add diagnostic dump step showing container status, network state, and recent logs on failure * ci: use pre-built Buildx images in smoke tests Add image tags to docker-compose.yml so compose services use the Buildx-built images instead of rebuilding from scratch. The compose rebuild was using a stale Buildx cache that pulled in conflicting otel schema versions, causing xatu-server to crash on startup. With --build, compose now builds and tags correctly. Without --build (CI), it uses the pre-built images directly. * fix(observability): tolerate otel schema URL conflicts in resource merge resource.Merge returns ErrSchemaURLConflict when resource.Default() and our semconv attributes use different schema versions (e.g. 1.39.0 vs 1.34.0 due to transitive dependency version skew). The merged resource is still valid and usable, so we tolerate this specific error. * ci: fix health check to use TCP port check instead of wget The xatu container doesn't have wget, and port 8080 serves gRPC not HTTP. Use bash /dev/tcp for a simple TCP connectivity check from the host instead of docker exec wget inside the container.
1 parent 9427d28 commit e9a1e46

10 files changed

Lines changed: 212 additions & 47 deletions

File tree

‎.github/workflows/buf.yaml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: Protobuf
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- '**/*.proto'
7+
- 'buf.yaml'
8+
- 'buf.gen.yaml'
9+
- 'buf.lock'
10+
workflow_dispatch:
11+
12+
concurrency:
13+
group: ${{ github.workflow }}-${{ github.ref }}
14+
cancel-in-progress: true
15+
16+
permissions:
17+
contents: read
18+
19+
jobs:
20+
buf:
21+
runs-on: ubuntu-latest
22+
steps:
23+
- name: checkout
24+
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
25+
26+
- name: Set up buf
27+
uses: bufbuild/buf-setup-action@a47c93e0b1648769eb9a2e1f98e7b4e3e13089f0 # v1.50.0
28+
29+
- name: Lint protos
30+
run: buf lint
31+
32+
- name: Check for breaking changes
33+
uses: bufbuild/buf-breaking-action@c57b3d842a5c3f3b454756ef65305a50a587c5ba # v1.1.4
34+
with:
35+
against: 'https://github.com/ethpandaops/xatu.git#branch=master'

‎.github/workflows/cannon-smoke-test.yaml‎

Lines changed: 61 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,15 @@ name: Cannon Smoke Test
22

33
on:
44
pull_request:
5+
paths:
6+
- '**.go'
7+
- 'go.mod'
8+
- 'go.sum'
9+
- 'Dockerfile'
10+
- 'docker-compose.yml'
11+
- 'deploy/**'
12+
- '.github/cannon/**'
13+
- '.github/workflows/cannon-smoke-test.yaml'
514
workflow_dispatch:
615

716
concurrency:
@@ -32,15 +41,21 @@ jobs:
3241
- name: Pull docker compose images in the background
3342
run: |
3443
docker compose pull &
44+
- name: Set up Docker Buildx
45+
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
3546
- name: Build xatu image
36-
run: |
37-
docker build -t ethpandaops/xatu:local .
38-
echo "Xatu image is built."
47+
uses: docker/build-push-action@14487ce63c7a62a4a324b0bfb37086795e31c6c1 # v6.16.0
48+
with:
49+
context: .
50+
load: true
51+
tags: ethpandaops/xatu:local
52+
cache-from: type=gha
53+
cache-to: type=gha,mode=max
3954
- name: Fetch finalized epoch from beacon node
4055
id: beacon
4156
run: |
4257
# Read config
43-
BEACON_NODE="https://mainnet-lb-bn.utility.production.platform.ethpandaops.io"
58+
BEACON_NODE="https://mainnet-archive-lb-bn.utility.production.platform.ethpandaops.io"
4459
EPOCH_OFFSET=$(yq '.epoch_offset' .github/cannon/seeding.yaml)
4560
NETWORK_ID=$(yq '.network.id' .github/cannon/seeding.yaml)
4661
NETWORK_NAME=$(yq '.network.name' .github/cannon/seeding.yaml)
@@ -78,7 +93,30 @@ jobs:
7893
timeout-minutes: 10
7994
shell: bash
8095
run: |
81-
docker compose up --build --detach --quiet-pull
96+
docker compose up --detach --quiet-pull
97+
- name: Wait for Xatu stack to be ready
98+
run: |
99+
echo "Waiting for xatu-server gRPC to be ready on port 8080..."
100+
for i in $(seq 1 30); do
101+
if bash -c 'echo > /dev/tcp/localhost/8080' 2>/dev/null; then
102+
echo "xatu-server is ready."
103+
break
104+
fi
105+
if [ "$i" -eq 30 ]; then
106+
echo "::error::xatu-server failed to become ready"
107+
echo "=== Container status ==="
108+
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
109+
echo "=== xatu-server logs ==="
110+
docker logs xatu-server 2>&1 | tail -50
111+
exit 1
112+
fi
113+
echo " attempt $i/30..."
114+
sleep 2
115+
done
116+
117+
echo ""
118+
echo "=== Container status ==="
119+
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
82120
- name: Seed Postgres with cannon data
83121
run: |
84122
set -e
@@ -153,7 +191,7 @@ jobs:
153191
ntpServer: time.google.com
154192
155193
ethereum:
156-
beaconNodeAddress: "https://mainnet-lb-bn.utility.production.platform.ethpandaops.io"
194+
beaconNodeAddress: "https://mainnet-archive-lb-bn.utility.production.platform.ethpandaops.io"
157195
beaconNodeHeaders:
158196
Authorization: "AUTH_HEADER"
159197
overrideNetworkName: "$NETWORK_NAME"
@@ -191,16 +229,30 @@ jobs:
191229
NETWORK_NAME: ${{ steps.beacon.outputs.network_name }}
192230
run: |
193231
.github/cannon/assert_clickhouse.sh .github/cannon/seeding.yaml
232+
- name: Dump diagnostic info on failure
233+
if: failure() || cancelled()
234+
run: |
235+
echo "=== Container status ==="
236+
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
237+
echo ""
238+
echo "=== Docker network ==="
239+
docker network inspect xatu_xatu-net --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}' 2>/dev/null || echo "Network not found"
240+
echo ""
241+
echo "=== xatu-server logs (last 50) ==="
242+
docker logs xatu-server 2>&1 | tail -50 || echo "Container not found"
243+
echo ""
244+
echo "=== xatu-cannon logs (last 50) ==="
245+
docker logs xatu-cannon 2>&1 | tail -50 || echo "Container not found"
194246
- name: Collect docker logs on failure
195-
if: failure()
247+
if: failure() || cancelled()
196248
uses: jwalton/gh-docker-logs@2741064ab9d7af54b0b1ffb6076cf64c16f0220e # v2.2.2
197249
with:
198250
dest: './logs'
199251
- name: Tar logs
200-
if: failure()
252+
if: failure() || cancelled()
201253
run: tar cvzf ./logs.tgz ./logs
202254
- name: Upload logs to GitHub
203-
if: failure()
255+
if: failure() || cancelled()
204256
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
205257
with:
206258
name: logs.tgz

‎.github/workflows/golangci-lint.yaml‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,13 @@
11
name: golangci-lint
22
on:
3-
# push:
4-
# tags:
5-
# - v*
6-
# branches:
7-
# - master
83
pull_request:
4+
5+
concurrency:
6+
group: ${{ github.workflow }}-${{ github.ref }}
7+
cancel-in-progress: true
8+
99
permissions:
1010
contents: read
11-
# Optional: allow read access to pull request. Use with `only-new-issues` option.
12-
# pull-requests: read
1311
jobs:
1412
golangci:
1513
name: lint

‎.github/workflows/sentry-smoke-test.yaml‎

Lines changed: 73 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,24 @@ name: Sentry Smoke Test
22

33
on:
44
pull_request:
5+
paths:
6+
- '**.go'
7+
- 'go.mod'
8+
- 'go.sum'
9+
- 'Dockerfile'
10+
- 'sentry-logs/**'
11+
- 'docker-compose.yml'
12+
- 'deploy/**'
13+
- '.github/sentry/**'
14+
- '.github/workflows/sentry-smoke-test.yaml'
515
workflow_dispatch:
616

717
concurrency:
818
group: ${{ github.workflow }}-${{ github.ref }}
919
cancel-in-progress: true
1020
jobs:
1121
sentry-smoke-test:
12-
timeout-minutes: 10
22+
timeout-minutes: 15
1323
runs-on: ubuntu-latest
1424
steps:
1525
- name: Checkout repository
@@ -50,16 +60,27 @@ jobs:
5060
if: steps.canary.outputs.skip != 'true'
5161
run: |
5262
docker compose pull &
63+
- name: Set up Docker Buildx
64+
if: steps.canary.outputs.skip != 'true'
65+
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
5366
- name: Build xatu image
5467
if: steps.canary.outputs.skip != 'true'
55-
run: |
56-
docker build -t ethpandaops/xatu:local .
57-
echo "Xatu image is built."
68+
uses: docker/build-push-action@14487ce63c7a62a4a324b0bfb37086795e31c6c1 # v6.16.0
69+
with:
70+
context: .
71+
load: true
72+
tags: ethpandaops/xatu:local
73+
cache-from: type=gha
74+
cache-to: type=gha,mode=max
5875
- name: Build sentry-logs image
5976
if: steps.canary.outputs.skip != 'true'
60-
run: |
61-
docker build -t ethpandaops/xatu-sentry-logs:local ./sentry-logs
62-
echo "Sentry-logs image is built."
77+
uses: docker/build-push-action@14487ce63c7a62a4a324b0bfb37086795e31c6c1 # v6.16.0
78+
with:
79+
context: ./sentry-logs
80+
load: true
81+
tags: ethpandaops/xatu-sentry-logs:local
82+
cache-from: type=gha,scope=sentry-logs
83+
cache-to: type=gha,mode=max,scope=sentry-logs
6384
- name: Create sentry-logs directory and log file
6485
if: steps.canary.outputs.skip != 'true'
6586
run: |
@@ -71,7 +92,31 @@ jobs:
7192
if: steps.canary.outputs.skip != 'true'
7293
timeout-minutes: 5
7394
run: |
74-
docker compose up --build --detach --quiet-pull
95+
docker compose up --detach --quiet-pull
96+
- name: Wait for Xatu stack to be ready
97+
if: steps.canary.outputs.skip != 'true'
98+
run: |
99+
echo "Waiting for xatu-server gRPC to be ready on port 8080..."
100+
for i in $(seq 1 30); do
101+
if bash -c 'echo > /dev/tcp/localhost/8080' 2>/dev/null; then
102+
echo "xatu-server is ready."
103+
break
104+
fi
105+
if [ "$i" -eq 30 ]; then
106+
echo "::error::xatu-server failed to become ready"
107+
echo "=== Container status ==="
108+
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
109+
echo "=== xatu-server logs ==="
110+
docker logs xatu-server 2>&1 | tail -50
111+
exit 1
112+
fi
113+
echo " attempt $i/30..."
114+
sleep 2
115+
done
116+
117+
echo ""
118+
echo "=== Container status ==="
119+
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
75120
- name: Inject sentry-logs test data
76121
if: steps.canary.outputs.skip != 'true'
77122
run: |
@@ -200,16 +245,33 @@ jobs:
200245
echo "Sample execution_block_metrics data:"
201246
docker exec xatu-clickhouse-01 clickhouse-client --query "SELECT block_number, total_ms, mgas_per_sec, meta_client_name FROM default.execution_block_metrics LIMIT 5"
202247
248+
- name: Dump diagnostic info on failure
249+
if: failure() || cancelled()
250+
run: |
251+
echo "=== Container status ==="
252+
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
253+
echo ""
254+
echo "=== Docker network ==="
255+
docker network inspect xatu_xatu-net --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}' 2>/dev/null || echo "Network not found"
256+
echo ""
257+
echo "=== xatu-server logs (last 50) ==="
258+
docker logs xatu-server 2>&1 | tail -50 || echo "Container not found"
259+
echo ""
260+
echo "=== xatu-sentry logs (last 50) ==="
261+
docker logs xatu-sentry 2>&1 | tail -50 || echo "Container not found"
262+
echo ""
263+
echo "=== xatu-sentry-logs logs (last 30) ==="
264+
docker logs xatu-sentry-logs 2>&1 | tail -30 || echo "Container not found"
203265
- name: Collect docker logs on failure
204-
if: failure()
266+
if: failure() || cancelled()
205267
uses: jwalton/gh-docker-logs@2741064ab9d7af54b0b1ffb6076cf64c16f0220e # v2.2.2
206268
with:
207269
dest: './logs'
208270
- name: Tar logs
209-
if: failure()
271+
if: failure() || cancelled()
210272
run: tar cvzf ./logs.tgz ./logs
211273
- name: Upload logs to GitHub
212-
if: failure()
274+
if: failure() || cancelled()
213275
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
214276
with:
215277
name: logs.tgz

‎.github/workflows/test-build.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ on:
44
pull_request:
55
workflow_dispatch:
66

7+
concurrency:
8+
group: ${{ github.workflow }}-${{ github.ref }}
9+
cancel-in-progress: true
10+
711
jobs:
812
goreleaser:
913
runs-on: ubuntu-latest

‎.github/workflows/test.yaml‎

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,15 @@ name: Go Test
22

33
on:
44
pull_request:
5-
65
workflow_dispatch:
76
branches: [ '**' ]
87

9-
jobs:
10-
full_ci:
11-
strategy:
12-
matrix:
13-
go_version: [ 1.25.x ]
8+
concurrency:
9+
group: ${{ github.workflow }}-${{ github.ref }}
10+
cancel-in-progress: true
1411

12+
jobs:
13+
test:
1514
runs-on: ubuntu-latest
1615

1716
steps:
@@ -21,10 +20,18 @@ jobs:
2120
- name: Set up Go
2221
uses: actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5 # v5.5.0
2322
with:
24-
go-version: ${{ matrix.go_version }}
25-
26-
- name: run tests
27-
run: go test -json ./... | tee test.json
23+
go-version: '1.25.x'
24+
25+
- name: Check go mod tidy
26+
run: |
27+
go mod tidy
28+
go mod verify
29+
git diff --exit-code go.mod go.sum
30+
31+
- name: Run tests with race detector
32+
env:
33+
CGO_ENABLED: "1"
34+
run: go test -race -json -timeout 10m ./... | tee test.json
2835

2936
- name: Annotate tests
3037
if: always()

‎.golangci.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@ linters:
3636
- tagliatelle
3737
- thelper
3838
- tparallel
39+
- exhaustive
40+
- protogetter
41+
- sloglint
3942
- unconvert
4043
- unused
4144
- whitespace
@@ -56,6 +59,8 @@ linters:
5659
govet:
5760
enable:
5861
- shadow
62+
exhaustive:
63+
default-signifies-exhaustive: true
5964
nolintlint:
6065
require-explanation: true
6166
require-specific: true

0 commit comments

Comments
 (0)