zpoline-snic

Transparent socket hook that lets an unmodified TCP application receive TCP payload from a ContaTOE-derived SmartNIC that terminates the TCP connection on-chip and delivers per-segment data with a 64-byte metadata header (struct toe_hdr) directly to a PF passed through to the container's netns.

Corresponds to the NIC Passthrough architecture in the ContaTOE paper (Fig. 3, right). No eBPF translator, no veth bridge — the PF is inside the container, our hook opens AF_PACKET SOCK_RAW on it, parses [toe_hdr | payload] frames, and turns them into a normal socket API.

Unlike the earlier LD_PRELOAD variant in nic-toe-sw/ld_preload, this one hooks at the syscall level via zpoline, and only adopts fds used server-side. Outbound connect()s pass straight through to the kernel TCP stack, so an application like nginx-thrift (DeathStarBench) can accept client traffic through the SmartNIC while its RPCs to backend services still use the normal stack.

Layout

zpoline-snic/
├── zpoline/          # upstream zpoline (submodule)
├── hook/             # our libzphook_nictoe.so
│   ├── main.c        #   zpoline syscall dispatcher
│   ├── nictoe.c/.h   #   fd-table + listener/accept + rx/tx logic
│   └── pf_io.c/.h    #   AF_PACKET SOCK_RAW on the PF, toe_hdr parse/build
├── scripts/
│   ├── setup_mmap_min_addr.sh
│   ├── nginx/                        # dual nginx use case
│   │   ├── nginx_a.conf, nginx_b.conf
│   │   ├── run_dual_nginx.sh
│   │   └── attach_pf.sh              # `ip link set <pf> netns <container>`
│   └── dsb/                          # DeathStarBench socialNetwork
│       └── docker-compose.override.yml
└── Makefile

Build

sudo apt install binutils-dev            # for libopcodes (zpoline)
sudo sh -c 'echo 0 > /proc/sys/vm/mmap_min_addr'
git submodule update --init --recursive  # pulls zpoline
make                                     # zpoline/libzpoline.so + hook/libzphook_nictoe.so

How the hook is selective

Which fds go through the hook is decided per-syscall, not per-process:

syscall	behavior
socket()	pass-through — always returns a real kernel fd
connect()	pass-through — outbound TCP is native kernel
bind()	pass-through
listen(fd)	adopts fd: replaces its kernel object with an eventfd via dup3, marks it as a hooked listener, and lazily brings up the AF_PACKET RX ring on $NICTOE_IFACE
accept(fd)	if fd is a hooked listener → returns an eventfd bound to a new TOE session; else pass-through
read/write/…	if fd is in our table (i.e. accepted from a hooked listener) → PF I/O; else pass-through
epoll_wait	wakes on both real fd events and PF frame arrivals; internal RX-fd events are filtered out

Wire format on the PF

Every frame on the passthrough'd PF is [struct toe_hdr | payload]. See hook/pf_io.h for the full 64-byte header. The fields we actually use on the RX side:

• sessionID — TCP session multiplexer (16 bit)
• ctrl_mode — 0 = stream data, 5 = close req to TOE, 6 = close req from TOE
• appNotification_length — payload byte count
• appNotification_dstPort, appNotification_ipAddress — for getpeername

On the TX side we build the same header with destCoreNum = DEST_TOE (0xfffe) and verify_pkt = 0xdeadbeef. All fields are native little-endian (matches what nic-toe/ebpf/toe.cpp writes/reads).

Use case 1 — two nginx containers, routed by Host header

The SmartNIC's ARM core matches the HTTP Host header and forwards the decapsulated TCP payload frames to one of two PFs. Two nginx containers, each with one PF passed through into its netns, receive their share.

# 0. Prereqs (see Build)
scripts/setup_mmap_min_addr.sh
make -C arm-sample all wasm     # builds wasm_loader + l7lb_nginx.wasm

# 1. Load the ARM-core WASM. MUST be done while the PF is still in the
#    host netns — after step 2 it moves into the container and disappears
#    from `ip link`. If that happens, `docker rm -f zp-nginx-{a,b}` returns
#    the PF to the host.
sudo arm-sample/wasm_loader enp129s0f0np0 0 arm-sample/l7lb_nginx.wasm

# 2. Launch both containers, move a PF into each
PF_A=enp129s0f0np0 PF_B=enp129s0f1np1 \
  scripts/nginx/run_dual_nginx.sh

# 3. Once ARM-side routing rules are in place, from another host:
curl -H 'Host: a.example' http://<smartnic-ip>/    # → nginx-a
curl -H 'Host: b.example' http://<smartnic-ip>/    # → nginx-b

nginx worker_processes for each container can be overridden at launch time via WORKERS_A / WORKERS_B (default 1) — the nginx_{a,b}.conf files intentionally omit the directive so this can win. Useful when sweeping backend parallelism against ARM-core parallelism:

PF_A=... PF_B=... WORKERS_A=4 WORKERS_B=4 \
  scripts/nginx/run_dual_nginx.sh

If you'd rather attach the PFs manually:

scripts/nginx/attach_pf.sh zp-nginx-a enp129s0f0np0
scripts/nginx/attach_pf.sh zp-nginx-b enp129s0f1np1

Use case 2 — two DeathStarBench socialNetwork stacks

Only the nginx-thrift frontend of each stack is hooked. Downstream thrift RPCs to user-service, post-storage-service, etc. use the normal kernel TCP stack (unchanged latency).

# Assume you have two DSB checkouts:
#   /opt/dsb-a/socialNetwork
#   /opt/dsb-b/socialNetwork
export ZPOLINE_ROOT=~/Projects/aist/zpoline-snic

for d in /opt/dsb-a/socialNetwork /opt/dsb-b/socialNetwork; do
  cp scripts/dsb/docker-compose.override.yml "$d/"
done

# NICTOE_IFACE must equal the PF name we'll attach — attach_pf.sh doesn't
# rename anymore, so the PF keeps its host name inside the container.
( cd /opt/dsb-a/socialNetwork && NICTOE_IFACE=enp129s0f0np0 docker-compose -p dsb-a up -d )
( cd /opt/dsb-b/socialNetwork && NICTOE_IFACE=enp129s0f1np1 docker-compose -p dsb-b up -d )

# Move a PF into each nginx-thrift container:
scripts/nginx/attach_pf.sh dsb-a_nginx-thrift_1 enp129s0f0np0
scripts/nginx/attach_pf.sh dsb-b_nginx-thrift_1 enp129s0f1np1

Runtime env vars

• NICTOE_IFACE — the PF interface name inside the container. Since attach_pf.sh no longer renames the PF, this must match the host-side PF name (run_dual_nginx.sh sets it to PF_A / PF_B per container).
• NICTOE_IFACE_WAIT_MS — how long to wait for the PF to appear in the container's netns after startup (default 30000).
• NICTOE_EAGER_INIT=1 — open the AF_PACKET ring at hook init rather than on first listen().
• NICTOE_DEBUG=1 — verbose hook log.
• PF_IO_DEBUG=1 — verbose L2 frame log.

Debugging

• Verify the hook loaded: sudo docker exec <name> cat /proc/1/maps | grep -E 'zpoline|nictoe'.
• Verify the PF is inside the container: sudo docker exec <name> ip -o link show.
• Verify frames arrive: run NICTOE_DEBUG=1 PF_IO_DEBUG=1; expect pf_io[…]: rx sid=… ctrl=… … on every incoming frame.
• If mmap failed at zpoline init: mmap_min_addr isn't 0 or SELinux is blocking mmap(0).