bgpd: accept extended-size BGP messages on receive (RFC 8654)

[rdemsystems]

The receive path (validate_header() / read_ibuf_work() in bgp_io.c) caps an incoming message at peer->max_packet_size and, when exceeded, drops it with NOTIFICATION Message Header Error / Bad Message Length.

peer->max_packet_size is the negotiated send limit: it is set to the extended size (65535) only when the Extended Message capability (RFC 8654) was both advertised and received, it is computed once while parsing the peer's OPEN (bgp_open_option_parse()), and it is never refreshed. If that value is left at the standard size (4096) on a connection -- e.g. our OPEN had not been built yet when the peer's OPEN was parsed, a state then preserved across collision resolution by peer_xfer_conn() -- it sticks for the life of the session.

RFC 8654 makes the limit asymmetric: a speaker that has advertised the capability MUST be able to receive messages up to 65535 octets; the peer's advertisement governs only what we may send. FRR advertises the capability unconditionally, so the receive limit must allow extended-size messages regardless of peer->max_packet_size. Two FRR speakers that both advertised and received the capability could therefore still tear each other down in a loop once one packed an UPDATE larger than 4096 octets (observed with a 7795-octet add-path UPDATE: NOTIFICATION data 0x1E73).

Use a dedicated, type-aware receive limit: the extended size for every message except OPEN and KEEPALIVE (which RFC 8654 never extends), independent of peer->max_packet_size which remains the send/update-group sizing field.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[greptile-apps]

Greptile Summary

This PR fixes an RFC 8654 compliance bug on the BGP receive path where incoming messages larger than 4096 bytes were incorrectly rejected with NOTIFICATION Message Header Error / Bad Message Length, even when the Extended Message capability had been (or should have been) negotiated. The root cause was that both read_ibuf_work and validate_header gated the receive limit on peer->max_packet_size, a field that tracks the send limit and can remain at the standard 4096 value during collision resolution.

• Introduces bgp_rx_max_packet_size(type): a pure, type-aware helper that returns 65535 for all message types that RFC 8654 allows to be extended (UPDATE, NOTIFY, ROUTE_REFRESH, CAPABILITY), and 4096 for OPEN and KEEPALIVE which are permanently fixed-size.
• Replaces both peer->max_packet_size comparisons in read_ibuf_work and validate_header with calls to this helper, decoupling the receive-side enforcement from the send-side negotiated limit.

Confidence Score: 4/5

The change is safe to merge; it fixes an observed session-teardown loop and correctly implements the RFC 8654 asymmetric receive semantics across both the size-check sites in the I/O thread.

The fix is narrowly scoped to the two places in bgp_io.c that gate incoming message size. The new helper is a pure, stateless function and the change does not touch state machines, peer state, or send paths. The unconditional 65535-byte receive allowance for non-OPEN/KEEPALIVE types is consistent with RFC 8654 Section 4 and with FRR's unconditional advertisement of the Extended Message capability. No existing checks are weakened; the only behavioral change is that oversized extended messages which were previously rejected with a spurious NOTIFICATION are now accepted.

bgpd/bgp_io.c is the only changed file; both modified sites (read_ibuf_work and validate_header) are straightforward one-line replacements of the size guard.

Important Files Changed

Filename	Overview
bgpd/bgp_io.c	Adds bgp_rx_max_packet_size() helper and replaces peer->max_packet_size with it in both read_ibuf_work and validate_header; logic is correct and consistent with RFC 8654.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Net as TCP / Network
    participant RH as read_ibuf_work
    participant VH as validate_header
    participant RX as bgp_rx_max_packet_size
    participant FSM as BGP FSM

    Net->>RH: bytes arrive in ibuf_work ringbuf
    RH->>VH: validate_header()
    VH->>VH: check marker (16 bytes)
    VH->>VH: peek type at offset 18
    VH->>RX: bgp_rx_max_packet_size(type)
    RX-->>VH: 4096 (OPEN/KEEPALIVE) or 65535 (UPDATE/NOTIFY/etc.)
    VH->>VH: "size > rx_max? NOTIFICATION + false"
    VH-->>RH: true (header valid)
    RH->>RH: peek pktsize, peek pkttype
    RH->>RX: bgp_rx_max_packet_size(pkttype)
    RX-->>RH: rx_max
    RH->>RH: stream_new(pktsize), ringbuf_get
    RH->>FSM: enqueue to ibuf (via io_mtx)
    FSM->>FSM: process message up to 65535 bytes

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Net as TCP / Network
    participant RH as read_ibuf_work
    participant VH as validate_header
    participant RX as bgp_rx_max_packet_size
    participant FSM as BGP FSM

    Net->>RH: bytes arrive in ibuf_work ringbuf
    RH->>VH: validate_header()
    VH->>VH: check marker (16 bytes)
    VH->>VH: peek type at offset 18
    VH->>RX: bgp_rx_max_packet_size(type)
    RX-->>VH: 4096 (OPEN/KEEPALIVE) or 65535 (UPDATE/NOTIFY/etc.)
    VH->>VH: "size > rx_max? NOTIFICATION + false"
    VH-->>RH: true (header valid)
    RH->>RH: peek pktsize, peek pkttype
    RH->>RX: bgp_rx_max_packet_size(pkttype)
    RX-->>RH: rx_max
    RH->>RH: stream_new(pktsize), ringbuf_get
    RH->>FSM: enqueue to ibuf (via io_mtx)
    FSM->>FSM: process message up to 65535 bytes

Loading

_{Reviews (1): Last reviewed commit: "bgpd: accept extended-size BGP messages ..." | Re-trigger Greptile}

[rdemsystems]

Full packet capture illustrating the issue: https://pkg.rdem-systems.com/ippi-bgp.pcap

[rdemsystems]

Just tested the build, and session works now, while non-upgraded router does not accept the same session.

IPPI Router <-> Rdem Systems 1 (patched) : OK
IPPI router <-> Rdem Systems 2 (not patched) : Not OK

[donaldsharp]

We must fix the problem in the resetting of the peer connection and before we construct a open message. That is the bug. I do not want to refigure the value every time we receive a packet that makes no sense.

[rdemsystems]

Dear @donaldsharp ;

Thanks for your comment.
Is the second patch better?

Regards,

[rdemsystems]

P.S : Why the OSPF check fails with BGP patch? :/ i don't see any OSPF patch in my patch :(

[ton31337]

LGTM

[ton31337]

Let's fix frrbot (styling) yet.

[rdemsystems]

Let's fix frrbot (styling) yet.

Dear,

Style applied.
Is it OK now?
Is there any other bad points?

[riw777]

looks good

[rdemsystems]

I'm not an expert in Github, must i create the same merge request for stable/10.5 and stable/10.6 ?
@riw777 @ton31337 @donaldsharp ?