Guard short RTPS messages in the DDSI security scan

[Carmel0]

Summary

I found a use of uninitialized memory while fuzzing CycloneDDS, then confirmed the same root cause on current master with Valgrind Memcheck and native packet tests.

In src/core/ddsi/src/ddsi_receive.c, handle_rtps_message() allows a packet that contains only the RTPS header to reach the DDS Security decode path, because it only rejects packets smaller than DDSI_RTPS_MESSAGE_HEADER_SIZE.

if (sz < DDSI_RTPS_MESSAGE_HEADER_SIZE || *(uint32_t *)msg != DDSI_PROTOCOLID_AS_UINT32)
{
  /* discard packets that are really too small or don't have magic cookie */
}
...
ddsi_rtps_msg_state_t res =
  ddsi_security_decode_rtps_message (thrst, gv, &rmsg, &hdr, &msg, &sz, rbpool, conn->m_stream);

In src/core/ddsi/src/ddsi_security_omg.c, ddsi_security_decode_rtps_message() receives size_t *sz, but the early security check is called without that length.

ret = check_rtps_message_is_secure (gv, *hdr, *buff, isstream, &proxypp);

In the same file, check_rtps_message_is_secure() then reads the first submessage byte without checking whether the byte is part of the current message.

const uint32_t offset = DDSI_RTPS_MESSAGE_HEADER_SIZE +
  (isstream ? sizeof (ddsi_rtps_msg_len_t) : 0);
const ddsi_rtps_submessage_header_t *submsg =
  (const ddsi_rtps_submessage_header_t *) (buff + offset);
if (submsg->submessageId != DDSI_RTPS_SMID_SRTPS_PREFIX)
  return DDSI_RTPS_MSG_STATE_PLAIN;

For UDP, offset is DDSI_RTPS_MESSAGE_HEADER_SIZE, which is 20. An RTPS packet that contains only the header is exactly 20 bytes long, so buff + 20 is outside the logical packet contents. In practice this address is still inside CycloneDDS' reused receive buffer, so the read consumes stale or uninitialized buffer contents rather than data from the current packet.

Native packet tests show the parser effect in normal execution. A stale 0x33 byte from previous receive buffer contents can make a malformed RTPS message that contains only the header look like it starts with DDSI_RTPS_SMID_SRTPS_PREFIX.

I also looked for stronger impact from this behavior. I did not find a practical path to information disclosure, authentication bypass, crash, or sustained resource exhaustion, so I am filing it as a robustness bug rather than as a security issue.

Version Notes

The unchecked early security check appears to have been introduced in 66c0d878 (Encoding preparations (#329), December 6, 2019), which added check_rtps_message_is_secure() with the buff + offset submessage read and no message length check.

The same pattern is still present in current master at 2b8dceb4 (Reduce stack size usage in ddsperf., May 26, 2026). The evidence below uses this commit.

Reproduction

The packet prefix below is a minimal RTPS header.

52 54 50 53 02 01 01 10 01 02 03 04 05 06 07 08 09 0a 0b 0c

I sent the following sequence to the same CycloneDDS receive port using two UDP sockets with different source ports.

1. Socket A sends the RTPS header followed by 0x55.
2. Socket B sends only the RTPS header.
3. Socket A sends the RTPS header followed by 0x33, which is DDSI_RTPS_SMID_SRTPS_PREFIX.
4. Socket B sends only the RTPS header.

With tracing enabled, the native trace shows the negative control first. After socket A sends 0x55, socket B's packet with only the RTPS header is logged as a header and is not classified as encoded.

recv: HDR(1020304:5060708:90a0b0c vendor 1.16) len 21 from udp/127.0.0.1:47903
recv: HDR(1020304:5060708:90a0b0c vendor 1.16) len 20 from udp/127.0.0.1:43023

After socket A sends 0x33, socket B's packet with only the RTPS header is classified as an encoded RTPS message even though socket B did not send a submessage byte.

recv: HDR(1020304:5060708:90a0b0c vendor 1.16) len 21 from udp/127.0.0.1:40045
recv: received encoded rtps message from unknown participant
recv: HDR(1020304:5060708:90a0b0c vendor 1.16) len 20 from udp/127.0.0.1:35424
recv: received encoded rtps message from unknown participant

This demonstrates that the stale byte is process receive buffer state, not data from socket B's packet itself.

Memcheck Check

Valgrind Memcheck reports the same parser decision as a use of uninitialized memory on current master.

Conditional jump or move depends on uninitialised value(s)
   at check_rtps_message_is_secure (ddsi_security_omg.c:3684)
   by ddsi_security_decode_rtps_message (ddsi_security_omg.c:3776)
   by handle_rtps_message (ddsi_receive.c:3299)
   by do_packet (ddsi_receive.c:3387)
   by ddsi_recv_thread (ddsi_receive.c:3672)

Uninitialised value was created by a heap allocation
   by ddsrt_malloc (heap.c:43)
   by ddsi_rbuf_alloc_new (ddsi_radmin.c:432)
   by ddsi_rbufpool_new (ddsi_radmin.c:363)

Patch Check

On the same current master commit, I also tested a local guard before reading submsg->submessageId.

const uint32_t offset = DDSI_RTPS_MESSAGE_HEADER_SIZE +
  (isstream ? sizeof (ddsi_rtps_msg_len_t) : 0);
if (sz <= offset)
  return DDSI_RTPS_MSG_STATE_PLAIN;

With this guard, an explicit packet containing the RTPS header followed by 0x33 is still classified as encoded, but the following packet from a different socket that contains only the RTPS header is not.

recv: HDR(1020304:5060708:90a0b0c vendor 1.16) len 21 from udp/127.0.0.1:40322
recv: received encoded rtps message from unknown participant
recv: HDR(1020304:5060708:90a0b0c vendor 1.16) len 20 from udp/127.0.0.1:45260

Impact

The demonstrated effect is a parser classification error for a malformed RTPS packet that contains only the header. A stale 0x33 can make such a packet enter the encoded message path, where it is rejected as an invalid encoded RTPS message.

Suggested Fix

Pass the message length into check_rtps_message_is_secure() and check that the first submessage byte is present before reading it. The minimal guard I tested was sz <= offset, because the current helper only reads submessageId.

[Carmel0]

@eboasson Hi, just following up on the issue I reported. Would this be considered for a fix? Thanks.

[eboasson]

Hi @Carmel0, thanks for reminding me! I completely agree with your impact analysis, but it should nonetheless be fixed.