IPv4 multicast TTL and loopback use a different ABI than std and system headers on BSD-family targets

[lopopolo]

Report provenance

This bug was found by OpenAI Codex and validated by @lopopolo.

Summary

socket2 uses c_int for IP_MULTICAST_TTL and IP_MULTICAST_LOOP everywhere.

The BSD system headers mandate unsigned char for these options and this is reflected by Rust std carrying a target-specific ABI split for these exact options to use c_uchar on:

• dragonfly
• freebsd
• openbsd
• netbsd
• solaris
• illumos
• nto

Root cause

The IPv4 multicast getters and setters in socket2 use c_int directly for:

• Socket::multicast_ttl_v4
• Socket::set_multicast_ttl_v4
• Socket::multicast_loop_v4
• Socket::set_multicast_loop_v4

Rust std instead carries an internal IpV4MultiCastType alias and makes it
c_uchar on the targets above:

• std source: https://doc.rust-lang.org/nightly/src/std/sys/net/connection/socket/mod.rs.html

The platform headers and manuals also point in that direction. For example:

• FreeBSD in.h annotates IP_MULTICAST_TTL and IP_MULTICAST_LOOP as u_char
• NetBSD ip(4) documents IP_MULTICAST_TTL as u_char ttl and IP_MULTICAST_LOOP as u_char loop: https://man.netbsd.org/NetBSD-9.2/emips/ip.4

Validation

I checked this on FreeBSD 15.0-RELEASE-p5 amd64. A raw getsockopt probe
returned a four-byte payload for both options:

IP_MULTICAST_TTL rc=0 len=4 bytes=01 00 00 00
IP_MULTICAST_LOOP rc=0 len=4 bytes=01 00 00 00

So I am not claiming a confirmed security issue on that host.

The bug here is the ABI divergence from std:

• socket2 and std disagree on the value type for these options
• BSD-family headers/docs suggest a narrower type
• at least one current FreeBSD kernel still returns a 4-byte payload in practice

Suggested direction

Match std's target-specific alias for these options:

• introduce an internal IpV4MultiCastType
• use u8 / c_uchar on dragonfly, freebsd, openbsd, netbsd, solaris, illumos, and nto
• use c_int elsewhere
• use that type in both the getters and setters for IP_MULTICAST_TTL and IP_MULTICAST_LOOP

[Thomasdezeeuw]

I'm not going to read this long AI generated report.

[lopopolo]

Hi Thomas, would you prefer I open a PR instead? There is a legitimate discrepancy between socket2 and the system headers.

I have validated this by hand and also disclosed my methodology intentionally.

[Thomasdezeeuw]

@lopopolo sure, ideally with a test to show it's actually wrong/broken.

[lopopolo]

I've validated the ABI mismatch prevents socket2 from round-tripping out-of-range socketopt values and have put this up as PR #647 with a regression test and fix.

The regression test only is gated on for freebsd because that is the VM I happened to have available for reproduction. I can widen this cfg if you'd like so it covers all of the relevant BSD OSes.