Quick note: nftables and TCP MSS clamping

Another short note to myself, and whomever cares/searches later for nft or nftables, tcp mss clamping.

Somewhat surprising, many/most of the instructions found by Google are still related to iptables. I guess people stopped writing blog posts by the time nftables became widely used? 🤷

The only official documentation I can find is in the official wiki, but it doesn’t list/explain exactly how does this work/in which conditions. I think this results in posts like this one that suggest additionally limiting the packets it acts on using a size limiter, in order to prevent changing small packets.

Looking at the code that actually implements this, in net/netfilter/xt_TCPMSS.c (and not in the lower case-named file, which is about matching, TIL), in the function tcpmss_mangle_packet, first there is this comment:

/* Never increase MSS, even when setting it, as
 * doing so results in problems for hosts that rely
 * on MSS being set correctly.
 */

So at least the intent is that this always does the right thing (only decrease).

Second, the code does correctly look at both directions of the packet when using auto-clamping (set rt mtu rather than set 1452), in the if branch for XT_TCPMSS_CLAMP_PMTU. This means, it’s safer to use auto-clamping, rather than manually set the value.

And finally, there is handling of some corner cases as well (syn packet with data, syn packet without the MSS option - unlikely for modern stacks - in which case it defaults to minimal values).

All in all, it seems to me that it should always be correct to simply do what the wiki recommends, setting this for all packets traversing the host:

nft add rule ip filter forward tcp flags syn tcp option maxseg size set rt mtu

Of course, if you’d rather not do it always, but only for external interfaces, make sure you set it in both directions:

nft add rule ip filter forward iifname ppp0 tcp flags syn tcp option maxseg size set rt mtu
nft add rule ip filter forward oifname ppp0 tcp flags syn tcp option maxseg size set rt mtu

And that should be it. Well, use iifgroup/oifgroup for better rules 😉.