Net Message Overflows

Hello Facepunch!
For a few years now, on-and-off I’ll have my server’s population slash in half because of players being kicked off from the server for the reason (Client -number- overflowed reliable channel). I have tried numerous approaches such as disabling networking-heavy addons, as well as recording all net messages being send to the server by the clients, however, I have been unable to determine the cause of this issue. I would appreciate any pointers.
Two separate occasions where this issue occurred with 100+ players online can be seen over here:
A)


B)

Looks like somebody is spamming net messages and causing crashes, you can install a net message limiter to fix this, here is one I found on github

I appreciate your help, but as you can see in the screenshot, my netmessage logger is not picking up any netmessages being spammed by a client, am I missing something?

PS, in my screeshots you can see that I already use Hexane’s net debugger with the threshhold at 100, “hexanenetlibrarydebug.threshold = 100”

A client likely isn’t spamming messages, but the server is spamming messages to a client, causing him to crash.
This is likely a bug in the addon’s code, could be also exploited though by a client.
Check all the net messages you saw in the log, try to remove these addons to see if the issue is resolved.

I figured as much, and I appreciate your response. I was wondering if anybody is familiar with a tool to potentially log netmessages that are server>client so that way I am able to potentially see a live feed, as opposed to removing addons until I find the correct one?

You can try detouring net.Incoming on the clientside, and log it there. Something like

function net.Incoming(len, p)
   local i = net.ReadHeader()
   local strName = util.NetworkIDToString( i )

   if ( !strName ) then return end

   local func = net.Receivers[ strName:lower() ]
   if ( !func ) then return end
   len = len - 16

   func( len, client )
   print("Received message from server: " .. strName)
end
1 Like

You’ll probably have more success if you detour net.Start (To grab the message name) and the various net.Send/Broadcast functions on the server instead. That way you don’t have to rely on the players going through and sending you their logs when they get affected.

Looking at the images you’ve sent it seems to trigger just after someone’s connected, based on that my first guesses would be that there’s either some wonky synchronization code that’s sending more data the more players are connected or there’s just so much data being networked regularly that that one player starts pushing people over the limit.

After talking with Garnet on steam I’ve taken some time to write a script that should (hopefully) make debugging this a bit easier, I’m just posting it here in-case anyone else has any use for it.

Example output:

1 Like