User messages without a size limit

I’m sure I’m not the only one who absolutely hates the 255 (or less sometimes) byte limit for user messages. Most lazy coders right now use datastream which produces extremely bloated output and the more experienced coders write complex systems to transfer data in multiple user messages.

Yesterday I started thinking of a solution that provides the efficiency of user messages and the power of datastream. I came up with a wrapper for user messages without a size limit. Although a wrapper sounds quite primitive, I actually had to rewrite most of the user message library to get it to work.

GLON vs User messages

Since I’m writing an alternative to Datastream, it’s good to know precisely why exactly it is better and how it works. Let’s see the difference between GLON and user messages when encoding two strings and a color:

GLON (59 bytes)
[lua]astring oneastring twoaa255ab255ag255ar255[/lua]

User messages (26 bytes)
[lua]string oneastring twoaaaaa[/lua]

We see that size of the user message is just 44% of the GLON output size. Add the dozen user messages datastream uses and it’s obvious how user messages are much more efficient.

But aren’t your special user messages less efficient than regular ones?

Unfortunately, yes, they are a tiny bit less efficient. This is because the contents of the big user message are automatically spread over multiple user messages, just like your complex system was doing! However, unlike datastream, it uses only two user messages. One to send a chunk and one to indicate that the full user message has been received. Other than that the transmission procedure is exactly the same.

How does it work?

It’s fairly simple. The server keeps an array with bytes. When you send a user message and you add all kinds of types, it is not sent, but instead added to that buffer as bytes. So, when the code calls umsg.End(), the buffer contains all of the data as a big chunk of binary data. This chunk is sent in steps of 200 bytes after which a second type of user message follows that indicates the end of the transfer.

How to use it in my code?

Using it is extremely simple. You don’t have to make any changes to your code at all. Instead of the error that indicates your user messages are too big, it’ll happily send them in chunks as specified above. All you need to do is add these two files to your server’s autorun directory or include them in your script somehow:

sv_umsg.lua
cl_umsg.lua

Usage sample


if ( SERVER ) then
	umsg.Start( "Test" )
		umsg.String( "Hello, there! This is quite a long string already, at least for one that's networked. I wonder when it will stop, although it can't be longer than 255 bytes of course. Hmm... we're almost there! I should come up with something interesting to finish this message. Wait. Did this string just get longer than 255 bytes? Damn this is cheesy :V What year is it?" )
		umsg.Short( os.date( "%Y" ) )
	umsg.End()
else
	usermessage.Hook( "Test", function( um )
		local str = um:ReadString()
		print( #str, str )
		print( um:ReadShort() )
	end )
end

Feedback

Tell me what you think of it, if you encounter any errors or tell me how I can make the code more efficient.

I already have some optimization planned, but since it’s user messages and not some rendering/think stuff, I don’t think it’ll really benefit from it. Because this completely does the type conversion in Lua, it’s possible to add umsg.Double (Higher precision floating point numbers) and umsg.MultiBool (Eight bools in one byte/char) if you guys want them.

Will be testing this.

Looks good.

I tried something like this a while back, and I couldn’t get client to server working, it kept getting corrupted.

I put tons of work into it and was planning on fixing it, but never got to it, here’s my usercommand encoding and concommand/usermessage stream libraries.

http://dl.dropbox.com/u/5766132/Pcwizdan/stream_library.zip

For float encoding, use this.

Encode:
[lua]
local frac, exp = math.frexp(value)
local integerFrac = math.floor(frac * 1000)
– Float now is two integers. Exp and integerFrac
[/lua]

Decode:
[lua]
local value = math.ldexp(integerFrac / 1000, exp)
– Now we has our float again.
[/lua]

I don’t understand the comparison between glon and your usermessage. Where did all the color values and the keys go?

Color is chars after “two”

Are the strings null terminated, or do they use a byte at the start to determine length? I see no such bytes in your example, so how does Source know how big the strings are?

They are null terminated, just like regular user messages. If it used the Pascal way of defining strings, their size would be limited to 255 bytes.

Not that the limitation on string size would come into effect in normal umsgs, being limited to that size anyway.

I guess this is what you do/would use for streaming level data for your GCraft?

Yes or for bans in an admin mod or something like that.

This is much cleaner looking than datastream. Now you make one of these going from client to server!

You can get a few extra bytes out if you use a single character concommand for sending data to the server. I prefer something like “`” or another unused character.

-snip-

Looks useful.

Doesn’t appear to work for me, I get this error



Warning: Unhandled usermessage 'UM_C'
Warning: Unhandled usermessage 'UM_D'


I assume its something to do with either the gamemode,
Although this really does look useful.

I was trying to use this because the value of ‘sv_usermessage_maxsize’ won’t change for me its stuck at 256

That means you didn’t receive cl_umsg.lua clientside.

This is useful thanks Overv.

Ah, I will try it again tomorrow.

i have an error on my server…

Hook ‘WirePlayerInitSpawn’ Failed: lua\autorun\sv_umsg.lua:144: attempt to perform arithmetic on a string value

It’s probably related to the code you used, passing a string where a number was expected? Check your arguments.