User messages without a size limit

[Overv]

I'm sure I'm not the only one who absolutely hates the 255 (or less sometimes) byte limit for user messages. Most lazy coders right now use datastream which produces extremely bloated output and the more experienced coders write complex systems to transfer data in multiple user messages.

Yesterday I started thinking of a solution that provides the efficiency of user messages and the power of datastream. I came up with a wrapper for user messages without a size limit. Although a wrapper sounds quite primitive, I actually had to rewrite most of the user message library to get it to work.

[b][u]GLON vs User messages[/u][/b]

Since I'm writing an alternative to Datastream, it's good to know precisely why exactly it is better and how it works. Let's see the difference between GLON and user messages when encoding two strings and a color:

[b]GLON (59 bytes)[/b]
[lua]string one
string two
a
255
b
255
g
255
r
255


[/lua]

[b]User messages (26 bytes)[/b]
[lua]string onestring two[/lua]

We see that size of the user message is just 44% of the GLON output size. Add the dozen user messages datastream uses and it's obvious how user messages are much more efficient.

[b][u]But aren't your special user messages less efficient than regular ones?[/u][/b]

Unfortunately, yes, they are a tiny bit less efficient. This is because the contents of the big user message are automatically spread over multiple user messages, just like your complex system was doing! However, unlike datastream, it uses only two user messages. One to send a chunk and one to indicate that the full user message has been received. Other than that the transmission procedure is exactly the same.

[b][u]How does it work?[/u][/b]

It's fairly simple. The server keeps an array with bytes. When you send a user message and you add all kinds of types, it is not sent, but instead added to that buffer as bytes. So, when the code calls umsg.End(), the buffer contains all of the data as a big chunk of binary data. This chunk is sent in steps of 200 bytes after which a second type of user message follows that indicates the end of the transfer.

[b][u]How to use it in my code?[/u][/b]

Using it is extremely simple. You don't have to make any changes to your code at all. Instead of the error that indicates your user messages are too big, it'll happily send them in chunks as specified above. All you need to do is add these two files to your server's autorun directory or include them in your script somehow:

[url=http://pastebin.com/qFqqUjPH][b]sv_umsg.lua[/b][/url]
[url=http://pastebin.com/Z2dweT7N][b]cl_umsg.lua[/b][/url]

[b][u]Usage sample[/u][/b]

if ( SERVER ) then
	umsg.Start( "Test" )
		umsg.String( "Hello, there! This is quite a long string already, at least for one that's networked. I wonder when it will stop, although it can't be longer than 255 bytes of course. Hmm... we're almost there! I should come up with something interesting to finish this message. Wait. Did this string just get longer than 255 bytes? Damn this is cheesy :V What year is it?" )
		umsg.Short( os.date( "%Y" ) )
	umsg.End()
else
	usermessage.Hook( "Test", function( um )
		local str = um:ReadString()
		print( #str, str )
		print( um:ReadShort() )
	end )
end

[b][u]Feedback[/u][/b]

Tell me what you think of it, if you encounter any errors or tell me how I can make the code more efficient. [img]http://sae.tweek.us/static/images/emoticons/biggrin.gif[/img]

I already have some optimization planned, but since it's user messages and not some rendering/think stuff, I don't think it'll really benefit from it. Because this completely does the type conversion in Lua, it's possible to add umsg.Double [i](Higher precision floating point numbers)[/i] and umsg.MultiBool [i](Eight bools in one byte/char)[/i] if you guys want them.

[Chewgum]

Will be testing this.

Looks good.

[Lau]

I tried something like this a while back, and I couldn't get client to server working, it kept getting corrupted.

I put tons of work into it and was planning on fixing it, but never got to it, here's my usercommand encoding and concommand/usermessage stream libraries.

[url]http://dl.dropbox.com/u/5766132/Pcwizdan/stream_library.zip[/url]

[haza55]

For float encoding, use this.

Encode:
[lua]
local frac, exp = math.frexp(value)
local integerFrac = math.floor(frac * 1000)
-- Float now is two integers. Exp and integerFrac
[/lua]

Decode:
[lua]
local value = math.ldexp(integerFrac / 1000, exp)
-- Now we has our float again.
[/lua]

[CapsAdmin]

I don't understand the comparison between glon and your usermessage. Where did all the color values and the keys go?

[pollyzoid]

[QUOTE=CapsAdmin;23823207]I don't understand the comparison between glon and your usermessage. Where did all the color values and the keys go?[/QUOTE]

Color is chars after "two"

[thomasfn]

Are the strings null terminated, or do they use a byte at the start to determine length? I see no such bytes in your example, so how does Source know how big the strings are?

[Overv]

They are null terminated, just like regular user messages. If it used the Pascal way of defining strings, their size would be limited to 255 bytes.

[thomasfn]

[QUOTE=Overv;23828481]They are null terminated, just like regular user messages. If it used the Pascal way of defining strings, their size would be limited to 255 bytes.[/QUOTE]

Not that the limitation on string size would come into effect in normal umsgs, being limited to that size anyway.

I guess this is what you do/would use for streaming level data for your GCraft?

[Overv]

Yes or for bans in an admin mod or something like that.

[Mr_Roberto]

This is much cleaner looking than datastream. Now you make one of these going from client to server!

[Gbps]

You can get a few extra bytes out if you use a single character concommand for sending data to the server. I prefer something like "`" or another unused character.

[Mr_Roberto]

-snip-

[Grea$eMonkey]

Looks useful.

[freefall1103]

Doesn't appear to work for me, I get this error

Warning: Unhandled usermessage 'UM_C'
Warning: Unhandled usermessage 'UM_D'

I assume its something to do with either the gamemode,
Although this really does look useful.

I was trying to use this because the value of 'sv_usermessage_maxsize' won't change for me its stuck at 256

[Overv]

That means you didn't receive cl_umsg.lua clientside.

[Cubar]

This is useful thanks Overv.

[freefall1103]

[QUOTE=Overv;24033447]That means you didn't receive cl_umsg.lua clientside.[/QUOTE]

Ah, I will try it again tomorrow.

[Fox-Warrior]

i have an error on my server...

Hook 'WirePlayerInitSpawn' Failed: lua\autorun\sv_umsg.lua:144: attempt to perform arithmetic on a string value

[Crazy Quebec]

It's probably related to the code you used, passing a string where a number was expected? Check your arguments.

[Fox-Warrior]

its not my hook...
i have installed sv_umsg.lua and cl_umsg.lua.

and i got that error

[editline]04:34PM[/editline]

when any player spawn on my server

[Crazy Quebec]

I am positive that the problem is not Overv's code. Somewhere in your code you are passing a string to the umsg.Short function. That's what's on line 144 and it's the only way that error could happen.

[editline]12:36PM[/editline]

[lua]function umsg.Short( s )
-- Convert to unsigned
s = ( s or 0 ) + 32768 -- This is line 144.

local a = math.modf( s / 256 )

umsg.Char( a - 128 )
umsg.Char( s - a * 256 - 128 )
end[/lua]

[freefall1103]

Works! This is win :)

[amcwatters]

This is so great.

[freefall1103]

Bah I've got an error, Causing everyones first names to show up as the default I put in place to the client.

Failed to receive a UMsg string.
autorun\cl_umsg.lua:80: attempt to perform arithmetic on local 'b' (a nil value)

[_nonSENSE]

Post the code that errors (where you are sending / receiving the usermessage).

[freefall1103]

[QUOTE=_nonSENSE;24181053]Post the code that errors (where you are sending / receiving the usermessage).[/QUOTE]

[lua]
function _R.bf_read:ReadString()
local s, b = "", self:ReadChar()

while ( b != 0 ) do
s = s .. string.char( b + 128 )--This is line 80
b = self:ReadChar()
end
[/lua]

[Overv]

What is your sending and receiving code?

[freefall1103]

[QUOTE=Overv;24261802]What is your sending and receiving code?[/QUOTE]

Its perp2, I'll have to read through it and get back to you.

EDIT:

Never mind I fixed it.

[Clavus]

As Overv stated, his system will always send at least two usermessages. Since I found this a bit unnecessary for usermessages that only send a few numbers, I took the liberty of renaming the functions so that Overv's system co-exists beside the normal system. I got his permission to post these:

[url=http://pastebin.com/JcxBqyVJ]cl_ext_umsg.lua[/url]
[url=http://pastebin.com/44uZV6nX]sv_ext_umsg.lua[/url]

Some quick code to demonstrate:

Serverside
[lua]
umsg.ExtStart("extended_test")
umsg.ExtVector( Vector(1,10,100) )
umsg.ExtShort( 99 )
umsg.ExtFloat( 100.12345 )
umsg.ExtLong( 211321312 )
umsg.ExtString( "HOLY BALLS WTF SERIOUSLY THIS IS SOME LONG ASS TEXT OH GOD HOW LONG DOES IT GO HOLY BALLS WTF SERIOUSLY THIS IS SOME LONG ASS TEXT OH GOD HOW LONG DOES IT GO HOLY BALLS WTF SERIOUSLY THIS IS SOME LONG ASS TEXT OH GOD HOW LONG DOES IT GO" )
umsg.ExtEnd()
[/lua]

Clientside
[lua]
local function UMSG_ReadExtended( um )

print(tostring(um:ExtReadVector()))
print(tostring(um:ExtReadShort()))
print(tostring(um:ExtReadFloat()))
print(tostring(um:ExtReadLong()))
print(tostring(um:ExtReadString()))

end
usermessage.Hook("extended_test", UMSG_ReadExtended)
[/lua]

This way you can still use normal usermessages for small transmissions. Be sure not to mix up with what system you send and which one you use to receive.