Useful networking wrappers

[Lau]

And now, for a shitty banner to waive our right of passage into the lua community:
[img]http://imgkk.com/i/XZEjtW.png[/img]

This is some useful networking shit from my library system that I've been using for the last year and a half now...

:siren:[b][u]USE the LZW compression file to compress your datastream shit if you wanna be cool[/u][/b]:siren:

This includes:
- [b]umsg.RegisterObject[/b] --a lua object with obj.MetaName as a string that can identify it in lua
- [b]umsg.Color[/b]
- [b]LibCompress[/b] ( [b]LibCompress.CompressLZW(strUncompressed)[/b] ) (from some random WoW site that hosts lua, I just fixed it up and and added substitutes for the bit library they had used in the code, aswell as taking out Huffman compression because it wasn't working due to improper coding on their part)

- [b]smsg[/b] and [b]servermessage[/b] library is basically a umsg and usermessage library but for client -> server; it includes full serialization and supports all 0 to 255 byte characters sent (bytes 0 and 10 are substituted for argument separation and separate commands)

- ucmd library that adds client and server hooks (PushUcmdData, PullUcmdData) which allows you to encode 32 bits of information into the left over buttons and the angle roll of your view angle (roll = 24 bit), 8 keys unused...etc (I dont know if its 32 bit, but whatever, you can test that out yourself)
HOW TO USE IT:
CLIENT: [b]hook "PushUcmdData" function () return < a 32 bit number >[/b]
SERVER: [b]hook "PullUcmdData" function (objPl,bitNumber)[/b]

Probably more stuff I left out...tell me if you have any errors, because I'll gladly fix it up for you to use separate from my phoenixlibrary.

SVN: [url]http://phxlibpartial.googlecode.com/svn/trunk/[/url]

Credits:
- Pcwizdan (me)
- Deco for most of the networking assistance
- Some guy like JSharpe for the hex2bin and other stuff (got it from his nice luadump)

[b]Example Usage for servermessage library:[/b] (Client to server, good for streaming data)
USES:
- Send larger amounts of data in the same amount of commands datastream's bloat uses.

[b]SERVERSIDE[/b][lua]
servermessage.Hook("example",function (objPl,sm)
print(sm:ReadBool(),sm:ReadString())
end)
[/lua]

[b]CLIENTSIDE[/b][lua]
smsg.Start("example")
smsg.Bool(true)
smsg.String("hello world")
smsg.End()
[/lua]

[b]OUTPUT[/b][lua]
true hello world
[/lua]

[b]Example usage for ucmd library:[/b] (Client to server but extremely unreliable)
USES:
- Add another input device seperate from the mouse or keyboard, like a joystick and have it update as fast as the keyboard and mouse does :D
- Simple state networking such as booleans (this is not very useful for streaming data sequentially, as it's not reliable)

[b]SERVERSIDE[/b]
[lua]
hook.Add("PullUcmdData","example",function (objPl,bitNumber)
--the bitnumber is the 32bit integer that we can use to decode information from.
end)
[/lua]

[b]CLIENTSIDE[/b]
[lua]
local tblData = {
true, false, true, false
}
local bitNumber
hook.Add("PushUcmdData","example",function ()
bitNumber = 0
for i=1, 32 do
bitNumber = bitNumber + (2^(i-1))*((tblData[i] and 1) or 0)
end
return bitNumber
end)
[/lua]

As for datastream, I just need to make a streaming library to properly use LZW after glon is used, then we can :hurr: and start a datastream rehab center.

[NullPoint]

Seems useful, haven't looked at it thoroughly though. useful'd

[Lau]

[QUOTE=NullPoint;19178082]Seems useful, haven't looked at it thoroughly though. useful'd[/QUOTE]
Do you think I should work on a streaming library to encapsulate usermessages and servermessages for streaming larger data?

[Deco Da Man]

Yay!

Ya finally released it.

Good work, pc.

[NullPoint]

[QUOTE=Lau;19182365]Do you think I should work on a streaming library to encapsulate usermessages and servermessages for streaming larger data?[/QUOTE]

Do eeeeet.

[editline]10:16AM[/editline]

Would be very useful.

[Lau]

[QUOTE=NullPoint;19192337]Do eeeeet.

[editline]10:16AM[/editline]

Would be very useful.[/QUOTE]

I'll do it tonight then, I'll just mimic datastream as everyone loves its interface, so you can just use the new networking library named datastream2 or whatever, maybe an override of datastream library would be ok.

Merry Christmas.

[NullPoint]

Nah, call it something different, datastream has a bad reputation (apparently for good reason).

Also, Merry Christmas to you too :smile:

[thomasfn]

I've been wanting to do something like this for ages, but you beat me to it and probably did a better job than I ever could. Useful'd.

[Lexic]

[QUOTE=Lau;19196462]I'll do it tonight then, I'll just mimic datastream as everyone loves its interface, so you can just use the new networking library named datastream2 or whatever, maybe an override of datastream library would be ok.

Merry Christmas.[/QUOTE]
Please don't. KISS
Something like this:
server:
stream.Send(crfilter, name, data)
stream.Hook(name,function(sender,name,data) end)
client:
stream.Send(name, data)
stream.Hook(name,function(name,data) end)
That's really all that's needed.

[Lau]

[QUOTE=Lexic;19200245]Please don't. KISS
Something like this:
server:
stream.Send(crfilter, name, data)
stream.Hook(name,function(sender,name,data) end)
client:
stream.Send(name, data)
stream.Hook(name,function(name,data) end)
That's really all that's needed.[/QUOTE]

You want me to redo datastream because you don't like it's interface...I have nothing wrong with opinions, but, if you want me to completely redo the interface of how datastream works,

People also like the progress reader (I do too) so I'll most likely just modify datastream to use smsg and LZW with optional arguments ofc.

Thank you for your suggestions, I'll be keeping datastream's interface, you can always write a wrapper library if it so suits your coding habits.

[Deco Da Man]

If you want, I can improve GLON to allow for built in LZW support
If I do this, I'll also add __glon_encode and __glon_decode metamethods.

I'll elaborate more later.

(ignore what I said about smsg)

[Lau]

[QUOTE=NullPoint;19198583]Nah, call it something different, datastream has a bad reputation (apparently for good reason).

Also, Merry Christmas to you too :smile:[/QUOTE]
maybe stream would do too (seeing that Lexic is gunning for that name)

Lexic wants:
[b]SV and CL[/b]

stream.Send
stream.Hook

I think I want to use datastream: (For reference goto: [url]http://wiki.garrysmod.com/?title=Datastream[/url])
[b]SV[/b]

datastream.StreamToClients

[b]CL[/b]

datastream.StreamToServer

[b]SHARED[/b]

datastream.DownstreamActive
datastream.GetProgress
datastream.Hook

Shall we settle for:
[b]SHARED[/b]
- with tempid's for the identification of streams, just like the rest.

stream.Hook
stream.Send
stream.GetProgress
stream.GetStats? --returns table of total transferred, just so we dont have to hack into the table to make our nifty developer tools?

[editline]12:35AM[/editline]

[QUOTE=Deco Da Man;19203739]If you want, I can improve GLON to allow for built in LZW support (and use smsg)
If I do this, I'll also add __glon_encode and __glon_decode metamethods.

I'll elaborate more later.[/QUOTE]
Well, if you want, that'd be helpful, but I'd rather like people to stop using keys in their parsing, such that GLON reinforces, I'd like to talk to you about that and some alternatives and options to that.

Elaboration:
- I want to send a simple table of vectors that will signify start positions for a bunch of random clientside objects in some random ass game, why would I want to parse the data down into strings so that we can read it being transferred. I can see LZW fixing that, but we could just stop using naming conventions from GLON and go straight to encoding color channels into character values which are 1:1 in size structure.

[PC Camp]

Yippie! Now I won't be made fun of by dan for using datastream! :D

[Nori]

I wanted to redo datastream now that I know a bit more about lua but you can go ahead and do it yourself.

[Lexic]

[QUOTE=Lau;19203809]Shall we settle for:
[b]SHARED[/b]
- with tempid's for the identification of streams, just like the rest.

stream.Hook
stream.Send
stream.GetProgress
stream.GetStats? --returns table of total transferred, just so we dont have to hack into the table to make our nifty developer tools?

[/QUOTE]
:dance:
Sounds good to me. However, could you pass the callback args in the order of sender, handler, data, id, encoded, whatever else you wanted, so those of us who do not wish to make nifty developer tools don't have to handle excess vars please? :3:

[editline]12:36PM[/editline]

(I would have suggested sender, data, handler, id, encoded, so most people only need to have two vars in their function definitions, but that would break the fine tradition of telling people they're in the hook they just defined that people seem to like so much.)

[Lau]

[QUOTE=PC Camp;19207311]Yippie! Now I won't be made fun of by dan for using datastream! :D[/QUOTE]
Problems with Datastream:
- 128 byte limit for concommands, it can be 254 (it just doesnt calculate the extra bytes used internally for ds)
- 128 byte limit or whatever for usermessages is absolute bullshit
- glon being glon

I need to get to it, I hope you guys can wait for me to properly implement fixes to datastream in a new library named stream. Thanks for all the support!

[ChewGum]

I'll wait.

[fishface60]

This looks quite useful, if only I had something to make use of it.

[aualin]

I demand awesome rating!
Can i send an entity from the client to the server now? (Without any datastream this is, or the entity index)

[Lau]

[QUOTE=aualin;19329745]I demand awesome rating!
Can i send an entity from the client to the server now? (Without any datastream this is, or the entity index)[/QUOTE]
Alright, I'm gonna try finishing the stream library to use smsg library...new years was cool, man I'm tired.

I'll be implementing LZW compression defaulted to on for server<->client transfers unless specified not to (It wont send LZW compressed data if its larger than the uncompressed format)

This stream will not send keys, will not send tables.

[Lau]

-.- for fuck sake
I am confused as hell right now, because LZW compression library doesn't allow me to easily compress a binary stream (thus the table -> table -> string -> string -> modify table -> string..........)
Thus I am fucked.
Here is what I got, it was going to support Datastream parsing just like glon, but with 4bit identification of object type rather than a full 8 bit and then some more (ex "r255g255b255a255")

[sp]I may pick this up again, but the LZW compression only working with strings and not binary streams, makes me rage because I have to put a 1 bit value in there somewhere to signify compressed or not compressed, I'm not going to waste a whole character (byte) by doing "\001" or "\002".[/sp]
I'm a dumbass, I'll just use the other 250 hooks in smsg available in 1 byte namespaces to signify that kinda shit...fucking idiot that I am.

[b]Anyway, here's the code I have going so far, I'll radically change over the next few days though, I'll try to get this done sometime in the next few days as I have time.[/b]
[lua]module('stream',package.seeall)
local Hooks = {}
local Streams = {
IN = {},
OUT = {}
}

local hex2bin = {
["0"] = "0000",
["1"] = "0001",
["2"] = "0010",
["3"] = "0011",
["4"] = "0100",
["5"] = "0101",
["6"] = "0110",
["7"] = "0111",
["8"] = "1000",
["9"] = "1001",
["a"] = "1010",
["b"] = "1011",
["c"] = "1100",
["d"] = "1101",
["e"] = "1110",
["f"] = "1111"
}

local function Hex2Bin(s)
local ret = ''
local i = 0
for i in string.gmatch(s, ".") do
i = string.lower(i)
ret = ret..hex2bin[i]
end
return ret
end

local function Dec2Bin(s, num)
local n
if (num == nil) then
n = 0
else
n = num
end
s = Hex2Bin(string.format("%x", tostring(s)))
while string.len(s) < n do
s = '0'..s
end
return s
end

local function EncodeBinaryToString(tblBits)
local str = ''
local byte
for i=1, #tblBits, 8 do
byte = 0
for bit=0, 7 do
byte = byte + ((tblBits[i+bit] and 2^bit) or 0)
end
str = str..string.char(byte)
end
return str
end
local function DecodeStringToBinary(str)
local bin = {}
local t
for i=1, #str do
t = Dec2Bin(string.byte(string.sub(str,i,i)),8)
for i=8, 1, -1 do
table.insert(bin,string.sub(t,i,i)=='1')
end
end
return bin
end

local function AppendBinary(bitsBuffer,strBin)
for i=string.len(strBin), 1, -1 do
table.insert(bitsBuffer,string.sub(strBin,i,i)=='1')
end
end

local tblSpecificType = {
['Short'] = function (bitStream,n)
AppendBinary(bitStream,Dec2Bin(math.Clamp(tonumber(n) or 0,-(2^15),2^15 - 1)+2^15,16))
end,
['UShort'] = function (bitStream,n)
AppendBinary(bitStream,Dec2Bin(math.Clamp(tonumber(n) or 0,0,2^15),16))
end,
['Long'] = function (bitStream,n)
AppendBinary(bitStream,Dec2Bin(math.Clamp(tonumber(n) or 0,-(2^31),2^31 - 1)+2^31,32))
end,
['ULong'] = function (bitStream,n)
AppendBinary(bitStream,Dec2Bin(math.Clamp(tonumber(n) or 0,0,2^31),32))
end,
['Char'] = function (bitStream,ch)
AppendBinary(bitStream,Dec2Bin(tostring(string.byte(ch)),8))
end,
['string'] = function (bitStream,str)
AppendBinary(bitStream,Dec2Bin(tostring(string.len(str)),8))
for i=1, string.len(str) do
AppendBinary(bitStream,Dec2Bin(tostring(string.byte(string.sub(str,i,i))),8))
end
end,
['boolean'] = function (bitStream,b)
table.insert(bitStream,b)
end
}
tblSpecificType.number = tblSpecificType.Long
local tblSpecificTypeID = {}
local numMaxID = 1
for k,v in pairs(tblSpecificType) do
tblSpecificTypeID[k] = numMaxID
numMaxID=numMaxID+1
end

local function ParseToString(tbl,bAllowCompress)
local tblBin = {}
for k, v in ipairs(tbl) do
if type(v) == 'table' then
local type = v[1]
local val = v[2]
if v.type and tblSpecificType[v.type] then
AppendBinary(bitStream,Dec2Bin(tostring(2^(tblSpecificTypeID[v.type] - 1)),4))
tblSpecificType[v.type](tblBin,v.var or v[1])
else
ErrorNoHalt("stream: Failed to convert to specific type: '"..tostring(v.type).."'\n",2)
end
elseif type(v)~='nil' and tblSpecificType[type(v)] then
tblSpecificType[type(v)](tblBin,v)
end
end
local str
if bAllowCompress then
str, bCompressed = LibCompress.CompressLZW(EncodeBinaryToString(tblBin))
table.insert(tblBin,bCompressed)
end
--[[
for k, v in pairs(tbl) do
if type(k) ~= 'number' then

end
end
]]
return str
end

if SERVER then
local CHUNK_SIZE = 252 --limit of the umsg library to send to clients in one package
local function FormatDestination(d)
if type(d) == 'table' then
local rp = RecipientFilter()
for k, v in pairs(d) do
rp:AddPlayer(v)
end
return rp
end
return d
end

function Send(valDest,objIdent,tblData, bNoCompress, funcCallback)
--if type(objIdent) == 'number' then objIdent = tostring(Dec2Bin(objIdent,16),2) end
objIdent = tostring(objIdent) --fuck me
return table.insert(Streams.OUT,{dest=FormatDestination(valDest),name=strName,data=ParseToString(tblData,not bNoCompress),callback=funcCallback,id=0})
end

hook.Add("Tick",NAME,function ()

end)
servermessage.Hook('\1',function (objPl,sm) --start stream
local str = ''
for i=1, CHUNK_SIZE do
sm:ReadChar()
end
local tblBin = DecodeStringToBinary(str)
local bCompressed = tblBin[1]
local

end)
servermessage.Hook('\2',function (objPl,sm) --continue stream (or last packet)

end)
end

if CLIENT then
local CHUNK_SIZE = 123 --limit of the smsg library to send to server in one package
function Send(strName,tblData, bNoCompress, funcCallback)
return table.insert(Streams.OUT,{name=strName,data=ParseToString(tblData,not bNoCompress),callback=funcCallback,id=0})
end

hook.Add("Tick",NAME,function ()

end)
usermessage.Hook('\1',function (um)

end)
end

--For use in stream.Send('test',{'hello world',stream.Object("Long",10123434)})
function Object(strType,val)
return {strType,val}
end

function Hook(strName,funcCallback)
Hooks[strName] = funcCallback
end

function GetProgress(numID)
--return percent
end

function IncomingMessage(numID,strName,nSize)
--bleh bleh bleh
end
[/lua]
*RAGE*

[infinitywrai]

So basically this is umsgs except from client to server? That's pretty tight.

Is the security of it tested?

[Lau]

[QUOTE=infinitywrai;19414096]So basically this is umsgs except from client to server? That's pretty tight.

Is the security of it tested?[/QUOTE]
Its as secure as the user of it makes it.
In comparison to datastream, the datastream AcceptStream Hook is utterly useless in my opinion.

And if there are DoS issues with it or instability problems, I'll patch them asap, though I don't expect that with simple lua and concommands in this module.

[haza55]

[QUOTE=Lau;19212894]Problems with Datastream:
- 128 byte limit for concommands, it can be 254 (it just doesnt calculate the extra bytes used internally for ds)
- 128 byte limit or whatever for usermessages is absolute bullshit
- glon being glon

I need to get to it, I hope you guys can wait for me to properly implement fixes to datastream in a new library named stream. Thanks for all the support![/QUOTE]

This is why I love forcing all my clients to have luasockets, it makes everything much more pleasant.

[Lau]

[QUOTE=haza55;19444649]This is why I love forcing all my clients to have luasockets, it makes everything much more pleasant.[/QUOTE]
Yeh, luasockets is hands down better than hacking up source engine networking like I did with ucmd, but I'm doing it for science.

[TomyLobo]

what gmod needs is something as efficient as umsgs, just client->server.
Is this something like that?

[Deco Da Man]

[QUOTE=TomyLobo;19500096]what gmod needs is something as efficient as umsgs, just client->server.
Is this something like that?[/QUOTE]

[QUOTE=Lau;19177959]
- [b]smsg[/b] and [b]servermessage[/b] library is basically a umsg and usermessage library but for client -> server; it includes full serialization and supports all 0 to 255 byte characters sent (bytes 0 and 10 are substituted for argument separation and separate commands)[/quote]

[TomyLobo]

yeah I read that. I should have emphasised on the "efficiency" part...
i.e. does it transfer a Char as a single byte, does it pool smsg names

[aualin]

Seems to be pretty much public domain, read the code?

[Lau]

[QUOTE=TomyLobo;19524872]yeah I read that. I should have emphasised on the "efficiency" part...
i.e. does it transfer a Char as a single byte, does it pool smsg names[/QUOTE]
It doesn't pool smsg names for the specific reason that pooling doesn't even work right in source <-> lua interface, if you notice: [b]umsg.PoolString does nothing, and it pools after a string has been sent 5 times.[/b]

Also, if you guys are still wondering if I am going to finish stream library, the answer is yes. I just ran into a ton of problems and I thought I was about 90% done, but there's some logical error mixing up the bitstream psuedo-system.