NetWrapper - a simple and lightweight networking library

[Mista Tea]

[B]Hello friends![/B]

This is my first publicly-released library. I originally wrote this to create a centralized way of networking data with net messages and without the use of ENTITY:SetNW* / ENTITY:SetDT*.

The code is available on [B]GitHub[/B]
[B]Addon version[/B]: [URL]https://github.com/Mista-Tea/netwrapper/tree/master[/URL]
[B]Single, shared file version[/B] (drop it into your own addon): [URL]https://github.com/Mista-Tea/netwrapper/tree/condensed[/URL]

[B]!!! *** Please view the readme file on GitHub *** !!!: [/B][url]https://github.com/Mista-Tea/netwrapper/blob/master/README.md[/url]
It has far prettier formatting than Facepunch can support, apparently.

[IMG]http://i.imgur.com/tQJt1ZG.png[/IMG]

The NetWrapper library is a simple wrapper over Garry's standard net library to provide lightweight networking without needing to care about the type of data you are networking (unlike the ENTITY:SetNetworked* library) and without needing to create dozens of networked strings for net messages.
There are 2 ways to network data with the NetWrapper library:
[B] • Net Vars
• Net Requests [/B]

[B]== Net Vars[/B] ==
If you are looking to replace your existing scripts' use of the ENTITY:SetNW*/ENTITY:SetDT* functions, Net Vars are the way to go. With Net Vars, data set on entities is only networked when the data is added or changed with ENTITY:SetNetVar( key, value ) from the server. By broadcasting net messages only when the data changes, this library has a relatively low impact on network traffic. Once these values have been broadcasted, all connected clients will be able to retrieve the values like you would with the standard networking libraries.

[B] • Setting networked values:[/B]

-- if run on the server, this key/value pair will be networked to all clients
ENTITY:SetNetVar( key, value )

[B] • Getting networked values:[/B]

-- if run on the client, this will attempt to grab the value stored at the key
ENTITY:GetNetVar( key, default )

Where 'default' is the default value you would like returned if the key doesn't exist. If a default value isn't provided and the key doesn't exist, nil will be returned.

[B]• Example:[/B] If you wanted to network a title on a player when they connect, you could do something like the following:

hook.Add( "PlayerInitialSpawn", "SetPlayerTitle", function( ply )
    local title = ... -- grab the title somewhere
    ply:SetNetVar( "Title", title )
end )

As soon as ply:SetNetVar() is called, a net message will be broadcasted to all connected clients with the key/value pair for the title.
If you wanted to show the player's title in a GM:PostPlayerDraw hook, you could do something like the following:

hook.Add( "PostPlayerDraw", "ShowPlayerTitle", function( ply )
    -- retrieve the player's title if one has been networked, otherwise returns nil
    -- if a title hasn't been networked yet, don't try drawing it
 
    local title = ply:GetNetVar( "Title" )
    if ( !title ) then return end 
 
    draw.SimpleText( title, ...  -- etc
 
end )

[B]== Net Requests ==[/B]
Net Requests are a new features in the NetWrapper library. They allow you to determine exactly when a client asks the server for a value to be networked to them by using ENTITY:SendNetRequest( key ). If the server has set data on the entity with ENTITY:SetNetRequest( key, value ), the value will be sent back to the client when they request it. If the server has not set any data on the entity at the given key, the client will keep sending requests (as long as you use ENTITY:SendNetRequest( key ) again) until they have either reached the maximum amount of requests that can be sent per entity+key (netwrapper_max_requests cvar) or the value has been set by the server.

This is especially useful if you have hundreds or thousands of entities spawned out when clients join the server. If you networked a value using ENTITY:SetNetVar() on every entity, that means that the client will receive hundreds or thousands of net messages to sync all of the Net Vars when they initialize during GM:InitPostEntity. However, by using Net Requests instead you can network data to the client only when they ask for it (such as when they look directly at it).

[B]• Setting net requests:[/B]

-- if run on the server, this key/value pair will be stored in a serverside table that the client can request from
ENTITY:SetNetRequest( key, value )

[B]• Getting net requests:[/B]

--  when run on the client, this will send a net message to the server  asking for the value stored on the entity at the given key
ENTITY:SendNetRequest( key )

-- once the client has received the value from the server, subsequent calls to ENTITY:GetNetRequest() will return the value
ENTITY:GetNetRequest( key, default )

Where 'default' is the default value you would like returned if the key doesn't exist. If a default value isn't provided and the key doesn't exist, nil will be returned.

[B][B]• [/B]Example:[/B] If you want to network the owner's name on props but don't want to flood connecting clients with hundreds of possible net messages, you can do something like the following:

-- some serverside function that pairs up the player with the entity they spawned
ent:SetNetRequest( "Owner", ply:Nick() )

Now the value has been stored in the netwrapper.requests table and can be accessed by clients when they request it:

-- somewhere clientside
local owner = ent:GetNetRequest( "Owner" )
if ( !owner ) then ent:SendNetRequest( "Owner" ) end

Assuming you use the above in a HUDPaint hook or something that gets repeatedly gets called, this will check to see if the 'Owner' value has already been requested from the server. If it hasn't (and therefore returns nil), ent:SendNetRequest( "Owner" ) is called which sends a request to the server asking for the value stored at the 'Owner' key.

Since the 'Owner' was set earlier, the server will reply to the client's request by sending a net message back with the entity and key/value pair. When the client receives the message, the value is stored in the netwrapper.requests table and will be retrieved with any subsequent calls to ent:GetNetRequest( "Owner" ).

[IMG]http://i.imgur.com/BQewYpe.png[/IMG]

[U][B]Q: What sort of data can I network with this library?[/B][/U][B]

A:[/B] Since this is a wrapper library over the standard net library, all limitations of the net library apply here. For example, you can't network functions or user data.

[B]What you CAN network:[/B]

nil
strings
numbers
tables
booleans
entities
vectors

angles

------------------------------------------------------------------------------------------------------------------[U][B]

Q: How often is the data networked?[/B][/U][B]

A:[/B] [B]For Net Vars[/B]:
Every time you use ENTITY:SetNetVar( key, value ) from the server, the data will be networked to any clients via net message. If you set a value on a player and then change that value 5 minutes later, the data will have been broadcasted only 2 times over the span of that 5 minutes. However, this does mean that if you use ENTITY:SetNetVar( key value ) in a think hook, it will be broadcasting net messages every frame.

As with any other function, be sure to set networked data only as often as you need to. Think hooks should typically be avoided if you plan on networking large amounts of data on a large amount of entities/players.
[B]
A: For Net Requests:[/B]
Whereas Net Vars are automatically broadcasted to connected clients, and synced to connecting clients during GM:InitPostEntity, Net Requests are only networked on a 'need-to-know' basis, which significantly reduces the amount of network traffic that connecting players receive.

------------------------------------------------------------------------------------------------------------------[U][B]

Q: What happens when clients connect after the data has already been broadcasted?[/B][/U][B]

A:[/B] [B]For Net Vars[/B]:
When a client fully initializes on the server (during the GM:InitPostEntity hook, clientside), they wi

[Tyler B.]

Nice release!

(the bbcode in your last code example doesn't work though, shows <b> </b>)

[Mista Tea]

Thank you for the notification! Whenever I edit my post, the text box is entirely empty, so I have to copy/paste the text from the current post and then redo all of the code tags, which is becoming a pain.

Have I accidentally done something to prevent the text box from letting me edit the previous post?

[rejax]

this looks excellent, i was just about to write my own too :eng101:

cheers!

[Mista Tea]

[QUOTE=rejax;44466982]this looks excellent, i was just about to write my own too :eng101:

cheers![/QUOTE]

Thank you! If you end up using it, let me know how things work out.

Also, I forgot to mention that I included an [B]example.lua[/B] file that isn't actually run by Lua but contains a few more examples and detailed Q&A's.

[Th3Angel]

Very nice. I love the thorough documentation as well. :)

[fnox]

do you have any statistics to show that this library is actually more efficient than NWVars?

like, I'll admit it, your code is impeccable, tbh it's one of the cleanest I've ever seen for GMod. but after all you're still using the net library, which is an implicit overhead, and I'm a bit skeptical if this is actually any more efficient. it's definitely more convenient than the default libraries but i'm not so sure if its efficiency surpasses them.

[.delay]

Nice work man

[SeriousLion]

keep up this work but like big bang said do you have any statics?

[Cardcapture1]

Very nice work. Got any statistics on this versus other networking methods?

[G😹nz😻]

Excellent! I'll use it

[Mista Tea]

When I was originally writing this in March, I was doing simple tests to compare SetNW* against SetNetVar, such as networking an integer on 100 entities, or 100 integers on a single entity (to quickly see results). When using SetNWInt and SetNWString in the above tests, net_graph was reporting high volumes of user messages being sent constantly even though I only networked the variables once.

When I compared it to my library by performing the same tests but with SetNetVar, the messages were only being sent out once, as expected.

I run a Sandbox and a Flood server, so having hundreds of entities out with multiple SetNW*'d variables was something I wanted to avoid. I mainly wrote this as an easier way to network with net messages, but when I saw SetNW* constantly sending out messages for variables that weren't changing, I started using my library to replace nearly everything that I was networking.

That being said, I've spent this morning running the same tests that I had back in March. I don't know whether something has changed internally since the last 2 updates, but SetNW* seemingly no longer constantly networks data when it isn't being changed. As such, I appear to have no real way to prove my library is any more or less efficient than SetNW* now, so I apologize. I am more than willing to perform whatever tests are suggested in this thread though.

However, as an alternative to creating dozens of different net messages to network data, using this library is still much simpler, so I will continue using it and updating it.

My only hope in making this public was to help someone in some way, even if it's just by looking at the concept and learning how to use net messages. Thanks for reading :)

[thelastpenguin]

from reading your code it's actually fairly short (with comments removed), it'd be nice to see this packaged into a single file for easy inclusion in projects rather than an addon as a dependency.

[Mista Tea]

I've created a branch named [I]condensed[/I] that contains the library in a single shared file: [URL]https://github.com/Mista-Tea/netwrapper/tree/condensed[/URL]

I've removed the documentation to keep it compact. The master branch is still the same, so you can always view it to see the documentation.

Thanks for the suggestion!

[Acecool]

I would've done it different; actually I did do it differently when I created my system.

Instead of SetNetVar, I use SetFlag, I also use GetFlag instead of GetNetVar.

You're missing an argument for your getter; a default value to be returned when none is read. You are also storing in a table based on ent, I'd recommend using EntIndex for the value so it can be read even after the entity is NULL'ed; so you have additional time to save values.

Benchmarking NWVars: [url]https://dl.dropboxusercontent.com/u/26074909/tutoring/benchmarking_tips/benchmarking_nw_vars.lua[/url]

Near the bottom you see it accessing standard variables. That's essentially what my system does, and apparently what yours does.

[MattJeanes]

Are we allowed to package this in our addon? Much like how vON is packaged with many addons, including wiremod itself.

[Mista Tea]

[QUOTE=MattJeanes;44481860]Are we allowed to package this in our addon? Much like how vON is packaged with many addons, including wiremod itself.[/QUOTE]

I have no problem with that!

I'll be honest in saying I'm rather clueless on the whole licensing part. I tried choosing a license with the most amount of flexibility to anyone that wants to use it.

I've put the license disclaimer in the file itself of the condensed version: [URL]https://github.com/Mista-Tea/netwrapper/blob/condensed/netwrapper.lua[/URL]

I think that means you can just put the file wherever you'd like and as long as the disclaimer isn't removed, there shouldn't be any problem.

I'm not really concerned about licensing or even credit, just as long as you can put the library to good use!

[editline]7th April 2014[/editline]

[QUOTE=Acecool;44481730]You're missing an argument for your getter; a default value to be returned when none is read.[/QUOTE]

Thanks! I've added a [B]default parameter[/B] to netwrapper.GetNetVar( key, default )

[QUOTE=Acecool;44481730]You are also storing in a table based on ent, I'd recommend using EntIndex for the value so it can be read even after the entity is NULL'ed; so you have additional time to save values.[/QUOTE]

I've gone through and updated both the master and condensed branches to store/send Entity Indexes now.

I no longer need the client to poll the server during [B]OnEntityCreated [/B]since the server can now just send the entity index instead of the entity itself. Even though the entity isn't initialized on the client at first, it will be able to assign the networked data just by grabbing its EntIndex.

Since entities reuse the same indexes when you constantly create/delete them, I've added the [B]EntityRemoved [/B]hook on both the client and server to remove the entity's index in [I]netwrapper.ents[/I]. This will prevent data corruption (i.e., a future entity using a previous entity's data because they share the same EntIndex).

[B]All of these changes are under the hood[/B] -- so, thankfully, nothing has changed about how to set and get the networked values (SetNetVar/GetNetVar)

Links once more (also available in OP)
Addon version: [URL]https://github.com/Mista-Tea/netwrapper/tree/master[/URL]
Single, shared file version: [URL]https://github.com/Mista-Tea/netwrapper/tree/condensed[/URL]

Thanks for the suggestions!

[Whiterabbit]

[QUOTE=Mista Tea;44482071]
I no longer need the client to poll the server during [B]OnEntityCreated [/B]since the server can now just send the entity index instead of the entity itself.

---

I've added the [B]EntityRemoved [/B]hook on both the client and server to remove the entity's index in [I]netwrapper.ents[/I].[/QUOTE]

I've had a quick look at the code and I think you'll get an issue with this.
EntityRemoved will be called clientside when an entity leaves the PVS, so the entities values get cleared.
When the entity re-enters the PVS, since you aren't polling the server it's values will not get resynced.
Or is there something I missed there?

[Mista Tea]

[QUOTE=wh1t3rabbit;44482988]I've had a quick look at the code and I think you'll get an issue with this.
EntityRemoved will be called clientside when an entity leaves the PVS, so the entities values get cleared.
When the entity re-enters the PVS, since you aren't polling the server it's values will not get resynced.
Or is there something I missed there?[/QUOTE]

I'm afraid I've never had any experience with PVS.

The new wiki doesn't say anything about that, but after reading the old wiki's EntityRemoved article, I saw a similar comment to what you're talking about: [URL]http://maurits.tv/data/garrysmod/wiki/wiki.garrysmod.com/indexc434.html[/URL]

[I]"Called right before an entity is removed. (Note that this isn't going to be totally reliable on the client since the client only knows about entities that it has had in its PVS.)"[/I]

I can't seem to find much information that actually explains PVS other than how maps use visleafs.

Do you have any sort of test I can perform to check if this wipes the values? If it does, I can take the EntityRemoved hook off of the client and have the server send a message containing the entity's ID when the entity is removed instead, which is a fairly simple and clean fix.

[Acecool]

Yeah, I've been letting people know about the PVS issue. It caused some crazy issues with my net-code. Basically when an entity leaves the nearby area ( little over 5000 units ), they are out of PVS and the hook gets executed on the client.

[lua]//
// EntityRemoved; Called on SERVER when an ENTITY is REMOVED; Called on CLIENT when ENTITY leaves PVS for that client.
//
function GM:EntityRemoved( _ent )
if ( SERVER ) then
local _id = _ent:GetID( );

if ( _ent:IsVehicle( ) ) then
VehicleOnRemove( _ent );
end

self:ResetEntityFlags( _id );
end
end[/lua]

[Mista Tea]

[QUOTE=Acecool;44483252]Yeah, I've been letting people know about the PVS issue. It caused some crazy issues with my net-code. Basically when an entity leaves the nearby area ([B] little over 5000 units[/B] ), they are out of PVS and the hook gets executed on the client.[/QUOTE]

I can't seem to get the EntityRemoved hook to be called on the client by going any amount of distance away from some entities I am networking, even when going so far that the physgun texture starts jumping around when you move your eyes.

[IMG]http://i.imgur.com/Ws5k1cS.png?1[/IMG]
( very, very far away from the entities I am networking )

Is there anything else I can try?

[Acecool]

Are you calling it in multiplayer with other players? The issue haunted me for about 2 weeks until I figured it out. I had players come in, then all of a sudden values were getting wiped. Otherwise it would randomly happen when I was on the server by myself.

[editline]8th April 2014[/editline]

Example using public values:

Entity 0, worldspawn, will always be "null" on client, it doesn't matter though.
[lua][ 0 :"[NULL ENTITY]": :"N/A CLASS": :"N/A MODEL"]
[env_skypaint] = 1406
[env_sun] = 399
[light_environment] = 402
[day_night_brush] = 433
[elevators]
[1]
[property] = 51
[id] = 1407
[2]
[property] = 58
[id] = 1408
[speedlimits]
[city] = 1
[country] = 4
[sickness_road] = 2
[lake_road] = 2
[country_tunnel] = 4
[serverip] = xx.xx.xx.xx:27015

[ 1 :"Player [1][Acecool][STEAM_0:1:4173055] ": :"player": :"models/player/group03/male_02.mdl"]
[accesslevel] = 9000
[inconsole] = true

[ 1407 :"[NULL ENTITY]": :"N/A CLASS": :"N/A MODEL"]
[direction] = 0
[floor] = 1
[called]

[ 1408 :"[NULL ENTITY]": :"N/A CLASS": :"N/A MODEL"]
[direction] = 0
[floor] = 1
[called]
[/lua]

The last two are Lua elevators. That's before I was close, and now after enabling admin mode and flying over:

[lua]] dev_getdata
[ 0 :"[NULL ENTITY]": :"N/A CLASS": :"N/A MODEL"]
[env_skypaint] = 1406
[env_sun] = 399
[light_environment] = 402
[day_night_brush] = 433
[elevators]
[1]
[property] = 51
[id] = 1407
[2]
[property] = 58
[id] = 1408
[speedlimits]
[city] = 1
[country] = 4
[sickness_road] = 2
[lake_road] = 2
[country_tunnel] = 4
[serverip] = xx.xx.xx.xx:27015

[ 1 :"Player [1][Acecool][STEAM_0:1:4173055] ": :"player": :"models/player/group03/male_02.mdl"]
[adminmode] = true
[inconsole] = true
[playercolor] = Vector( 0.24310000240803, 0.34509998559952, 0.41569998860359 )
[accesslevel] = 9000
[weaponcolor] = Vector( 0.30000001192093, 1.7999999523163, 2.0999999046326 )

[ 1408 :"Entity [1408][func_brush] ": :"func_brush": :"*227"]
[direction] = 0
[floor] = 1
[called]

[ 1407 :"Entity [1407][func_brush] ": :"func_brush": :"*43"]
[direction] = 0
[floor] = 1
[called]
[/lua]
When I leave the area, the entity retains the data / type etc... It mainly happens on players / vehicles for me for PVS data.

I'd also recommend setting up a private flag; basically a third optional for set, and a third optional for get:

[lua]] dev_getpdata
[ 1 :"Player [1][Acecool][STEAM_0:1:4173055] ": :"player": :"models/player/group03/male_02.mdl"]
[view_angle] = Angle( 21.339998245239, -91.980049133301, 0 )
[timeconnected] = 88.787879943848
[view_angle_old] = Angle( 21.339998245239, -91.980049133301, 0 )
[oldstamina] = 100
[job_role] = 1
[total_playtime] = 191709
[stamina] = 100
[firstlogin] = 1368849186
[lastlogin] = 1396909771
[lastdrawhudtime] = 5488.03515625
[name] = Acecool
[username] = Acecool
[userid] = 1
[user_connects] = 1232
[/lua]

Basically if it's private data, then only the server and client know about it; or the server/client/and vehicle client is in. This reduces the amount of data networked for certain things. My system does the same thing as yours where if you set a public or private variable on the client, it's only clientside for that player.

[Mista Tea]

I am testing it in multiplayer, but it's just me connected.

I'm assuming bots don't count either, since I haven't seen anything change when spawning a few.

[Acecool]

Better be safe than sorry; have the server network when an entity is gone. This is why using the EntIndex would be better than using the ENT itself for the table key. If you move out of PVS and it becomes null, then another becomes null, then the tables will either be removes, or merge..

[Mista Tea]

[QUOTE=Acecool;44483718]This is why using the EntIndex would be better than using the ENT itself for the table key.[/QUOTE]

A few posts back I replied to your first post and stated that I had changed everything from using entities as the index to EntIndex.

[Acecool]

Ah, I missed that. Excellent! It's good to hear, it'll be more reliable with that method.

As long as only the server calls for resetting data, you should have no hidden bugs. But, if the code is running shared for EntityRemoved; you'll encounter bugs. Add me on steam if you want : id/Acecool

[Mista Tea]

I still haven't been able to conclusively find any issues with PVS.

At any rate, I've updated both branches to let the server handle the EntityRemoved hook.

A net message containing the entity index of the entity being removed is sent to the client that makes the client remove the entity data from the netwrapper.ents table.

[Acecool]

It didn't happen all the time for me either while playing with just myself in the server; the issue became much more noticeable with more players. Since you updated it to only be handled by server, there should be no other issues.

[Mista Tea]

[QUOTE=Acecool;44485105]It didn't happen all the time for me either while playing with just myself in the server; the issue became much more noticeable with more players. Since you updated it to only be handled by server, there should be no other issues.[/QUOTE]

Thanks for the suggestions and the heads up!

[Abdul]

Do I need to use AddNetworkString for GetNetVar/SetNetVar to work?