vON: Vercas' Object Notation [Release for developers]

[KatNotDinner]

Really nice work vercas. Quite interesting library. You should ask garry about including it in gmod. It'e definately an improvement.

[QUOTE=rebel1324;36538065]No more string.Explode![/QUOTE]

It has always been a huge mystery to me why people love string.Explode since string.split is faster and does the same thing.

[vercas]

[QUOTE=KatNotDinner;36549087]Really nice work vercas. Quite interesting library. You should ask garry about including it in gmod. It'e definately an improvement.[/QUOTE]

Well, maybe I'll ask him.
But definitely not in this state. It still needs a lot of improvement.
I'm separating the serializer into a normal one and a "nice" one for a speed gain and I'm trying to think of a better way of separating the table to serialize into the two components.
And I still need to add GMod-specific types... And I'll also add support for big numbers and I'll look into number compression.

[KatNotDinner]

Well you are right. Don't you think that the stuff used only in the script should all be local? I didn't stop and pay atention to how everything works but I noticed that the first 2 functions are global. Although it will probably be a minor improvement it should be still looked at.

Also do you have any idea how you are going to compress gmod only types? For example entities. i had the most generic idea - get the ent index - but it will mostly likely fail since as far as I remember Source sets all world entities' indexes to 0.

[Divran]

[QUOTE=KatNotDinner;36549087]It has always been a huge mystery to me why people love string.Explode since string.split is faster and does the same thing.[/QUOTE]

What? You do know string.Split is this

function string.Split( str, delimiter )
	return string.Explode( delimiter, str )
end

right?

By the way, Tomylobo (and me a little) have optimized string.Explode, and Overv put it in gmod back when he had access. So it's no longer super slow.

[QUOTE=KatNotDinner;36549182]Also do you have any idea how you are going to compress gmod only types? For example entities. i had the most generic idea - get the ent index - but it will mostly likely fail since as far as I remember Source sets all world entities' indexes to 0.[/QUOTE]

Saving entities to a file doesn't make sense, unless you do it adv dupe style.
Saving players is possible, but you'd have to set the player entity to NULL if that player isn't connected to the server when you decode the string.

[vercas]

[QUOTE=Divran;36549275]What? You do know string.Split is this

function string.Split( str, delimiter )
	return string.Explode( delimiter, str )
end

right?

By the way, Tomylobo (and me a little) have optimized string.Explode, and Overv put it in gmod back when he had access. So it's no longer super slow.

Saving entities to a file doesn't make sense, unless you do it adv dupe style.
Saving players is possible, but you'd have to set the player entity to NULL if that player isn't connected to the server when you decode the string.[/QUOTE]

[QUOTE=KatNotDinner;36549182]Well you are right. Don't you think that the stuff used only in the script should all be local? I didn't stop and pay atention to how everything works but I noticed that the first 2 functions are global. Although it will probably be a minor improvement it should be still looked at.

Also do you have any idea how you are going to compress gmod only types? For example entities. i had the most generic idea - get the ent index - but it will mostly likely fail since as far as I remember Source sets all world entities' indexes to 0.[/QUOTE]

Everything is localized but the [i]von[/i] table, which is supposed to be global.
The name of those functions is declared in the first line of locals.

Uh, entities, NPCs and players are and will always be saved as a number (the entity id), simply.
It won't serialize entities with the purpose of storing them, but for sending them.
MAYBE I'll do adv-dupe style entity saving... But only if it's actually needed.

About compression, I'll look into the built-in util.Compress function and I'll also attempt to compress numbers to base 64, or to base 86 (the amount of printable characters on my keyboard that are not used to mark the end of the number).

[b]Edit[/b]: Only if I knew how to actually convert decimal points to other bases.
I'd gladly accept suggestions for compression.

[KatNotDinner]

[QUOTE=Divran;36549275]What? You do know string.Split is this right?[/QUOTE]

I'm surprised. You seem to be a decent lua coder but don't know the difference between string.split and string.Split?

string.split IS faster because it's a C++ function.

And I know that string.Split is like string.Explode because you are not the only one who browses the GCF files.

[Gran PC]

What are you talking about, string.split doesn't even exist.

[vercas]

[QUOTE=KatNotDinner;36549747]I'm surprised. You seem to be a decent lua coder but don't know the difference between string.split and string.Split?

string.split IS faster because it's a C++ function.

And I know that string.Split is like string.Explode because you are not the only one who browses the GCF files.[/QUOTE]

There is no [i]string.split[/i] in any [URL="http://www.lua.org/manual/5.1/manual.html#5.4"]manual[/URL] or [URL="http://lua-users.org/wiki/StringLibraryTutorial"]tutorial[/URL], and [i]print(string.split)[/i] prints [b]nil[/b].
Can't find it in Lua 5.0 or 5.2 either.

If it's faster or not is a question of philosophy. Is doing nothing faster than doing something? :v:

[TGiFallen]

[QUOTE=vercas;36549371]
Uh, entities, NPCs and players are and will always be saved as a number (the entity id), simply.
It won't serialize entities with the purpose of storing them, but for sending them.
[B]MAYBE I'll do adv-dupe style entity saving... But only if it's actually needed.[/B][/QUOTE]
Doing this is [I]way[/I] out of the scope of what vON does. Leave this to tools like advanced dupe.

[QUOTE=vercas;36549371]
About compression, I'll look into the built-in util.Compress function and I'll also attempt to compress numbers to base 64, or to base 86 (the amount of printable characters on my keyboard that are not used to mark the end of the number).

[b]Edit[/b]: Only if I knew how to actually convert decimal points to other bases.
I'd gladly accept suggestions for compression.[/QUOTE]

Look up [URL="http://maurits.tv/data/garrysmod/wiki/wiki.garrysmod.com/index1dd1.html"]bitshifting[/URL] and [URL="http://www.lua.org/manual/5.1/manual.html/#pdf-string.format"]string.format[/URL]. These should get you starting with compressing numbers.

[Chief Tiger]

[QUOTE=vercas;36549778]There is no [i]string.split[/i] in any [URL="http://www.lua.org/manual/5.1/manual.html#5.4"]manual[/URL] or [URL="http://lua-users.org/wiki/StringLibraryTutorial"]tutorial[/URL], and [i]print(string.split)[/i] prints [b]nil[/b].
Can't find it in Lua 5.0 or 5.2 either.

If it's faster or not is a question of philosophy. Is doing nothing faster than doing something? :v:[/QUOTE]

One can never truly be doing nothing, the word is merely used for explanations of things that are doing so little of something noticeable that it is negligible whatever else they are actually doing.

Take for instance a person standing and "doing nothing" on a street corner. For one, we just noted that they were standing, that's something, they are also breathing, seeing, thinking, etc.

/ontopic

This library is awesome. I think I might try to make a new datastream using it instead of glon to see the benchmarks there as well.

[vercas]

[QUOTE=Chief Tiger;36552465]One can never truly be doing nothing, the word is merely used for explanations of things that are doing so little of something noticeable that it is negligible whatever else they are actually doing.

Take for instance a person standing and "doing nothing" on a street corner. For one, we just noted that they were standing, that's something, they are also breathing, seeing, thinking, etc.[/QUOTE]

I meant computers. :wink:
A thread can do nothing (sleep).

[QUOTE=Chief Tiger;36552465]This library is awesome. I think I might try to make a new datastream using it instead of glon to see the benchmarks there as well.[/QUOTE]

Heh, with the net library in GMod 13, datastream is no longer required.
But yeah, I'm curious how well would it perform.

[QUOTE=TGiFallen;36552366]Doing this is [I]way[/I] out of the scope of what vON does. Leave this to tools like advanced dupe.[/QUOTE]

I didn't say I'm going to do it in the vON library, did I? :wink:

[QUOTE=TGiFallen;36552366]Look up [URL="http://maurits.tv/data/garrysmod/wiki/wiki.garrysmod.com/index1dd1.html"]bitshifting[/URL] and [URL="http://www.lua.org/manual/5.1/manual.html/#pdf-string.format"]string.format[/URL]. These should get you starting with compressing numbers.[/QUOTE]

Yeah... I really have no clue how to use those to compress numbers.

[TGiFallen]

[QUOTE=vercas;36552895]I didn't say I'm going to do it in the vON library, did I? :wink:
[/QUOTE]
No, but you said you might so I was telling you that it's a bad idea to.

[QUOTE=vercas;36552895]
Yeah... I really have no clue how to use those to compress numbers.[/QUOTE]
Simplest way for me to put bit shifting for saving info would be an example with a short (0-255)
you can save a char in a file as a plain number, such as

100

Or you can save it as a character in the ascii set like this

string.char(100) --> would come out as "d" since d is mapped to 100 on the ascii chart

It gets more complicated with larger numbers and I suck at explaining it. [URL="http://www.litfuel.net/tutorials/bitwise.htm"]this[/URL] and this [URL="http://en.wikipedia.org/wiki/Bitwise_operation"]http://en.wikipedia.org/wiki/Bitwise_operation[/URL] should help more. Keep in mind that the bit wise functions that come with gmod are made for signed numbers iirc.

[LauScript]

keeping keys in order would be nice

[Map in a box]

I've been working on a Lua serialization library, glad to see others are too!

[vercas]

[QUOTE=LauScript;36553975]keeping keys in order would be nice[/QUOTE]

Why? The numeric keys [b]are[/b] in order. Why would you need the others?

Well, it's actually up to [b]you[/b].
The [URL="http://lua-users.org/wiki/GeneralizedPairsAndIpairs"][b]__next[/b] metamethod[/URL] of your table will take care of that.

I've updated the code with some speed improvements, and a completely new way of splitting the table.

[Donkie]

I've seen string.split somewhere, I think it was Löve2D, must've been implemented by them then.

[LauScript]

[QUOTE=vercas;36554146]Why? The numeric keys [b]are[/b] in order. Why would you need the others?

Well, it's actually up to [b]you[/b].
The [URL="http://lua-users.org/wiki/GeneralizedPairsAndIpairs"][b]__next[/b] metamethod[/URL] of your table will take care of that.

I've updated the code with some speed improvements, and a completely new way of splitting the table.[/QUOTE]

Oh i misunderstood, for some reason i thought you said that numeric keys didn't get put in order or something, sorry.

[vercas]

I'm looking for help. My current method of escaping strings is proving outrageously slow.
Does anyone know a better way of escaping the strings?

	json_example = {
		menu = {
			header = "SVG Viewer",
			items = {
				{ id = "Open" },
				{ id = "OpenNew", label = "Open New" },
				nil,
				{ id = "ZoomIn", label = "Zoom In" },
				{ id = "ZoomOut", label = "Zoom Out" },
				{ id = "OriginalView", label = "Original View" },
				nil,
				{ id = "Quality" },
				{ id = "OpenNew" },
				{ id = "Mute" },
				nil,
				{ id = "Find", label = "Find..." },
				{ id = "FindAgain", label = "Find Again" },
				{ id = "Copy" },
				{ id = "CopyAgain", label = "Copy Again" },
				{ id = "CopySVG", label = "Copy SVG" },
				{ id = "ViewSVG", label = "View SVG" },
				{ id = "ViewSource", label = "View Source" },
				{ id = "SaveAs", label = "Save As" },
				nil,
				{ id = "Help" },
				{ id = "About", label = "About Adobe CVG Viewer..." }
			}
		}
	}

Takes 1.9 seconds to complete 5000 cycles.

von_complex_example = {
	    1, -1337, -99.99, 2, 3, 100, 101, 121, 143, 144, "ma\"ra", "are", "mere",
	    {
	        500,600,700,800,900,9001,
	        TROLOLOLOLOLOOOO = 666,
	        [true] = false,
	        [false] = "lol?",
	        pere = true,
	        [1997] = "vasile",
	        [{ [true] = false, [false] = true }] = { [true] = "true", ["false"] = false }
	    },
	    true, false, false, true, false, true, true, false, true,
	    [1337] = 1338,
	    mara = "are",
	    mere = false,
	    [true] = false,
	    [{ [true] = false, [false] = true }] = { [true] = "true", ["false"] = false }
	}

Takes 1.3 seconds to complete 5000 cycles.

[KatNotDinner]

[QUOTE=Gran PC;36549769]What are you talking about, string.split doesn't even exist.[/QUOTE]

Then you say that the function is brought by datastream? Cause there's no other explanation for it being in the wiki.

[URL]http://maurits.tv/data/garrysmod/wiki/wiki.garrysmod.com/index9f52.html[/URL]

[vercas]

[QUOTE=KatNotDinner;36567314]Then you say that the function is brought by datastream? Cause there's no other explanation for it being in the wiki.

[URL]http://maurits.tv/data/garrysmod/wiki/wiki.garrysmod.com/index9f52.html[/URL][/QUOTE]

Why would that even matter?
Nothing useful can come out of debating over this.

[Gran PC]

[QUOTE=KatNotDinner;36567314]Then you say that the function is brought by datastream? Cause there's no other explanation for it being in the wiki.

[URL]http://maurits.tv/data/garrysmod/wiki/wiki.garrysmod.com/index9f52.html[/URL][/QUOTE]

How about you go ingame and test if it exists?!

[KatNotDinner]

How about you calm down and think that I might've used this function in GMod 12 when datastream still came with gmod? I used to use the function and I found out that it came with datastream recently.

[Remscar]

VON seems to fail when i run
[lua]
print(von.deserialize(von.serialize({1,2,3})))
[/lua]
line 134: Reached end of string, incomplete number definition.

[vercas]

[QUOTE=Remscar;36582631]VON seems to fail when i run
[lua]
print(von.deserialize(von.serialize({1,2,3})))
[/lua]
line 134: Reached end of string, incomplete number definition.[/QUOTE]

Fixed. Thanks for reporting! :smile:

[Remscar]

[QUOTE=vercas;36582671]Fixed. Thanks for reporting! :smile:[/QUOTE]

Awesome on the fast fix, i was working on my gamemode and was using vON to store character data in a MySQL database.

[Divran]

Found a bug. When using this to transfer data from the client to the server, all " are changed into ' (It seems console commands do it, don't know why)
This makes von print this error:

[...\von.lua:81] vON: Malformed data... Can't find a proper type definition. Char#1:'

And I don't think you have any code for escaping ', if that's necessary.

EDIT: A temporary fix I will use until you fix this is gsub:
[lua]datastr = datastr:gsub( "'", '"' )[/lua]
but I'd rather not have to do that.

And what if the data I send is supposed to have both a ' and a ", and they have a different meaning?

[vercas]

[QUOTE=Divran;36588667]Found a bug. When using this to transfer data from the client to the server, all " are changed into ' (It seems console commands do it, don't know why)
This makes von print this error:

[...\von.lua:81] vON: Malformed data... Can't find a proper type definition. Char#1:'

And I don't think you have any code for escaping ', if that's necessary.[/QUOTE]

's are not escaped and they're not used as type definitions. This is very weird.
Can you show me the table you're serializing and the resulted vON code?

[Divran]

[QUOTE=vercas;36588680]'s are not escaped and they're not used as type definitions. This is very weird.
Can you show me the table you're serializing and the resulted vON code?[/QUOTE]

Any table of strings does the same. The resulting vON code is fine. It only breaks when sending it through console commands.

{"test","hurr durr"}

EDIT: I think I have a solution. I'll use E2Lib's E2Lib.encode. A function which was made for encoding chars that normally break when sending over console commands. This should fix all problems.
EDIT 2: Yep that fixed it.

[vercas]

[QUOTE=Divran;36588690]Any table of strings does the same. The resulting vON code is fine. It only breaks when sending it through console commands.

{"test","hurr durr"}

[/QUOTE]

I'm sure the problem is not with vON.
If you send vON code through a console command, won't you have to re-escape it?

[b]Edit[/b]: Oh, now I've seen your edit.

[Divran]

Yeah it never was your fault :) Ah well, talking (edit: err typing?) out loud helped me figure it out, as usual.