vON: Vercas' Object Notation [Release for developers]

[vercas]

[QUOTE=Divran;36588743]Yeah it never was your fault :) Ah well, talking (edit: err typing?) out loud helped me figure it out, as usual.[/QUOTE]

Are you going to use vON in Wiremod?

[Divran]

I'm looking for a replacement for glon, yes. When I added includes to E2, I needed to be able to send multiple files. The easiest way to do this is to encode it in glon, split up the glon string into umsg (or console command) sized chunks, and send them. But if your E2 has approximately 5000 lines of code, glon.decode causes the entire server to freeze for several seconds. I wanted to try your vON in the hope that it didn't have this problem. It does... I tried sending 10 000 lines of

#AAAAAAAAAAAAAASSSSSSSSSSSSSSDDDDDDDDDDDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

and after freezing gmod for over 30 seconds, it errored with

Expression 2 upload failed! Error message:
not enough memory

Help?

EDIT: If you're traversing the entire string, looking for separators, you need to stop doing that and instead use string.find or match or similar. That could fix it.

[vercas]

[QUOTE=Divran;36588764]I'm looking for a replacement for glon, yes. When I added includes to E2, I needed to be able to send multiple files. The easiest way to do this is to encode it in glon, split up the glon string into umsg (or console command) sized chunks, and send them. But if your E2 has approximately 5000 lines of code, glon.decode causes the entire server to freeze for several seconds. I wanted to try your vON in the hope that it didn't have this problem. It does... I tried sending 10 000 lines of

#AAAAAAAAAAAAAASSSSSSSSSSSSSSDDDDDDDDDDDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

and after freezing gmod for over 30 seconds, it errored with

Expression 2 upload failed! Error message:
not enough memory

Help?[/QUOTE]

Ah damn it. Looping so much shit character by character is obviously going to be slow. :C
For storing this much data, you need something else.

If you wish, I'll start working on a filesystem or archive-like format for serialization of large datas..
It will be slow-ish for small tables, but I promise good speeds for such large data.
It's funny because I've actually planned to do it this afternoon.

[b]Edit[/b]: Wait, before that, I will try your idea.

[Divran]

[QUOTE=vercas;36588804]Ah damn it. Looping so much shit character by character is obviously going to be slow. :C
For storing this much data, you need something else.

If you wish, I'll start working on a filesystem or archive-like format for serialization of large datas..
It will be slow-ish for small tables, but I promise good speeds for such large data.
It's funny because I've actually planned to do it this afternoon.[/QUOTE]

Couldn't you just edit vON to handle long strings as good as small ones?

EDIT:
[QUOTE=vercas;36588804][b]Edit[/b]: Wait, before that, I will try your idea.[/QUOTE]

ok

[vercas]

[QUOTE=Divran;36588823]Couldn't you just edit vON to handle long strings as good as small ones?

EDIT:

ok[/QUOTE]

Well, I'll be damned. It's actually faster than the old method. The JSON example scored 0.15 seconds less.
I've updated the file. I hope it works.

I'll look into using the same method to parse the numbers. :D

[B]Also to note, it broke compatibility with old vON code.
\ is no longer escaped.
The nice serializer no longer exists. It's useless anyway.[/B]

[b]Edit[/b]: There goes the first Dumb rating.

[Divran]

vON is now in wiremod. :)

[vercas]

Old:

GLON:
	5000 encoding/decoding successions took 4.3902492523193 seconds to finish.
	Length of final (probably overmutilated) data: 597.

JSON:
	5000 encoding/decoding successions took 2.1211223602295 seconds to finish.
	Length of the final (100% mutilated) data: 545.

vON:
	5000 encoding/decoding successions took 2.0071144104004 seconds to finish.
	Length of the final (100% healthy) data: 583.
[Finished in 8.6s]

New:

GLON:
	5000 encoding/decoding successions took 4.2862453460693 seconds to finish.
	Length of final (probably overmutilated) data: 597.

JSON:
	5000 encoding/decoding successions took 2.1321220397949 seconds to finish.
	Length of the final (100% mutilated) data: 545.

vON:
	5000 encoding/decoding successions took 1.4760837554932 seconds to finish.
	Length of the final (100% healthy) data: 583.
[Finished in 7.9s]

25% faster thanks to Divran.

[Python1320]

--snip--

[hydral1k]

Very neat. Would be really nice if you added support for other Gmod datatypes. :)

Edit:
[QUOTE=pennerlord;36547935]Maybe this could help you:
[url]http://trac.caspring.org/wiki/LuaPerformance[/url][/QUOTE]

Nice find.

[vercas]

[QUOTE=hydral1k;36593835]Very neat. Would be really nice if you added support for other Gmod datatypes. :)[/QUOTE]

It's on my to-do list. :smile:

[T3hGamerDK]

I'm glad to see things such as these come up, instead of people constantly relying on older technologies as time passes by (I'm looking at you, X.Org).
Hopefully, developers around will adapt this, or include their own and equally fast versions, instead of still relying on GLON or the like.

[Chewgum]

Got an error trying to serialize a table containing vectors:

] lua_run PrintTable(dusk.zombieSpawnpoints)
> PrintTable(dusk.zombieSpawnpoints)...
1	=	7806.8506 7943.6479 992.0000
2	=	7655.5376 8504.5400 992.8785
3	=	7739.3364 8230.2705 992.0000
4	=	8018.0410 7831.1504 992.0000
] lua_run dusk:SaveZombieSpawnpoints()
> dusk:SaveZombieSpawnpoints()...
[gamemodes\dusk\gamemode\modules\sv_von.lua:99] attempt to call field '?' (a nil value)

[vercas]

[QUOTE=_Chewgum;36610252]Got an error trying to serialize a table containing vectors:

] lua_run PrintTable(dusk.zombieSpawnpoints)
> PrintTable(dusk.zombieSpawnpoints)...
1	=	7806.8506 7943.6479 992.0000
2	=	7655.5376 8504.5400 992.8785
3	=	7739.3364 8230.2705 992.0000
4	=	8018.0410 7831.1504 992.0000
] lua_run dusk:SaveZombieSpawnpoints()
> dusk:SaveZombieSpawnpoints()...
[gamemodes\dusk\gamemode\modules\sv_von.lua:99] attempt to call field '?' (a nil value)

[/QUOTE]

I didn't add support for GMod types yet.
Imma do it now, tho. Don't know how long is it going to take.

[b]Edit[/b]:
[URL="https://dl.dropbox.com/u/1217587/GMod/Lua/von%20for%20GMOD%20-%20development.lua"]Here[/URL] is the development version of vON for GMod.
Please tell me the results.

[hydral1k]

Looking good. Is there any plans for Color obj?

[vercas]

[QUOTE=hydral1k;36611586]Looking good. Is there any plans for Color obj?[/QUOTE]

As far as I know, [b]Color[/b]s are just tables, so they should be safely serialized and deserialized.

[b]Edit[/b]:
I need feedback on the GMod development version.
Please tell me if there are any errors or more types you need in.

[b]Edit2[/b]:
I have to go to sleep, so I'll read the feedback tomorrow (in aprox. 10 hours).

[Chewgum]

[QUOTE=vercas;36611601]As far as I know, [b]Color[/b]s are just tables, so they should be safely serialized and deserialized.

[b]Edit[/b]:
I need feedback on the GMod development version.
Please tell me if there are any errors or more types you need in.

[b]Edit2[/b]:
I have to go to sleep, so I'll read the feedback tomorrow (in aprox. 10 hours).[/QUOTE]
You got 2 Vector (one is supposed to Angle) in serialize.

[KatNotDinner]

[QUOTE=vercas;36611601]As far as I know, [b]Color[/b]s are just tables, so they should be safely serialized and deserialized.[/QUOTE]

True. The Color() function is defined in the lua files and it just creates a table.

[James]

[QUOTE=vercas;36610309]I didn't add support for GMod types yet.
Imma do it now, tho. Don't know how long is it going to take.

[b]Edit[/b]:
[URL="https://dl.dropbox.com/u/1217587/GMod/Lua/von%20for%20GMOD%20-%20development.lua"]Here[/URL] is the development version of vON for GMod.
Please tell me the results.[/QUOTE]

[lua]
-- 3 numbers separated by a comma.
["Vector"] = function(data, mustInitiate, isNumeric, isKey, isLast)
if mustInitiate then
if isKey or isLast then
return "v"..data.x..","..data.y..","..data.z
else
return "v"..data.x..","..data.y..","..data.z..";"
end
end

if isKey or isLast then
return "v"..data.x..","..data.y..","..data.z
else
return "v"..data.x..","..data.y..","..data.z..";"
end
end,

-- 3 numbers separated by a comma.
["Vector"] = function(data, mustInitiate, isNumeric, isKey, isLast)
if mustInitiate then
if isKey or isLast then
return "a"..data.p..","..data.y..","..data.r
else
return "a"..data.p..","..data.y..","..data.r..";"
end
end

if isKey or isLast then
return "a"..data.p..","..data.y..","..data.r
else
return "a"..data.p..","..data.y..","..data.r..";"
end
end
[/lua]

just browsing through and noticed you had 2 Vector keys. I think the second one is supposed to be an angle :3

[vercas]

Ow darn.
Fixed it.

So, uh, how well does it work with the GMod types?
I need to know so I can declare it a stable version.

[Chewgum]

[QUOTE=vercas;36617392]Ow darn.
Fixed it.

So, uh, how well does it work with the GMod types?
I need to know so I can declare it a stable version.[/QUOTE]
I tested the vector, and it works, so I guess angle should work too.

[vercas]

Would you say I can declare it a stable version for GMod?

[Chewgum]

[QUOTE=vercas;36619456]Would you say I can declare it a stable version for GMod?[/QUOTE]

Assuming the entity encoding works, yes.

[vercas]

[b]Emergency update![/b]

Divran lead me to finding a catastrophic flaw in the vON code regarding strings.
[b]Any strings ending in a \ would break the parser![/b]

Please download the latest version!

Now strings end in [b]v"[/b]. That means one character extra for every string... [i](It's obviously removed when deserializing!)[/i]
Sorry, but it's the best way to keep the speed.

[Map in a box]

No offense, but vON doesn't have much use in the beta update. Do keep up the good work, though!

[editline]4th July 2012[/editline]

It could be used in non-gmod projects such as Love2D if you remove the gmod specific syntax! Which I'd use it for. After the beta release, you should look into writing the data as binary.

[vercas]

[QUOTE=Map in a box;36623360]No offense, but vON doesn't have much use in the beta update. Do keep up the good work, though!

[editline]4th July 2012[/editline]

It could be used in non-gmod projects such as Love2D if you remove the gmod specific syntax! Which I'd use it for. After the beta release, you should look into writing the data as binary.[/QUOTE]

There is no GMod-specific syntax.
I've said clearly, it's written in pure Lua.

Also, it is of much use!
I'll use it to save item data in the database. (Most items have different proprietes and I don't want to end up with a table with 100 columns)

[Divran]

[QUOTE=Map in a box;36623360]No offense, but vON doesn't have much use in the beta update.[/QUOTE]

You can still use it to save data to files, but without glon's insanely slow speeds.

[Map in a box]

You have binary data access in beta. It's very inefficient to use this in the beta.

[vercas]

[QUOTE=Map in a box;36626019]You have binary data access in beta. It's very inefficient to use this in the beta.[/QUOTE]

Why would I use non-printable characters?
I want to keep this compatible with as much as possible. There aren't going to be binary characters unless the strings you serialize contain those.

[Map in a box]

Because it's much more efficient. You won't have to worry about keeping things compatible once the beta is released. You can always make a serialization to binary blob converter.

[vercas]

[QUOTE=Map in a box;36627260]Because it's much more efficient. You won't have to worry about keeping things compatible once the beta is released. You can always make a serialization to binary blob converter.[/QUOTE]

Will SQL and MySQL handle binary characters better?
And how are they more efficient?