HON Solo (Henry's Object Notation)... solo

This table serializer, with a name based on a notable Star Wars character, focuses on readability. And this is as readable as it gets… I hope.
This actually translates the table into something that lua can parse and run. There isn’t a decoder written, not yet, since all you need to do is RunString the encoded text.

Speed Test:


Start HON Solo Deserializer
===================================
    Test [1]: 0.304469
    Test [2]: 0.315747
    Test [3]: 0.301665
    Test [4]: 0.303268
    Test [5]: 0.303499
    Test [6]: 0.299851
    Test [7]: 0.319738
    Test [8]: 0.303351
    Test [9]: 0.301949
    Test [10]: 0.304352
    Length: 899
    Average Time: 0.3057889
    Accuracy: pretty darn accurate
===================================

Start VON Deserializer
===================================
    Test [1]: 1.318248
    Test [2]: 1.302694
    Test [3]: 1.322732
    Test [4]: 1.330334
    Test [5]: 1.319991
    Test [6]: 1.329704
    Test [7]: 1.314146
    Test [8]: 1.323041
    Test [9]: 1.401673
    Test [10]: 1.348156
    Length: 967
    Average Time: 1.3310719
    Accuracy: pretty darn accurate
===================================


example.lua:
[lua]require(“honsolo”)

local test = {
rawr = “blah\b”;
test = [=[a\bsd\afsdf()
test()]==]]]]
test “”’’]=];
zombie = { test = “hi”; hello = “brah”; bloo = {}; ack = ‘aaasdasd’ };
[1] = “test”;
[“hello there!”] = “zz”;
}

print( honsolo.encode(test, true) );

[/lua]

output:



{
    [1] = "test",
    ["hello there!"] = "zz",
    test = [=[a\bsd\afsdf()
test()]==]]]]
    test ""'']=],
    rawr = "blah\b",
    zombie = 
    {
        test = "hi",
        hello = "brah",
        ack = "aaasdasd",
        bloo = 
        {
        }
    }
}

if you want to create a more compact one with less readability, don’t compile with the “pretty” argument on:
[lua]
– don’t add the second argument.
honsolo.encode(test);
[/lua]

the output will have no indentation and other unnecessary stuff

honsolo.lua:
[lua]-- “Beer-Ware License” (Revision: Pabst Blue Ribbon)
– <qwookiejar@gmail.com> wrote this file. As long as you retain this notice you
– can do whatever with this stuff. Unlike Poul-Henning Kamp’s license, if we
– meet some day, and you think this stuff is worth it, you can buy me a PBR
– in return.

– version 0.0.4 added booleans

local string, table, type, pairs, tonumber = string, table, type, pairs, tonumber;
local print = print;

module( “honsolo” );

local regex_endquotes = “](=)]";
local regex_unescaped = "^[%a_][%a%d_]
"; local regex_digits = "^%d+”;
local norm_dquotes = “”";
local norm_quotes = “’”;
local norm_newline = "
";

extensions = {};

– Add your extensions here I guess.
– Name the child after the datatype. AKA whatever you get with type().
– eg. string, number, Vector, Entity, Angle

– The below extension is not guaranteed to work, this is due to me not having GMOD to test it out on.
function extensions.Vector( data )
return “Vector(” … data.x … “,” … data.y … “,” … data.z … “)”
end

function escape_key( str )
if string.find(str, regex_unescaped) then
return str;
else
if tonumber(str) then
return “[” … str … “]”;
else
return “[” … escape_string(str) … “]”;
end
end
end

local controls = {
["""] = “\”";
["\a"] = “\a”;
["\b"] = “\b”;
["\f"] = “\”";
["
“] = “\””;
["\r"] = “\”";
[" “] = “\””;
["\v"] = “\”";
["\"] = “\\”;
["’"] = “\’”;
["\0"] = “\0”;
}
local function control_escape( char )
return controls[char];
end

function escape_string( str, out )
local use_big_string_escape = false;

local dquotes = string.find(str, norm_dquotes);
local quotes = string.find(str, norm_quotes);

if ( not quotes and dquotes ) then
    return norm_quotes .. str:gsub("%c", control_escape) .. norm_quotes;
end

if ( not dquotes ) then
    return norm_dquotes .. str:gsub("%c", control_escape) .. norm_dquotes;
end

local escape_count = grab_string_escape_count( str );

return "[" .. ("="):rep(escape_count) .. "[" .. str .. "]" .. ("="):rep(escape_count) .. "]";

end

function grab_string_escape_count( str )
local max_num = 0;
local unescaped = {};
local finding_unescapers = true
local gend = 0;
while ( finding_unescapers ) do
local start, fend, match = string.find(str, regex_endquotes, gend );
gend = fend;
if not match then
finding_unescapers = false;
else
unescaped[match:len()] = true;
end
while ( unescaped[max_num] ) do
max_num = max_num + 1;
end
end
return max_num
end

function encode( data, pretty, indent, out, child )
– first time calling initiate all variables
indent = indent or 0;
out = out or {};

local t = type( data );
if t == "string" then
    return escape_string(data);
elseif t == "number" then
    return data;
elseif t == "boolean" then
    if data then
        return "true";
    end
    return "false";
elseif t == "table" then
    if ( pretty ) then table.insert( out, "

" ); table.insert(out, (" "):rep(indent)) end
table.insert( out, “{” );

    local first = true;
    for key, value in pairs( data ) do
        indent = indent + 1;
        if not first then
            table.insert( out, "," );
        end
        if ( pretty ) then table.insert( out, "

" ); table.insert(out, (" "):rep(indent)) end
table.insert( out, escape_key(key) );
if ( pretty ) then table.insert( out, " " ); end
table.insert( out, “=” );
if ( pretty ) then table.insert( out, " " ); end
table.insert( out, encode(value, pretty, indent, out, true) );
indent = indent - 1;
first = false;
end

    if ( pretty ) then table.insert( out, "

" ); table.insert(out, (" "):rep(indent)) end
table.insert( out, “}” );

    if child then
        return "";
    end
elseif extensions[t] then
    return extensions[t]( data, pretty, indent );
end

return table.concat(out);

end

[/lua]

I quickly wrote this module on a macbook… in VIM. There’s no “decode” function, it converts the table into lua syntax. You have to RunString the text.

How does it compare to VoN?
:v:

i dunno, i wrote it for fun.
for one, the string escaping isn’t broken.

vON is better since it adds vector and angle support. i can probably do that but garry’s mod wont load on my mac :frowning:

Write it in plain lua then convert to gmod?

just updated it with a quick extensions system for the honsolo.lua file.
now you can add different datatypes, like Vector or Angle.
I added Vector as an example.

Updated it once again. It’s faster now.

Benchmark test:


Start HON Solo Serializer
===================================
    Test [1]: 1.126522
    Test [2]: 1.152416
    Test [3]: 1.106659
    Test [4]: 1.101698
    Test [5]: 1.116938
    Test [6]: 1.094186
    Test [7]: 1.149754
    Test [8]: 1.099651
    Test [9]: 1.105124
    Test [10]: 1.124522
    Length: 861
    Average Time: 1.117747
===================================

Start VON Serializer
===================================
    Test [1]: 1.188559
    Test [2]: 1.121239
    Test [3]: 1.117582
    Test [4]: 1.124909
    Test [5]: 1.148265
    Test [6]: 1.146918
    Test [7]: 1.120226
    Test [8]: 1.118492
    Test [9]: 1.126816
    Test [10]: 1.166944
    Length: 930
    Average Time: 1.137995
===================================

Benchmark source:
[lua]require(“honsolo”)
require(“von”)

local test = {
rawr = “blah\b”;
test = [=[a\bsd\afsdf()
test()]==]]]]
test “”’’]=];
zombie = { test = “hi”; hello = “brah”; bloo = {}; ack = ‘aaasdasd’; z = {}; };
[1] = “test”;
[“hello there!”] = “zz”;
again = {
rawr = “blah\b”;
test = [=[a\bsd\afsdf()
test()]==]]]]
test “”’’]=];
zombie = { test = “hi”; hello = “brah”; bloo = {}; ack = ‘aaasdasd’; z = {}; };
[1] = “test”;
[“hello there!”] = “zz”;
again = {
rawr = “blah\b”;
test = [=[a\bsd\afsdf()
test()]==]]]]
test “”’’]=];
zombie = { test = “hi”; hello = “brah”; bloo = {}; ack = ‘aaasdasd’; z = {}; };
[1] = “test”;
[“hello there!”] = “zz”;
big_string = [====[

local string, table, type, pairs, tonumber = string, table, type, pairs, tonumber;
local print = print;

module( “honsolo” );

local regex_endquotes = “](=)]";
local regex_unescaped = "^[%a_][%a%d_]
"; local regex_digits = "^%d+”;
local norm_dquotes = “”";
local norm_quotes = “’”;
local norm_newline = "
";

extensions = {};

]====];
};
};
}

print( “” );
print( “Start HON Solo Serializer” );
print( “===================================” );
–print( " Output: " … honsolo.encode(test) );
local num = 0;
for i = 1, 10 do
local x = os.clock();
for i = 1, 10000 do
honsolo.encode(test);
end
local y = os.clock();
print( " Test ["…i…"]: " … y - x );
num = num + ( y - x );
end
print( " Length: " … honsolo.encode(test):len() );
print( " Average Time: " … num / 10 );
print( “===================================” );
print( “” );
print( “Start VON Serializer” );
print( “===================================” );
–print( " Output: " … von.serialize(test) );
local num = 0;
for i = 1, 10 do
local x = os.clock();
for i = 1, 10000 do
von.serialize(test);
end
local y = os.clock();
print( " Test ["…i…"]: " … y - x );
num = num + ( y - x );
end
print( " Length: " … von.serialize(test):len() );
print( " Average Time: " … num / 10 );
print( “===================================” );
print( “” );
[/lua]

LuaData.

Please benchmark the deserialization too. I’m curious about the results. On vON, my main focus was deserialization.

Edit: From the source, I see you didn’t localize some functions like table.insert. Doing so will make it even faster.

There is no deserialization
you just do RunString(“blah =” … encodedstuff)

It just translates the table into something readable by lua.

Well, test that. I’m curious but too lazy to test for myself. xD


Start HON Solo Deserializer
===================================
    Test [1]: 0.304469
    Test [2]: 0.315747
    Test [3]: 0.301665
    Test [4]: 0.303268
    Test [5]: 0.303499
    Test [6]: 0.299851
    Test [7]: 0.319738
    Test [8]: 0.303351
    Test [9]: 0.301949
    Test [10]: 0.304352
    Length: 899
    Average Time: 0.3057889
    Accuracy: pretty darn accurate
===================================

Start VON Deserializer
===================================
    Test [1]: 1.318248
    Test [2]: 1.302694
    Test [3]: 1.322732
    Test [4]: 1.330334
    Test [5]: 1.319991
    Test [6]: 1.329704
    Test [7]: 1.314146
    Test [8]: 1.323041
    Test [9]: 1.401673
    Test [10]: 1.348156
    Length: 967
    Average Time: 1.3310719
    Accuracy: pretty darn accurate
===================================


this is using
[lua]loadstring(“z=” … hon_todecode)();[/lua]
as the deserializer for HON, since that’s lua’s native way of evaluating strings.

Benchmark Source:
[lua]require(“honsolo”)
require(“von”)

local test = {
rawr = “blah\b”;
test = [=[a\bsd\afsdf()
test()]==]]]]
test “”’’]=];
zombie = { test = “hi”; hello = “brah”; bloo = {}; ack = ‘aaasdasd’; z = {}; };
[1] = “test”;
[“hello there!”] = “zz”;
again = {
rawr = “blah\b”;
test = [=[a\bsd\afsdf()
test()]==]]]]
test “”’’]=];
zombie = { test = “hi”; hello = “brah”; bloo = {}; ack = ‘aaasdasd’; z = {}; };
[1] = “test”;
[“hello there!”] = “zz”;
again = {
rawr = “blah\b”;
test = [=[a\bsd\afsdf()
test()]==]]]]
test “”’’]=];
zombie = { test = “hi”; hello = “brah”; bloo = {}; ack = ‘aaasdasd’; z = {}; };
[1] = “test”;
[“hello there!”] = “zz”;
[“acur a see”] = “pretty darn accurate”;
big_string = [====[

local string, table, type, pairs, tonumber = string, table, type, pairs, tonumber;
local print = print;

module( “honsolo” );

local regex_endquotes = “](=)]";
local regex_unescaped = "^[%a_][%a%d_]
"; local regex_digits = "^%d+”;
local norm_dquotes = “”";
local norm_quotes = “’”;
local norm_newline = "
";

extensions = {};

]====];
};
};
}

local hon_todecode = honsolo.encode(test);
local von_todecode = von.serialize(test);

print( “” );
print( “Start HON Solo Deserializer” );
print( “===================================” );
local num = 0;
for i = 1, 10 do
local x = os.clock();
for i = 1, 10000 do
loadstring(“z=” … hon_todecode)();
end
local y = os.clock();
print( " Test ["…i…"]: " … y - x );
num = num + ( y - x );
end
print( " Length: " … honsolo.encode(test):len() );
print( " Average Time: " … num / 10 );
print( " Accuracy: " … z.again.again[“acur a see”] );
print( “===================================” );
print( “” );
print( “Start VON Deserializer” );
print( “===================================” );
local num = 0;
for i = 1, 10 do
local x = os.clock();
for i = 1, 10000 do
z = von.deserialize(von_todecode);
end
local y = os.clock();
print( " Test ["…i…"]: " … y - x );
num = num + ( y - x );
end
print( " Length: " … von.serialize(test):len() );
print( " Average Time: " … num / 10 );
print( " Accuracy: " … z.again.again[“acur a see”] );
print( “===================================” );
print( “” );
[/lua]

needs a better way of detecting accuracy though

I also gotta write something for booleans.

As I expected. :v:
Your deserializer is basically written in C. Mine is in Lua… :C
Nice work! :slight_smile:

waiting for an even faster serialization code to appear

Why not use JSon, it looks easier to read and does the same function?

JSON is like 3x slower than this.

And it’s bound to the JSON standard, which says stuff like all indices must be strings. Tables with numeric indices are turned to arrays and the rest of the keys are lost.

also, on your thing about readability, it is pretty readable.

input:
[lua]
local test = {
so = “you want readability?”;
blah = {
this_is = “pretty readable to me.”;
what_more = “can you want?”;
[“if you know lua”] = “it’s pretty easy to figure it out.”;
}
}
[/lua]

out:


{
    blah = 
    {
        what_more = "can you want?",
        ["if you know lua"] = "it's pretty easy to figure it out.",
        this_is = "pretty readable to me."
    },
    so = "you want readability?"
}

order gets messed up on the output though.

Looks like i am switching to HON from VON.
-.-

Extremely useful, thanks. One minor optimization though. Localize only the functions you are using, not the entire libraries, that way there will be no table lookups.

I misread a post.

you can always re-rate