gm_zlib_b64.dll - ZLib Compression and Base64 Encoding

[ihenry]

[b]GMod In-Memory Compression and Base-64 Encoding module[/b]
:siren: Version 1.2 released - download below :siren:

[b]Download:[/b]
1.1: [url=http://91.90.24.164/d/f32f229a6fe4538bfeaa1f748359fb47]http://upl.vs-hs.com/d/f32f229a6fe4538bfeaa1f748359fb47[/url]
[b]1.2:[/b]
[b][url]http://files.filefront.com/gm+zlib+b64zip/;10113797;/fileinfo.html[/url][/b]
[url=http://91.90.24.164/d/c9cdea15d9b545cb0a42bc19d06d9b1b]http://upl.vs-hs.com/d/c9cdea15d9b545cb0a42bc19d06d9b1b[/url]
Source and documentation included in download.

[b]Features:[/b]
- In-memory compression and decompression of strings
- In-memory Base-64 encoding and decoding of compressed data
- Reading and writing compressed data to and from files
- Now threaded for slow operations (1.2)

[b]Other code used:[/b]
zlib: [url]http://www.zlib.net/[/url]
base-64 code: http://base 64.sourceforge.net/ (remove the space)

[b]Compression Statistics (compression level 6):[/b] [i](multiply by 4/3 to get Base-64 encoded size)[/i]
Advanced Duplicator v0.83 save file: 142KB => 34.9KB (25%)
Advanced Duplicator v0.63 save file: 665KB => 31.9KB (5%)
Wire CPU (assembly) code: 177KB => 23.8KB (13%)
Contraption Saver save file: 83KB => 6.8KB (8%)

[b]Changelog:[/b]

Version 1.0
    - Initial release
    - Changed zlib code to remove VC++ 2008 CRT dependency (now uses kernel32.dll heap functions for memory allocation)
    - Changed Base-64 code for in-memory encoding and decoding, and for module use
    - Removed GZip support from zlib code (not used in this module)
Version 1.1
    Added:
        - Headers. See CompBuffer:SetHeader and CompBuffer:GetHeader functions.
        - Optional strategy parameter to zlib.Compress
        - Added Z_DEFAULT_STRATEGY, Z_FILTERED, Z_HUFFMAN_ONLY, Z_RLE and Z_FIXED strategy constants.
    Changed:
        - 'compress' zlib function to use deflateInit2 (for zlib.Compress strategy parameter).
        - Moved zlib source into the main project.
    Fixed:
        - Completely restructured the Base-64 functions and the file format.
        - Crash on disconnect (the garbage collection functions are called *after* the gmod_close function...)
        - Bug where execution would continue if there was an error (in zlib.Compress and zlib.Decompress).
        - A few buffer overrun bugs.
    Removed:
        - More unused zlib functions.
Version 1.2
    Added:
        - Asynchronous functions (zlib.CompressAsync, zlib.DecompressAsync, b64.EncodeAsync, b64.DecodeAsync,
              CompBuffer:DecompressAsync, CompBuffer:EncodeAsync, zlib.ReadAsync, zlib.WriteAsync)
        - Added TestZlibAsync(f, complevel, strategy) function to the test script (to test the new asynchronous functions).
              The timing uses UnPredictedCurTime() before and after the function call.
        - Documented zlib.version (not available in version 1.0, available but undocumented in 1.1)
        - Wrote new documentation
    Fixed:
        - Bug in zlib.Decompress where the error code was being returned (so Lua was looking for 'err' return values),
              instead of jumping to a gLua->Error call (this would only happen if there was an error in deflateInit2))
        - Missing local variable in test script (reported by tad2020)
        - Crash on disconnect when the module was loaded into both the client and server Lua states (in singleplayer and
          for listen server hosts).
        - Handling of multiple Lua states for callbacks.
        - Occasional 'stream error' messages when decompressing
    Changed:
        - Now using fastcall calling convention for some internal functions
        - Made some zlib internal functions inline

[kevkev]

So this allows us to zip up files inside gmod?
Awesome, they need to add this to the Adv. Duplicator.

[ConnaCook]

I'm adding this to Contraption Saver, definatly.

[cold12141]

[QUOTE=ConnaCook]I'm adding this to Contraption Saver, definatly.[/QUOTE]

Adv. Duplicator > Dirt > Contraption Save

[ihenry]

It got removed from garrysmod.org for some reason, so I re-uploaded it to [url=http://upl.vs-hs.com/d/c2ae84b78c7f7035eb1bd8cf82f38399]VS-HS Upload[/url].

[ConnaCook]

[QUOTE=cold12141]

Adv. Duplicator > Dirt > Contraption Save

[/QUOTE]

Hehe yeah, dirt is a better way of saving contraptions.

[superadamwo]

Yeah, I agree.

[ConnaCook]

I heard if you shovel dirt onto a crate, you can re-use it over a map change!

[ubermensch]

[QUOTE=cold12141]

Adv. Duplicator > Dirt > Contraption Save

[/QUOTE]

More like:

Dirt > Adv. Duplicator > Contraption Saver

Because you can physically touch/throw/eat dirt. Plus dirt doesn't break every time Gmod is updated...

Anyway back on topic.

If I wanted to send a compressed string to the client, would I need the client to have this installed?

[ihenry]

[quote=UberMensch]If I wanted to send a compressed string to the client, would I need the client to have this installed?[/quote]
Yes, the client would have to have this installed.

[HTF]

Cool beans, though zlib is a sucky type of compression to use for text, Huffman would be better, maybe consider adding that in the future? The In-memory compression and decompression of strings is epic nice one :D

[tad2020]

How the fuck did I miss this, awesome.

[tad2020]

On one of my computers, when I have this installed and I start a new game for the seconds time, gmod crashes.

[ihenry]

[quote=tad2020]On one of my computers, when I have this installed and I start a new game for the seconds time, gmod crashes.[/quote]
Fixed in version 1.1. The problem was that it was destroying the heap in gmod_close, then the __gc metamethods were called and were trying to free from it.

[quote=HTF]zlib is a sucky type of compression to use for text, Huffman would be better[/quote]
zlib uses Huffman already, but I added the strategy parameter from deflateInit2 to the zlib.Compress function (see [url]http://zlib.net/manual.html#deflateInit2)[/url].

[ERROR1234]

[url]http://www.paulschou.com/tools/xlate/[/url]

[tad2020]

I love this:
[quote]
"adv_duplicator/aircraftcarrier.txt"
Compressed (level 9) to 81.48KB from 2000.44KB (compression ratio: 95.93%)
Compressed data written to "temp.z"
Data read from file "temp.z", matches original compressed data.
Base64 encoded (108.65KB)
Base64 decoded. Decoded data matches compressed data
[/quote]
From over 2MB down to just 108KB, awesome. That's going to speed up file transfers shitloads.

I found an error in your test script: datalen should be data:len()

[QUOTE=ihenry]Fixed in version 1.1. The problem was that it was destroying the heap in gmod_close, then the __gc metamethods were called and were trying to free from it.[/QUOTE]
Cool, that also seemed to fix another crash problem I had with some files.

[Killers2]

Tried to download but the host keeps timing out for me :(.

[Terminal5]

Any noticeable FPS difference?

[LuaPineapple]

Good god, [B]GIVE THIS MAN LUA KINGS YOU BUNCH OF WEAKLINGS[/B]!

[b]Edit:[/b]

[QUOTE=Terminal5]Any noticeable FPS difference?[/QUOTE]
:downs:
Only when you compress/decompress.

[Terminal5]

[QUOTE=LuaPineapple]:downs:
Only when you compress/decompress.[/QUOTE]

I kind of figured that?

I'm asking if it decreases your FPS to 1-2 for a second while it loads or does it go pretty instant.

I'm pretty aware stuff runs when stuff runs Apple.

:downs:

[ihenry]

[quote=Terminal5]I'm asking if it decreases your FPS to 1-2 for a second while it loads or does it go pretty instant.[/quote]
Depends how much you compress/decompress, it usually takes less than half a second for about 200KB. The whole game stops when it's compressing or decompressing.
Decompressing is usually much faster than compressing.

If you want to test it, bind a key to the compression function, e.g.

bind x "lua_run zlib.Compress(file.Read('your_file_here.txt'))"

replacing your_file_here.txt with the path to a large file in the data folder (such as an Advanced Duplicator file), unpause and press the key, and see how long it takes.

[HTF]

If the whole game stops you may want to look into make it compress/decompress in a worker thread. Similar to the way the mysql module is threaded for querying. It should be simple enough.

[ihenry]

[b]Version 1.2 has been released[/b]
[quote=HTF]make it compress/decompress in a worker thread[/quote]
Version 1.2 adds asynchronous functions, that do the work in worker threads, then call a callback function when they finish.
Compression, decompression, encoding, decoding, reading and writing can now be done in worker threads (by calling the functions ending with 'Async'), but the slowest operations are compression and decompression, where decompression is more than 3 times faster than compression.
[lua]-- example - 'data' is the string to compress
if (zlib.version >= 1.2) then
zlib.CompressAsync(data, function(cmp)
-- do stuff with compressed data here
end)
else
local cmp = zlib.Compress(data)
-- do stuff with compressed data here
end[/lua]
The FPS hit of these new functions is very small, and even smaller on multi-core processors.

It also fixes a bug where some compressed streams would cause a stream error on decompression, and a crash on disconnect when it was loaded by both client and server Lua states in the same process (single player and listen server hosts).

The variable [b]zlib.version[/b] has also been added. It contains the version number.
This was in version 1.1, but as a string. It is now a number.

[b]EDIT:[/b] If anyone got "stream error" in the console when decompressing some things, version 1.2 will be able to decompress them.

[Termy58]

[QUOTE=ihenry][b]Version 1.2 has been released[/b]

Version 1.2 adds asynchronous functions[/QUOTE]
Thanks for the update.

[tad2020]

Awesome, thanks!

[HTF]

Very nice, no more game freezing. Cheers for the update.

[ihenry]

VS-HS is down, I re-uploaded it to Filefront.
[url]http://files.filefront.com/gm+zlib+b64zip/;10113797;/fileinfo.html[/url]

[b]EDIT:[/b] If I look up "upl.vs-hs.com" at [url]http://www.dnswatch.info/[/url], and use the IP address in the address bar ([url]http://91.90.24.164/[/url]), it works...

[NINTENDUDECT]

Sorry if this is a stupid question, but, what exactly does this do? I understand it compresses something, but what exactly? The way I see it, it compresses addons to save memory? I don't really know much about this.

Thanks.

[kevkev]

[QUOTE=NINTENDUDECT]Sorry if this is a stupid question, but, what exactly does this do? I understand it compresses something, but what exactly? The way I see it, it compresses addons to save memory? I don't really know much about this.

Thanks.[/QUOTE]

For example the advance duplicator uses this to save space on your server's drive, it compresses adv, dupe saves.

[ihenry]

[quote=kevkev]For example the advance duplicator uses this to save space on your server's drive, it compresses adv, dupe saves.[/quote]
And to speed up uploads and downloads between the client and the server.
[quote=NINTENDUDECT]I understand it compresses something, but what exactly?[/quote]
It provides Lua functions to compress strings.

[b]EDIT:[/b] Why is

B a s e 6 4

(without spaces) censored?