gm_heap

If you don’t know what a pointer is, or what the heap is, don’t bother with this thread.

So yea, this module lets you allocate and deallocate memory from the heap.

Download: http://www.fortfn.co.uk/files/gm_heap.dll
Source: http://pastebin.com/Nm4tdyJn

Usage:

[lua]local ptr = heap.Allocate( 10 ) – Allocate 10 bytes
heap.Set( ptr, 0, 10 ) – Set all of our newly allocated memory to 0
for i=1, 10 do
heap.SetByte( ptr+i - 1, i ) – Set each byte to a value
end
for i=1, 10 do
print( heap.GetByte( ptr+i - 1 ) ) – Print it
end

local ptr2 = heap.Allocate( 10 ) – Allocate another 10 bytes
heap.Set( ptr2, 0, 10 ) – Set all of our newly allocated memory to 0
heap.Copy( ptr, ptr2, 5 ) – Copy 5 bytes across

print( “—” )

heap.Deallocate( ptr ) – Deallocate our original 10 bytes
for i=1, 10 do
print( heap.GetByte( ptr2+i - 1 ) ) – Print it
end

heap.Deallocate( ptr2 ) – Deallocate our last 10 bytes[/lua]


1
2
3
4
5
6
7
8
10
---
1
2
3
4
5
0
0
0
0
0

Now of course, using a module for storing an array of 10 bytes is completely stupid, but suppose you want to store an array of 1,000,000 bytes (~1 mb of data). Lua will not offer you the speed to manage, access and set the data in a table of such size but this module should do.

Can we get perf times for both a 1mb table(4 bytes per element) and 1mb of data with your module.

And perhaps 10mb, 100mb and 500mb tests.

[editline]02:32AM[/editline]

Also you need to encapsulate the pointers, so the do bounds checking or something.

This could be very unsafe if it wanted to be.

Awsome module, but i cant find any use for it

[editline]11:18AM[/editline]

Actaully lets store the players data in the heap! :pseudo:

As haza said we need proper testing before :slight_smile:

It’s only as unsafe as the scripts which use it. And I’ll do some speed tests later.

I will use this for storing items in my gamemode, as i will have around… 5k+ items placed on the map it will be a good test of this

[editline]04:07PM[/editline]

Deallocating seems to take about… forever on 1mb data

I did some basic testing.

[lua]
require( “heap” )

local B = 1
local KB = B * 1024
local MB = KB * 1024

local cnt = 10 * MB

local function cmdTestLuaTable( ply, com, args )
debug.sethook()
local start = os.time()
local tbl = {}
for i=1, cnt do
tbl* = i % 255
end
local o = 0
for i=1, cnt do
o = o + tbl*
end
print( o )
print( “Lua table took " … (os.time()-start) … " seconds to test!” )
end
concommand.Add( “heap_testluatable”, cmdTestLuaTable )

local function cmdTestHeapTable( ply, com, args )
debug.sethook()
local start = os.time()
local ptr = heap.Allocate( cnt )
local Set = heap.SetByte
local Get = heap.GetByte
for i=1, cnt do
Set( ptr + i - 1, i % 255 )
end
local o = 0
for i=1, cnt do
o = o + Get( ptr + i - 1 )
end
print( o )
heap.Deallocate( ptr )
print( “Heap array took " … (os.time()-start) … " seconds to test!” )
end
concommand.Add( “heap_testarray”, cmdTestHeapTable )[/lua]



] heap_testluatable 
not enough memory
] heap_testarray 
1331684080
Heap array took 5 seconds to test!


Seems like the limit for the amount of data to be stored in a lua table is between 1mb and 10mb. Both loops cause Garry’s infinite loop detector to be triggered off (hence debug.sethook). I guess there’s no way to make looping ~1,000,000,000 times in Lua any faster (without LuaJIT). So I think this module is only good if you need to store ALOT of memory and need to access it fast (like mine and overv’s minecraft in gmod projects).

I’m not gonna bother going higher than 10mb since obviously you can’t test lua tables, and I’d get bored waiting.

I just got a hl2.exe error after testing that one, its pretty unstable, i think its the deallocating

I hope you don’t mind, but I was already working on a module that will probably depreciate this, so don’t be offended if it does :saddowns:.

Go right ahead, my implementation is pretty crappy to be fair.

[editline]08:08PM[/editline]

Can you show me your code?

All i did was some heap.Allocate some heap.SetByte and some heap.Deallocate, it crashes when you Dealloc it, your example crashes too, i think its your de-allocation that crash it

It could be, but my example works fine on my PC. Are you testing on a win32 machine?

You should use malloc and free. new and delete are probally performing rtti checks.

This module seems rather useless, because unless you’re going to read a value from memory, Lua can do the exact thing. It abstracts the ability to call malloc/free by giving you garbage collected values like tables.

If you want to store 5000 items use a table with unique ids as keys. It will be many times faster than any system you could possibly create with the scope of this module. If you’re worried about memory constraints, move it into SQL and memcached so you can cache item data and anything else.

Also, as for your benchmark about not being able to allocate the table, it’s because of the way tier0 handles memory allocation. It’s not optimized for the way Lua works, which was one of the issues being investigated. (Many allocations of small objects will take up larger buckets, large allocations will fail because of this. The bucket system isn’t exactly suited for Lua)

This is better in some cases when lua runs out of memory, its just that its… crashy

Yeap, the source engine mallocs hunks(heap chunk). It then deals out small proportions of that memory. This is generally faster because we don’t have to do a syscall, wait for the operating system to allocate a virtual block of memory and return(and all the extra descriptors placed on that memory etc etc).

It’s not useless when you want to store 250,000+ items.

Cool, this is just like ( warning ) [SP] the cheat engine [/SP] and you can ( Warning ) [SP] speed hack by changing the speed values and update timers [/SP] but sadly i think garry or valve fixed it :(:

What?

If you want to use this to cheat, you still have to find the memory addresses of stuff you want to change and get them into lua. Using cheat engine would be easier than using this to cheat.