Increase performance many small files

I don’t think many are going to disagree that addons such as Wire and PHX and used by a large number of people. The issue with such addons is that they use many small files which becomes a performance bottleneck when loading. (Due to the harddrive’s limited random access performance.)

Additionally there are two types of files which case these problems, Lua files and Assets. (Model / texture files and so on.)

I’ll start off with the Assets: If they could be loaded as a single large file, the hard drive performance would no longer be a substantial bottle neck. This could be a single zip archive which can be either compressed or not depending on the user’s system. The archive would be “mounted” in the system memory and accessed using file mapping. If the user had lots of available memory, the archive(s) may be fully loaded into memory and therefore compressed archives may be used to save space. (As decompression would only be needed during loading.) If the user has limited memory, the archives may only be able to be partially loaded as required, requiring decompression during the game and hence no compression may be better. This system is like that of the Game Cache Files.

Next, Lua files: These not only suffer from the problems that Assets have, but also suffer from execution performance issues when used with complex addons. These issues could both be solved by writing entire addons as binary modules instead of collections of Lua files. This could probably already be done with the current setup, but would be easier for addon developers if had a native interface which could allow more direct access to entities and the like. In other words, the power and simplicity of the Lua interface but with native performance. Also as all of the code would be combined into a single .dll for the addon, it would not suffer the multiple files issues mentioned above.

The problem with binary modules is that they require compiling and restarting in order to take effect. It’d take too long to test for errors, especially to new scripters.

Well with a change in the way they are loaded, the restarting part can be removed.

Also while I agree that it would take longer to do testing, keep in mind that Source Mods take just as long to make a change and there are plenty of them which get made. Additionally the binary module setup would be as well as the Lua option - Most addons would still be sets of Lua files (Possibly in an archive like the Assets.) and only computationally expensive addons would need to be done as modules.

You sound very smart so I’m going to agree.

Typical 12 year old’s reaction

I’m not 12 >_> it sounds like a good idea, though its not in my field so i can’t be talking. I thought saying that would make it easier to convey the general idea easier.

If you don’t understand it, what makes you agree with it?

Apparently, Garry is going to recreate the addon system. It should work better for more addons and files. If and when he does, this shouldn’t be required.

You make me have to type too much. I understand it vaguely but I am no expert in the area to say for sure, but it seems that it might be a cool idea. Wasn’t it easier to say it sounds cool so I agree?

Also, it would be nice if he could make gmod ignore hidden files so you can svn straight to addons without a performance hit.
I think it would take him longer to remake the addon system so he could do that to help us in the meantime.

If custom content was packed into one file then the client would need to redownload everything again if you make a single change. This isn’t a problem for Lua files because they’re relatively small.

Take a look at gcf’s. There all packed into 1 file, however if needed, individual files can be updated. If you use some system similar to this then it should be fairly effective.

Source isn’t capable of streaming certain parts of a GCF to the client.

The client would have to state how much of the GCF it has, if any, then the server would send what’s missing or outdated. Modifying the clients connection process like that isn’t possible without touching the engine code. Even if it was, you would still need to go further to allow client-specific file downloads due to the unpredictable content requests.

And in the end, that still requires each single file to be downloaded individually. Sure, it’s archived nice and neatly, but you still have the slow download issue.

Okay that’s an interesting point about limitations of the engine, while the Lua files are done by hand I forgot that the asset downloading was handled by the engine. (In fact I forgot about downloading in-game altogether, so keep that in mind when reading the OP. Obviously it still needs a good amount of thought.) I’m not sure on the specific limitations so if anyone has any ideas on how the asset management could be changed that would solve these issues.

The whole point of this was not download performance, but loading (For pre-installed addons.) and run-time performance.

You have typed a few extra sentences because of me, I am truly sorry to waste your time like that

Saying ‘myeah sounds cool’ sounds like ‘myeah, don’t know what the fuck you’re talking about but I’ll just agree’

Perhaps it’d be better to say something like ‘I’m not really sure, but I do think op has got a point’.

But I guess it’s time for me to stop derailing