I don’t play Garry’s Mod anymore and I already long abandoned this project, I am going to leave some technical explanations in case someone asks me in the future when I will have likely forgotten entirely.
I have got wind that someone else was going to make their own duplicator as well (calling my duplicator library “flawed”). Surely duplicating my work would be a waste of effort and time, especially if you are going to avoid what I’ve done and end up with something inherently inferior.
First off, all four operations (copying, pasting, saving, loading) chunk themselves into a number of smaller “tasks” that are expected to finish fairly quickly. Provided that the tasks are evenly-sized, this allows for a very controlled and predictable run. It is the reason that I am able to say that Photocopy serializes at such a specific and exact rate and it is the reason that I am able to reduce the amount of server freezing instances caused by a duplicator. However, the implementation of this was made immeasurably complicated by the fact that Lua co-routines do not work server-side. Thus, in lieu of being able to suddenly yield and maintain stack state, I had to manually re-implement some of that functionality at at the cost of greatly complicating the code.
The overview of how this works is as follows. A set of functions that make up the process are queued in line for execution. However, this line is never generated in full any time. Rather, the first function passes control to the next function that should run, followed by that function passing control to the next function, continuing until the process is finished. To pass control is to store a reference to the next function to be called – not to actually call it. Advancing to the next function actually requires calling an appropriate method from outside the operation’s code. This could occur in a Think hook, for example. Thus, the operation can be split over an indefinite period of time because it is expected that no single function should a long time to finish and will depend on time.
If you look through the code for the operations, you will see self:SetNext() calls spread throughout the code. The first argument is the amount of time to wait before the next function is called. Rather than splitting tasks into even further smaller sub-tasks, I added delays between larger tasks as another approach to the chunking problem. Chunking those larger operations would complicate the code even further and that would be the reason for it.
However, there is one task that cannot be split into smaller sub-tasks and delaying doesn’t really help: copying the table. A copy of the entity tables is required in order to maintain a consistent state of an entity. Splitting this task over several ticks would seriously risk the integrity of the copy.
Adv. Duplicator and TB’s duplicator do indeed do some chunking on their own. However, they do not do as a fine of a chunking as Photocopy. I have frozen servers for at least 10 seconds at a time with Adv. Duplicator so I feel that it is an important problem to tackle. Of course, once again, one limitation still is the table.Copy (a deep table copy written in pure-Lua).
The Photocopy is a part-binary part-text file format. I chose to implement some structures as partial-binary to reduce file size and ease parsing. However, as you may be aware, it is not possible to write or read null bytes to and from files and so I needed a workaround. The part-binary structures that I did use are essentially 4-byte integers; however, one major difference is that 0 will never be read or written. You can find the implementation code in the photocopy_dupe.lua file.
That aside, the format is as follows. The header consists of the bytes:
0x89 P C O P Y 0x1A 0x0D 0x0A
(PCOPY is in ASCII: 0x50 0x43 0x4F 0x50 0x59)
0x1A is so that you can use “type” in MS-DOS on the file (as similar with PNG). The following characters are for detection of new line corruption caused by bad file transfers (such as over FTP), which the .txt files Gmod write are particularly susceptible to. Unfortunately 0x1A should really go after the newline/carriage return characters as the version byte will come after, making detection difficult. If you use Photocopy, you should fix this and also mention it.
What comes after is the version. This is one byte.
Everything after this point is stored as a “chunk.” Chunks consist of a 4-byte ASCII string that indicate the type of chunk, followed by the length of the chunk written as the 4-byte “integer” described above, and then followed by the chunk body. Because chunks are completely independent and store their length in their chunk headers, you do not have to worry about escaping the data. Photocopy files can contain up to as many chunks as the 4-byte “integer” can handle, though you are obvious limited by other factors. The order of chunks does not matter.
Photocopy uses these four chunks:
[li]info - Stores information about the file[/li][li]ents - Stores entity information[/li][li]cons - Stores constraint information[/li][li]strs - Stores strings pooled from the entity and constraint data structures[/li][/ul]
The format of those chunks vary. You can read the code to figure out how they are stored as they are pretty simple.
Fixing Saggy Constraints
The physics engine in the Source engine places a high emphasis on constraint order, as with many physics engines. The reason TB’s duplicator does not have saggy constraints, for the most part, is because it stores the order in which the constraints were created. However, the lack of this feature in other duplicators alone does not explain the severity of the saggy constraint problems in them. The sag created by Adv. Duplicator and Gmod’s duplicator is especially bad because both (Garry’s duplicator library being the original) use a hash map when tabulating the list of constraints. If this alone is fixed, a lot of saggy constraint problems are fixed; however, due to the issues of cross-constraints, it is not a complete solution. Photocopy, like TB’s Duplicator, stores the order that the constraints are created in. Every time constraints are added to the clipboard, the array is re-sorted (wholly re-sorted via Lua, rather than added using a binary search).
Photocopy does not use a hash map to store the list of constraints. However, it still saves the constraint creation times to file, wasting some space. You may want to fix this.
On another note, here are some feature ideas that I wanted to implement if I had made an STool:
[li]Client-side duplicator file viewing (as a constraint graph and list of entities and constraints)[/li][li]Client-side duplicator editing (to an extent)[/li][li]Detailed duplication restriction on the server (Adv. Duplicator has a number of unfixed exploits) – Photocopy already has some hooks to allow this[/li][li]Controllable duplicator file pasting, allowing you to paste only certain entities/constraints with very fine controls – The hooks were added for this too[/li][li]Bundled “RP profile” to allow relatively safe use of a duplicator on RP servers[/li][li]Paste position restrictions (for RP servers mostly)[/li][li]Much better file transfer / management dialog[/li][li]Storing original constraint positions like TB’s Duplicator[/li][/ul]