Photocopy - Duplication / saving library w/ adv. dupe support

This is a contraption duplication library that I wrote some time ago and did not finish entirely. As it looks like that I will not be finishing it, I’d like to post my progress. It may be useful to some of you if you want to procedurally save things and put it into a file. A sandbox server I ran had a “quick save” feature but it was a big hack on Adv. Dupe, so I was hoping to eventually replace it with this.

It is currently only a library (for programmers). There is no STool. (Though you are free to make one.)

Features:
[ul]
[li]Reads Adv. Duplicator files[/li][li]Writes in its own format that has some advantages over Adv. Dupe’s[/li][li]Written to reduce choking the server for entities that maintain a lot of data[/li][li]Serializes at a fixed rate, measured as number of individual items serialized/sec[/li][li]Stores constraint creation times to prevent saggy constraints[/li][li]I’ve run Photocopy through all my adv. dupes to make sure that it correctly serializes everything[/li][/ul]

Advantages over Adv. Dupe:
[ul]
[li]Writes in its own format that serializes new line characters correctly[/li][li]Photocopy’s files are 20-40% smaller than Adv. Duplicator’s files[/li][li]Does not create saggy constraints like Adv. Dupe[/li][li]Library is very reusable and is not tied to the tool[/li][li]Will not choke the server when serializing large tables like Adv. Dupe[/li][li]May save facial expressions, unlike Adv. Dupe (this may longer be true as I might have broken that part of the code)[/li][/ul]

Advantages over TB’s Duplicator:
[ul]
[li]Will not permanently freeze the server on badly parented props[/li][li]Written to not freeze the server on large tables[/li][li]Has more of Adv. Dupe’s legacy compatibility fixes[/li][li]Photocopy’s format and Adv. Dupe’s format create much smaller files than TB’s Duplicator as TB’s Duplicator does not pool strings[/li][/ul]

Here’s example code:
[lua]include(“photocopy/photocopy.lua”)

– Put some entities onto the clipboard
local headEnt = Entity(64)
local clipboard = photocopy.Clipboard(headEnt:GetPos())
clipboard:Copy(headEnt)
clipboard:Copy(Entity(24))

– Save it
local writer = photocopy.GetWriter(“PhotocopyDupe”)(clipboard)
writer:Complete()
local data = writer:GetOutput()

– Read Adv. Dupe file
local data = file.Read(“adv_duplicator/some_dupe.txt”)
local reader = photocopy.GetReader(“AdvDupe”)(data)
reader:Complete()

– Spawn it
local clipboard = reader:GetClipboard()
local paster = photocopy.Paster(clipboard, ply, Vector(0, 0, 300), Angle(0, 0, 0))
paster:SetSpawnFrozen(true)
paster:Start(function()
ply:PrintMessage(HUD_PRINTTALK, “Paste done!”)
end)
[/lua]

What I didn’t finish:
[ul]
[li]Not store constraint creation time in the file (redundant, as array-type tables maintain order)[/li][li]Going through the code and re-organizing it[/li][li]Ghosting functions[/li][/ul]

You will want to read this, as a programmer:
http://www.facepunch.com/showpost.php?p=25232445&postcount=6

You can find the code here:

I have not touched this for a few months.

Absolutely awesome! Thanks.

I partly read through it and it is awesome! :smiley:

Nice work, if you really have managed to eliminate saggy constraints then I’ll take this over adv dupe if someone wants to make this a STool, although you might have trouble gaining ground over adv dupe because of its sheer popularity. Good luck though!

[editline]12:47AM[/editline]

Tip to anyone planning to make a STool: please add a “save to client” option, and make deleting old photocopies easier than adv dupe, damn it takes a while.

Photocopy does read Adv. Dupe files, which should make adopting it much easier if someone does make a STool.

I don’t play Garry’s Mod anymore and I already long abandoned this project, I am going to leave some technical explanations in case someone asks me in the future when I will have likely forgotten entirely.

I have got wind that someone else was going to make their own duplicator as well (calling my duplicator library “flawed”). Surely duplicating my work would be a waste of effort and time, especially if you are going to avoid what I’ve done and end up with something inherently inferior.

Process Chunking

First off, all four operations (copying, pasting, saving, loading) chunk themselves into a number of smaller “tasks” that are expected to finish fairly quickly. Provided that the tasks are evenly-sized, this allows for a very controlled and predictable run. It is the reason that I am able to say that Photocopy serializes at such a specific and exact rate and it is the reason that I am able to reduce the amount of server freezing instances caused by a duplicator. However, the implementation of this was made immeasurably complicated by the fact that Lua co-routines do not work server-side. Thus, in lieu of being able to suddenly yield and maintain stack state, I had to manually re-implement some of that functionality at at the cost of greatly complicating the code.

The overview of how this works is as follows. A set of functions that make up the process are queued in line for execution. However, this line is never generated in full any time. Rather, the first function passes control to the next function that should run, followed by that function passing control to the next function, continuing until the process is finished. To pass control is to store a reference to the next function to be called – not to actually call it. Advancing to the next function actually requires calling an appropriate method from outside the operation’s code. This could occur in a Think hook, for example. Thus, the operation can be split over an indefinite period of time because it is expected that no single function should a long time to finish and will depend on time.

If you look through the code for the operations, you will see self:SetNext() calls spread throughout the code. The first argument is the amount of time to wait before the next function is called. Rather than splitting tasks into even further smaller sub-tasks, I added delays between larger tasks as another approach to the chunking problem. Chunking those larger operations would complicate the code even further and that would be the reason for it.

However, there is one task that cannot be split into smaller sub-tasks and delaying doesn’t really help: copying the table. A copy of the entity tables is required in order to maintain a consistent state of an entity. Splitting this task over several ticks would seriously risk the integrity of the copy.

Adv. Duplicator and TB’s duplicator do indeed do some chunking on their own. However, they do not do as a fine of a chunking as Photocopy. I have frozen servers for at least 10 seconds at a time with Adv. Duplicator so I feel that it is an important problem to tackle. Of course, once again, one limitation still is the table.Copy (a deep table copy written in pure-Lua).

Format

The Photocopy is a part-binary part-text file format. I chose to implement some structures as partial-binary to reduce file size and ease parsing. However, as you may be aware, it is not possible to write or read null bytes to and from files and so I needed a workaround. The part-binary structures that I did use are essentially 4-byte integers; however, one major difference is that 0 will never be read or written. You can find the implementation code in the photocopy_dupe.lua file.

That aside, the format is as follows. The header consists of the bytes:

0x89 P C O P Y 0x1A 0x0D 0x0A

(PCOPY is in ASCII: 0x50 0x43 0x4F 0x50 0x59)

0x1A is so that you can use “type” in MS-DOS on the file (as similar with PNG). The following characters are for detection of new line corruption caused by bad file transfers (such as over FTP), which the .txt files Gmod write are particularly susceptible to. Unfortunately 0x1A should really go after the newline/carriage return characters as the version byte will come after, making detection difficult. If you use Photocopy, you should fix this and also mention it.

What comes after is the version. This is one byte.

Everything after this point is stored as a “chunk.” Chunks consist of a 4-byte ASCII string that indicate the type of chunk, followed by the length of the chunk written as the 4-byte “integer” described above, and then followed by the chunk body. Because chunks are completely independent and store their length in their chunk headers, you do not have to worry about escaping the data. Photocopy files can contain up to as many chunks as the 4-byte “integer” can handle, though you are obvious limited by other factors. The order of chunks does not matter.

Photocopy uses these four chunks:
[ul]
[li]info - Stores information about the file[/li][li]ents - Stores entity information[/li][li]cons - Stores constraint information[/li][li]strs - Stores strings pooled from the entity and constraint data structures[/li][/ul]

The format of those chunks vary. You can read the code to figure out how they are stored as they are pretty simple.

Fixing Saggy Constraints

The physics engine in the Source engine places a high emphasis on constraint order, as with many physics engines. The reason TB’s duplicator does not have saggy constraints, for the most part, is because it stores the order in which the constraints were created. However, the lack of this feature in other duplicators alone does not explain the severity of the saggy constraint problems in them. The sag created by Adv. Duplicator and Gmod’s duplicator is especially bad because both (Garry’s duplicator library being the original) use a hash map when tabulating the list of constraints. If this alone is fixed, a lot of saggy constraint problems are fixed; however, due to the issues of cross-constraints, it is not a complete solution. Photocopy, like TB’s Duplicator, stores the order that the constraints are created in. Every time constraints are added to the clipboard, the array is re-sorted (wholly re-sorted via Lua, rather than added using a binary search).

Photocopy does not use a hash map to store the list of constraints. However, it still saves the constraint creation times to file, wasting some space. You may want to fix this.

Features

On another note, here are some feature ideas that I wanted to implement if I had made an STool:

[ul]
[li]Client-side duplicator file viewing (as a constraint graph and list of entities and constraints)[/li][li]Client-side duplicator editing (to an extent)[/li][li]Detailed duplication restriction on the server (Adv. Duplicator has a number of unfixed exploits) – Photocopy already has some hooks to allow this[/li][li]Controllable duplicator file pasting, allowing you to paste only certain entities/constraints with very fine controls – The hooks were added for this too[/li][li]Bundled “RP profile” to allow relatively safe use of a duplicator on RP servers[/li][li]Paste position restrictions (for RP servers mostly)[/li][li]Much better file transfer / management dialog[/li][li]Storing original constraint positions like TB’s Duplicator[/li][/ul]

ill make a stool for it later today then

It’ll be good to see this completed, we quite badly need a decent adv dupe replacement since old adv dupe is riddled with bugs to this day.

adv dup isnt buggy at all minus the fact that it duplicates constraints poorly.

Well that is a bug then now isn’t it?

And a rather important one at that.

Since it still hasn’t been fixed and it’s a big problem I bet it’s some huge flaw in the code that would require a lot of rewriting which no one has obviously not bothered with yet.

it seems pretty cool, if only someone would be willing to fix it up a bit