Slate - Backend STALKER Framework

Hello, would just like to share something I’ve been working on in anticipation of Sandbox. I’m making this as a backend element to a STALKER-based gamemode I’m planning for Sandbox.

SLATE

Slate is a Rust-based application meant to allow a dedicated Sandbox server to offload/coordinate certain tasks with a remote server. Slate sits in front of a MongoDB instance to handle persistent data storage.

Why?

A few reasons:

  • For now, there doesn’t appear to be a native database connector in Sandbox. I’m fine with that- It allows you to offload menial data tasks to a specialized application while letting the Sandbox server handle more important, user-facing stuff. You’re more likely going to saturate CPU usage before you start saturating a network link.
  • Big picture, I’d like to use this to allow for cross-server activities. In other words, multiple separate Sandbox servers using the same database, but also able to communicate between themselves.
  • I don’t have Sandbox access, and I’ve been meaning to learn Rust, so this seemed like a good learning experience.
  • MongoDB is another technology I’ve heard of but never used, and I was drawn to a no-schema database. I spent a lot of time in my Garry’s Mod days writing code to try and automate/abstract away MySQL schema management and it was pretty painful. I’m pretty happy with how MongoDB works in this instance.

What?

Rust runs on a remote Linux server and allows for Sandbox servers to establish TCP connections with it. It can handle multiple simultaneous connections. Sandbox servers send payloads to the Slate server, which parses them and either deals with them internally or broadcasts back out to all connected clients.

An example request from a Sandbox server to Slate might look like:

{
    "dispatcher": "database",
    "payload": {
        "operation": "read",
        "collection": "characters",
        "data": {
            "steam_id": "STEAM_0:1:00000000"
        }
    }
}

This would send the ‘payload’ to the ‘database’ dispatcher. The inner payload is further parsed by the dispatcher that receives it. In this case it would return a JSON object containing the entries for all documents in the ‘characters’ collection that match the given Steam ID.

Slate is built in a modular way such that there can be multiple different dispatchers available for clients to send messages to. The ‘database’ dispatcher is the only one right now, but I’m planning on a few more.

Okay… And?

I have some ideas that I’d like to implement for a STALKER-based gamemode by leveraging Slate.

  • C# properties to denote certain classes as integrated into the Slate database. When a marked object is instantiated, updated, or deleted, it would push their states to the Slate database automagically.
  • Multiple dedicated servers for different parts of the Zone. The multi-server support will enable people to use the same characters and travel across the Zone (server-to-server).
  • A Slate dispatcher for simulating flora/fauna population in different areas of the Zone. Based off of some simple Lotka-Volterra equations and with some static configuration values per area, Slate would regularly update servers with spawn rates for NPC mutants.
  • Same idea as NPC mutant spawn rates, but for in-game factions and their relations/movements across the Zone.
  • Global PDA network. Players with a working PDA item could post alerts/adverts that could be seen real-time across multiple servers.

Does it work?

I whipped up a quick Python script to demonstrate how a Sandbox server would be communicating with the Slate server. Here I’m just connecting to the Slate server (local to the machine) and demonstrating some basic CRUD (create, read, update, delete) operations against the database. Left is Slate, right is Python.
(It’s called ‘Rebar’ here, going through some rebranding lol. Excess logging just for debug purposes)
(Open image in a new tab for full size)

So far I’ve identified the following pros and cons.

Pros:
  • Pretty dang fast.
  • No-schema database means you don’t have to worry about columns matching up.
  • JSON is a familiar format, easy to work with and serialize/deserialize.
  • World is your oyster. (You can do most MongoDB operations through the JSON interface)
Cons:
  • Added system complexity (something has to run Slate).
  • World is your oyster. (You’re responsible for enforcing any rules on your data/operations. Slate is just a conduit, it doesn’t inspect your queries)

Roadmap

Basic Slate framework (TCP interface, dispatcher system)
Basic authentication (IP whitelist, connection password)
Basic MongoDB integration (CRUD)
Basic broadcast capability
General cleanup
More niche database operations (indices, collection management)
More security options (? May not really be necessary. Seems excessive)
Zone fauna simulator dispatcher
Zone faction simulator dispatcher
Sandbox implementation

I’ll probably end up open-sourcing it once I can get my hands on Sandbox and round out that side of the implementation.

2 Likes

Neat. How do you plan on doing communication between different S&box servers?

1 Like

It would just be a matter of crafting the right Slate message. It would work something like this:

The origin server would send something like this:

{
    "is_broadcast": true,
    "payload": {
        ... whatever you want ... 
    }
}

When Slate receives the message and tries to figure out what dispatcher it’s meant for, it sees ‘broadcast’ and then immediately spits it back out to all other connected clients. This would work well for the PDA broadcast message I mentioned in the OP.

A more complicated example may look something like this:

{
    "dispatcher": "dispatcher_a",
    "is_broadcast": true,
    "payload": { ... }
}

This would be something like a database update, whose result is relevant to all connected servers. Or maybe you want to save your PDA broadcasts into the database before sending them out.

At the end of the day, Slate just behaves like a router between a bunch of different endpoints. It’s up to the Sandbox servers to figure out what they want to send and when, and what to do with incoming messages that they may have not been expecting.

I can’t imagine a use case for non-broadcast, direct server-to-server messages. Maybe for cross-server PMs? Even then, Slate has no idea what clients are connected to the Sandbox servers its serving, so it would probably have to be a broadcast message regardless.

EDIT:
Here is a .gif of the broadcasting in action. Left is Slate, top-right is broadcaster and bottom-right is broadcast receiver.

2 Likes

Cool that’s pretty flexible. I think instead of having an is_broadcast option the dispatcher should be able to decide where it should respond to (“dispatcher”: “broadcast”).

For knowing which players are connected to what Sandbox server there could be some authentication dispatcher which maintains that table. Might be good to have direct server-to-server messages to reduce load instead of broadcasting all the time.

Have you thought about what could be done to scale a system like this out? Or how to make everything more fault tolerant in case a Slate server goes offline, or a Sandbox server goes offline?

1 Like

Yeah, doing PMs could be its own dispatcher that regularly queries/gets updated by the Sandbox server(s) when player count changes. It could keep track of that internal to the dispatcher as well.

As for the last part, that’s a good question. Frankly I didn’t put too much thought into reliability since I was just going to use it for my individual use case. However, I think the technology stack already lends itself to fault-tolerance pretty well.

The MongoDB connection that any given Slate server maintains is just the standard mongodb://host:port protocol. Right now its pointing internally (mongodb://127.0.0.1:port) but it could easily be moved to another remote server without any additional work, just increased latency.

That would then enable multiple Slate servers for redundancy. They all point at the remote MongoDB instance, and maybe have a list of ‘peer’ Slate servers in a config file that they attempt to establish connections to on startup. Would probably look something like this:
(Red is no connection possible, blue is connect request sent, green is connection established)

Still should work relatively similarly, only broadcast messages would also have to be relayed across all active peer links. Sandbox servers would probably maintain the same list of Slate servers and choose one at random to connect to, and attempt to connect to any of the others if the connection ever goes down. Would have to think about how more specific messages get shared, the peers would probably actively relay information about their clients to each other, kind of like an STP protocol of sorts. That would be cool to do, but pretty far out right now.

If a Sandbox server goes offline, Slate doesn’t really care. It simply manages the connections while they’re alive and tosses them out when they die. Here’s me creating and killing the connection a bunch and Slate just keeps doing its thing.

I’d like to get the C# side of things rounded out before I go overboard with scalability, if I can’t even make TCP sockets server-side in Sandbox this idea kind of dies lol.

1 Like

Plain TCP sockets are not available and probably wont be. WebSockets are available right now though which shouldn’t be a huge shift from what you have now.

1 Like

Interesting, luckily there is a Websocket Crate that looks like it’s a drop-in replacement to the TcpListener/TcpStream with TLS support. That would solve any security concerns for the most part. I’m currently rewriting this with the correct naming and error handling so I’ll do that conversion while I’m at it. Thanks for the heads up!

1 Like