Client-Server Communication programming

[Frable]

As the title states I am starting to venture into client-server communication programming in research for my thesis and would love to hear your literature and example suggestions. Any advice on how to approach this are welcome as well.
So far I have mainly worked only on GUI/Algorithm/graphic pipeline implementation C++/Qt/VTK.

Final goal is to implement a real time communication between a server and client. I guess looking at video conference program samples might be a good approach here?

Greetz,
Frable

[AtomiCasd]

There are lots of network APIs out there. You can select for example ENet, SDL_Net or SFML.
Networking isn't that hard in theory, it's how you implement it that is the hard part.
You're basically just sending a stream of bytes to eachother so a simple way to figure out which packet is being sent to you is to read the first byte as the ID of the packet, this can be a type stored in an enum like PACKET_LOGIN, and then typecast it to the correct struct.

Simple login packet sent from client to server.

struct LoginPacket
{
    char id;
    char username[20];
    char password[20];

};

LoginPacket login = new LoginPacket();
login->id = PACKET_LOGIN;
login->username = "AtomiC";
login->password = "123";
net->send(login,sizeof(login));

[Deco Da Man]

There are many "layers" of networking, ranging from low-level physical signals to high-level RPC interfaces.
Take a look at the [url=http://en.wikipedia.org/wiki/OSI_model]OSI model[/url].

The physical and data link layers are quite low-level and reside in hardware, operating system internals, and drivers. These involve things like the frequencies of digital signals over Ethernet, addressing of individual computers, and traffic routing decisions.

The network layer is about getting the information from one place to another. It takes the source data, fragments it, routes it, reassembles it, and reports any delivery problems. (Fun fact: many routers have silly default network layer settings which result in over fragmentation and poor connection speeds!)

The transport and session layers are the most familiar to people.
The transport layer provides an easy way to send data without having to go into specific connection details; you only need to provide how you want to send it (reliably or unreliably), what you want to send, and where you want to send it (including the port). It handles the rest!
TCP provides reliable connection-based messaging. This means that the order and content of data is usually guaranteed (often forced) and that packet 49 will not be readable until packet 48 is received intact. Also, each end has a rough idea of the status of the other end and notifies of disconnection.
UDP provides unreliable connection-less messaging. Delivery order and data is not guaranteed. Each end is totally unaware if the other end has received it intact. This is commonly used in games, as by the time packet 49 is received, it is irrelevant to the current game state!
(Minecraft uses TCP, which is why a poor connection can result in jittering.)
SCTP provides both options, as well as a lot of other features and optimisation. However, it is still being "worn in"; most routers don't even understand the protocol.

Unless you've been asked to make your own transport protocol (or lower!) you'll probably be working on the session layer, which consists of sending and receiving packets of bytes over UDP or streams of bytes over TCP.

A: Hello.
B: Hello. User: Bob. Password: 12345.
A: Accepted.
B: List online users.
A: 2 users online. Bob, Jane.
B: Bye.
A: Bye.

That's one of the simplest protocol you can have. In fact, nearly all protocols work of this idea: "command:data; command:data; etc". HTTP, Source networking protocol, RMTP, Gobby collab protocol, etc.
Now, they are not usually "text" protocols. Instead, you define something like the following:
(DISCLAIMER: I'VE NEVER DONE THIS FOR C; THIS IS PROBABLY BAD CODE.)

struct netcomm {
    short command;
    union {
        struct command_1_Hello {
            long protocol_version;
            int secret;
            int timeout;
        };
        struct command_2_ListUsers {
            int filter;
            long etc;
        };
        ...
    }
}

You would simply serialise that struct to a string (flattening should work, unless there are pointers), and then read X bytes on the other end and cast it to a struct of the same type.
If `command` is 1, read a 'Hello' command, and so on!

The easiest thing to start with is a chat program. From there, you can extend the protocol to include formatting, pings, inquires, etc.
This is all the presentation and application layers.
The presentation layer is the raw data you send: "H E L L O ." or "0x01 0xFF 0x00" (same meaning, different representation).
The application layer is what the data means: "HELLO" means run the 'Hello' command, whereas "PING 123" means ping someone called 123, etc.

Every layer is incredibly complex when it comes to advanced serialisation, transformation (encryption), and optimisation. And don't forget that changes on one layer may benefit another!
For example, using UDP means you don't have to randomly insert "updates" into the TCP byte stream.
("This is a message that is very very HALT... player 1 moved ...RESUME very very long".)
On the other hand, using UDP means you don't have a single byte stream! Instead, you handle each packet (which is limited to about 60000 bytes). So then you have to manually reassemble them to construct the original data.

If you were to add video abilities to your chat program, you would have to add the following commands at least:
"VIDEO INIT" (id, image size)
"VIDEO FRAME" (id, raw image data)
"VIDEO HALT" (id)

Things to consider:
- Should you send the whole frame or just the changes?
- What if they miss a change? Maybe you should use an ordered stream.
- What if their internet is too slow for an ordered stream? Should I decrease the resolution, or send updates less often?
- What is the best way to compress the image data? Lossy (JPEG)? Loseless (GZIP)?
- How can I secure the connection? (look up TLS/SSL)
- How can I ensure video updates do not slow down chat updates? (UDP/SCTP/ENet channels, or an entirely new connection)

It's... a very complex field!
If you're looking for a library that's really nice to use, take a glance at ENet.
It "sits" between the session and presentation layers and provides many features: connections, reliable or unreliable messages, packet optimisation, etc.
Unfortunately, it leaves security up to you though it's pretty easy to use existing handshake/encryption libraries.

There are plenty of tutorials out there on the topic of networking... Google for them!

[b]Edit:[/b]
Oh, I forgot about connection topology! (NOT the same as network topology, which is the physical connection layout of your local computers).
A centralised system (what you want) is when multiple clients connect to a central server (like Source games). Here's an example with some different optimisations:

Client A (v1) connects to Server.
Client B (v2) connects to Server.
Client C (v3) connects to Server.
Client A sends a message to Server; Server handles it and sends any new changes to Clients A, B and C.
Client B sends a message to Server and PREDICTS the outcome (if possible); Server handles it and sends a confirmation to Client B and any new changes to Clients A and C.
Client C predicts what Client A and Client B are doing; Server sends periodic updates to inform Client C what is actually going on.

A peer-to-peer system is when the clients connect to eachother. This sort of system is VERY complex, so start with a centralised system!

[Frable]

@AtomiCasd: Thanks for the API suggestions. Having a look at them now.

@Deco Da Man: Huge thanks for the break down.

I will definitely start in the session layer to get a comfortable feel of this area and have a skeleton program ready to start testing. Optimizing and venturing down the layers is way in the future for now.

For the first big milestone I won't dive into connection topology as I want a client-server relationship but am focusing on only one connection for now. Client sends tracked interactions and gets a picture back that depends on the interactions. Later on it could be something like a MPEG-4 stream comming from the server. As I can imagine the frame and motion vector structure of it could be a very effective compression for my visualisations.
Based on the option of slow internet or client's computational power I thought about trying a double pipeline. So in general you work on a "decimated" model that satisfies the network and hardware restrictions given and only when no interactions are present from the client you send additional information from the original model to the cient to increase the LOD. Well or any other kind of progressive rendering. Either way, as you see it's all just ideas for now. Pretty sure some of my ideas are just fancies born out of stupidity.

What I tried up till now was a bit of rereading about the layers and various protocols (studying electrical engineering) but it's still very theoretical for me and I guess I'll get a better Idea what I need and what knowledge I lack when I get hands on. For now each of the various protocols seem to have their pros and cons when I brainstorm implementations. But have the feeling my "judgment" might be off due to lack of deeper knowledge.
More or less the only hands on experience right now is a simple chat program with Winsocks I did for a general sockets understanding.
Thanks to your post I realise that for now I need to pick up one or two libraries and tinker around. Stack up basic knowledge and hands on exp. . After that it should be easier to judge what I should focus on and decide which parts in my implementation will be essential and need extra consideration.
Will start with ENet and see where it takes me.

Thanks again for the input!