A few questions on network programming

#1 Could you explain your reasoning for using UDP a bit more? MMORPG's aren't traditionally 'fast paced' enough to warrent it really. I went through this loop myself and eventually decided for a non-immediate environment TCP just removed so much hassle.

This eventually evolved into using both a TCP and a UDP connection. So the core server messages (that weren't particulary time critical) were TCP'd and all the other fluff I really didn't mind losing some of went via UDP.

In my mind it's far better to use a single port and loop through your connections in a single thread than use multiple ones.

If might also be worth looking at a suitable framework like Twisted or browsing the solution that ActionRPG.com have come up with. Depends how much you enjoy programming vs getting the game done ;)

Good luck.

#2 I'll chime in, as I know a bit about network programming, but I'm not really a MMO guy.

If most of your messages require acknowledgement, I'd recommend going back to TCP. My feeling here is you're trying to implement TCP features on top of UDP, which is a losing proposition. Besides, I know a couple of the big name MMORPGS use TCP communications these days.

If you're mostly not sending guaranteed delivery stuff, or if you don't care about message ordering, stick with UDP.

If you want to go this way and need some guaranteed delivery, consider researching a 'sliding window' method. What you've got here will stall sending until acks come back.

I'd recommend you stick with a small set of ports for your UDP traffic. It's not really necessary to give every client a port, and that smacks of TCP over UDP again.

I'd recommend _NOT_ giving a thread to every client. If you're expecting MMORPG like load, the context switching is going to kill you. If you want to take advantage of multiple processor servers, though, consider partitioning your server along whatever lines make sense to you the most. Seperating communications and game logic wouldn't be a bad idea, since you won't be blocking on network sends so much.

You shouldn't need very heavy encryption, so you can probably get away with a diffie-helman type key exchange when your client is initializing, then a message by message DES crypt, say. Don't spend too much time or CPU on this. You're not coding for a bank, and you're dealing with an untrusted client anyway, so it really doesn't buy you very much. You may end up cutting crypto back to just log in information, depending on wether you end up needing to cut CPU usage.

#3 Just thought I'd chime in expand on the client issue a bit:

The client is not to be trusted period, no exceptions. If you can truely accept that then there's no need to excrypt your datastream. If you think of your client as nothing more than a dumb terminal and only send it the information it NEEDS in order to present the world to the user it can really free up your bandwidth / CPU on the server. Don't send the client every object in the simulation, send it information about things that are close to / can effect it.

#4 Donovan, there is absolutely reason to encrypt the data stream.

1) IP sniffers looking for login info

2) Man in the middle attacks.

3) (Since we're talking UDP here) 3rd party command spoofing.

#5 First of all, thanks for your speedy replys...
Secondly, I think I should clarify my game idea: it's a space simulator (very similar to Elite by David Braben/Iain Bell), but multiplayer... That makes the game easy to partition (a process per star system, or cluster of star systems, maybe, since they're self contained)...

Now, for more specific answers:

Gareth:

Quote:
Could you explain your reasoning for using UDP a bit more? MMORPG's aren't traditionally 'fast paced' enough to warrent it really. I went through this loop myself and eventually decided for a non-immediate environment TCP just removed so much hassle.

Well, I now standart (fantasy) MMORPGs aren't very fast paced, but my reasoning is that a space sim is... Of course, I think that prediction is also easier in that case, so I'm not really sure now if UDP is the way to go...

Quote:
If might also be worth looking at a suitable framework like Twisted or browsing the solution that ActionRPG.com have come up with. Depends how much you enjoy programming vs getting the game done ;)

I would prefer to get the game finished, really... I know my idea isn't so original that I'm going to make money out of it or win any prizes, it's more like a learning experience, and a way for me to try some stuff I'd like to see in a game like this (like you could control just a turret and a friend of yours a pilot in the same ship)...

Mark:

Quote:
If most of your messages require acknowledgement, I'd recommend going back to TCP. My feeling here is you're trying to implement TCP features on top of UDP, which is a losing proposition. Besides, I know a couple of the big name MMORPGS use TCP communications these days.

Well, I think the only messages that require acknowledgement are the login process messages, and the "entering new star system" messages... And an occasional ping or so... :)

Quote:
I'd recommend _NOT_ giving a thread to every client. If you're expecting MMORPG like load, the context switching is going to kill you. If you want to take advantage of multiple processor servers, though, consider partitioning your server along whatever lines make sense to you the most.

That was already in my plans, partioning the server in star systems or clusters of star systems (using a master server so that the client knows where to connect).

Quote:
Seperating communications and game logic wouldn't be a bad idea, since you won't be blocking on network sends so much.

Could you elaborate on this?

Mark, J.:

Quote:
You shouldn't need very heavy encryption, so you can probably get away with a diffie-helman type key exchange when your client is initializing, then a message by message DES crypt, say. Don't spend too much time or CPU on this. You're not coding for a bank, and you're dealing with an untrusted client anyway, so it really doesn't buy you very much. You may end up cutting crypto back to just log in information, depending on wether you end up needing to cut CPU usage.

and

Quote:
Donovan, there is absolutely reason to encrypt the data stream.

1) IP sniffers looking for login info

2) Man in the middle attacks.

3) (Since we're talking UDP here) 3rd party command spoofing.

I think the need for encryption is very great, specially because of the IP sniffers and man in the middle attacks (I heard about this in my OS classes)... Of course, I'm very worried about the performance issues, so I'll probably just split the messages packets in packets with encription and packets without (some messages aren't critical!)...

Thanks again for all your input! Keep it comming!
Now I have to decide if I should use UDP or TCP... I'm more inclined to TCP again, since I already programmed lots of stuff with that...

Diogo de Andrade

#6 Excellent that's a lot clearer. I've recently been doing the networking code for my space sim and 99% of the work is in the prediction. Infact even with updates as low as once a second it's really not bad at all. If you're using large slow moving capital ships then this would work even better. It's more interesting when you're tracking a lot of projectiles.. but if you keep weapon fire rates down then that's not bad ;)

UDP is nice as it's more 'raw' than TCP and allows you to deal with 'missing' data in a better way. But as mentioned earlier if you segment the network system and the game logic it's a choice (upgrade?) you can make later. Use TCP and if that causes problems change it to UDP on a required basis but I don't think it'll be a problem in your case.

I'd still be mighty tempted if I was trying for a MMORPG to not do any of it and try and leverage some other open source work. But i'm just lazy... :)

#7 You will definitely want to use UDP for the bulk of traffic in a space sim. I also use HTTP over TCP for lobby and admin functions.

You will find that almost every type of UDP message in the game really does need an acknowledgement. You need to ack the packet whenever it is important that the clients and server agree on something. Joins and quits obviously need agreement. But so do warp transitions, damage, communications, any weapon launch where ammo is expended, decoys, probes, etc. etc.

About the only things that don't need ack packets are position updates and laser shots that don't use up ammo. For packets that don't need acks (I call these "unreliable" packets) you will want a timestamp on each message. Keep track of the most recent timestamp you have received and remember to discard "unreliable" packets that arrive out-of-order with older timestamps.

Regarding encryption - if it really is a persistent MMO game with user accounts, then yes, you will want to encrypt sensitive packets. But if it's a more conventional multiplayer game with no persistent data on the server, then encryption probably isn't that critical. If there is no information to protect, why protect it?

--milo
www.starshatter.com

#8 Hum... The plot thickens... I'm still a bit undecided what to use, but I think I'll make a new OO approach that enables to switch from one to another with ease...

I think I'm going to create two classes: Server and Client... Each can be implemented with it's own mechanisms (TCP/UDP)... and I'll use that class as a RPC class... What do you think?

#9 Well... Here's the catalog of classes I use in my networking layer:

NetLayer - wraps winsock setup / shutdown
NetAddr  - encapsulates IP address
NetHost  - used to lookup addresses by host name or dotted quad
NetSock  - wrapper for a socket, either datagram or stream
NetGram  - UDP datagram
NetMsg   - application-level message, several go in each datagram
NetPeer  - one side of a UDP connection
NetLink  - used to send and receive messages to all connected peers
.
NetClient  - stream-oriented client
HttpClient - specializes NetClient to add HTTP support
.
NetServer  - network message pump for request/response server
HttpServer - specializes NetServer to add HTTP support
.
HttpServlet     - generates HTTP response to a request
HttpServletExec - calls the appropriate servlet based on URL
HttpParam       - name/value pair from request
HttpRequest     - parses GET/POST request into params and cookies
HttpResponse    - assembles response

Per-packet overhead is much higher for UDP than it is for TCP. You generally cannot afford to place each application level message in its own packet. So the application sends NetMsgs to NetPeers. When the NetPeer has enough messages saved up to send, it composes a NetGram containing all of them and sends them in a single network operation.

Notice that the client and server classes are for stream oriented request/response stuff. The UDP protocol stuff does not really distinguish between client and server at the network layer. Clients send messages to servers that must be acknowledged and processed, and servers send messages to clients in the same manner. The distinction between client and server happens at the next layer up, the application layer.

NetGame        - manages a NetLink and NetPlayers
NetGameClient  - specializes NetGame for client stuff
NetGameServer  - specializes NetGame for server stuff

The NetGameServer class is responsible for informing clients of game state, and distributing messages between clients. For client A to chat with client B, a message must be sent from client A to the server, and then resent from the server down to client B.

Also note that there is also a distinction between a NetPeer and a NetPlayer. The former represents a computer on the network, while the latter represents an agent in the game world (either a human player or a remote NPC).

Now, you're probably not going to need all of that stuff, especially the HTTP support. But it might give you some ideas for abstractions you may have missed.

--milo
www.starshatter.com

#10 > Per-packet overhead is much higher for UDP than it is for TCP.

What overhead? I'm assuming size here...

> Now, you're probably not going to need all of that stuff,
> especially the HTTP support. But it might give you some ideas
> for abstractions you may have missed.

I'm at a loss why do I need HTTP support... Is there something I'm missing?

#11 I just wanted to add that I bought my Torque license primarly to get a look at the Network layer after reading the White paper about how it is done.

So at least go and READ the White Paper about their method of handling the network code, I do alot of distrbutited and parallel programing for a living and think that it is still probably the slickest most well designed game network code on the market.

Nothing is off topic just because it is not Torque related Oh yeah, and you suck you moron ;)

#12 I'm finding this to be a very interesting discussion. Thanks for making it so.......

#13 One advantage to UDP that I don't think has been touched on yet is that it is connectionless - meaning there is no overhead related to the number of users that you do not yourself inflict.

Whenever a TCP/IP connection is made, the TCP/IP stack on your system has to allocate memory, keep track of state information, etc. - with a lot of clients connected at once, this can really add up, especially if your OS's implementation isn't fast and optimized.

Some OSes also have (relatively low) hardcoded limits of simultaneous connections, which means you have to recompile the kernel or apply a patch (or pay the vendor more money) to support more users.

UDP, on the other hand, lets you do your own connection handling... The beauty of this is that (if you do a good job) you control which packets are going to get effort spent to send, and which will not. For instance, if you're sending positional data, you don't care about where stuff was - but if you use TCP/IP you are potentially wasting packets making sure old data was sent, and worse, if newer packet is received while you're waiting for an older one, the OS won't give you the new data until the old one comes!

That means a single difficult packet can introduce significant lag to your game. This is fine if every packet has to be received in order (like a telnet or HTTP session), but for a game, where (most) packets are basically seperate entities, it's ludicrous.

In a real time situation, you want to only worry about the most recent information. So that means that for most packet tpes, you don't want to worry about resending missed packets. If a position data packet gets lost, who cares? A new one will come along pretty soon. If a weapon-firing packet disappears... that's a little worse, but it can be dealt with. You can either require those to be acknowledged, or just let the user live with it. The server is the authority in these matters anyway, so it's annoying but not unacceptable for a user to get hit by an unseen projectile.

(Note that when you're dealing with relatively rare projectiles, like a rocket launcher's projectile, it's much more important that it gets transmitted than, say, a burst of machine gun fire.)

On the other hand, you DO want stuff like chat to always be delivered. You can either open a seperate TCP/IP connection for that sort of thing, which isn't as real time but is important to get in order and accurately, or require every UDP packet you send the data over to be acknowledged.

Anyway... There are tradeoffs aplenty between UDP and TCP/IP, and many a detail to watch out for, but in essence:

UDP is fast and unreliable.

TCP/IP is slow and reliable.

And with UDP, you can be selectively reliable (if you are willing to put in a little work).

I also strongly suggest reading at least the Torque Networking white paper, if you don't have access to the Torque source. It's very instructional.

#14 > What overhead? I'm assuming size here...

Yes, exactly. Because TCP is a stream-oriented protocol, you don't send "packets" yourself. You just write bytes into the stream and the TCP protocol assembles them into packets automatically in a way that tries to minimize the ratio of packet header length to payload length.

UDP is a connectionless protocol, and each packet needs to have complete source and destination address information in the packet header. Something like 40 bytes per packet. This is invisible overhead that is added to your packets by the winsock library, so it isn't something you need to manage.

> I'm at a loss why do I need HTTP support...
> Is there something I'm missing?

As I said, you probably don't need HTTP support.

I use it for a variety of things - remote administration of the server via a standard web browser from anywhere on the Internet can be a handy thing to have. HTTP is also a convenient protocol to download large-ish datafiles. I use it to download distributable mod archive files so that all connecting players have the same set of game data, for example.

It is handy to have and easy to write, but I don't know that I would do it the same way again. The trouble with HTTP is that it is strictly request-response. There is no way to push data from the server to the client unless the client asks for it. This means that you need to poll the server every so often to get a state update (e.g. download all recent chat messages to a web browser). You end up needing lots more threads than you would otherwise because the client has to wait for the server to respond to each message.

--milo
www.starshatter.com

#15 UDP packet header is 8 bytes
TCP packet header is 20 bytes

They both based on IP header, which is another 20 bytes.

Destination address is part of IP header, that say, no matter you chose TCP or UDP, the destination address is always part of the every packet.

The ineffeciency comes when you try to emulate the QoS of TCP on UDP.

You could always use both TCP and UDP in your network code.

The differences on resource overhead between TCP and UDP is neligitble.

constantly polling on non-blocking socket status is evil and should be avoid at all cost.

1 or more thread for each socket is evil

Latence (Lag) is always there regardless what you use.

Choose the overall synchronisation architecture carefully, handle the client inconsistancy properly.

NEVER RUSH INTO CODE!

#16 Jarod:

Quote:
So at least go and READ the White Paper about their method of handling the network code, I do alot of distrbutited and parallel programing for a living and think that it is still probably the slickest most well designed game network code on the market.

Nothing is off topic just because it is not Torque related Oh yeah, and you suck you moron ;)

I will, thanks for the tip... And The "You suck" and "You are a moron" were exclusive! You can't call me both! :P

Mike:

Quote:
I'm finding this to be a very interesting discussion. Thanks for making it so.......

It's things like this that make up the community! Thanks for the support! :)

Ben: I thought of all those things and you've made my case better than I could! It's exactly that that I was thinking and couldn't put to words! Although I didn't thought of the diferent weapons needing or not the acknowledgement... that's a very good tip!

fred:

Quote:
NEVER RUSH INTO CODE!

That's great advice... I really could follow it more times, but I get excited... ;)

Anyway, thanks all for your help... I'll read the Torque networking whitepaper, but I'll probably stick to UDP (see Ben's explanation of why! ;) )... I need to code UDP support into one of my commercial projects, since I have to code SNMP support in it... :) Joining the usefull ($$$$) to the fun (games, games, games!)...

Thanks all! But keep that input comming! :)

#17 Hello,

Resurrecting an old thread here, but I had a few comments to add.

Encryption:

YES! Even a simple RNG-stream cypher with an overhead of ~5 assembly instructions per long is acceptable. The data is real-time, so your encryption only needs to slow an opponent for a minute or two to be valuable.

Login Security:

Don't use Diffie-Helman, it is weak against Man-in-the-Middle. You can perform an interlock-Diffie-Helman, but that's a LOT of code for something that doesn't need it.

Use this protocol instead:
- Server creates random cookie (128+ bits).
- Server sends cookie + date stamp to client.
- Client hashes (SHA-1, MD5, etc.) their password, the cookie and the timestamp.
- Client returns the User Name, hash, timestamp, and cookie to server.
- Server checks the hash against the user's password and timestamp.

This protocol happens to be immune to all forms of attacks (man in middle, replay, spoof, brute force), and it is also stateless. The server does not require keeping track of the cookie or the user. You can make it a little more secure by tossing in the client IP address, and by returning a truncated hash (64+ bits).

UDP vs TCP:

For any real-time game, you will want to use UDP for client updates. Lobby/Login are easier to write in TCP, but once you're in-game, just drop the connection and go with UDP.

For games, TCP is only useful for traversing firewalls. All real-time games on the market which use TCP are consistantly hurt by lag (Ultima Online is a classic example). UDP allows you to optimize dropped packets and replace them with fresh data instead of resending stale data.

Ports:

Use a *very* small port range. This allows firewall operators to open the fewest number of ports possible to allow your game through. You may even ride on top of "common" ports, although this goes against an unwritten rule of network programming.

Updates:

Only send the data a client can render. For instance, don't send the position of a monster until the client would actually render it. This avoids cheats (although it requires a lot more CPU time on the server side).

Send absolute values when possible ("you have 20 gold") instead of differences ("you gain 5 gold"), since lost packets will change the meaning of them.

Timestamp everything by the server's time. This allows clients to perform dead-reckoning and prediction.

Don't let things "build up" - if there's a long enough network spike, just take it like a man and drop the packets. It's easier on the server, easier on the client, and chances are, it won't change the outcome of most events. (ie: make your outgoing queue only so deep, and drop things off the front of the queue)

System Architecture/Software:

Real-time systems are the optimal programming architecture for network programming. Unfortunately they're also the most difficult to write. Basically the server has a single thread. Each itteration it reads all packets from the socket until a time limit, updates the state of the system, then sends out updates to all clients.

A simpler architecture is a message-passing system. A network thread reads packets and passes them to a handler thread. The handler thread processes each packet and updates the state of the system. A timer passes a message to the handler every update-tick, which causes the handler thread to send out updates to all clients.

System Architecture/Hardware:

Today, most massive game worlds are run by a layered system of machines. Here's an example deployment:

Internet->Fat Switch->Connection Servers->Fat Switch->Content Servers->Fat Switch->Database Servers.

The Connection Servers hold the client connections and perform the client update loop. The Fat Switch in front of them keeps the incomming connections load balanced across them (once connected, you remain connected to that connection server).

The next layer is the content servers, they handle receiving the updates from all the clients, modifying the world database, and passing the updated information back to the connection servers. Typically a content server serves a geographic area of the game and all object interactions within it.

Interestingly enough, this is also how massive Web sites deploy their resources.

For a indie game system of 100-300 people per server, you could probably get away with a fat internet connection (2.5-5 Mbs per zone) and a fast machine. Toss a few of these servers connected by a medium switch fabric, and you could have several "Zone Servers". Then you need an in-game representation of exiting one zone and entering the next.

Hope that helps!
--Bryan
bryan.turner@pobox.com

#18 There are a few things that spring to mind, havent read through every huge post for this, but here are a few reasons I'd choose UDP.

1) TCP under windows isnt controllable enough to be able to throttle data and control its importance to your app, in TCP you shove it in a stream, it gets buffered, it waits for a while, it gets ack'd etc. In udp, you can choose whats most important to send, and send it. At least under Winsock 1.1 youre stuffed if youre interested in priority ordering.

2) Making a guaranteed protocol on top of UDP is simple enough, and you can then tweak its capabilities to your hearts content.

3) UDP requires a LOT less maintenance on server code (no problems handing off threads to deal with incoming tcp connects etc). PS: Thread per connect dies! :)

I always read that UDP overhead was greater per packet, but that compression made up for it. Needless to say a properly constructed server comms protocol makes this a moot point (other games use it).

As for architecture, I'd suggest making a base class that is network serializable, then use that base class to serialize decendants onto the network and de-serialize. You can then build into the base class all sorts of cool stuff.

Think of it this way.. you write a base "Message" class..

the Message class can be written to a Network "Stream" classs with << operator.

At the other end, you have another stream class that reads messages with >> operator.

When the message at the other end is created, it first de-serializes the packet type id. From that it calls the message constructor which de-serializes the rest of the packet.

From there, you have a CLASS that has your data in it. It can do all sorts of cool stuff, for instance, it can register itself with a debug stream.

I remember David Weinstein of Red Storm writing a doc about this kind of architecutre, and I used it once for a game (in a slightly different form, but roughly similar) and it worked REALLY nicely.

Phil.

#19 "Hope this isn't offtopic, since it isn't related to Torque..."

Actually, being about torque would make it OFF topic - they've got their own area, though I often wonder why... ;)

Is there a 3rd party library for setting up socket communications ? I've used the amazing CURL for client-server stuff in the past but I want to add simple pier-pier networking to my mini-golf game and make it work over the net as well as a lan.

I've tried writing this stuff before and I can remember there being a lot of gotchas before you even get to making your own game network playable.