Amazon EC2 Torque Test Follow Up

DISCLAIMER: This is a personal project undertaken by me, David Wyand. It does not represent work being performed by GarageGames, nor is it an indication of any future product releases.

Introduction
Last week I wrote a blog describing how I set up a dedicated TGE 1.5.2 server using Amazon's EC2 service. I then went on to ask the community to help in stress testing the server. This blog gives the results of this testing.

Executive Summary
The Amazon EC2 virtual server hosting the TGE 1.5.2 dedicated game server remained operational for the testing period (and has been up for nearly a month), and the TGE server process itself ran without any issues. Unfortunately only 4 (!) people showed up for the stress test. The good news is that it appears the Amazon EC2 server had no problems with hosting an active FPS game for that number of players and provided an experience on par with a dedicated server.

Server Data Collection
To help collect game server statistics during the stress test, I set up a separate Amazon EC2 server with MySQL and an Apache web server to display the real time data. This was accessible through the forwarding address of www.torquetest.info for anyone to see.

Charts of the data collected during the stress testing hour

The data was collected on the game server and sent to the MySQL server every 10 seconds using a separate Python process. A diagram of this setup may be found on my previous blog. I'll talk a little about each of the charts and what they are showing.

Torque EC2 Network Traffic
This graph shows the transmitted and received bytes per second for the gaming server itself (the server's eth0 port). This is compared against the number of connected players as reported by the TGE server process (the $Server::PlayerCount global script variable). You may see the expected correlation between connected players and bandwidth used.

Bandwidth and player chart produced from the collected statistics in Microsoft Excel

During the testing hour we tried to keep fairly active. Constant firing of crossbow bolts and running around. You'll notice that around 18:25 a lowering of the bandwidth used. I believe during this time we stopped shooting at each and had a chat. Of course the truce didn't last long.

Are you feeling lucky, punk?

This ended up as the only server-side graph to produce any practical results.

Torque Server Packet Round Trip Time
The idea behind this chart is to display the minimum and maximum round trip (ping) packet time for all clients. I was hoping it would provide some indication of when a player could be lagging. Unfortunately it didn't work out as well as I hoped.

The main problem lies with the fact that if the TGE client is not the front window it introduces a Sleep() into the event loop. This causes it to process and acknowledge network packets at a much slower rate than normal. So if someone brings a web page window forward for example, then from the server's point of view that game client will begin to lag. This makes it impossible to differentiate between true client lag, and just an end user's normal use of their computer, using this method.

Torque Server Main Loop
This chart displays how each 10 second time slice is divided between the tasks of the server's main event loop as found in DemoGame::main()'s while(Game->isRunning()){} section. The thought behind this is that as the game server is loaded down, some of these tasks will take more than their fair share of the processing time. In the chart the greedy task's colour area would increase at the expense of the other tasks.

Unfortunately with the tiny number of players during the stress test time I was never able to test this hypothesis. As you may see in the chart above, the time allocated to each event loop task remained fairly constant throughout the hour.

Torque Server Tick Load
This one is similar to the main loop chart above but focuses on 'ticks'. Over any given 10 second period the number of ticks processed on the server should be fairly consistent. If for some reason a tick has been missed, then all queued ticks are processed at once (but in order) until the server catches up. This is done in ProcessList::advanceServerTime() where advanceObjects() is called. What I was attempting to determine here was when the server was slipping due to load and had to catch up.

Interestingly when you look at the chart the number of tick loop calls is consistently less than the number of ticks processed. This means that more than one tick is processed per loop call as the server catches up. I hadn't expected this and don't know if this is a normal operation. I would have to test this on a dedicated server to know for sure.

Unfortunately as with the main loop chart above, the number of players was not enough to place a real load on the server. So I don't know if this set of statistics would provide any useful information while under load.

Client Data Collection
For the stress test I modified my TGE 1.5.2 client to write out all data as produced by NetGraph::updateStats() for the NetGraph GUI (press 'n' to open and close the graph). This was written to a comma delimited file and included the following information:

- Active Ghosts
- Ghost Updates
- Bits Sent
- Bits Received
- Ping
- Packet Loss

I imported this data into Excel and decided to produce a graph of Active Ghosts, Ping and number of clients on the server to see if there was any correlation. It is worth noting that Packet Loss was zero during the stress test.

Client data plotted with number of connected clients on the server

As I would expect, the number of active ghosts fluctuated during the test as people fired their weapons and caused general mayhem. Also my ping (as calculated by TGE using packet time to acknowledge) didn't appear to vary by the number of players. Note that where there are gaps in my client's ping data the next mission was loading. So we played two full Stronghold missions during the stress test.

As with all of the other graphs, the number of players doesn't appear to have loaded my client to any degree. So the collected data doesn't help with determining the Amazon virtual server's capabilities.

TGE Issues
Two issues did crop up with TGE during my month long run with the Amazon EC2 servers, but I have not investigated them:

Negative Connected Players
While leaving the TGE 1.5.2 game server up and running for a number of days, I would see a phantom disconnect in the console.log when no one is actually connected. This would drop $Server::PlayerCount to a negative value. When this occurred the server would no longer show up in a TGE client's server list.

In looking on my GG Master Server Monitor at www.gnometech.com the server would show up as '255/64' players. I often see a number of servers in this state within my monitor, so I suspect this is a general problem somewhere in the net code.

Ghost Count Always Increasing Between Missions
If you take a look at my Client Activity graph above you'll notice something wrong with the Active Ghosts value. It does not reset itself between mission restarts. I don't know if this is an actual problem with not releasing ghost objects or just a reporting issue with the NetGraph code. I only came across this while producing my graph.

Conclusions
I've come to three conclusion based on the stress test:

1. It is possible to run a TGE 1.5.2 dedicated server on the Amazon EC2 service.
2. The small instance Amazon EC2 virtual server successfully handles up to 4 players.
3. The upper limits of the small instance Amazon EC2 virtual server were not reached.

Available Data
I'm making available the data I used to produce my graphs above. It is in Microsoft Excel format and includes all collected server data for January 17th (24 hours worth) and all of my collected client data during the stress test. You may download the ZIP file here.

Thank-you
I wanted to thank those that came out for the stress test:

- warme (MEX)
- Dunsany
- JeremyA
- Gnometech (that's me! :o)

Looking to the Future
I'd like to attempt another stress test in the near future and hopefully get more people involved. I have another project that requires my attention for the next month or two, so any additional testing will have to wait until then.

I'll leave my two servers up until Monday for anyone that wants to try them out. The game server is Amazon EC2 Test and the real-time web statistics are at www.torquetest.info where you'll be forwarded to the actual Amazon web server.

Thanks!

- LightWave Dave