Server Compromised - CONCLUDED
by Randel Reiss · in Technical Issues · 03/04/2011 (4:07 pm) · 13 replies
PLEASE DO NOT ADD ANY MORE SECURITY PRACTICE & PROCEDURE ADVICE TO THIS THREAD.
THIS THREAD IS FOR ANYONE WHO MAY KNOW, OR SOMEDAY KNOW, IF A TORQUE PORT CAN BE COMPROMISED.
One of our Torque Servers was compromised. This thread is strictly to get word out to advanced projects to share information - not to cause panic or controversy in the Torque community. Due to the small number of ports on the machine and the lack of any running services, our security experts advised that we publicize this information to add brain power to a potential solution.
On the evening of Wednesday, March 2, around 11pm (Pacific), one of our Minions of Mirth virtual world servers in San Jose, CA, went offline with no notice. By morning it was discovered that although the machine was still running, the Administrator password had been changed. Our hosting service was forced to do a reinstall of the OS. We found, after recovery, a ZS folder containing an install of Zombie Service with a typical welcome.txt containing a typical script-kiddie F'the world message.
The server ran 2 years problem free. Here's the stats on the machine:
Windows Server 2003 Web Edition
Only a single user account, Administrator
All security updates implemented on a bi-weekly basis
No web service
No Telnet service
No file sharing service
We run Remote Desktop (has no documented record of being compromised)
14 instances of Torque 1.3.5
Python 2.5
Twisted Python
SQLite (built into code, not as a service)
This is not the actual port analysis from that night, but a sample snap shot from one of our other live servers:
Not shown: 992 closed ports
PORT STATE SERVICE
135/tcp open msrpc
445/tcp closed microsoft-ds
1025/tcp open NFS-or-IIS
2008/tcp open conf
2009/tcp open news
3389/tcp open ms-term-serv
7000/tcp open afs3-fileserver
7001/tcp open afs3-callback
7002/tcp open afs3-prserver
7000 - 7002 are Torque instances.
3389 is the Remote Desktop.
We have no idea how the machine was compromised. At this time we're only interested in any feedback concerning Torque as a potential vulnerability - not general security discussions or plugs for Linux, T3D, or newer OS.
Thank you!
Randel Reiss
Director Product Development
Prairie Games, Inc.
mailto:RandelR@PrairieGames.com
Moderator: Updated the thread title to better reflect the outcome of the thread.
THIS THREAD IS FOR ANYONE WHO MAY KNOW, OR SOMEDAY KNOW, IF A TORQUE PORT CAN BE COMPROMISED.
One of our Torque Servers was compromised. This thread is strictly to get word out to advanced projects to share information - not to cause panic or controversy in the Torque community. Due to the small number of ports on the machine and the lack of any running services, our security experts advised that we publicize this information to add brain power to a potential solution.
On the evening of Wednesday, March 2, around 11pm (Pacific), one of our Minions of Mirth virtual world servers in San Jose, CA, went offline with no notice. By morning it was discovered that although the machine was still running, the Administrator password had been changed. Our hosting service was forced to do a reinstall of the OS. We found, after recovery, a ZS folder containing an install of Zombie Service with a typical welcome.txt containing a typical script-kiddie F'the world message.
The server ran 2 years problem free. Here's the stats on the machine:
Windows Server 2003 Web Edition
Only a single user account, Administrator
All security updates implemented on a bi-weekly basis
No web service
No Telnet service
No file sharing service
We run Remote Desktop (has no documented record of being compromised)
14 instances of Torque 1.3.5
Python 2.5
Twisted Python
SQLite (built into code, not as a service)
This is not the actual port analysis from that night, but a sample snap shot from one of our other live servers:
Not shown: 992 closed ports
PORT STATE SERVICE
135/tcp open msrpc
445/tcp closed microsoft-ds
1025/tcp open NFS-or-IIS
2008/tcp open conf
2009/tcp open news
3389/tcp open ms-term-serv
7000/tcp open afs3-fileserver
7001/tcp open afs3-callback
7002/tcp open afs3-prserver
7000 - 7002 are Torque instances.
3389 is the Remote Desktop.
We have no idea how the machine was compromised. At this time we're only interested in any feedback concerning Torque as a potential vulnerability - not general security discussions or plugs for Linux, T3D, or newer OS.
Thank you!
Randel Reiss
Director Product Development
Prairie Games, Inc.
mailto:RandelR@PrairieGames.com
Moderator: Updated the thread title to better reflect the outcome of the thread.
About the author
Making great games so you don't have to.
#2
03/04/2011 (5:42 pm)
Also, worth doing an anti-virus/spyware sweep to make sure you have no key-loggers on your machines (desktop and server), and change passwords regularly (every few months) and if you have employees that leave the company, then change again.
#3
only allow game servers to root through to master servers and back, prevents any external servers that are not authorised to access to master server out.
Ensure any communication on the network layer for file transfer/comms are also locked down to specific IP addresses and ports.
Worth following a firewall setup of something like this if you have your own colo rackspace, or managed hosted solution:
Untrusted Network (internet) -> DMZ (for web/game servers) and databases behind the firewall in a trusted zone. Maybe add in a VPN for desktops (generally slower) or locked down IP addresses.
Security may be tight on your network/servers, but worth checking it all again via an audit tool as some rules could have been changed.
03/04/2011 (5:46 pm)
Firewall rules should also be locked down;only allow game servers to root through to master servers and back, prevents any external servers that are not authorised to access to master server out.
Ensure any communication on the network layer for file transfer/comms are also locked down to specific IP addresses and ports.
Worth following a firewall setup of something like this if you have your own colo rackspace, or managed hosted solution:
Untrusted Network (internet) -> DMZ (for web/game servers) and databases behind the firewall in a trusted zone. Maybe add in a VPN for desktops (generally slower) or locked down IP addresses.
Security may be tight on your network/servers, but worth checking it all again via an audit tool as some rules could have been changed.
#4
03/05/2011 (12:35 am)
Thanks for your quoted message via my website. As I said, look at your security first! I very much doubt it's Torque, and you'll get no other reply than that, so why wait and get to the point. 
#5
03/13/2011 (11:22 pm)
I had a simular issue on my Win 2003 server with simular specs. I belive we just turned off a few services and "upped" security. Might consider looking at some new snap-in network tools for better secruity.
#6
Found this article, I'd pass it along to your MCSE on staff. Should be helpful info. I remember the concept from college in my MCSE class. Hope it helps
03/13/2011 (11:26 pm)
http://technet.microsoft.com/en-us/library/cc163140.aspxFound this article, I'd pass it along to your MCSE on staff. Should be helpful info. I remember the concept from college in my MCSE class. Hope it helps
#7
1. Never ever, ever, ever put a windows computer connected straight to the internet. This included one with firewall software! A firewall is only as good as the OS it runs on and pure hardware firewalls tend to be outdated before they hit the market. What I would recommend is pfsense it uses FreeBSD, link http://www.pfsense.org/. Take a old computer slam a bunch of nicks in it. Install the pfsense ISO. It is easy to set up and get running. Best of all it does not require a lot of knowledge or a beefy computer. It is also very flexible and can do VPN, DMZs and all your Routing just to name a few.
2. Stop using remote desktop from the outside world. Remote desktop can transmit user name and password in clear text. All you have to do is use a program like wireshark to grab it. Before others chime I have seen it. Best thing to do is cut off access to that port with the firewall then VPN into the network then remote desktop. This way you get a layered approach.
3. Move the port on remote desktop this can be done with regedit and a quick search on google will show you how. One of the most scanned for ports is the remote desktop port along with MsSQL ports. I see it in my logs every day.
4. Make a account for each and every person on the box and make who needs the access an admin then disable the Administrator account. Everyone and there dog knows that windows uses "administrator" for the admin account and if you are going to use a brute force attack that would be the account to go after. Also by giving everyone their own log in if the box is hacked you can tell what account was used by the logs.
03/20/2011 (10:46 pm)
As a Network Admin by trade first thing I would recommend:1. Never ever, ever, ever put a windows computer connected straight to the internet. This included one with firewall software! A firewall is only as good as the OS it runs on and pure hardware firewalls tend to be outdated before they hit the market. What I would recommend is pfsense it uses FreeBSD, link http://www.pfsense.org/. Take a old computer slam a bunch of nicks in it. Install the pfsense ISO. It is easy to set up and get running. Best of all it does not require a lot of knowledge or a beefy computer. It is also very flexible and can do VPN, DMZs and all your Routing just to name a few.
2. Stop using remote desktop from the outside world. Remote desktop can transmit user name and password in clear text. All you have to do is use a program like wireshark to grab it. Before others chime I have seen it. Best thing to do is cut off access to that port with the firewall then VPN into the network then remote desktop. This way you get a layered approach.
3. Move the port on remote desktop this can be done with regedit and a quick search on google will show you how. One of the most scanned for ports is the remote desktop port along with MsSQL ports. I see it in my logs every day.
4. Make a account for each and every person on the box and make who needs the access an admin then disable the Administrator account. Everyone and there dog knows that windows uses "administrator" for the admin account and if you are going to use a brute force attack that would be the account to go after. Also by giving everyone their own log in if the box is hacked you can tell what account was used by the logs.
#8
6. If one box has been compromised, assume ALL your computers have been compromised. If someone goes through the trouble of rooting your box it is a safe bet they will take steps to make things easier for them next time. I know it will be a pain but trust nothing and start redoing production computers and keeping the old ones isolated from the ones that you have redone. I know this may seem like over kill how ever better safe than sorry. Also I would check out each and every box that has access to the production computer. Could be an office computer was compromised and from there they gained access to other boxes.
7. Grate time to go over your security both at the computer level and personal level. This may sound strange but the easiest thing to hack is a person. People are the weakest link. Browsing the web, testing new software, and even copying software with out checking it first are never a good idea on production servers (I know we all have done it). Go over with employees about security and things like clicking attachments in emails, using good secure passwords (8 length at least 1 number and 1 letter to start with), different passwords for email and logins and changing them at least every 6 months, for a start. You have a lot of groaning how ever it is worth it in the long run. Lets say one box gets compromised and the hacker finds a way to infect your clients computers when they connect. Now your not just redoing a server but you have to deal with damage control with your users.
8. Save your logs to a locked down box and back them up (like a planes black box). That way if a box does get compromised you have logs to find out how, when, what and why. You may not be able to track down who did it, how ever it will let you know where the whole is and what you need to do to fix it. Also will allow you to check your other computers to see if they have the same issues.
9. And very important!!! Get ready for a storm. Rather it be a scrip kiddy or a true blue hacker once they gain access 10 to 1 they will be back and in force. Like a kid tossing a little fit you should expect DoS attacks, port scans, and other nasty things. We had a mail server that due to an update to the mail server program it had a open relay. It did not take long before someone found (our Nagios alarms went off) we shut it down. For over a month they keep trying different ways to gain access before giving up.
The best thing about all these is that other than time it will take, and some old hardware (prob have it laying around) it is free. All the software (pfsense, Nagios and Cacti) are free. So there is no reason not to lock things down.
At a major Colo here in New York, just for fun we hook a fully patched Windows 2003 box to the internet (raw connection) and it was compromised in less than 10min. So if you got by for 2 years you where lucky.
Background: Network Admin for a company that does the website, email and booking eng for over 300 of the top hotels.
P.S. - Tell Savage Hi!!! from GrayWidow and GothicQueen, if you need help let us know....
03/20/2011 (10:47 pm)
5. Behind your firewall install a Nagios and Cacti box. Both can go one box and can be real pain to set up but worth every min of it. This will do two things for you. First it will let you know how your servers are doing and give you early warning if something is wrong. After a while of running Cacti you will be able to see what is normal for your servers operations and can set Nagios send an alert when something is not right by email text message and more. The sooner that you know there is an issue the more damage you can prevent. Also you will soon find small issues before they become a problem like servers running out of memory or disk space to the CPU is getting over loaded.6. If one box has been compromised, assume ALL your computers have been compromised. If someone goes through the trouble of rooting your box it is a safe bet they will take steps to make things easier for them next time. I know it will be a pain but trust nothing and start redoing production computers and keeping the old ones isolated from the ones that you have redone. I know this may seem like over kill how ever better safe than sorry. Also I would check out each and every box that has access to the production computer. Could be an office computer was compromised and from there they gained access to other boxes.
7. Grate time to go over your security both at the computer level and personal level. This may sound strange but the easiest thing to hack is a person. People are the weakest link. Browsing the web, testing new software, and even copying software with out checking it first are never a good idea on production servers (I know we all have done it). Go over with employees about security and things like clicking attachments in emails, using good secure passwords (8 length at least 1 number and 1 letter to start with), different passwords for email and logins and changing them at least every 6 months, for a start. You have a lot of groaning how ever it is worth it in the long run. Lets say one box gets compromised and the hacker finds a way to infect your clients computers when they connect. Now your not just redoing a server but you have to deal with damage control with your users.
8. Save your logs to a locked down box and back them up (like a planes black box). That way if a box does get compromised you have logs to find out how, when, what and why. You may not be able to track down who did it, how ever it will let you know where the whole is and what you need to do to fix it. Also will allow you to check your other computers to see if they have the same issues.
9. And very important!!! Get ready for a storm. Rather it be a scrip kiddy or a true blue hacker once they gain access 10 to 1 they will be back and in force. Like a kid tossing a little fit you should expect DoS attacks, port scans, and other nasty things. We had a mail server that due to an update to the mail server program it had a open relay. It did not take long before someone found (our Nagios alarms went off) we shut it down. For over a month they keep trying different ways to gain access before giving up.
The best thing about all these is that other than time it will take, and some old hardware (prob have it laying around) it is free. All the software (pfsense, Nagios and Cacti) are free. So there is no reason not to lock things down.
At a major Colo here in New York, just for fun we hook a fully patched Windows 2003 box to the internet (raw connection) and it was compromised in less than 10min. So if you got by for 2 years you where lucky.
Background: Network Admin for a company that does the website, email and booking eng for over 300 of the top hotels.
P.S. - Tell Savage Hi!!! from GrayWidow and GothicQueen, if you need help let us know....
#9
* This thread is for anyone who may know, or will know, a way to compromise a system through a Torque port.
Thank you.
Randel Reiss
Director Product Development
Prairie Games, Inc.
mailto:RandelR@PrairieGames.com
03/21/2011 (11:19 am)
* PLEASE DO NOT ADD ANY MORE SECURITY PRACTICE & PROCEDURE ADVICE TO THIS THREAD.* This thread is for anyone who may know, or will know, a way to compromise a system through a Torque port.
Thank you.
Randel Reiss
Director Product Development
Prairie Games, Inc.
mailto:RandelR@PrairieGames.com
#10
03/21/2011 (11:46 am)
If there was a way, it shouldn't be posted here - else you are putting everyones games at risk just by requesting this. I'd suggest that you contact Randel and GG directly.
#11
That's a good point. I hadn't thought about the repercussions of the answer to port-circumvention enabling other good people to try very bad things.
Randel Reiss
Director Product Development
Prairie Games, Inc.
mailto:RandelR@PrairieGames.com
03/21/2011 (9:10 pm)
Julian,That's a good point. I hadn't thought about the repercussions of the answer to port-circumvention enabling other good people to try very bad things.
Randel Reiss
Director Product Development
Prairie Games, Inc.
mailto:RandelR@PrairieGames.com
#12
05/22/2011 (7:04 pm)
Another good point is if there is a security hole in Torque everybody should know so we can fix our code.
#13
Thank you, everyone, for your patience and support.
Randel Reiss
Director Product Development
Prairie Games, Inc.
mailto:RandelR@PrairieGames.com
07/16/2011 (3:42 pm)
After extensive investigation, and what appears to be a lack of breach after some modifications, we're ready to declare that the system was most likely circumvented through Remote Desktop login. We'll update if that theory doesn't survive the test of time.Thank you, everyone, for your patience and support.
Randel Reiss
Director Product Development
Prairie Games, Inc.
mailto:RandelR@PrairieGames.com
Torque Owner Jules
Something2Play
It would also be wise to lock down the 3389 remote desktop port to your broadbands IP address, or group of addresses to add further security, as allowing an outside connection is asking for trouble and if you use a generic password or something that can be cracked by a hacker easily using some number or dictionary cruncher then this will not help things. Also, if you allow port 80 on the server as a web service then this would be only as secure as the code written on it. Logs can be wiped, or replaced by a hacker.
This may not be what you are looking for as an answer, but stopping this from further intrusion then all measures should be taken. If you have then taken all of these measures then you can start looking at 3rd party software having back doors.