If you reading this you must have a good reason to suspect your server has been compromised in some way. If that is the case, your first objective is to immediately secure the server from new access. I would begin by changing the server and all account/ftp passwords to something very secure. A great tool for secure password generation is located here.
The next thing to do is to scan your server for exploits using clamscan (if you have it installed… most servers should have this installed by default as a basic security measure along with rkhunter and chrootkit)
freshclam && clamscan -ir --log=/root/clamscan.log /home*/*/public_html &
This will update clam and run a scan and dump the results into a file in the /root directory named clamscan.log. I will not go into installing and updating these programs as it is covered elsewhere in this blog.
Once the scan is complete, you can check the log to see if anything (shells, trojans or anything else) shows up and deal with them accordingly; chmod 000 filename, chattr +i filename… ect…) If your hosting company has a security response team, notify them immediately and hold off on the chmod/chattr as they may want to check the logs and files themselves for more information.
While the scan is running, see if your data is still intact. Check your backups to make sure they are current. I would download a copy of the latest backups (daily, weekly and monthly if you have them
–you DO have backups running right???–
via ftp or ssh to your local computer) and scan them locally for malware. I would also make sure you scan your local drive as well, in case you picked a keylogger somewhere and you unknowingly provided the attacker the info he needed to access the server.
Once you know you have a clean set of backups, provide that information to your hosts system-restore/migrations team who should be able to get your data re-imported quickly once the server has been checked by security.
If it is found that the server has been rooted, you will need a re-image of the server. No questions asked. If your hosting company does not recommend this, I would find a different host. Once a server is rooted, it is no longer considered secure and needs to be re-imaged. Period. No questions asked.
if a single account was compromised, you can basically follow the same steps as above;
Please do not go off on your host and assail them with complaints about how much time it is taking or some other such nonsense, and blaming them for the compromise. The hard truth is that 99.999% of the time, the issue will be related to poor password policy on the server, sql injections, keyloggers on a local users computer or simply not having the basic security measures in place on the server which basically means that it is your fault. Yea, I said it… your fault. Take it like a man and learn from the experience, move on and be better prepared in the future. Secure your server like you secure your house, because at the end of the day, if you make your living from this box, it deserves the same protection your home and family does because guess what… those server are putting food on the table, paying the bills and providing little Johnny the Wii he keeps bugging you about.
I had a client remove mod_security because they felt it was slowing down their server, needless to say within a week of removing it, they were “pwned”; 850 sites were compromised, backups were deleted and shells to numerous to count were uploaded to their server as well as dark mailers which spammed the known universe for 2 days before the exploits were detected. The client was blacklisted on Sorbs, Google and many other places which also resulted in their IP’s being dirtied.
The client then called in to complain because the repair of the server was taking to long. (sigh)
A penny’s worth of prevention can save several thousand dollars worth of business but in many cases that is a hard lesson learned at the expense of a few milliseconds of speed. It is your choice…
Further checks can be made to make sure the kernel and other software is updated to the latest version. This does not mean moving your apache and mysql to the latest alpha release. Run a nmap scan of the server to make sure there are no ports that are open to exploit. You can also use netstat to list active connections to/from system
netstat -anp --udp --tcp | grep LISTEN
If you search the site for “BETTER THAN NOTHING SECURITY”, you will get a good start on securing your server correctly.
Basic precautions should prevent and maintain your server security. There are many tutorials and walk throughs out there which address securing your server… heed them well unless you want to be talking to me one morning at 4:00 a.m. hollering about how your server has been hacked to pieces… and I, being the good and wise admin that I am, will direct you to this article…
