CentOS, How to tell if I've been hacked?

CentOS] How to tell if I’ve been hacked?

Dave tdbtdb+centos at gmail.com
Sat Aug 22 09:37:49 UTC 2009

On Tue, Aug 18, 2009 at 3:53 PM, Scott Ehrlich<srehrlich at gmail.com> wrote:
> There is a lot of talk about the vulnerable Linux kernel.   I'm simply
> wondering the telltale signs if a given system has been hacked?
> What, specifically, does a person look for?

This is an interesting and frustrating question. Perfect security is
impossible, but maybe we can achieve 'good enough'.

On Tue, Aug 18, 2009 at 5:14 PM, Christopher
Chan<christopher.chan at bradbury.edu.hk> wrote:
> rpm -Va is a good start for modified binaries/libraries.

Problems with rpm: 1) Many files on a system did not come from an rpm
or had good reasons to change, so dealing with false positives is a
problem 2) Some packages are sloppy and no longer verify immediately
after installation 3) rpm cannot address memory-only attacks or bios
attacks. Still, if rpm tells you some binary has changed since
installation, you know you're in trouble. Or you're using prelink.

> rootkit detectors is another thing you can try.

Problem with rootkit detectors I have used: many false positives. At
least, I hope they were false! Googling around, I found that once you
had a positive result, it was sort of complicated process to figure
out whether you'd really been hacked or were just having timing
problems or had an unusual configuration.

> Other than that, it is checking your logs and looking for odd files
> lying around...

And prayer.

On Tue, Aug 18, 2009 at 5:22 PM, Ryan Pugatch<rpug at tripadvisor.com> wrote:
> Also, processes running that you don't recognize.

Unfortunately, if you're like me there are really a lot of processes
running on a virgin linux box (never touched the internet) that I
don't recognize. I once tried just making a big file of them and
having a cron job send me email when the list changed significantly.
This could have caught an unlucky or inept cracker who launched some
process named "meEvilCrackhead", but wouldn't have done much to catch
someone using an innocuous name, like say 'grep'.

>Users you don't
> recognize.

Again, it is possible to catch someone who doesn't bother to get rid
of the smoking gun. Someone who has root on your system can create a
new user, or they could use a pre-existing user. You can keep an eye
out for strange users, but the real problem is spotting familiar users
doing stuff they ought not. Even that can be covered if the cracker
replaces your tools or hacks the kernel.

> Logged in sessions that you don't recognize.

I'm not sure what Ryan means here, unless he is assuming only one
person (you) has authorized access to your machine, and you see
sessions logged in as you that you know nothing about. Yeah, that
would tip you off. If lots of users can log in, there's not much

> Free space
> shrinking abnormally.

Again not really sure what this would mean. Too high a load, too many
programs running? Again, someone with root access could hack the tools
you use to monitor this, or even the kernel, and make it really hard
to see. Assuming you really know how much free space you ought to have
at a given moment, which, for me, I am ashamed to admit, would be
quite rare.

> An increase in bandwidth usage that is unexpected.

Now we're talking! Well, I am still pretty damn ignorant of what a
system's bandwidth demand ought to be, but at least you could see the
stuff actually happening and make a sort of reasonable investigation
of 'what do I have running that would possibly want to talk to IP
xxx.yyy.zzz.aaa?'  And for once, no matter how good the intruder is,
they won't be able to get your own system to lie to you for them
(assuming you're using a different system to do the network analysis).
But while you analize the traffic, the bad guys has more time to
damage your data.

On Tue, Aug 18, 2009 at 5:58 PM, Christopher
Chan<christopher.chan at bradbury.edu.hk> wrote:
> Yeah...one should not assume that those will be hidden by rogue
> libraries/binaries. Not every case will be taken that far or unspotted
> before it gets that far.

Every intrusion is vulnerable for a while at least, while the intruder
is trying to get in and get root. After that they will probably try to
cover their tracks.

On Tue, Aug 18, 2009 at 6:57 PM, Bill Campbell<centos at celestial.com> wrote:
> To really know whether a system has been hacked, it's necessary
> to use something like Tripwire or Aide,

And very carefully. Only that won't help you with memory-only attacks,
or bios stuff, etc. These tools concentrate on verifying that your
disk files have not been altered. I don't think they would help with
an attack that uses free space (guessing here). Also, they are a pain,
unless your system stays absolutely static, which in effect means, if
you never use it. Have them ignore your data space, and the hacker can
exploit that. And even then, linux is constantly updating various
files in the background, and of course you need to update software to
keep up with the security patches. You need to track every change of
every file. I doubt many people have the patience.


> It's also a good idea to check for executables in places they
> normally shouldn't be, /tmp, /dev/shm on SuSE systems, /var/tmp,
> and similar directories where crackers like to hide their work.
> Often these executes will be in directories with names like ``.. ''
> (note the trailing space) that look legitimate.

I like this, because it might actually be automated. Of course, you're
trusting stat or whatever.


> You cannot trust tools like ``ps'', ``find'', ``netstat'', and
> ``lsof'' as these are frequently replaced by ones that are
> modified to hide the cracker's work.

Naturally we are running aide and tripwire from a CD or other
read-only medium, why not toss in a copy of these tools as well? Of
course, if the kernel has been hacked, even that won't save us, but we
have to take what we can get.

On Wed, Aug 19, 2009 at 1:03 AM, Eduardo
Grosclaude<eduardo.grosclaude at gmail.com> wrote:
> As a corollary, the only safe way to audit a suspected system is
> booting your diagnostic tool from known good media (eg try a security
> Live CD distro)

Safe in what sense? When you rebooted, you may have erased from memory
the only remaining trace of the intrusion and you are still
vulnerable. But at least you *can* trust what the tools are telling
you - I hope.

On Thu, Aug 20, 2009 at 12:59 PM, Magnus
Holmström<magnus.holmstrom at gmail.com> wrote:

Oops, someone is going to scold you for top-posting.

> Check for failed logins in /var/log/messages

Kind of useless, don't you think?  Failed logins I can live with. It's
successful ones that hurt.

> Check if the /etc/passwd file have been changed

Certainly, no decent detection system should overlook changes to
/etc/passwd. But that's just one of many important files.

> Use commands like last, w and uptime.

These tools are kind of weak. If you have a single-user system or the
intruder is doing something really obvious (what? rm -rf /?  wall 'I
am an intruder!'?) they would help. With multiple users they won't
tell you enough, and if the intruder is good they will lie to you.

Thanks to all for stimulating thoughts. I've been thinking about this
for a while. I need to think some more.

Server Security Related articles


Major IP Addresses Blocks By Country

Stop Traffic From China IP Address Blocks To Protect Your Web Server From Chinese Hackers

20 Linux Server Hardening Security Tips

How do I know if my Linux server has been hacked?

Setup SSH to run on a non-standard port

HowTo: Install Adobe Flash Player on CentOS 6.2

Install Apache, MySQL 5.5.30 & PHP 5.4.12 on RHEL/CentOS 6.4/5.9 & Fedora 18-12

Linux Firewalls Using iptables

Protected: Cpanel Resource Monitor

AWS Free Usage Tier

esxi where are snapshots stored

CentOS] How to tell if I’ve been hacked?

Install Skype 4.1/4.0 on Fedora 18/17, CentOS/RHEL/SL 6.3

How To Configure a Comcast Business Class Static IP Address

what is VPS hosting

ConfigServer Security & Firewall : CSF

Securing OpenSSH

psad: Intrusion Detection and Log Analysis with iptables

What is port scan attack

Moving or copying a virtual machine within a VMware environment (1000936)

vSphere 5.1 Complete install and client – How to Install and Configure VMware

what is distributed denial-of-service attack (DDoS)?

How to Login as a Different User in Linux

Parallels Plesk Panel – Summit 2013 – Birger Steen Keynote

How to transfer my osCommerce from one host to another?

7 Chmod Command Examples for Beginners

PDOException: SQLSTATE[HY000] [2002] Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2) in lock_may_be_available() (line 165

CageFS is a virtualized file system and a set of tools to contain each user in its own ‘cage’.

Checking what PHP version I’m running on Linux?

what can you do with vmware esxi

WHM/Cpanel in Amazon EC2 server

EC2 CPU benchmark

How to Install wget on CentOS 6.2

How to Install zPanel on CentOS 6

Amazon EC2 micro instance type

2013 AWS Summit Keynote — San Francisco

Building a Home Virtual Host Server

How to Build a Supercomputer

Using Apache Bench for Simple Load Testing

CPU Usage HIGH – spamd child?

What is Mod_Security

Turn Windows Firewall on or off

How to Configure Your Mac’s Firewall


How to Install dig on a CentOS Linux Server

How to change hostname of a CentOS system?

Install Linux Rkhunter (Rootkit Hunter) in RHEL, CentOS and Fedora

cPanel Server Service Package – ConfigServer Services


visitors counter: since Sep 4th, 2012