This details the events of the security breach and gives a rather lengthy anaylsis of our present situation at the end.
On or about Friday of last week we received an anonymous e-mail from a hacker who had managed to get access to a user's account through a stolen password. He told us which account he used, but did not tell us how he got the password.
The hacker told us that he had managed to get root access to light through the use of a system command I had created for some of our users that allows them to restart the web server when it is jammed. He detailed exactly how he got root access and told us that we should check our system for other similar scripts. He also asked that we destroy his letter in return for the information he had provided us, which I did.
I checked out the script that had given him access and used the exploit he detailed and indeed it gave me instant root access. I immediately fixed the script and others like it, finding a number I had created in my ignorance, long ago for similar special purpose uses and had forgotten about completely.
We then locked out the account he had come in through and informed the user of the breach of his account.
Although this hacker seemed to be a decent fellow, we were naturally suspicious that maybe he had left some back doors in the system just for fun. No matter how decent a hacker may be, it is often hard for them to restrain themselves from their shenanigans.
So we started a thorough sweep of light using the oldest possible system binaries that we had on backup tapes from a few months ago.
We loaded the tape backups onto a hard drive so that the directory structure exactly matched our present directory structure, then we wrote a program to compare every program on the hard drive with the present system binaries.
The first thing we found was a file called logfile.dat hidden deep in the system binary directories that contained a long list of user names and cleartext passwords! The entries looked like
I immediately contacted our security staff and they recognized it instantly for what it was, a rather crude 'rootkit'. A rootkit is a hacker tool-set for accumulating passwords on compromised systems. Such rootkits abound on the internet and are generally freely available.
Mind you a rootkit won't GET you access to a site, but once you have access, it will allow you get as many passwords as there are people signing onto shell. This particular rootkit didn't get slip/ppp signons, only direct shell logins to light.
In the rootkit directory was a number of backed up copies of login, ps, ifconfig and netstat, which are important system programs that allow the user to monitor what is going on.
These programs had been replaced by hacked versions of themselves, and I presume the backed up unhacked versions were there in case the hacker ever wanted to clean up and leave without a trace.
We compared the backed up versions with known good versions from backup tapes and majesty (which was apparently not touched), and replaced the bad versions with their proper counterparts.
All of the hacked programs had had their timestamps surgically altered to look like the time stamps had not been changed from when they were first installed on the system in Oct 1994.
A program for doing this timestamp tampering was in the rootkit directory.
The hacked login program that was installed kept a clear text record in a log file of everyone who logged into light as a shell user, including myself as root. It also had a back door with its own password giving the intruder a root shell any time he wanted it.
The ps (process stats), ifconfig (interface config) and netstat (network stats) programs were altered to not show certain processes related to the attack and to not log the root logins in various databases that keep track of who logs in when.
The date on one of the rootkit directories was Aug 16, 2:11am, so we can conclude that the hacker was on the system from at least that time to present if not earlier.
There were about 5000 logins to light recorded for about 300 different shell users. Slip/PPP users were not recorded.
However the hacker had complete access to the system. Thus we can assume he has the encrypted password file which he can run through a cracking program if he wishes to extract weak passwords (more on this below), and he had access to everyone's home directories, mail, web and ftp sites for the duration of his stay, as anyone with root access does.
CHANGING YOUR PASSWORD TO A STRONG PASSWORD WILL DISABLE THE HACKER FROM USING YOUR ACCOUNT.
You should know that we don't keep user's credit cards on the system, so he did not have access to our credit card base, however people often have lots of credit cards in their incoming and outgoing mail, so he may have grabbed a few if that was his mission.
We have found no obvious damage done to the system, although there have been sporadic reports of accounts being used by unauthorized parties which may or may not be related.
During our scan of the system, we did find a number of security outnesses, and checking into the credit card number of one suspicious account showed that it was being paid with a stolen credit card. We have contacted the real owner of the card and will be refunding her the amount charged for the rogue account which has been closed.
The rootkit site itself looks like it was installed and then abandoned a few days later leaving the password logger to run unattended. So its not clear that the hacker actually did much with his access to the system. However he had the ability to come and go undetected, and his intent clearly was not to leave large footprints or to do much damage.
We were not able to find evidence of packet sniffing which would have compromised passwords on remote systems as people telneted from light to elsewhere, however there was a packet sniffer in the rootkit, so this has to be considered a possibility.
There are a number of measures that will be put in place to guard against this again in the future, which I am sure doesn't make anyone feel better about the present incident. More on this below.
For your information, there are 5 people on this system who are supposed to have root access.
Homer Wilson Smith firstname.lastname@example.org Owner Jane Elizabeth Staller email@example.com Owner Doc Doolittle firstname.lastname@example.org Co-Sysop John Paine email@example.com Security Max Parke firstname.lastname@example.org Systems Programmer
ANAYLSIS OF OUR PRESENT SITUATION
OPEN AND CLOSED OPERATING SYSTEMS
Security on a Sun based system is an interesting paradox. The operating system is more than 20 years old and has been tested so thoroughly it is one of the most rock stable systems on the planet for large scale multi user use.
Further it is an open system, which means that the code is publically available, so programmers can study the internals of the system kernel and find weaknesses in it and patch them before they cause any damage. This provides a sort of long term planet wide beta testing of the operating system guaranteeing to a large extent that if there are any holes in it, they will be found and fixed.
The paradox is that because it is an open system, hackers have the same access to the software to find those holes first and exploit them before the rest of the world becomes aware of them.
Holes in fact are often first found by the very large hacker community and are both exploited and reported to the public at large at the same time. Sometimes they are reported BY publishing the exploit in hacker magazines and on the internet. Whether or not this is a public service is open to opinion :).
An "exploit" is a short demonstration program that takes advantage of the hole and shows what can be done through it.
The system however is very complex, and over time it is changing as software is upgraded, not only fix old holes but also to add new functionality. This inevitably creates new holes. The usual cycle goes something like this.
The software is upgraded and published. Hackers comb through it looking for security problems. When they find them, if they are good guys, they write exploits that demonstrate the problem or if they are bad guys, that take advantage of them.
In either case the exploits become well known on the internet and in hacker circles and eventually make their way to CERT, the world wide Computer Emergency Response Team. CERT then publishes official warnings, with the exploits and the fixes included, and e-mails them to everyone who has subscribed to their mailing lists. It is then up to the respective operators to close the holes and secure their system.
This process is a never ending battle.
There is something to be said about using a non open software system to provide internet access. Other platforms such as Windows NT use proprietary software where the code is not openly available to the public to scrutinize for holes. This does not mean that such platforms do not have any holes, only that no one ever finds them. However when they do find them, they don't get broadcast, and you can't go in and fix them yourself once you find out about it.
Some would tell you that with closed software systems, only the hackers have the source code and no one else does!
So there is a trade off here. Closed software systems make it harder to find these holes, but also harder to fix them.
You have to ask yourself, do you want an operating system where every line of code has stood the test of public scrutiny for 20 years or do you want code that only the company that sold it to you knows?
Hackers are not a small group of kids who have too much time on their hands. Hackers are a VERY large and influential community planet wide in the computer world. Some are good guys and some are bad guys, often the bad guys are called crackers to distinguish them from the the good guys.
The term TO HACK means to put together a new and ingenious solution to a problem. That hacking is sometimes used to gain unauthorized access to systems where they do not belong is incidental.
Anyhow most hackers are benign, only a very few of them are out to cause any harm, although a large number of them are definitely into getting into where they don't belong.
I imagine somewhere in their minds they have a huge wall map of every system in the known universe, and little pins in all the machines where they have gained root access. They are like homeless crabs looking for another shell to live in.
No pun intended.
The main thing that hackers want root access for is, it makes it much easier for them to gain root access on the next machine over. With root access they can watch tcp/ip packets come and go, mostly unencrypted, they can see people sign on to the remote machines using telnet and ftp and they can gather more passwords in this way.
They don't really want to DO anything to the system they have inhabited, for that would ruin their game of getting to the next system. So in this sense many hackers are a very scary but semi benign virus.
However that they then have access to all your mail, home directories and the rest does not make anyone feel any better about them. If you were to go scanning through ALL the extant mail on our system, not only in the mailboxes of new incoming mail, but also in the saved mail in people's home directories, how many credit card numbers do you think you would come up with?
What good is having a secure web server to transmit encrypted credit card transactions across the net, when some hacker is merely picking up the UNENCRYPTED mail at the destination end, credit card numbers and all?
There are 4 broad areas of security when it comes to hackers.
1.) Keeping the hacker off the system entirely 2.) Keeping the hacker out of other's accounts. 3.) Keeping the hacker out of root. 4.) Finding and purging the hacker once he has root.
1.) Keeping the hacker off the system entirely.
This is pretty much impossible because all he has to do is call us with a credit card, and get an account. There is no way to know who is a hacker and who isn't.
It is little solace to know that as a matter of pride hackers don't like to purchase accounts, they would much rather sniff or social engineer a password out of some hapless user.
Hackers have certain profiles, for example they are more likely to be coming in from remote sites using a telnet only account than from our local dialups. It is tempting to just say we won't accept any more telnet only accounts, but a lot of very good people will be hurt by this.
2.) Keeping the hacker out of accounts that are not his.
This is a bit easier but in the long run a hopeless case also. There are many ways that a hacker can get into an account that is not his. Almost all of them involve getting a password that is not his to use for that account.
There are a number of ways that a hacker can get a password to an account that does not belong to him.
a.) Social engineering. This means interacting with the owner of the account in such a way as to trick him into giving you the password. Someone calls you up and says "Hi this is the root administrator, we are doing some special testing and we need your password to verify the security on your account..."
b.) Brute force guessing. They try your name, your name with a 1 at the end of it, your kid's names, your birth date, your favorite color, your town, anything they know about you.
c.) The get a hold of the system wide encrypted password file and run crack against it for a week and come up with 100 passwords that were not as strong as they should have been.
Passwords are stored on our system in encrypted form. For example you type in homer2 as your password and the system encrypts this to ..SHFJ678LKSDD and stores this in the password file.
When you go to log on, you type in homer3 by accident, and it encrypts homer3 to a different string which does not match the one in the password file and that way the system knows it is an incorrect password.
It is not possible to unencrypt the password once it is encrypted, which is why when you change your password and loose it, we can't tell you what it is because we don't know what it is, we can only assign you a new one. As root administrator we don't need passwords to get around on the system, so this is not really a problem (unless of course someone gets root who shouldn't!).
However it is a very easy task for a hacker to take a 10 million word electronic dictionary and encrypt every single word in the dictionary and compare the result with your encrypted password. If they get a match, they have a password.
Clearly if you have a password like sailboat, then a hacker is going to get it very fast by this method. Even putting a single star in it, row*boat, will make it *MUCH* harder for a cracker to get a hold of it. However most cracking programs know to take ALL special characters out of the password, and THEN check it against the dictionary! Bang, your dead.
They also know to change all upper case to lowercase first, and to get rid of ly, ing, ed, er at the end of words. They also know that EVERYONE uses words with numbers at the end, so they just get rid of the number, look it up in the dictionary, and they have your password.
Crack programs are programmed to try millions of combinations on words, they will try them backwards, they will try two words in a row they will try every known proper name in existence, so really you have to have a really good password to escape this kind of analysis.
Clearly the problem then is how to keep the encrypted password file out of the crackers hands. Normally the file is not world readable, only those with root access can get at it. But sometimes it can leak out, slip ups happen, it becomes world readable for a short while and you don't notice due to some stupid programming error.
However even barring operator stupidity, there are numerous weaknesses in the operating system that will sometimes allow a hacker to get all or part of a password file. He doesn't need too many entries seeing how poor most people's passwords really are. All he needs is one cracked password to get into an account and start working on root.
d.) Sniffing packets from another system.
This is one of the most common ways that a hacker gets into other machines. He has a packet sniffer running on hacked.com internet service. Some hapless user has an account on hacked.com and gets a telnet or web account on lightlink.com. Now every time the user telnet's to his lightlink account, or uses ftp to upload his web pages from hacked.com, the hacker sees the transaction of signing on and records the lightlink password.
THERE IS ABSOLUTELY NOTHING THAT CAN BE DONE ABOUT THIS!
Or say someone from lightlink.com telnet's to hacked.com. The sniffer on hacked.com will pick up the user's hacked.com password which you might think is no big deal to lightlink. BUT MANY USERS USE THE SAME PASSWORD ON ALL SYSTEMS. The Hacker is sure to try out the hacked.com password on the guy's lightlink account with a high probability of getting in.
Considering how many systems our users telnet and ftp to, and considering how high the probabilities are that any one of those systems might have been hacked, it is pretty clear that at any one time quite a number of local lightlink passwords are in the hands of any number of people they don't belong in.
Once the hacker has a password on lightlink he can sign on at will to the user's lightlink account and start to work on getting root. If he gets root access to lightlink through some security 'feature' of the operating system (not!), he then can set up a packet sniffer to find out the passwords of everyone signing onto light AND all the remote machines that THEY sign onto.
If they are so foolish as to signon to a remote machine as root adminstrator, then he will have root on that remote machine, with no trouble at all.
At this point security becomes a joke. No matter where you are telneting or ftping to or from, if either machine has a sniffer on it, then your security is gone.
For example, if I happened to be over at hacked.com, and I telnet back to light and signed on to root, this whole system would be compromised instantly. Thus one has to take great care WHERE one signs onto root from.
The hacker also knows that some system administrators use the same password for their home account as for the root account, which is a VERY BAD practice. So if he gets my home account password, and he knows I am administrator of lightlink because I have a big mouth on usenet, then he is sure to try my home account password on root. (In this case he would have come up with squat).
I estimate that between 1 to 10 percent of all public internet systems have sniffers on them at any one time, and about 50 percent have had one at one time or another.
This estimate includes the many hundreds of thousands of university and corporate systems that we all telnet and ftp to and from daily.
It is probably impossible to fully eradicate the hacker virus from the public access planetary grid.
It is clear from the above analysis that keeping the hacker off the system is impossible, as he can just buy an account. Barring that, he can very easily get access to another's password through various subtrefuges, so he is going to get on to SOME account no matter what if he really wants to.
The problem then becomes one of keeping the hacker out of root.
3.) Keeping the hacker out of root.
This is a multifaceted problem that involves a number of areas of competency that unfortunately I have been woefully lacking in.
First off, there are a number of things that you just don't do as a system administrator. If you do these things, the hacker will have root access in about 10 seconds.
I did one of these things, so end of story. Actually we don't really know if he took advantage of one of my bugaboos, but if he didn't that was his stupidity.
Second of all, there are a number of long standing holes in the SunOS 4.1.4 that people have been aware of for a while. We have fixed some of them, but not all of them. One needs a $2500 contract with Sun to get the patch, which we just ordered a few weeks ago.
The truth is though, I just did not put enough importance on fixing these things, I was putting my attention on getting people on line, and I figured we were relatively safe.
THE HACKER'S MOTTO: "ROOT IS ALWAYS OPEN"
Anyhow, sometimes holes are easy to get into, sometimes they are hard or impossible to get into without a LOT of work and persistence.
A common hole results from something called a race condition. A race condition is when two separate processes on the same machine race together to get at a file.
Say one process wants to read a file and checks that it belongs to the person wanting to read it. There is a millionth of a second between that check for correct permissions and when the process actually goes to read the file.
Say during that millionth of a second another process erases the file that was just checked but not yet opened, and replaces the file with a link to the secure password file. Now the program has full read access to the secure password file because it doesn't see that the file it checked was changed out from under it.
User goober now has access to the encrypted password list, he runs crack on it over a weekend, gets 90 passwords out of 900, and he's in.
But he had to already be in, in order to do this. So really what he wants to do is create a race condition that WRITES to the encrypted password file ADDING a root entry for himself with a known password, so that he can just sign onto root and do what he wants.
Our security guy, John Paine, did this to light when we first started many many moons ago, to show me how weak Sun's original sendmail was that came with our system.
I said "John, here's a free account, go terrorize it. Get me root." He did.
He has handed me root on a platter a number of times.
We hired him permanently shortly afterwards.
Operating systems are FULL of these problems. The reason is that almost every one of the 150 processes that are running simulataneously on our system has to deal with user files or root files and deciding whether it has permission to read or write them.
All these processes are constantly going through this cycle of
1.) who has permission to the file? 2.) reading or writing the file.
This happens many times a second on the machine, and any one of them is open to a cleverly designed race condition if not programmed absolutely perfectly.
It takes intelligence, knowledge, skill and determination to take advantage of these holes, but that's just what hackers are good at.
The various mail handling facilities for example are some of the most heavily used programs on the system. They are CONSTANTLY opening and closing files belonging to all people on the system including root.
These programs are NOTORIOUS for race conditions, and in fact the main sendmail program and its slaves have been ugraded many times just to fix such holes.
But every time it is upgraded, the hackers start scouring the new source code to find yet another bug that no one has even dreamed of before including the writer.
THE HACKERS DO THIS FOR A LIVING! :)
The hacker can install the latest version of sendmail on his linux box, go pouring through the source code, find a potential weakness, and set his system up to run a race condition 10 million times over a week until he figures out exactly how to do it.
It is impossible to write new code for operating systems as complex as these and not introduce new bugs. And these aren't amatures writing this code either, sendmail was written by someone who has been doing this FOR YEARS.
Unfortunately the hackers are usually the first to find those bugs because that's what they do all day long, find bugs in code.
Hackers are professionals at doing this, they are not stupid, they are more like Atomic Physicists trying to bust open the next level sub atomic particle in a 10 giga watt atom smasher. Only they are doing it out of the comfort of their home, and they got all the time in the world on multiple dedicated computers in their bedrooms.
Firewalls are interesting devices. A fire wall is a machine that protects your machine by preventing anyone from accessing it!
A firewall looks at all incoming and outgoing packets, and determines where they are going and where they are coming from, and what they are doing (telnet, ftp, web etc) and rejects any that are not approved.
This allows a corporate office for example to grant access into their local network to remote offices with certain IP numbers, but not to the internet as a whole. It might allow employees to telnet out to the internet but not allow anyone to telnet in to the corporate lan unless they are coming in from an approved IP address.
If the company is really paranoid, they can refuse all telnet and ftp connections from the outside world and only allow e-mail and web connections.
So firewalls are useful, but they work mainly by disallowing access on a selective basis, which is inappropriate for a system designed for broad public access.
Firewalls are not conscious intelligent machines, they can't tell good guys from bad guys. They can however tell where you are coming in from, and if it isn't a trusted location or over an allowed protocol, they can keep you out.
This is in part why our end customers have so little to worry about regarding the security of their own pcs they use to connect to lightlink. They are running CLIENTS but are not running SERVERS.
They ftp and telnet OUT of their personal machines, but they never allow anyone to ftp or telnet INTO their machines. If they did, they would potentially be open to as much trouble as any other server like lightlink.
Basically the ultimate in having a firewall is to not have a machine at all, or if you do, not let anybody on.
One firewall that might have prevented the present security breach at lightlink would have been to not allow shell access to anyone.
If there are no passwords on our machine, and no one can login, then they certainly are never going to get root!
Thus the fastest way to secure root for us is to get rid of shell.
Sounds good to me.
But that ain't going to happen. (More on this later)
So in the long run one has to expect that hackers will get root access no matter how careful or up to date you are, if only because they are the only ones TRYING to get root and are therefore usually the first to do so.
Our policy on this then is that root is always open, and its only a matter of time that someone will find the door and come in.
This does not excuse the absurd set of situations that may have led to the present debacle, but it does mean that I can not make any promises of a safe and secure environment especially as long as we are running shell.
4.) Finding and purging the hacker once he has root.
There are a number of manifestations that occur when a hacker inhabits a system. Hackers can hide their presence in cunning ways, but ultimately he has to change SOMETHING or why be on the system at all?
The main thing the hacker wants is passwords to accounts so he can always have access should he loose his foothold, and he wants to sniff packets so he can find other passwords to remote systems.
His game, like that of a virus, is mainly to inhabit. Beyond that he doesn't care much.
It's like they compare score cards with each other, "Oh you got 20 machines, well I only have 18, we are going to have to do something about this!"
There are only so many ways you can do these things, mainly by corrupting the system binaries that monitor logins and tcp/ip connections.
So the first step in guarding against inhabitation, is to run a check sum on all critical system binaries and put them in a safe place, called a database.
A check sum is a sort of numerical finger print taken from each system binary that uniquely describes the material in it. We have a program for doing just this called Tripwire. If the hacker changes even one byte in the binary, the finger print will not match and tripwire wire will scream.
Then every hour or day, you redo the checksum on the same binaries and compare them against the database, and if anything changes, set off every siren in the house.
Tripwire can't prevent a hacker from coming in, but it will more than likely spot the changes that he makes soon after he comes in. We have had tripwire in fact for a long time, but I just never installed it. Like a number of other things that should have been done a long time ago, it was put off and just not done.
There is one problem to tripwire which is that it is VERY cpu intensive as there are a lot of critical system binaries and it takes a long time to compute the check sum on each one. This means that you can't just run it all day long as there would be no cpu left to serve pages or do mail with! On the other hand if you are going to run it once a day, to compare the present binaries with the stored data base, that means a hacker could be on the system for a whole 24 hours, before he is caught, which is plenty of time to grab everyone's passwords, read their mail, and generally create havoc with their credit cards.
So tripwire is not an end all solution. At best it shortens the window of exposure, but the hacker doesn't need much of a window to get what he needs.
More to follow.
------------------------------------------------------------------------------ Homer Wilson Smith News, Web, Telnet Art Matrix - Lightlink (607) 277-0959 SunOS 4.1.4 Sparc 20 Internet Access, Ithaca NY email@example.com firstname.lastname@example.org http://www.lightlink.com