PCA Forums

HELPPP....

My ISP has been hacked, they are down!! Servers wiped out!

Last week they went down and thought it was hardware problems, they got it back up for a few days maybe a week, now they got trashed again. Everythng gone, hackers left the console up enough to see it was hacked this time and that's all they have. T1's are running but they can't even get to them theirselfs.

Talking about shutting it down permanant now if we don't find a fix by Monday.

What can be done to prevent this from happening again? They can set it back up, but as they say why bother if it will be wiped out again in a few days. I suggested for the time being we set up a system OFF the net and lan as a duplicate system we can use for back up and copy everything for now so if it is wiped out we only have to re-install a backup till we find a fix.
But every-one using the ISP e-mail would constantly lose e-mails that way although the system would be up 90% for surfing still.

I think they were using RedHat Linux on the servers. I still never messed with linux but have Mandrake, about a year old, the free download.

This is a small ISP, maybe 300-500 users, they cannot afford $5,000 software fixes or such.

Why they don't have a backup I don't know!! They will have to set up the system fresh manually, AGAIN.
What would be involved in setting up the system on one of my extra computers and backing up to an image to install onto the severs so we always have a backup? They said something about it being Linux and imaging not working??? I am sure we can, but I am not familar with Linux.

And most iportant of course is how to keep it up. This time they think the hackers were from korea.

Thanks all help needed and apreciated.

That is where I have my office and a Lan connection to the net servers myself. Luckily I have not been using it so my system was off and could not be hacked, I think one of their personal Pcs was running on the lan and got wiped also. I have other options for dialup available but realy want this ISP to stay running!!

you mean dial up ? hmmm what is the company name ?

don't you use a firewall, a good hardware solution. If your servers are making you money, it would behoove you to invest in an excellent solution (Cisco, Firebox, etc...)

Not mine, as in belongs to me, mine as the one I use and also have a lan connection to at the office (if I ever use it).

They have some stuff, not sure what all, but obviousilly it did not work well or was not set up properly.

I just got online again this morning, went to another ISP and set up an account for now. Looking for suggestions to fix that one, and will try to get answers to questions like what they have later tonight.

As for firewalls, all my systems on the net and off have software firewalls installed and running at all times!
Never boot up without one! Even if the only connection is the local home lan

well software firewalls could be the first problem. I never liked them. Hardware just seems so much more rigid when compared to software.

Hardware just seems so much more rigid when compared to software.

Well ya, but we're talking thier system not mine. Mine works fine, I have the software stuff, thiers is down and I don't know what they have at the moment.

Tried to call a little while ago, but no answer. They were working on it earlier, (I think) when I called. But main person i needed to talk to was not there,

Is anyone else thoroughly confused by this post???

Lets start with the basics. I understand your ISP went down. I'm assuming that there are some files, etc, that were hosted by the company that are now gone? I assume you need them hosted elsewhere? Did you not have your own backups? What exactly is the problem here?

FWIW, the operating systems, firewalls, etc, are completely irrelevant if you have really good or really bad IT people. A skilled professional can lock down any box.

Is anyone else thoroughly confused by this post???

Ya me too

Ok, not my ISP as in belonging to me, My ISP as in the one I use. The best one locally, and I have an office there I rarely use that had a lan connection to thier servers. All my stuff works, nothing was turned on when they got hacked.
All I lost is my connection to the net, nothing else. These are friends and they are about to shut down their business which is something I really don't want to see happen.

A skilled professional can lock down any box.

Ya that's what I basically am looking for, answers from a skilled pro how to lock down a linux server and keep these hackers out so my friends can stay in business. This is harder to do since I don't know linux at all or exactly what they have to work with.

Though they are my friends, I must say I am quite suprised they did not have any kind of backups it seems, so I would rank that more on the bad side.

So provided they put the server back online and running on the T1 lines, where and what should we start looking for to lock down?? Obviousily something they have not done twice now!!!

Any sofware we can run on a linux server on our end to find security holes where the hackers could have came in?? Other holes etc...

I am googling my butt off here, but 90% of what I find is garbage and 10% I don't understand. Most of the 10% is interesting but does not apply to the problem anyway.

I'm what you may consider a skilled professional. I know a lot about linux and locking it down. First of all what version of Redhat are they using? You said they don't have money for software so I'm assuming the worst, they have some old version like 9 or maybe if they are lucky one of the server editions but don't have the continuing contract for security upgrades.

If thats the case the first thing I would do is find someone who can manage security updates for you or go back to Redhat and pay for it because there is no other way. Second turn everything off you don't need. Sounds like they are running some sort of mail server. I think redhat uses sendmail and a quick look through Redhats updates, http://rhn.redhat.com/errata/ , to the different versions lists no sendmail vulnerabilities for the sendmail included in Enterprise Linux since version 2.1. After you make sure sendmail is good tripple check that no other port is remotely accessible. I would also take a look and see if the version of the kernel your running has any know remote exploits.

Lastly if this is a sophisticated attacker, doubtful because someone good wouldn't kill the server they'd have much better uses for it, theres very little you can do but be extreamly vigilant. If theres an exploit no one knows but the attacker the best you can do is clean up afterwards. But having all ports closed except for the mail ports would mitigate a lot of that.

How exactly are they connected to the internet so that if this one server goes down the whole thing is not accessable? They may want to use more than one server to do all of this.

PS. I forgot to tell pugsley he's an idiot, what does he thing is running one of those "hardware" firewalls. If he thinks his WRT54GS is any different than some smoothwall box he's wrong. They both run the same software.

Yea i know that i am talking about the software running on the box that its supposed to be protecting. Thats the type of software firewall i dont like. By hardware fire wall i mean a dedicated computer (smoothwall) or a box (linksys) that does nothing but firewall stuffs.

I'm going to disagree a little with rndmtask. *ANY* distro will work fine, provided that you stay on top of security updates. We run all Red Hat servers at work, ranging from Red Hat 9 through FC3 (the free versions). All of them have recompiled kernels, recompiled software, etc, etc, etc with the most current versions available. RPMs are nice, but they can be a PITA to manage on some systems (although tools like yum on Fedora make things easier). They all have iptables firewalls along with file integrity checkers (e.g. Tripwire) that run every night. We also spend an hour every day getting up-to-date on security holes, breaches, etc. and patch the servers as needed.

Your friends need to start with a fresh install. Once a box has been compromised, it cannot be trusted, period, the end. Tell them not to do something stupid like recycle the root password. Apply any/all available security updates *BEFORE* putting it on the network. Like rndmtask said, shutdown any/all services that aren't needed. For the love of all that is good, don't run sendmail!

Of course all of this is irrelevant if your friends can't pay someone to manage security or do it themselves. They may fend off the attackers for now, but you can bet they will be back. Once an IP is known to be comprisable (is that a word? ), the script kiddies and all of their friends will be pounding on that IP for some time trying to get back in.

BTW, what the hell kind of a ****ty ISP only has 1 server???

Skilled professional ? Your initial post was a bit confusing, that's why I suggested a hardware solution.

First, as mentioned, whever was hosting your site should have know to keep up with security, which brings to question, why would they be a host if they could not offer protection. Friends or not, any host worth their salt would keep up with security. Vigilance is the key word. Never taking anything for granted.

I second the "don't use sendmail" statement. it's really unsecure, and can easily become a problem.

Backups...sounds like your covered on this one, luckily you had one.

Well I think I got them covered on the backups IF they get it set backup.

They have me a bit confused also, I just don't uderstand why the entire system was not backed up in the first place. 20 Minutes to re-install an image, then look for holes and change all passwords etc.. and they should have been back online easily and searching for any needed patches or updates. Why no backup??

So if they don't use the send mail, what else is there for e-mail? I told them not to use it, I think they are gonna send users to yahoo and such to get email accounts for now. At least till everything else is solved.

So what besides the obvious can be done, or should we look for to lock this thing down. No send mail, Extremly hard passwords, lock all un-needed ports.

Now what about re-naming directories or moving files to unknown places? Somthing i am thinking here is if the hackers can't find it they can't hack it, to a point. So if required files are moved to a new directory that is unknown to a hacker, they can't just delete /bin if say it was renamed to somthing like /nib .
Of course the server has to find anything it needs,

What I been reading up on for linux is somethings can be moved or renamed ok. Any suggestions on this that might help.
Not knowing myself what they did or how they got in for sure means I don't have a clue what to fix either.
So, might as well go for broke and do everything posible right!!

So in order to hack the system, did they somehow have to get root access? And run remotely. Can we somehow lock the system directories to prevent remote access? Not sure what all HTaccess does, is it any good for somethng like this or just concerning websites? I know I used it for my sites, but I never messed with a server.

By the way, when I have a site online I go with the bigger guys, same with e-mail. Too many small Isps go down for good so I don't rely on them for anything except net access myself.
It's been a couple years since I even built a website from scratch, so I have forgot most of that even now

Originally posted by canton_kid
20 Minutes to re-install an image, then look for holes and change all passwords etc.. and they should have been back online easily and searching for any needed patches or updates. Why no backup??

Most backup imaging software (e.g. Ghost) don't handle journaling file systems (e.g. ext3) well. Images are not a viable solution on most linux platforms. And even if they DID have an image, as soon as the box goes back online, it would get broken into again (read: you don't have time to find the holes after it is online, they need to be found before hand).

Originally posted by canton_kid
So if they don't use the send mail, what else is there for e-mail?

Qmail, Postfix, Exim, Courier.....

Originally posted by canton_kid
Now what about re-naming directories or moving files to unknown places? Somthing i am thinking here is if the hackers can't find it they can't hack it, to a point. So if required files are moved to a new directory that is unknown to a hacker, they can't just delete /bin if say it was renamed to somthing like /nib .
Of course the server has to find anything it needs,

OK, I don't want you to take this the wrong way, but you need to hire someone to handle security for you. This is one of the dumbest things I've read in this thread. Lets put aside the facts that 1) you would be breaking every *nix standard in the books, and 2) Lots of (admittedly poorly-written) software that looks for files in a specific locations would suddenly break. Setting these items aside, you are still left with the fact that any cracker worth a darn would find these files in less than 5 minutes. Of course, he might then die of laughter at this "security measure" thus ending the attack.

Originally posted by canton_kid
Not knowing myself what they did or how they got in for sure means I don't have a clue what to fix either.

That's why there are log files. Read them.

Originally posted by canton_kid
So in order to hack the system, did they somehow have to get root access? And run remotely.

Not neccesarily. Depends completely on the type of exploit that was used to gain access to the server.

Originally posted by canton_kid
Can we somehow lock the system directories to prevent remote access? Not sure what all HTaccess does, is it any good for somethng like this or just concerning websites? I know I used it for my sites, but I never messed with a server.

hame Seriously: HIRE SOMEONE.

Can I ask why you think this box has broken into at all? Are you sure this isn't just a hardware failure or something easier?

" Seriously: HIRE SOMEONE.

Can I ask why you think this box has broken into at all? Are you sure this isn't just a hardware failure or something easier?"

Well the first time they thought the hard drive crashed, from what I was told this second time I think they found an Korean IP for a hack attack, the console comes up enough that they said it shows they were hacked. Kinda "I was here and now your not" message maybe from the hackers!

It's not just down not working, the console comes up to a certain point and they are locked out or whatever, it does nothing from that point and they could not get into it.

If this were my system and I owned the place I could do what I want, since it's not I can only try to help them anyway I can. So I am shooting in the dark here looking for about everything to do or try to fix this.
Maybe things that are not just obvious lock the ports answers.

My big concern here is that I do not want them to go out of business for alot of reasons. They been just fine for many years untill this last 2 weeks, So I geuss in general they know a bit of what they are doing really. But they went down 2 times in a little over a week now. This time they are still down, been days.


As for the thought on moving or re-naming things I had mentioned, was a crap shoot question. But yes I have done similar things in the past when I had problems with local networks and users getting into things they should not. Course that was MS O/S totally different and they were amatures also.
In that case I had 2 O/s folders installed. I installed to something like C:\Canton for the actaull operationing system, then copied it to C:\windows as a dummy copy. Do anything you wanted to C:\windows and the files in it or subdiriectories and it did not effect the system because the system was running from c:\canton files. So it would look like you did something to the system if you changed files, added, or deleted files, but in reality you only messed with one unused folder that did nothing. Course I did have to carefully install programs that looked for c:\windows and change them to look at c:\canton instead durring install, and change alot of .cfg files to look in the correct place also.
I suppose in that methode, if a person was hacking a program, the program would find the real O/S but when just hacking the hard drive files directly nothing really happened.

Doing the above and changing things to read only and also hidden attributes worked then. If an amature sees what he is looking for right where it belongs then they may not think to look for a read ony hidden copy of the same thing in a different folder. Course there are many things you cannot do that with. Reasons I asked!

Far as I know maybe this is a script kiddy that knows nothing either?

As for hiring someone, well around this area there is no-one to hire in peson I am sure, so it would have to be remotely worked.
And I think it was mentioned, if put online with the same problems, it will probably be hacked again before you could fix it from your end!

SO, what would it cost for a pro to look this over remotely and fix it if that could be done? Not my system, but I could relay a message.

Pm's welcome