I reported yesterday (http://www.webhostingtalk.com/showthread.php?t=729230) that our recent downtime was due to issues with our backup servers followed by the corruption of some db tables from a hack attempt.

We've since learned that this very deliberate, sophisticated and calculated hack against Web Hosting Talk was carried out by gaining access to our offsite backup servers. From our backup servers, the hacker gained access to the WHT db server. The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.

This individual is still in possession of our user table that includes all user names, email addresses and hashed passwords. Absolutely no credit card or PayPal data was compromised.

Passwords are hashed with salt. It would be an unprecedented event to reverse engineer our passwords. I change my password periodically though, so maybe today is a good day for that. Go here (http://www.webhostingtalk.com/profile.php?do=editpassword) to change your password.

My concern is the distribution of your email addresses and the potential spam you may receive. We know the hacker has posted the user table containing email addresses to various places (file sharing sites) and we're working diligently to remove the tables as we find them. If you see the user table posted anywhere, please let us know so we can get it taken off line.

We are working on recovering the deleted data. In the meantime, we've restored to an old db. We cannot yet determine if we can restore to a more recent db backup.

If you have any clues as to the individual who caused this malicious attack on the Web Hosting Talk community, please let me know.

At least it's back, I guess. I've only lost 800 posts and countless topics of interest to me... :(

Good luck :)!

I saw the uploads that you are referring to, I wanted to see how much of my information was there and it's 5400+ pages of account information but only usernames/e-mails/hashed passwords + salt.

Luckily I use a secondary address for forum notifications so I can set it to :blackhole: and just create a new forwarder.

My personal advise is that *EVERYBODY* change their passwords.

My personal advise is that *EVERYBODY* change their passwords.

My personal advice is that WHT should secure their stuff properly and not just backup to one location.

It's ridiculous!

My personal advice is that WHT should secure their stuff properly and not just backup to one location.

It's ridiculous!

What has been done, is done - and hopefully it will be a learning experience.

Ouchie. Best of luck.

What has been done, is done - and hopefully it will be a learning experience.

oh come on, you're not serious, right? You're comfortable knowing that there's hundreds/thousands of people sitting in front of their computers with a copy of your password, and every other members? I know I'm not.

Wow, this is disappointing. I hope the lost data can be recovered some how and that you have some luck limiting the distribution of all our email addresses. Major blow to WHT.

Good luck.

My personal advice is that WHT should secure their stuff properly and not just backup to one location.

It's ridiculous!

and how many different backup locations do you use?

Saying "this is unforgivable" may sound too hard. But it really is. WebHostingTalk, a place where we often read "make backup of backup" got hacked and lost their only backup. Great.

My personal advice is that WHT should secure their stuff properly and not just backup to one location.

It's ridiculous!

I hate be like this but I agree.

WHT has has issues like this before if I member correctly.

So now I could be spammed great.

Password changed.

I'm curious as to how they got into the backup server? software, password, or other exploit?

What has been done, is done - and hopefully it will be a learning experience.

Mike is right but I'm still furious that this happened.

I understand people can get hacked, problems happen. But i would figure there would be at least two back up servers for the forum. Seeing as the forum has been DDoSS or attacked before if I remember correctly.

I know this is no ones fault. But steps need to be taken so this doesn't happen again.

I hate to sound like a whinner but this could happen again.

This is serious breach of security.

I've received about 5 spam e-mails today, I hope it isn't due to this.

THE BEST THING YOU CAN DO DENNIS IS CHECK THE IP LOGS AND FIND OUT WHO DID THIS AND GO FROM THERE!!

Go back thru EVERY IP UNTIL YOU GET TO THE SCUMBAG WHO DID THIS!! (Its not impossible my friend)

Good luck!

oh come on, you're not serious, right? You're comfortable knowing that there's hundreds/thousands of people sitting in front of their computers with a copy of your password, and every other members? I know I'm not.

Welcome to the Internet. :cool:

There's really no reason to make a huge issue out of this. Simply change your password(s) and move on.

I wonder how people find time to do such things and for what reason.
Chickens.

I get spammed every day, these things unfortunately do happen.

Hopefully wht will learn from this, and take any action required.

lol, can we just purge the entire forum? 90% of this crap is outdated anyways :D

Does the DB include a copy of our PM's etc?

I could not log in with the password I know was set as it was saved in firefox. Well I was able to log in after using the recovery thing.
So I now have a new password.
I also have a new password for almost every thing else.

Oh wow, I never thought about PM's... likely some extremely sensitive info being exchanged.

I wonder how people find time to do such things and for what reason.
Chickens.

I'm guessing either spite or profit. Either way it sucks for us.

Hopefully wht will learn from this, and take any action required.What action??

This is a stupid hacker with NO LIFE,you cant predict what they might do ESPECIALLY IF THEY THINK THEY ARE UNSTOPPABLE...

The truth is: THEY ARE NOT.. IF ENOUGH TIME WAS DEVOTED,THIER IP CAN BE TRACKED DOWN!! (Logs,etc) People just dont seem to care enough to track anyone down and its sad...... (I HOPE DENNIS WILL TAKE MY ADVICE AND TRY)

and how many different backup locations do you use?

Three, thanks for asking.

What action??

This is a stupid hacker with NO LIFE,you cant predict what they might do ESPECIALLY IF THEY THINK THEY ARE UNSTOPPABLE...

The truth is: THEY ARE NOT.. IF ENOUGH TIME WAS DEVOTED,THIER IP CAN BE TRACKED DOWN!! (Logs,etc) People just dont seem to care enough to track anyone down and its sad...... (I HOPE DENNIS WILL TAKE MY ADVICE AND TRY)






No need to shout friend

I'm just guessing here, but any hacker worth their salt probably at a minimum uses a chain of proxy addresses so they can't be tracked. I'm sure other methods were used as well.

The real question is how the heck did they get in?

Thats where the real question lies.

No need to shout friend

I'm just guessing here but any hacker worth their salt probably at a minimum uses a chain of proxy addresses so they can't be tracked and I'm other methods.

The real question is how the heck did they get in?

Thats where the real question lies.
I agree with you.

What has been done, is done - and hopefully it will be a learning experience.

The thing is, this isn't the first time that WHT has been compromised. Remember them having C99 on their site? ;)

Having one back up server is a pretty big mistake. Figured that having their one and only backup server going offline at the same time was a bit strange.

There's really no reason to make a huge issue out of this. Simply change your password(s) and move on.
Password compromise is unfortunately commonplace. I've had my ATM card replaced twice in a year's time because their database was compromised.

The bigger issue, of course, is the massive data loss that has apparently occurred and the fact that a sophisticated hacker could take out a million-dollar business.

I think WHT will be able to pick up the pieces but their credibility is definitely taking a huge hit from this - in my estimate anyway. This kind of thing can shutter a business quite easily. If they can get their data back - great - but if they find they just have to roll back 7 months - we're all going to be scratching our heads and wondering if this is the best place to do business. Well, maybe I'll be the only one, but I doubt it.

WHT/iNET will take away a lot of tough lessons from this issue I'm sure. I can't imagine the post-mortem will be pretty at all.

In any case thanks for the updates.

I guess the backup should be locked out of public access and via private VPN to access it. I guess RackSpace can arrange this ? =)

Well, thanks for the hard work to bring this back online.

Great this is the second time a major forum ive been on has been hacked and user tables distributed e.g recent phpbb.com hack fiasco

As for the loss of data im not bothered too much regarding the number of posts ive made - while i do think this does provide 'rep' on forums like this im more annoyed over the actual content lost, wht is like a web hosting encyclopaedia and alot of people effort has just been wasted building up this knowledge bank.

As for the backup situation inet and rackspace should be ashamed of themselves really only having one on site backup server. WHT preaches about having backup procedures but yet it has a crap one of that and rackspace are supposed to be managed specialist why didnt they recommend more than one server or a off site server, would have been a easy $$$ for them knowing Inet can obviously afford it

Hopefully the advertisers will be credited for the downtime, funny thing is i discovered the downtime by coming hereto buy premium membership but now im not too sure i want it now,knowing that some retard has all my user details

---Edit---
Just read some post while making mine regarding the pm tables compromised? if so that is unforgivable the amount of sensitive data stored there isnt worth thinnking about

I'm just glad you're back. Happily, I use gmail for all forums etc. so the spam doesn't matter. I'd never sign up to a forum with one of my domains' addresses.

Whats going on with those of us that have become premium after this backup?

I've seen the files too, and the password hashes.

vBulletin uses a sophisticated hashing algorithm, it uses md5 to hash the passwords once, then adds a salt next to it, and hashes again.

Although the stolen info has these hashes, it is absolutely impossible to recover your passwords from these hashes. I haven't changed my password (really) and I see no reason to do so.

However, the stolen table makes your passwords vulnerable to dictionary attacks. If your password is a dictionary word (one word) or a simple series of numbers (like 654321), then your passwords are somewhat vulnerable.

Still, the salting mechanism in vBulletin adds 2^18 = 262144 times more difficulty to a general dictionary based attack, so the chances of your passwords being revealed is extremely low.

For example, the stolen info about MY account is this:
Password: 3248b2676776395e4336b32b862f1301 Salt: "%M

My password is a complex one (with numbers, capitals and punctuations), it is first hashed to some large string, then the salt added, then hashed again.

I have no worry revealing my stolen password details here, if any have the stolen data, you can easily publicly verify that the information I posted here is true. I can post my own details here because there is no way a hacker can break it with the current amount of technology he might possibly have (even a cluster of PS3s).

However as I said, dictionary words have a higher chance of being cracked, but this possibility always exists with weak passwords, stolen or not.

As for the email addresses, yes they are revealed too, but I already get a lot of spam everyday and I don't think I'll be effected too much :)

And as for the recent posts and information, I share your concerns and I hope some data recovery company can recover the data in the corrupted servers.

Well, I've lost my Premium Membership, posts & changed username :(

I do not know vbulletin that well, but my guess is that only way to get passwords is to test them against some dictionary / common password list and then compare hashes to hashes stored in the database. In short, if you think your password might appear on such list at this moment someone already knows what it is and they have your email address too, so if you used the same password for anything else, it is time to start panicking.

<edit>Ok, I was right, but too late it seems :) </edit>

I'm just guessing here, but any hacker worth their salt probably at a minimum uses a chain of proxy addresses so they can't be tracked.Yes well you go thru EVERY IP YOU GET and continue until you get his.. (It can be done)

Yes well you go thru EVERY IP YOU GET and continue until you get his.. (It can be done)






Good luck with that even if they do log them. ;)

What action??

I meant they can try to find out how they did it, close the vulnerability, and possibly take other measures.

Oh, and try to find the lame/useless/layabout/scum* who did this.

*Choose the most apropriate.

FOOTNOTE:

I would definetly change your password as a precaution, and if you use the same password elsewhere change that as well.

Better safe than sorry:mad:

I've seen the files too, and the password hashes.

vBulletin uses a sophisticated hashing algorithm, it uses md5 to hash the passwords once, then adds a salt next to it, and hashes again.

Although the stolen info has these hashes, it is absolutely impossible to recover your passwords from these hashes. I haven't changed my password (really) and I see no reason to do so.

However, the stolen table makes your passwords vulnerable to dictionary attacks. If your password is a dictionary word (one word) or a simple series of numbers (like 654321), then your passwords are somewhat vulnerable.

Still, the salting mechanism in vBulletin adds 2^18 = 262144 times more difficulty to a general dictionary based attack, so the chances of your passwords being revealed is extremely low.

For example, the stolen info about MY account is this:
Password: 3248b2676776395e4336b32b862f1301 Salt: "%M

My password is a complex one (with numbers, capitals and punctuations), it is first hashed to some large string, then the salt added, then hashed again.

I have no worry revealing my stolen password details here, if any have the stolen data, you can easily publicly verify that the information I posted here is true. I can post my own details here because there is no way a hacker can break it with the current amount of technology he might possibly have (even a cluster of PS3s).

However as I said, dictionary words have a higher chance of being cracked, but this possibility always exists with weak passwords, stolen or not.

As for the email addresses, yes they are revealed too, but I already get a lot of spam everyday and I don't think I'll be effected too much :)

And as for the recent posts and information, I share your concerns and I hope some data recovery company can recover the data in the corrupted servers.


I used a random password generator.

So, the last offsite backup was made in October 2008?

Its really sad. WHT is a great place, it should have more security and hope things should be up fast.

I lost around 130 posts and my premium membership. :(

With a forum this size, and considering it's primary purpose is web hosting talk (dun dun) I would have thought the people in charge would keep at least a monthly offsite offnetwork backup of everything.

If that's not the case, then I hope you start doing this now. A backup server is fine for daily backups etc, but having your files secured where no one can reach them every once in a while is essential for anything of importance you put online.

So, the last offsite backup was made in October 2008?

No apparently the backups where deleted.

Yes well you go thru EVERY IP YOU GET and continue until you get his.. (It can be done)


Ok, so you are an english speaking administrator and your server has been compromised.

Now you have to:

- Contact the first bounced IP's ISP in Japan (have fun)
- They say the connection came from an ISP in Africa (have fun)
- Well that connection came from Israel (have fun)
- Well that came from Germany..you get the idea...

and this can go on for as many IPs as he bounced through. Have fun dealing with those ISPs that are in jurisdictions that do not care about north american law.

No apparently the backups where deleted.

Sorry. Perhaps wasn't clear.

The last "backup of backup" was made on October 14 2008.

Also to people calling someone who pulled off a hack like this a person with "no life" is absolutely retarded.

If the guy can take down a huge site and/or buisness, what makes you think that a huge buisness would not employ him for a good salary to handle security? They are obviously (and moreso) up to date on the latest security flaws and strategies, and also have access to exploits distributed amongst the underworld.

Most of the "top hackers" dont go to jail, they get good jobs instead.

I'd really like to know about the PM issue.

Have our private messages been compromised? There's a lot of sensitive data there....

I am wondering.. does this mean all those who had signup recently have lost their accounts as well? Just a bit curious..

Atleast we are lucky to have our usernames and few posts saved.. Think about those who lost their accounts!

I'd really like to know about the PM issue.

Have our private messages been compromised? There's a lot of sensitive data there....

user table was taken, nobody knows if the posts/pm etc db was touched, there could be a dump somewhere.

I'd really like to know about the PM issue.

Have our private messages been compromised? There's a lot of sensitive data there....

Unless the guy is too good not to poke in there, they're NOT safe.

Yes recently registered accounts were lost too. However the work on recovering the backups from corrupted servers is on the way, it make take a while and cost some money, but we all know it is possible, unless the hacker wiped the disk clean, which would possibly take days, and someone would have noticed it befor it's complete.

Of course if anyone is interested there are some jobs still open from 2003
in the employments forum.

I'd really like to know about the PM issue.

Have our private messages been compromised? There's a lot of sensitive data there....

Can't even say about that

Does the DB include a copy of our PM's etc?The tables affected include user, post and thread. Your PMs should be current.

The real question is how the heck did they get in?As I mentioned, they gained access via our db servers that they wiped clean (well, maybe I wasn't clear on that). Hopefully forensics can pull enough information off to clear that bit up though.

The tables affected include user, post and thread. Your PMs should be current.

I think people are asking was the pm table taken, not corrupted?

The tables affected include user, post and thread. Your PMs should be current.


Where not asking if there current where asking weather the hacker has a dump of this data

The tables affected include user, post and thread. Your PMs should be current.

They are current but they might've been stolen.

Good luck in catching the idiot(s) who did this - there are so many pathetic morons around.

To clear one thing up - change all passwords which match your WHT one, and obviously change your WHT one too. The fact that they are encrypted with a salt means little since md5 was broken a few years back (I'm surprised vBulletin still use it). It doesn't mean that a hacker can get the plain text (i.e. your actual password) out of the hash, although this has other implications.

Ok, so you are an english speaking administrator and your server has been compromised.

Now you have to:

- Contact the first bounced IP's ISP in Japan (have fun)
- They say the connection came from an ISP in Africa (have fun)
- Well that connection came from Israel (have fun)
- Well that came from Germany..you get the idea...

and this can go on for as many IPs as he bounced through. Have fun dealing with those ISPs that are in jurisdictions that do not care about north american law.

Even then with how a lot of proxies work, how do you plan on tracking down the exact user when they might only have encrypted addresses stored in the logs? It'd be quite a feat to get to the end of things. Heck, they might not have even been using their own net connection to begin with. Wireless? Pinched cable? Easy enough.

Let's hope WHT secure themselves and don't have a third (that I'm aware of) serious security breach too soon

Ive never understood why hackers once they found a exploit f*ck around with it, id email inet with sensitive data retrieved through the exploit then making them pay me to let them know where it is.

Hackers are dumb :dunce:

Ive never understood why hackers once they found a exploit f*ck around with it, id email inet with sensitive data retrieved through the exploit then making them pay me to let them know where it is.

Hackers are dumb :dunce:

That's not how it'd work, they'd email iNet and be told to **** off, then if something happens inet know exactly who it was who leaked the info.

Where not asking if there current where asking weather the hacker has a dump of this data

The PM table has around 1,400,000 entries, averaging at 500 bytes per message. This would make the total dump 667 MBs of data.

The reaction time to the hack attempt was short enough to prevent generation of such a large dump, and prevent it from being downloaded it.

By comparison, the stolen user table was only 20 MBs in size. And the stolne version was a stripped down version of the user table, to decrease the size.

So, it's safe to assume the PMs were not stolen in bulk (unless they targeted specific PMs). But looking at the damage done, the hacker was not insterested in a targeted hack, but was interested a bulk clean up. So I would assume the PMs are safe, they are too much to steal.

And by thw way, what the heck do you share on PMs? Root passwords, credit card numbers? Most important data I transmit is my paypal address, which the hacker can use to make some donation :P

Whats going on with those of us that have become premium after this backup?Sales will be going through receipts and contacting those members.

I don't know how you stored the backups, but it shouldn't be much of an issue to recover the deleted data if the hacker didn't do too many overwrites. I'm sure you already have a data recovery company on the job though.

By comparison, the stolen user table was only 20 MBs in size. And the stolne version was a stripped down version of the user table, to decrease the size.


That's because it was stripped down to only username, salt, email and pw, in reality I'm sure the table is at least 10x that size. The hacker managed to destroy the backups and take a dump of the user table, what's to say they didn't dump the entire DB, but only posted the user table publicly?

The scariest part of all of this is we saw a noticable drop in the traffic to our website. Time to diversify my forum whoring :)

(It can be done)




If there's retrievable info on the drive. ;)

hmm...

good luck wht

you have my support as always.

owm

Hi, what about the off site tape backups, say weekly means you might loose only a week data.

Also i wonder how exactly a backup server can be known to a outsider person and initiate a hack attempt from that host unless he is insider or friend to insider.

I am wondering.. does this mean all those who had signup recently have lost their accounts as well?
If they signed up after October, of course they haven't signed up. ;)

oh come on, you're not serious, right? You're comfortable knowing that there's hundreds/thousands of people sitting in front of their computers with a copy of your password, and every other members? I know I'm not.
What scares me more than a double-hashed password of mine being out there (a password I use in only one place and change regularly IMHO) is the fact that if they had full access to the DB servers they *very* easily could have dumped every table and gotten PM's, Premium Member PayPal addresses, etc... There are surely root passwords in PMs from other members - why on earth you would send a root PW in a PM is beyond me but I know a few individuals I speak with on WHT and outside of WHT have sent this information.

This could end up being catastrophic for iNet/WebHostingTalk if the hackers actually dumped more confidential information - I hope that it's not going to be any larger an issue than it already is but if businesses begin to be compromised due to this and damages caused iNet could find themselves in some *very* hot water very quickly.

Does the DB include a copy of our PM's etc?They most certainly had access to the DB server so they theoretically could have dumped any tables they wanted - I wouldn't hold my breath in saying that it it wasn't copied.

If you passed any sensitive information via PMs or other means on the forum I recommend anybody who did so to *change* the passwords or to take any other necessary security actions as soon as possible.

Is this a attack on VB.. some 0day exploit?

lol each time i post my posts get edited.. ???

Is this a attack on VB.. some 0day exploit?

lol each time i post my posts get edited.. ???

It was a direct hack on the database servers, not a vBulletin exploit.

Hi, what about the off site tape backups, say weekly means you might loose only a week data.

Also i wonder how exactly a backup server can be known to a outsider person and initiate a hack attempt from that host unless he is insider or friend to insider.

Thats a good point how would a outside hacker find your database server. Im sure you cant publicly access it through protocols such as http. unless they did extensive network scanning and sniffing

It was a direct hack on the database servers, not a vBulletin exploit.

Thought so.. just checking.. thanks :)

This is very said.... not the hack but the backup part....

The hack can happen to anyone but not having backups and making it securely it's a BIG mistake....

Mable it's a fall that needed to be taken to learn the lesson.


Any way good luck guys!

PS: Mind telling us how heavy is wht (backup wise curios)?

Thats a good point how would a outside hacker find your database server. Im sure you cant publicly access it through protocols such as http. unless they did extensive network scanning and sniffingOr the security wasn't as good as it should have been (i.e. publicly available DB servers). Who knows, it's pure speculation at this point and I'm sure WebHostingTalk isn't going to publicly admit to any fault - it's not a wise business decision to do so.

Thought so.. just checking.. thanks :)
Sure :)

My question is, if you flick your mind back to the password reset it was never answered weather inet wanted to steal our passwords or if someone accessed the server and changed login.php. I guess we have the answer to this now but there are some serious questions inet need to answer for themselves,

What was actually done when the page was first compromised back then, from here it just sounds like it was brushed off.

How on earth was someone able to access your database and backup systems, why are these even accessible to the public internet?

What is the point in me even typing more, frankly you didn't take the first warnings seriously enough and you are solely to blame for this compromise. That is all there is to it.

O no bunny infractions are back !! Opening a ticket now, what did bunny do to deserve this :(

My question is, if you flick your mind back to the password reset it was never answered weather inet wanted to steal our passwords or if someone accessed the server and changed login.php. I guess we have the answer to this now but there are some serious questions inet need to answer for themselves,

What was actually done when the page was first compromised back then, from here it just sounds like it was brushed off.I don't know that you'll get any useful information.

How on earth was someone able to access your database and backup systems, why are these even accessible to the public internet?My guess is that the webserver/s was/were compromised and then used to access the backup servers. This would have allowed the hacker access to the backup servers over private lan and allowed them to do what they needed.

What is the point in me even typing more, frankly you didn't take the first warnings seriously enough and you are solely to blame for this compromise. That is all there is to it.Who knows, this could be an entirely different attack of an entirely different manner. If it is the same thing then perhaps - but I'm sure nobody is going to say it is.

I don't know that you'll get any useful information.

My guess is that the webserver/s was/were compromised and then used to access the backup servers. This would have allowed the hacker access to the backup servers over private lan and allowed them to do what they needed.

Who knows, this could be an entirely different attack of an entirely different manner. If it is the same thing then perhaps - but I'm sure nobody is going to say it is.

We will never know but can make most likely valid assumptions, given the way these "incidents" have been handled in the past no doubt some half truth story will come out and blame everyone and everything else.

Also to people calling someone who pulled off a hack like this a person with "no life" is absolutely retarded.

If the guy can take down a huge site and/or buisness, what makes you think that a huge buisness would not employ him for a good salary to handle security? They are obviously (and moreso) up to date on the latest security flaws and strategies, and also have access to exploits distributed amongst the underworld.

Most of the "top hackers" dont go to jail, they get good jobs instead.

It's because of this that people have no respect our morals anymore.

I will place my bet on that girl that found the Intel bug..... (That hacked wht)

Also I would place the image verification stuff everywhere in WHT (search/login/register and etc) to stop the automated vul. searches.


Good luck and hope the person that did this get's little punished like Kevin did in the 90’s.

O no bunny infractions are back !! Opening a ticket now, what did bunny do to deserve this :(

you left the top secret high security back door open to the all important backups naughty bunny heres a infraction

My question is, if you flick your mind back to the password reset it was never answered weather inet wanted to steal our passwords or if someone accessed the server and changed login.php. I guess we have the answer to this now but there are some serious questions inet need to answer for themselves,


Didn't that happen at the same time that Google Cache helpfully showed that WHT had C99 (shell script) on the forum? Goodness knows what was compromised that time.

We will never know but can make most likely valid assumptions
Well, you know what they say about assumptions ...

And you're right. I'm not going to address your posts. You obviously didn't read mine.

I have no qualms about stating exactly what happened. But the truth is, we may never know.

Of course, we hope we can get the information off the drives. But I'm certainly not going to state that we will. And without that information, I can't state that we'll post how someone gained access, because it's possible we won't know.

Now move on with the conspiracy theories, please.

Any plans to update vB if it's not already the latest?

Of course, we hope we can get the information off the drives. But I'm certainly not going to state that we will. And without that information, I can't state that we'll post how someone gained access, because it's possible we won't know.That in and of itself is a very dangerous statement to make because if you don't know how it happened, you don't know how to prevent it.

Well, you know what they say about assumptions ...

And you're right. I'm not going to address your posts. You obviously didn't read mine.

I have no qualms about stating exactly what happened. But the truth is, we may never know.

Of course, we hope we can get the information off the drives. But I'm certainly not going to state that we will. And without that information, I can't state that we'll post how someone gained access, because it's possible we won't know.

Now move on with the conspiracy theories, please.
I've read all of this thread (and the 17 page monster before, the one which the idiot hacker posted in), however I'm still a little unclear on the following (sorry if I missed it):

*If* the data in the backup server cannot be salvaged for whatever reason (which really wouldn't be your fault), would WHT stay as it is currently (i.e. at an October 2008 revision on many things)?

Well, you know what they say about assumptions ...

And you're right. I'm not going to address your posts. You obviously didn't read mine.

I have no qualms about stating exactly what happened. But the truth is, we may never know.

Of course, we hope we can get the information off the drives. But I'm certainly not going to state that we will. And without that information, I can't state that we'll post how someone gained access, because it's possible we won't know.

Now move on with the conspiracy theories, please.

All I can say is this can not happen again.

I understand the staff is doing there best to fix things as quickly and completely as possible.

But really I don't think many will tolerate another issues like this.

I know if I was new member I might leave because of this.

I'm not of course. I know stuff can happen.

But i'm just repeating myself.

The negative feedback here is a waste of time. WHT is more aware of the **** up then anyone else since its happened to them. If you not here to offer help then why bother posting? We get it, your upset, get over it and help fix things.

Site guy whoever you are, its a long shot but see if you can retrieve anything using tools like gpart you might get lucky. Do it on the backup server as well. Tracing them...you'll have more luck telling us where the dbase has been posted even more luck finding them if you can tell us where it was posted first. Someone will be bragging about this sooner or later but even if you do find them its not gonna help much.
In case you don't already do this... make a simple local backup nightly as well holding back 7 days if you can. Remove any trusted ssh keys from the backup server to this one unless they are totally nessessary and lastly... think about hiding this server from both the backup server and the rest of the world it will give you an extra layer of protection and you no doubt daily hacker wannabe attempts a dud host to exploit that holds no information whatsoever of any importance. Also try not to use remote sql backup software as this often requires too much access level to complete its jobs, rather do em locally and have the backup pick them up securely with no relationship to sql.

For everyone it's easy to criticize someone else's security until it happens to you and until it does you have no idea how vulnerable you really are. Remember, if someone wants to hack you and they are good enough... they will hack you eventually. After all, isnt that how we progress in the field of security..

If theres anything you need at WHT, help or whatever, im available and no this aint a sales pitch lol i just want to see your vast database restored to its former glory it would be a shame to permanently loose all that data.

I am happy to see WHT back online (again)

But it is really depressing to see that WHT has backed up DB using such an old backup. (I mean, com’on; Octoer 2008 :confused:) I am sure it’s not just the post count that others are complaining about; we have lost so many valuable and informative threads.

I still hope that iNet staff will be able to recover most (if not all) data.

That in and of itself is a very dangerous statement to make because if you don't know how it happened, you don't know how to prevent it.

Agreed it doesn't sound good.

I would argue its time for a complete review of the setup WHT uses.



The negative feedback here is a waste of time. WHT is more aware of the **** up then anyone else since its happened to them. If you not here to offer help then why bother posting? We get it, your upset, get over it and help fix things.

Site guy whoever you are, its a long shot but see if you can retrieve anything using tools like gpart you might get lucky. Do it on the backup server as well. Tracing them...you'll have more luck telling us where the dbase has been posted even more luck finding them if you can tell us where it was posted first. Someone will be bragging about this sooner or later but even if you do find them its not gonna help much.
In case you don't already do this... make a simple local backup nightly as well holding back 7 days if you can. Remove any trusted ssh keys from the backup server to this one unless they are totally nessessary and lastly... think about hiding this server from both the backup server and the rest of the world it will give you an extra layer of protection and you no doubt daily hacker wannabe attempts a dud host to exploit that holds no information whatsoever of any importance. Also try not to use remote sql backup software as this often requires too much access level to complete its jobs, rather do em locally and have the backup pick them up securely with no relationship to sql.

For everyone it's easy to criticize someone else's security until it happens to you and until it does you have no idea how vulnerable you really are. Remember, if someone wants to hack you and they are good enough... they will hack you eventually. After all, isnt that how we progress in the field of security..

If theres anything you need at WHT, help or whatever, im available and no this aint a sales pitch lol i just want to see your vast database restored to its former glory it would be a shame to permanently loose all that data.

With all do respect. WHT should not have this kind of issue. They have more than enough resources at their disposal.

Rack space and iNet should be able to prevent issues like this from happening. Or at least have better ways to deal with it. Its obvious one back up server isn't enough. Among various other problems.

This is at least the second time WHT has been hacked. I think we all know that WHT is targeted. We can assume that partly from he use of proxy shield.

This is most defiantly a big black eye to the forum.

I guess iNet is working to make sure it doesn't happen to any of their other sites as well now, since I would imagine some of them use similar setups :).

I noticed a lot of people complaining about their post counts and a lot of people telling them that it's merely a number and there's more to WHT. But, it's not the number that mattered, it's the information that was posted and is now lost. I've lost almost 20+ reviews...

I wish all the luck to the WHT staff in getting things restored because there was a lot of valuable information lost. Not just the post number...

If I ran a company that has as much earning power as WHT, I would make sure to invest a good amount into state of the art security and backup solutions.

Dear Dennis,

Take a look into /var/logs may be you can get some informative information there. I know you have good team but if you need assistance do let me know.

Best Regards.

I noticed a lot of people complaining about their post counts and a lot of people telling them that it's merely a number and there's more to WHT. But, it's not the number that mattered, it's the information that was posted and is now lost. I've lost almost 20+ reviews...

I wish all the luck to the WHT staff in getting things restored because there was a lot of valuable information lost. Not just the post number...

I'd like to know why WHT is still running vB version 3.6?
Also, why were we able to recover a version 6 months old, and not a version 1 week old?

If I ran a company that has as much earning power as WHT, I would make sure to invest a good amount into state of the art security and backup solutions.

I agree on the VB issues.

As for the security I guess the standards put in place weren't good enough. I'm sure they where expensive knowing Rackspace.

If a "hacker" (I dont think that that term qualifies here!) doesnt delete the logs after cleaning the house he/she got to be quite retarded...

Good luck to the site team to recover this forum!
Sh*t happends, unfortunately. I just hope I dont get swamped with spam...


Sincerely,
- Liroy

I can't believe there was no recent local backup of the database. :eek: Hope you can recover most of the data.

This surely reflects negatively against RackSpace. Nobody knows exactly what Rackspace provide iNet with (if anything? I don't know) but having that lovely "powered by" icon now doesn't look so good.

I'm wondering where the 6 month backup was stored and why there couldn't have been at least a monthly backup along with it.

This is all very worrying, alot of information lost and the company image has a big dent in it. When was that last hacking attempt, you know when we had to change our passwords all of a sudden? Not so long ago!

Ah well, it doesn't effect me too much just p***ed me off a bit.

I look forward to seeing the missing posts put into place but by that time no doubt more discussions will have made me forget about whatever was here previously.

[QUOTE=DedicatedBox;5360202]
Sh*t happends, unfortunately. I just hope I dont get swamped with spam...

You mean 'more' spam right? lol

I'm wondering where the 6 month backup was stored and why there couldn't have been at least a monthly backup along with it.



This is the true question.

I can't believe there was no recent local backup of the database. :eek: Hope you can recover most of the data.

I agree here.

Where are the hard backups, you know, in your office and perhaps at an off-site location? (on removable media)

I run a very small business but keep backup DVD's every month or so in addition to backups made to remote machines.

Of course WHT is massive but I see no reason why they can't cope with at least monthly hard backups, tape drives or something - whatever the big guys do with masses of data!

Folks, I'm just as annoyed as the next person is (Remember, I had a killer thread going in the VPS Hosting forum that was directed to providers and customers), but I'm not going to sit here and complain about missing posts, or my post count.

Am I annoyed that I lost a few posts and my post count? Yes.
Am I upset that this even happened? Yes.
Am I wondering what's going on about the backups? Yes.
Am I going to raise a stink about this? No.

iNet/WHT has some damned capable people on their team. If there's a way to recover the missing data, you can bet your last dollar that the guys working behind the scenes will do whatever they have to do in order to get it back.

You can also bet your last penny that going forward, the folks that admin the site will ensure that they are better protected against a situation like this.

Dennis has already said what was going on, and what the team is doing to try to rectify the problem.

The twit that did this gained access through a backup system. This tells me, right off the bat, that no matter what backup methodology that was used (even multi-tiered/separate systems) would have been at risk for fodder. If the backups were automated (which they should be), this clown would have been able to exploit it to his or her advantage. The only safe backup would have been manual ones where there was no path to follow (such as someone copying a tarball onto a local machine).

I'm not shaking my finger at anyone in this situation, except for the twit that caused all of this.

Everyone needs to take a step back and look at this situaiton logically.

The folks behind WHT need our support, our patience and our understanding. The last thing they need to do is to be distracted from what they're doing and dealing with folks just yammering about this.

Give them a chance to do what they do best, please.

Glad to see WHT back, good luck with catching this person and restoring the databases!

I sure hope that this wasn't all caused by saving money on security and backup resources to make more profit off of ad revenue...

The twit that did this gained access through a backup system. This tells me, right off the bat, that no matter what backup methodology that was used (even multi-tiered/separate systems) would have been at risk for fodder. If the backups were automated (which they should be), this clown would have been able to exploit it to his or her advantage. The only safe backup would have been manual ones where there was no path to follow (such as someone copying a tarball onto a local machine).



I'll make sure never to use your hosting company then; you're saying it's okay to only have one method of backup? Come on, I'd have multiple backup locations distributed throughout the world, along with offline backups distributed through at least 2 location, especially when stuff like this has happened before and WHT matters to a lot of businesses.

well said. There's a brightside to every bad situation and it will be found eventually here ;)

The folks behind WHT need our support, our patience and our understanding. The last thing they need to do is to be distracted from what they're doing and dealing with folks just yammering about this.


I understand completely, but isn't this the Forum Announcements, Feedback, and Questions forum? Feedback is a good thing.

Folks, I'm just as annoyed as the next person is (Remember, I had a killer thread going in the VPS Hosting forum that was directed to providers and customers), but I'm not going to sit here and complain about missing posts, or my post count.

Am I annoyed that I lost a few posts and my post count? Yes.
Am I upset that this even happened? Yes.
Am I wondering what's going on about the backups? Yes.
Am I going to raise a stink about this? No.

iNet/WHT has some damned capable people on their team. If there's a way to recover the missing data, you can bet your last dollar that the guys working behind the scenes will do whatever they have to do in order to get it back.

You can also bet your last penny that going forward, the folks that admin the site will ensure that they are better protected against a situation like this.

Dennis has already said what was going on, and what the team is doing to try to rectify the problem.

The twit that did this gained access through a backup system. This tells me, right off the bat, that no matter what backup methodology that was used (even multi-tiered/separate systems) would have been at risk for fodder. If the backups were automated (which they should be), this clown would have been able to exploit it to his or her advantage. The only safe backup would have been manual ones where there was no path to follow (such as someone copying a tarball onto a local machine).

I'm not shaking my finger at anyone in this situation, except for the twit that caused all of this.

Everyone needs to take a step back and look at this situaiton logically.

The folks behind WHT need our support, our patience and our understanding. The last thing they need to do is to be distracted from what they're doing and dealing with folks just yammering about this.

Give them a chance to do what they do best, please.

True but this never should have happened in the first place.

Again this isn't the first time.

There should have been a hard back up somewhere.

I understand that complaining doesn't help. I usually don't agree with such but I feel like people have at least a little right to be angry.

Especially seeing as we don't know how much was really taken.

I sure hope that this wasn't all caused by saving money on security and backup resources to make more profit off of ad revenue...

I doubt it.
And all the vague assumptions that nobody can backup are not really contributing any help to the current problem. ;)

I fully agree with Douglas.



Sincerely,
- Liroy

If it weren't for the complaining/feedback the boards would have a whole 10 posts since coming online. Gotta let us all Vent :D

I'm suprised with a board of this size and revenue there is no continuous backup solution in place (r1soft, ect). Off-site to some FTP account? Wow.

Well this sucks. For members and iNet (who I'm sure are now paying the price for not having multiple backup locations!).

Hope you're able to recover that DB. 6 months ago my account was unused for over a year. Since then its been filled with several hundred decent posts. I suppose this also means that all our reviews got wiped, along with my premium membership and username change.

The person who thought it would be a good idea to have a single backup shouldn't be working for iNet. Seriously a company with that kind of money shouldn't be shirking on backups.

Maybe an explanation as to why they felt only 1 backup was needed would be useful.

I'll make sure never to use your hosting company then; you're saying it's okay to only have one method of backup? Come on, I'd have multiple backup locations distributed throughout the world, along with offline backups distributed through at least 2 location, especially when stuff like this has happened before and WHT matters to a lot of businesses.

What he's saying is, even if there were more backups, the intruder made it from the backup server across to the main server. Kind of suggests they'd have been able to make it anywhere else backups were stored, no? You can have 12 backup copies spread all over the globe - if an intruder deletes them all you're still dead in the water.

As far as the people complaining about having "their password out there" - it's 2009 people! It's time we acted a little more educated about security isn't it?

I used a throwaway password for this forum and the only thing I'm upset about is that now I gotta pick another throwaway password for all the other "trivial" websites I go to. If your WHT password is used for banking, email, servers, or anything of that nature, I for one don't think you should even be in the web hosting business.

Passwords are out there, albeit in hashed format, but you should consider them compromised anyway, particularly given the minimal effort it'll take to pick a new throwaway password and memorize it. Your password being compromised isn't the big deal here - the spam list thing is a bigger deal IMHO. If you're not using the password for something stupid then what's the big deal?

Glad to see the site back - went from addict to newbie :(

I'll make sure never to use your hosting company then; you're saying it's okay to only have one method of backup? Come on, I'd have multiple backup locations distributed throughout the world, along with offline backups distributed through at least 2 location, especially when stuff like this has happened before and WHT matters to a lot of businesses.If we had all of my posts, I could have referred you to several posts where I comment about backups. :)

I'm one of the biggest advocates of backup policies around. One of my own personal customers had SIX different backup methods. Four of them available via internet methodology, two of them not.

You completely misunderstood/misread what I said. Let me clarify exactly what I mean:

The twit that did this gained access through a backup system. This tells me, right off the bat, that no matter what backup methodology that was used (even multi-tiered/separate systems) would have been at risk for fodder. If the backups were automated (which they should be), this clown would have been able to exploit it to his or her advantage. The only safe backup would have been manual ones where there was no path to follow (such as someone copying a tarball onto a local machine).

Let's use my former personal customer (We'll call him "Michel") as an example. We'll use the WHT breach as the defining factor for this situation.

* Michel has RAID1 protection: Hacker wipes the DB. This affects both drives.

* Michel has a rsync going over to a third drive on his local machine: Hacker wipes that out too.

* Michel has an automated FTP backup going: hacker locates the PW to that system and wipes that out, as well.

* Michel has an off-site (different facility) rsync backup that's automated: Hacker gets that info and wipes it out, as well.

* Michel has a tape backup, manually rotated out by DC staff, cycled every three days amongst five backups: Hacker could wipe the current one out, but has no physical access to that backup.

* Michel keeps an updated (once a week) backup, done manually onto his own local machine: Hacker doesn't have access to that, ergo no wiping capabilities.

In the section taht I quoted above, WHT had a manual backup that was stored locally somewhere. Granted, it was out of date, but that's the only safe backup that wasn't wiped. That's what I meant by my quoted part.

Everyone that has ever interacted with me with regards to my backup posts/philosophies can vouch for the fact that I always advocate at least THREE of the methods above being used at the same time, though I will always recommend all six, every time.

Please do not mis-read what I posted as me advocating only one backup methodology. Again, I meant that in WHT's situation, the one backup that they have was safe from being deleted, because the twit that did this couldn't gain access to a localized backup, ergo, meaning that it was the only safe backup that couldn't be deleted.

I hope this explains it a bit more clearly, citricsquid.

Just a little though about those backup methods. The rysncs and ftp's going out? Thats never going to be secure as it requires storing remote server information locally which kinda defeats the object of backup in these scenarios. You should always have a delivery guy.

rsync and FTP can both be secured (rsync of an SSL tunnel, and use sFTP instead).

FYI, I would recommend one more backup method, as well (Thanks to Mike V for reminding me): R1Soft's CDP Backup.

There, 7 layers of backups advocated. :D

I wonder if this is a known exploit - last week some dbs (less than 5) on Hostgator had corrupted data - many servers were affected and unscheduled maintenance occured as I had read on their forums. The administrator managed to repair some dbs I pointed out, while I had to restore a few tables from backups (it turned out to be quicker to restore them than open a ticket).

I completely disagree with all the backups options state till now.

you don't need more than 1 offsite remote backup to make it secure.

Any plans to update vB if it's not already the latest?Yes. Likely will be a mid-week thing.

Its easy to say what you think should have been done after its happened, and its a different story when it happens to you.

That in and of itself is a very dangerous statement to make because if you don't know how it happened, you don't know how to prevent it.Dangerous or not, I'm not going to lie to you.

*If* the data in the backup server cannot be salvaged for whatever reason (which really wouldn't be your fault), would WHT stay as it is currently (i.e. at an October 2008 revision on many things)?Of course that's a possibility. But I just don't want to run with every possibility. All things are possible, so the list could get a little long. ;)

I guess the option to hide my e-mail address on my profile no longer matters... because after 1 minute of Googling a list of everybody's e-mail address from WebHostingTalk can be downloaded straight to your desktop... :mad:

I'm not sure why users are being so forgiving of this following the other recent "hacking". WHT is a *big* website with *alot* of traffic - iNet should be able to keep the community safe but it would seem they cannot.

Obviously not on the same scale but imagine if Google leaked our e-mail addresses and some sort of hashed passwords from accounts they held? There would be chaos, WHT has done the same for a certain niche area of the internet.

Someone getting in from the remote backup server is inexcusable. I am talking junior sysadmin stuff here. Obviously nothing is hack proof but you sure can make it 99% hack proof with some minor changes.

* change ssh port to a random high port
* disallow root logins
* disable password authentication - use keys
* firewall off all access except to the ips and ports that need it.
* one way ssh key mechanism would not allow a hacker into the WHT boxes when doing backups.
* do not allow connections to WHT from the backup boxes...
* etc

I think there are some questions that need answered:

1) It seems the extent of the data loss is unknown

2) Was it the entire database that was compromised?

3) Why were the backup servers setup in such a way that more than one machine can access it? If you have a backup server that has a sole purpose (backup WHT.com) then why on earth was it accessible to other machine's login attempts? Was there no system (IPTables) setup to block good IPs from bad?

it seems an entire lack of planning on behalf of iNET staff has led to a seriously potentially dangerous situation for many providers.

The countless number of private conversations going on via PM that has been exposed is just really and truly alarming. I know for one that my previous business conversations with various partners, customers, and service providers that I had here are NOT public.

I would also not appreciate receiving spam on my business e-mail, which happens to be linked to a blackberry.

I guess the option to hide my e-mail address on my profile no longer matters... because after 1 minute of Googling a list of everybody's e-mail address from WebHostingTalk can be downloaded straight to your desktop... :mad:


Well, report the link so that action can be taken ;)

If the backups cannot be recovered, there could be a solution to recover the posts from google cache? An automated script should be created quickly.

magnolia tried something similar for the public bookmarks.

Well, report the link so that action can be taken ;)

I'm sure the data is well on its way, it is already hosted across multiple locations. We're forgetting these "underground" places too... you know, "security" or "hacking" forums etc.

The data has gone, iNet lost it - not good, not forgivable.

I am staying out of this thread besides this post, but SoftwareRevue, don't you feel bad now that the hack from last year was covered up? last year you guys were running a vulnerable kernel on your dev box, according to google caches, makes me wonder about the rest of your infrastructure.

Now move on with the conspiracy theories, please.

I heard what really happened from my cousin who heard from her dentist who heard from his wife who bought flowers from a guy who sat next to some lady on a bus who knew the culprits. The details are a little hazy (and sometimes nonexistent), so I'll just make them up as I go along ... as is standard practice in situations like these.

As some of you know, The Ann Arbor News will be publishing their last daily edition in July of this year. To those not living near Ann Arbor, this isn't too significant, but to Rutherford Steinjack of Burns Park, Michigan, this news was catastrophic ... you see, Mr. Steinjack has a very picky guinea pig named Boris who burrows exclusively in Stefanie Murray's articles (Steinjack once tried to sneak in a page of coupons from Village Corner and Boris bit off his pinky).

Afraid of what Boris might do when The Ann Arbor News did not show up on his doorstep every morning, Mr. Steinjack decided that drastic times called for drastic measures so he called his grandson in West Philadelphia (born and raised) to wage war against this "Internet" thing for him ... as the most current piece of technology in Mr. Steinjack's split-level house was a toaster built in 1986 which he avoided like the plague because it cooked toast unnecessarily fast. Because Steinjack's grandson had been busy with homework from Ms. Bailson's fourth grade math class, he had to outsource this "war on the Internet" to a team of huckleberry pickers living just south of Santa Barbara, CA.

Around this point in the story, the woman on the bus sitting next my cousin's dentist's wife's flower salesman had to disembark, but she promised to explain the mechanics of the hack tomorrow evening between Wheel of Fortune and Bingo on the #9 bus between Huron, SD and Wausau, WI.

Sorry I couldn't be of any help on that side, but at least we know a little of the back-story now. :stickout:

In all seriousness, I hope everything gets restored as soon as possible and you guys can track down the folks behind it.

* Michel has an off-site (different facility) rsync backup that's automated: Hacker gets that info and wipes it out, as well.


Easy fix for that is to run rysnc from the backup server. No way to hack that, especially when your backup server should work via private network only.

I can think of several backup methods that are fool proof and can not be accessed from the server being backed up.

Its too late to make excuses or gripe about what happened. The data on the drives needs to be recovered by a data recovery company.

I just found a method to recover all the posts. I remember recently several sites using wht rss to draw all the posts into their forums. Now all wht has to do is use rss to draw them back. :eek:

What he's saying is, even if there were more backups, the intruder made it from the backup server across to the main server. Kind of suggests they'd have been able to make it anywhere else backups were stored, no? You can have 12 backup copies spread all over the globe - if an intruder deletes them all you're still dead in the water.

As far as the people complaining about having "their password out there" - it's 2009 people! It's time we acted a little more educated about security isn't it?

I used a throwaway password for this forum and the only thing I'm upset about is that now I gotta pick another throwaway password for all the other "trivial" websites I go to. If your WHT password is used for banking, email, servers, or anything of that nature, I for one don't think you should even be in the web hosting business.

Passwords are out there, albeit in hashed format, but you should consider them compromised anyway, particularly given the minimal effort it'll take to pick a new throwaway password and memorize it. Your password being compromised isn't the big deal here - the spam list thing is a bigger deal IMHO. If you're not using the password for something stupid then what's the big deal?


I think the point was there there should be more than one method not more than one server. I.E CD's Tapedrives, etc.

Wow - major blow to WHT...

But - sometimes it is good to purge, out with the old, in with the new...

Welcome back WHT!
Missed you :P

But - sometimes it is good to purge, out with the old, in with the new...

Unfortunately it is the other way around at the moment :-X

Wow - major blow to WHT...

But - sometimes it is good to purge, out with the old, in with the new...

Except all the old stuff is still here and its the new stuff thats gone...so its more out with the new, in with the old :smash:

What he's saying is, even if there were more backups, the intruder made it from the backup server across to the main server. Kind of suggests they'd have been able to make it anywhere else backups were stored, no? You can have 12 backup copies spread all over the globe - if an intruder deletes them all you're still dead in the water.


I don't use a real password for WHT and I don't care who has my email, I'm just trying to make a point. If I pretend I care maybe inet will take their heads out of their asses and learn about real backups... haha I can't believe I just said that, that'll never happen; remember last time there was a **** storm about passwords being stolen they didn't care and denied it.

The point is, if you're running a site that relies upon data, as any forum does, why on earth do you keep a single backup? If I ran WHT, I'd have a secondary server, for switching in if stuff went down, I'd have remote backups around the world, I'd have offline backups taken weekly with copies of those at different locations, etc etc. inet is worth millions, I could do something like that for $500/month and in the long run, it saves users data. I wouldn't be surprised if they lose the coporate members from this, faith in WHT is being lost.

I think the point was there there should be more than one method not more than one server. I.E CD's Tapedrives, etc.

Exactly.

I don't use a real password for WHT and I don't care who has my email, I'm just trying to make a point. If I pretend I care maybe inet will take their heads out of their asses and learn about real backups... haha I can't believe I just said that, that'll never happen; remember last time there was a **** storm about passwords being stolen they didn't care and denied it.

The point is, if you're running a site that relies upon data, as any forum does, why on earth do you keep a single backup? If I ran WHT, I'd have a secondary server, for switching in if stuff went down, I'd have remote backups around the world, I'd have offline backups taken weekly with copies of those at different locations, etc etc. inet is worth millions, I could do something like that for $500/month and in the long run, it saves users data. I wouldn't be surprised if they lose the coporate members from this, faith in WHT is being lost.



Exactly.


For a forum like this I would have at least 3 backup methods.

I don't use a real password for WHT and I don't care who has my email, I'm just trying to make a point. If I pretend I care maybe inet will take their heads out of their asses and learn about real backups... haha I can't believe I just said that, that'll never happen; remember last time there was a **** storm about passwords being stolen they didn't care and denied it.

The point is, if you're running a site that relies upon data, as any forum does, why on earth do you keep a single backup? If I ran WHT, I'd have a secondary server, for switching in if stuff went down, I'd have remote backups around the world, I'd have offline backups taken weekly with copies of those at different locations, etc etc. inet is worth millions, I could do something like that for $500/month and in the long run, it saves users data. I wouldn't be surprised if they lose the coporate members from this, faith in WHT is being lost.



Exactly.

Ha sam i dont think a $500/mo budget would cover whts infrastructure

This only affected WHT or other inet databases (hotscripts and etc) are stolen too?

Ha sam i dont think a $500/mo budget would cover whts infrastructure

For backups? sure it would. Their database can't be that big. One of my clients is 5 times larger in terms of post count and users, and its 25gb.

Ha sam i dont think a $500/mo budget would cover whts infrastructure
Just proves the point the data is more important for the cost.

=(

I only lost roughly all of my posts, not to mention my account.

Oh well, hopefully I won't get in trouble for making this account again, I just wanted to post and chat with the community again.

The twit that did this gained access through a backup system. This tells me, right off the bat, that no matter what backup methodology that was used (even multi-tiered/separate systems) would have been at risk for fodder. If the backups were automated (which they should be), this clown would have been able to exploit it to his or her advantage. The only safe backup would have been manual ones where there was no path to follow (such as someone copying a tarball onto a local machine).

Should iNET look into using a pull backup solution, rather than push? This way your database/webservers do not store login details for the backup servers.

The way we have backups done is via completely locked down (ie, ALL inbound traffic firewalled off at software and hardware level) machines which SSH into our servers to download incremental backups daily.

Have a look at BackupPC! It'll do what you guys need for the future :)

For a forum like this I would have at least 3 backup methods.
i'd have 25 backups :)

--
anyway, whatever happened, happened.

one strange thing i saw today, since i was not logging into forums last fews days, today i saw wht came back, i was logged out of the board, so went to login again with the old credentials, however i wasn't able to do so, like password was incorrect/change, so i did email password recovery which went smooth, and i was logged in again.

i'm not sure did i missed something at all this posts regarding this, but if i remember there were no posts saying that all wht account password were set to reset so that we have to recover via email recovery?

some other users might be in this same situation like me *cannot login with the old credentials*, so it might be a good idea to make an annoucement about this.

For a forum like this I would have at least 3 backup methods.
I agree, and I would also make sure to have a weekly local backup as well. Though, I hope from this they will learn and make sure it won't happen again. A big loss won't be the data only, but also the Search Engine listing.

Should iNET look into using a pull backup solution, rather than push? This way your database/webservers do not store login details for the backup servers.

The way we have backups done is via completely locked down (ie, ALL inbound traffic firewalled off at software and hardware level) machines which SSH into our servers to download incremental backups daily.

Have a look at BackupPC! It'll do what you guys need for the future :)

I would personally run a multiple level backup, with the primary being a server running a replication slave.

Stop the slave
backup
start the slave

I backup most databases that way. This way the only access to the backup server is a replication user over a mysql connection.

My backup script alerts me if the slave is out of sync and the backup is going to be bad.

example:



echo "Checking Slave Status"
SLAVESEC=`mysql -e "show slave status\G"|grep Seconds_Behind_Master |awk '{print $2}'`
if [ $SLAVESEC == '0' ]; then
echo "Slave Status Ok"
else
echo "xxxxxx Mysql Backup Could not be completed" | mail -s "xxxxx Mysql Backup Could not be completed" xxxxxx
echo "Slave Status Bad"
exit
fi

Question,

SWR do you guys not have binary logs (from replication) to prior to the table dropping?

For backups? sure it would. Their database can't be that big. One of my clients is 5 times larger in terms of post count and users, and its 25gb.

I was talking complete infrastructure not backups. sure you could run a solid backup solution on that price

Ha sam i dont think a $500/mo budget would cover whts infrastructure

i'm talking about just backing up the wht database, not their entire infrastructure.

Dangerous or not, I'm not going to lie to you.

Of course not SoftWareRevue - I wasn't trying to criticize but simply stating that it's not a good situation to be in. I can understand that if the hacker was as smart as they think they are that there may be virtually no trace of how they got in or what they did exactly. I wasn't making a strike against you, or not trying to.

My apologies.

THE BEST THING YOU CAN DO DENNIS IS CHECK THE IP LOGS AND FIND OUT WHO DID THIS AND GO FROM THERE!!

Go back thru EVERY IP UNTIL YOU GET TO THE SCUMBAG WHO DID THIS!! (Its not impossible my friend)

Good luck!





you do know that is over 5 thousand ips or more since before the hack there was 1500 people on at any one time. stretch that the whole day



and.... ive seen mods etc state that you should make backups on multiple sites not just one. and wht doesnt do that lol. I lost 500 posts and cant read a few threads I was reading because I needed to use them to secure an older server. to late for that.


how the heck did the hacker get to the hardrive the backup is on if its done manualy there should be no logs besides the ip downloading it correct? which should not be to bad but how the hell did he find the ip to a offsite backup.



WHT has been hacked before and it seems they have not learned from there mistakes. Im still staying but you would think for a site that talks about webhosting that they would make extra backups.

wow, all topics in same areas 150+ days old without any replies. I hope it will be restored the whole database.

LOOK: I found this not sure if its a prank or what but http://www.webhostingtalk.nl/hacked.php

LOOK: I found this not sure if its a prank or what but http://www.webhostingtalk.nl/hacked.php

You know this is www.webhostingtalk.COM and not www.webhostingtalk.NL .... the two sites have nothing to do with each other.

Sirius

Since I can't quote yet, this relates to the above post -.-


And the main site works why?

That seems like more of a April Fools joke...

I finally got in yesterday using password recovery but when I tried it today I had to change my password again.

Anyway, regardless of whether INet should or should not have taken more precautions, what's done is done so I'm just hoping they are able to recover the lost data.

I am getting a lot more spam that's getting past Spam Assassin today but it could be coincidence. Crossing fingers.

I am wondering.. does this mean all those who had signup recently have lost their accounts as well? Just a bit curious..

Atleast we are lucky to have our usernames and few posts saved.. Think about those who lost their accounts!

lucky... my account was deleted, which really sucks. Luckily the password I used was only on this site, and since my account was deleted no one can really hack it, now that I've signed up with a completely different password.
Anyhow, couldn't Inet set it up to only have the back up server be accessed from an Office IP address, blocking all other ones? Sure would help.

edit: wonder if Inet will now add in their terms " not responsible for compromised passwords, emails, or confidential Private messages if hacked".

THE BEST THING YOU CAN DO DENNIS IS CHECK THE IP LOGS AND FIND OUT WHO DID THIS AND GO FROM THERE!!

Go back thru EVERY IP UNTIL YOU GET TO THE SCUMBAG WHO DID THIS!! (Its not impossible my friend)

Good luck!






You know this is www.webhostingtalk.COM and not www.webhostingtalk.NL .... the two sites have nothing to do with each other.

Sirius
I know but if they where hacked in the same time zone and in webhostingtalk.nl he posts what name he goes by then it may be easyer to trace. all im saying is if 2+ webhosting sites get hacked at about the same time usualy its from the same person or people. never said webhostingtalk.nl was the same either

Isn't it better to stop new user signups until recovery is completed?
If someone register again with one of the lost usernames, it may cause problems for data recovery because usernames should be unique.

Edit:
This includes topics and posts tables too. auto-number field get the old values again and when you try to add lost topics and posts, id fields are already existed and may cause problem.

Maybe iNet need to buy SecurityTalk.com and BackupTalk.com, setup some boards and read them. :D

All jokes aside, i really hope they can recover the last 6 months of posts it's a massive blow to lose them.

Edit:
This includes topics and posts tables too. auto-number field get the old values again and when you try to add lost topics and posts, id fields are already existed and may cause problem.
I'm pretty sure the auto_increment value was set to the most recent thing possible. I agree, if this wasn't done properly then the database will just corrupt again.

It really bums me out... my business had a great thread here with a lot of reviews. Now it looks like I'm starting from my very first day on here ;(

Arggg, Lost my account! :(

I hope WHT will take better measures to ensure this does not happen again. :)

Well at least it is back. I was starting to go nuts without my daily dose of WHT and the drama lol

This only affected WHT or other inet databases (hotscripts and etc) are stolen too?This was an attack aimed only at WHT.

I'm not happy. I put in a ticket asking my name to be changed to what I had it changed to a few months ago, and my Premium Membership re-added and it was denied as they said they were looking at restoring the db....make up your minds!

The longer you leave the site like this, the bigger problem you create. Either close everything down and import, or fix our accounts and move on. Don't deny things without providing reason!

As I replied in the support ticket, I assume you will be extending Premium memberships for free since your effectively denying people the premium membership they have paid for.

I'm a bit lost with all this people crying here about their post lost and I'm too lazy to read the 12 pages with those complains, I have just one question, that probably was already answered so forgive me in advance, is this how WHT is going to stay? I'm not complaining, just want to know if is it.

I'm a bit lost with all this people crying here about their post lost and I'm too lazy to read the 12 pages with those complains, I have just one question, that probably was already answered so forgive me in advance, is this how WHT is going to stay? I'm not complaining, just want to know if is it.

Pretty much, unless they can get the backup back.

I'm pretty sure the auto_increment value was set to the most recent thing possible. I agree, if this wasn't done properly then the database will just corrupt again.

I'm not sure if this is directly related, but today I got a couple of notification of updates of thread created today that I was never subscribed, I assume that this happened because I were subscribed to old threads with the same number.

I'm not happy. I put in a ticket asking my name to be changed to what I had it changed to a few months ago, and my Premium Membership re-added and it was denied as they said they were looking at restoring the db....make up your minds!

The longer you leave the site like this, the bigger problem you create. Either close everything down and import, or fix our accounts and move on. Don't deny things without providing reason!

As I replied in the support ticket, I assume you will be extending Premium memberships for free since your effectively denying people the premium membership they have paid for.Please supply a ticket ID so I can look at it. I find it hard to believe a staff member would respond like that. And I will take action if it is so.

You should have been contacted by iNET regarding your Premium membership. If you haven't been, please let me know.

WHT,

I wish you all the luck restoring your db if possible.

I'm a bit lost with all this people crying here about their post lost and I'm too lazy to read the 12 pages with those complains, I have just one question, that probably was already answered so forgive me in advance, is this how WHT is going to stay? I'm not complaining, just want to know if is it.It's too early to tell. We simply don't know at this point. But it's not out of the realm of possibilities that this will be as good as it gets. :S

... I assume that this happened because I were subscribed to old threads with the same number.That's likely it. :)

LOOK: I found this not sure if its a prank or what but http://www.webhostingtalk.nl/hacked.php

That has been there for ages.
I have no idea why. Even google indexed it in the meanwhile.

I can't say I'm not gutted about the leak but...

Good luck everyone working on the restoration. Hopefully it works as it will be a shame to lose all those irreplaceable posts - however, we can always replace them with new irreplaceable ones.

Is it possible to completely delete one's profile from the WHT database? I can change email and password in the user's control panel, but I would prefer to completely erase my profile in case something similar happens again.

Is it possible to completely delete one's profile from the WHT database? I can change email and password in the user's control panel, but I would prefer to completely erase my profile in case something similar happens again.

No, that is not possible.

No, that is not possible.

Well why not? This is quite an issue and has caused grief for many providers and people. I think WHT should make that an option.

Man, I hope iNet is able to restore all of this! :)

No, that is not possible.

Technically, it should be. Can't a iNet Member disable the account; then go find the db_user table and db_email and whatnot and delete that line? It should work. ;)

No, that is not possible.

Thanks for confirming this Christian, I'll just set both email and password to some really random string that I won't remember or save. I'll probably won't be back here, at least not anytime soon, good luck in getting things back into shape.

I reported yesterday (http://www.webhostingtalk.com/showthread.php?t=729230) that our recent downtime was due to issues with our backup servers followed by the corruption of some db tables from a hack attempt.

We've since learned that this very deliberate, sophisticated and calculated hack against Web Hosting Talk was carried out by gaining access to our offsite backup servers. From our backup servers, the hacker gained access to the WHT db server. The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.

This individual is still in possession of our user table that includes all user names, email addresses and hashed passwords. Absolutely no credit card or PayPal data was compromised.

Passwords are hashed with salt. It would be an unprecedented event to reverse engineer our passwords. I change my password periodically though, so maybe today is a good day for that. Go here (http://www.webhostingtalk.com/profile.php?do=editpassword) to change your password.

My concern is the distribution of your email addresses and the potential spam you may receive. We know the hacker has posted the user table containing email addresses to various places (file sharing sites) and we're working diligently to remove the tables as we find them. If you see the user table posted anywhere, please let us know so we can get it taken off line.

We are working on recovering the deleted data. In the meantime, we've restored to an old db. We cannot yet determine if we can restore to a more recent db backup.

If you have any clues as to the individual who caused this malicious attack on the Web Hosting Talk community, please let me know.

holy crap.. i log in and found the forumhome page messed up - see screenshot (http://img99.imageshack.us/img99/8490/forumhomeindexmessedup.jpg) but now reading this, the front page is the least of your worries!

This really does force me to also look at my own back up methods as well. Scary.. just scary

Hope you get it all sorted out !

Technically, it should be. Can't a iNet Member disable the account; then go find the db_user table and db_email and whatnot and delete that line? It should work. ;)

Of course it isn't impossible to do, but we don't per policy. ;)

What's done is done. Everyone, in the course of shared hosting will go through something like this. Yes, they should have had better backups, yes, they've got a bigger budget ... shudda cudda wudda... didn't happen, move on.

According to softwarereview, only the post tables were compromised, which unfortunately included email addresses.

Time to move on folks, drop the conspiracy theories, stop thinking your bank accounts are going to be wiped out and that the earth is going to stop because a table of posts was wiped/compromised.

Let's go forth and rebuild!

I just upgraded to "premium" to show my continued support for WHT. They've been a valuable resource over the years, before and since iNet ... they'll be here long after most of us are gone. Show your support and let's move forward!

Of course it isn't impossible to do, but we don't per policy. ;)

Per policy? What is this, I am not saying I want my account deleted but If I did, I'd expect like any other site that'd be a possibility.

To clear things up, if you make a support ticket, will your account be able to be deleted?

It's too early to tell. We simply don't know at this point. But it's not out of the realm of possibilities that this will be as good as it gets. :S
What will happen with the current posts making the users? The new offers on the Webhosting forums, will they get erased?

I just dont understand that inet has over 20 forums and has yet to decide to make more then 1 backup spot. and what about the premium users. will they have to pay again

you left the top secret high security back door open to the all important backups naughty bunny heres a infraction

Oo yaa true... but innocent people might have got banned for their old infractions coming back. And i better make sure i do not do something to get more infractions for who knows how long wht will get around to fix the infractions bug, it will only pile up and increase the issue, so wht mods should not start giving out infractions till its all restored, else we could see and infinite loop kick into action resulting into weird behavior and angry whters.

holy crap.. i log in and found the forumhome page messed up - see screenshot (http://img99.imageshack.us/img99/8490/forumhomeindexmessedup.jpg) but now reading this, the front page is the least of your worries!

This really does force me to also look at my own back up methods as well. Scary.. just scary

Hope you get it all sorted out !

I thought I messed it up just on my browser until I looked at other sites.

I'm guessing with the time that's passed it isn't going to be restored to where it was and we are stuck in the past ?

I hope not but it would be good to know one way or another.

No, that is not possible.
It's funny - because technically if a restore cannot be done there are quite a few people who have been "deleted" :) Not intentionally of course.

I'm guessing with the time that's passed it isn't going to be restored to where it was and we are stuck in the past ?

I hope not but it would be good to know one way or another.

I hope it can be, I miss the good ol' wht :(

It's funny - because technically if a restore cannot be done there are quite a few people who have been "deleted" :) Not intentionally of course.



I hope it can be, I miss the good ol' wht :(

I just managed to find what password I used to use :D I though I was lost.

Any disgruntled former iNET employee's?

It's funny - because technically if a restore cannot be done there are quite a few people who have been "deleted"

I believe the attack didn't delete anything from the user's table:

... The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. ...

This individual is still in possession of our user table that includes all user names, email addresses and hashed passwords. Absolutely no credit card or PayPal data was compromised.

Per policy? What is this, I am not saying I want my account deleted but If I did, I'd expect like any other site that'd be a possibility.

To clear things up, if you make a support ticket, will your account be able to be deleted?

There was a great recent post by bear on our policy regarding deletion of accounts. In short, we do not delete accounts to preserve the integrity of the information on the forum.

We can disable accounts on request, but we do not recommend it. The only "deleting" we do is merging accounts, outside of that, absolutely no accounts are "deleted".

I suggest anyone who lost threads with positive company reviews etc go to Google and type:

site:webhostingtalk.com + descriptive text in your thread

Click the "Cached" link, and copy the thread, screenshot it or which ever and add it to your own site so it's not lost before Google drops the cache.

wat if that person contributed no worthy information?

wat if that person contributed no worthy information?

We will still not delete the account. It's unfair if we delete one person's account, and not another's. And then we get into gray areas about what "worthy information" is and isn't. So it's a policy for all, regardless of post count. :)

Hopefully this script kiddie can be caught and prosecuted. This really sucks.

So was this hacked backup server the only source of backups, or did the admin also do backups to his house and keep on separate media?

Also sucks about the email addresses.

spamblockingtalk.com anyone? :p We'll need it.

Please supply a ticket ID so I can look at it. I find it hard to believe a staff member would respond like that. And I will take action if it is so.

You should have been contacted by iNET regarding your Premium membership. If you haven't been, please let me know.

PM Sent.

iNet hasn't contacted me either.

I suggest anyone who lost threads with positive company reviews etc go to Google and type:



Click the "Cached" link, and copy the thread, screenshot it or which ever and add it to your own site so it's not lost before Google drops the cache.

Excellent post, everyone should do this.

I miss my 500 posts in under 4 months. thats an acomplishment for me usualy i get 1 post a year in alot of forums. ohwell gotta start over the only think I dont like is there lack of security when it comes to backups. I still dont understand how the hacker got the backup servers ip as it was never posted on webhostingtalk?

digg has the database info but I think it just got deleted which is a good thing. but over 100 people have seen the digg so in that time who knows how many downloaded the db

so am I to assume that the current state of the board is the go forward state and that nothing can be done to go back to friday pre hack?

heres an idea. depending on the os couldnt you theareticly do a system restore back to before it was hacked or is that impossible. * never had to do a system restore so would not know*

What's done is done. Everyone, in the course of shared hosting will go through something like this.

Yeah, that's a really carefree and ignorant attitude to take towards this, especially since this isn't the first time that WHT has been attacked this viciously this year!!! Let's go forth and learn. Oh, wait, we've HAD that opportunity, haven't we?


I just dont understand that inet has over 20 forums and has yet to decide to make more then 1 backup spot. and what about the premium users. will they have to pay again

A site the size of WHT should be storing information on multiple (private access) servers as well as in multiple drives which are only mounted when the system is backing up. This is child's play here. ANY time you have MySQL databases that are critical, you back them up daily (sometimes multiple times daily) to multiple locations. That's just a given.


heres an idea. depending on the os couldnt you theareticly do a system restore back to before it was hacked or is that impossible.
With Windows, yes, with Linux, no.

Here's another thing to make you think:
The db was NOT lost. If it were, PMs would be lost, everything would be lost. Instead, only PART of the db was lost (or corrupted). PM's still exist, other profile related things still exist. It's hit and miss.

heres an idea. depending on the os couldnt you theareticly do a system restore back to before it was hacked or is that impossible. * never had to do a system restore so would not know*

Well, this is a bit more complicated. Think of it this way, the server with all the restore points is inaccessible, so no restoring back at the moment.. :(

If they were using r1soft they could do in time restoration.

Just a putting my two cents in :P

Glad I use a generic email that anyone can spam at any time... I do that for a reason... wait:: for reasons like this...

Not sure why anyone would want their account deleted after this... simply change your name, email, etc don't log on and move on.

I'd hope no one uses the same password on a forum that they use to get into their bank... lol

Far as a hacker... he/she will never be caught... unless they brag about it someplace. Not sure what glory they get out of this... or if just a reaction to some of the mods comments / infractions... who knows... oh, well...

Who will care about this in two months anyway? The boards will populate themselves again in a week anyway...

Windows restore only does registry I think.

Shadow file copy is another idea, but if the hacker was smart enough to kill the backup server think he would have easily been able to kill the shadow copy volume as well. (seperate partition that stores the shadow copies). Come to think of it, would be nice if Linux had this feature.


Also if they do get a backup that works, they have to decide if they wipe our posts as of now and go back to prehack, or just rebuild our post counts/threads etc. You can't really keep our existing posts + restore as the auto increments would mess up.

Now an idea would be to maybe modify the backup to increment the auto increments by whatever values required so it hits new numbers. Not an easy task (they'd have to match up everywhere).

In the end, I just hope this hacker is caught and dealt with legally. It's sad the grief a script kiddie can cause to a site owner. As a user it's not so bad, we just have to sit and wait. As the site owner/admins, it's go go stress time. No fun at all to deal with stuff like this.

If they were using r1soft they could do in time restoration.

Just a putting my two cents in :P

True, but in this case, if the backup (AKA CDP Server) was compromised, then what good would it be?

My main concern is that the user table should be restored. I had to re-register my account and now cannot even post in half the forums due to this.

Also noticed some bugs with the mixed tables, like posts saying they were edited by other users.

Windows restore only does registry I think.

It does more... but it is worthless and not worth talking about... I've never seen it work for anyone... and I happen to like Windows over most OSs....

What is needed is a better backup plan... but that costs money... backup solutions are driven by money many times...

We lose a couple of months... is it really that big a deal? In two months, I'd say no one will hardly be talking about this... unless issues are not taken care of right away...

Who will care about this in two months anyway?

Honestly? The people who actually care about security will. How long's it been, what, 6 months (if that) since the last time we were forced to change passwords? Someone didn't learn then?

Sure, I'm down almost 1000 posts, and I've got this annoying "You have a new private message" thing that I can't get rid of (I have no new PM's), but that's all inconsequential stuff. I mean, that's just superficial garbage. What ISN'T inconsequential is the lackluster attitude towards security and backups here.

Sure, it's a public site, we pay nothing for it, we should expect nothing. On the other hand, this site and it's members pay iNet's bills through advertisement, premium memberships, premium forum posts (sticky, etc), so iNet should be concerned about what is happening, yet, apparently, aren't so concerned. If they were, this wouldn't have happened, or it wouldn't have been as bad as it was.

Glad I use a generic email that anyone can spam at any time... I do that for a reason... wait:: for reasons like this...

Not sure why anyone would want their account deleted after this... simply change your name, email, etc don't log on and move on.

I'd hope no one uses the same password on a forum that they use to get into their bank... lol

Far as a hacker... he/she will never be caught... unless they brag about it someplace. Not sure what glory they get out of this... or if just a reaction to some of the mods comments / infractions... who knows... oh, well...

Who will care about this in two months anyway? The boards will populate themselves again in a week anyway...

Well hopefully there may be some logs somewhere, but even then they probably hacked through some zombie computer. Another reason I think people with infected PCs should be held liable for any damage their PCs cause. People need to learn to not leave their computers infected.

As for not using forum password for bank, yeah that's a good idea to not do that. :D In fact, any password used over an unencrypted channel should never be reused on a service that is encrypted, such as bank, ssh, etc... as you pretty much defeat the purpose of that encryption.

I thought it was standard to have a spam e-mail now of days?

Glad I use a generic email that anyone can spam at any time... I do that for a reason... wait:: for reasons like this...

Not sure why anyone would want their account deleted after this... simply change your name, email, etc don't log on and move on.

I'd hope no one uses the same password on a forum that they use to get into their bank... lol

Far as a hacker... he/she will never be caught... unless they brag about it someplace. Not sure what glory they get out of this... or if just a reaction to some of the mods comments / infractions... who knows... oh, well...

Who will care about this in two months anyway? The boards will populate themselves again in a week anyway...

True, but in this case, if the backup (AKA CDP Server) was compromised, then what good would it be?

well you see,

the cdp server has a different access then their backup server probably did. All there would be is an agent -> server connection.

Im going to assume they were doing backups via rsync over ssh.

I'm sitting here scratching my head wondering what the hell are they doing with all the money the advertisers put into this forum.

Why is the backup and database server not behind hardware firewalls which limit the IP access to those boxes? The DDOS attacks and slash dot I understand. That is uncontrolable. This seems like a complete lack of security policies on how to manage boxes. In my opinion, this looks bad as hell from my standpoint on Rack Space. Clearly Inet cannot be trusted with sensitive information if they cannot secure properly. This isn't like it was preventable by taking other steps. Its not so much about even the lost of data because of backups this is a lack of security.

Sure, I'm down almost 1000 posts, and I've got this annoying "You have a new private message" thing that I can't get rid of (I have no new PM's), but that's all inconsequential stuff. I mean, that's just superficial garbage. What ISN'T inconsequential is the lackluster attitude towards security and backups here.



have someone send you a PM, and delete it. It will go away.

Lets hope that they learn from this and can repair the tables while implementing better security and access policies on each server.

Honestly? The people who actually care about security will. How long's it been, what, 6 months (if that) since the last time we were forced to change passwords? Someone didn't learn then?

Sure, I'm down almost 1000 posts, and I've got this annoying "You have a new private message" thing that I can't get rid of (I have no new PM's), but that's all inconsequential stuff. I mean, that's just superficial garbage. What ISN'T inconsequential is the lackluster attitude towards security and backups here.

Sure, it's a public site, we pay nothing for it, we should expect nothing. On the other hand, this site and it's members pay iNet's bills through advertisement, premium memberships, premium forum posts (sticky, etc), so iNet should be concerned about what is happening, yet, apparently, aren't so concerned. If they were, this wouldn't have happened, or it wouldn't have been as bad as it was.

I honestly don't see how you can say that, they had back-up servers, firewall servers, file servers, and the server that WHT vBulletin is actually on.

The hacker attacked all of them, at once, since WHT is a free website, it's not like they have a team on 24/7 stand-by for these attacks.


*EDIT: They do all this for free, **** happens, get over it. WHT will be back to normal as soon as possible, if not we'll just have to rebuild the past 6 months and try harder to prevent this from happening again.

I honestly don't see how you can say that, they had back-up servers, firewall servers, file servers, and the server that WHT vBulletin is actually on.

The hacker attacked all of them, at once, since WHT is a free website, it's not like they have a team on 24/7 stand-by for these attacks.


*EDIT: They do all this for free, **** happens, get over it. WHT will be back to normal as soon as possible, if not we'll just have to rebuild the past 6 months and try harder to prevent this from happening again.

WHT is not a free website, it has a full staff that works to keep it online. It is paid for by ads.

so iNet should be concerned about what is happening, yet, apparently, aren't so concerned. If they were, this wouldn't have happened, or it wouldn't have been as bad as it was.

Go start your own site and see how easy it is... I'm sure the money will be flowing and you will have backups on 7 continents. Until the hacker finds a way to delete all 7. :D

I run a company so I know how hard it is to pay the bills and do everything the right way... unfortunately, sometimes hard decisions have to be made... not saying this was the case, but at least, I can understand backups... they are expensive... even though I have backups in three states now (really). I'm in a different business as WHT though...

Anyway, I hope they do learn from this... but you have to admit, hackers are not the script kiddies we joked about in the 80s and 90s... those kids grew up...

WHT is not a free website, it has a full staff that works to keep it online. It is paid for by ads.

What I meant is it's public, I understand it has it's ways of making money, and I know the iNet staff is there. I was giving the generalization that WHT doesn't have a entire NOC ready to defend something against that, yes they have techs that are ready to respond, but I was only saying that when a hacker knows what they are doing and is diligent enough to not give in, you can't do a ton to stop them.

Nothing is "100% secure".

This has really made me second-guess the security of my own backups.

I presently use a remote server for my backup (at an entirely different datacenter from my website) and it never occured to me, that my database is completely vulnerable to an attack, should that backup server get compromised.

The software I use on my website stores the database user & pass in plain text in the configuration file. If an attacker got the tarball with my server backups, he has all he needs to get in! (Except my db does not allow access to anything but localhost.. but providing he spoofed, he'd likely gain access that way)

What a terrifying thought! I just changed the db user & pass and I intend to do this with every backup, from now on.

This actually reflects more on Rackspace, after all it is their network. Not saying Inet is not to blame, but these are managed servers. Plus, as I tell all my clients, even if your hosts provide backups, you should still backup regularly yourself. And decide in your backup routine how much data you can afford to loose.

And never, ever leave backups on the main server, but on a protected server (for external servers through a tunnel of some kind). and even then keep in an offsite safe area.

I have seen too many times when a backup is made to a directory on the main server, data gets corrupted or a disk dies and whoa, no backups.

Not making any assumptions on what actually happened, but just my philosophy both private and at work above.

What I meant is it's public, I understand it has it's ways of making money, and I know the iNet staff is there. I was giving the generalization that WHT doesn't have a entire NOC ready to defend something against that, yes they have techs that are ready to respond, but I was only saying that when a hacker knows what they are doing and is diligent enough to not give in, you can't do a ton to stop them.

Nothing is "100% secure".

nothing is 100% secure, but i saw a posting by the hacker, if what was said is true, then inet seriously lacks at administration skill.

Lets hope that they learn from this and can repair the tables while implementing better security and access policies on each server.
Yeah, you would have thought they learned this the last time.

I honestly don't see how you can say that, they had back-up servers, firewall servers, file servers, and the server that WHT vBulletin is actually on.

Yet it wasn't enough, or we wouldn't be here.


The hacker attacked all of them, at once, since WHT is a free website, it's not like they have a team on 24/7 stand-by for these attacks.

Do you even grasp the concept people are trying to demonstrate here? This is ENTIRELY possible to do, and it's ENTIRELY possible to keep people out of it.

Step 1: unmount backup drives when not in use. Out of sight, out of mind
Step 2: Use vpn backup servers, so that NOTHING can get to them. No outside access at all.
Step 3: Use VPN DB servers, again, so that nothing can get to them at all. No outside access
Step 4: Control all access to said VPN servers through firewalls and other rules, so that ONLY one ip (or a set of them) can get in and out of the server.
Step 5: REDUNDANCY! Multiple backups, multiple servers.

Of course the above isn't exactly a guarantee, there ARE no guarantees, but by god, if you have the above in place, you should be fine.

Of course, server security is key here, keeping things updated is as well.

If all of the above are done correctly, no problems will ensue that can't be reasonably blocked.


*EDIT: They do all this for free,

OOOOOH no they don't. WHT charges in the form of advertisements, premium memberships, sticky threads, etc. I'm not saying that's bad at all, but they're not doing this "for free".


try harder to prevent this from happening again.
Yeah, I think Santa still exists too. That Easter Bunny, he's right around the corner isn't he? The tooth fairy owes me a fortune and a half by now!

It's nice to dream that this is going to be done, but, really, what do 2 critical attacks like this in less than a year tell you? TRYING isn't happening, or if it is it just aint cutting it.


Go start your own site and see how easy it is...

I do have "my own site", multiple ones even, and believe me, I practice what I preach (within reason of course). A site on this scale should be able to put those above steps into play easily, and should have done so long, long ago.

People you need to quit complaining. To recover a Corrupted Database is a hell of a lot of work. TO also find out how they gained access can be even worse. I have been hacked before. It sucks. But it was easy for me to recover because i am an Ex-Hacker. If i was iNet i would hire a very good hacker to help them out.

Okay -- listen. RackSpace is a HUGE company, and they're really not to blame -- nor is iNet. The hacker gained access to the backup servers -- there were probably multiple. From there he gained access to the main DB. This indicates that the stature was set right (keeping many backups, etc.) but there's a flaw somewhere. Somewhere there must have been a way the hacker went from getting access to Backup "A" to "B" to "C" then Main "D" and so on.

I'd think this is more of a security breach; and they should investigate the servers right away. But, the main reason this has affected me is the rollback was before I opened Sarora Hosting (the domain was registered Nov 2008 :P!); so I have to get the three customers to re-post their testimonials.

However, this poses a good thing (I guess..) as my few angry posts at IGSoBE were deleted *causing quite a debate* and some more things. It's very unfortunate this happened, and it did affect me, but I think iNet should first SECURE servers then back it up..also the new Premiums in 2009 need to be handeled and whatnot. I'm sure iNet lost something financially of course, and they are probably under a lot of stress from 200,000 members plus these huge companies paying thousands for banners and advertisements.


Overall the team has been great responding to helpdesk tickets (http://webhostingtalk.com/helpdesk) and they seem to be doing EVERYTHING they can. Just waiting for Mat and iNet to post if they do have a recent backup or not, as we all are. Thanks WHT. :D.

People you need to quit complaining. To recover a Corrupted Database is a hell of a lot of work. TO also find out how they gained access can be even worse. I have been hacked before. It sucks. But it was easy for me to recover because i am an Ex-Hacker. If i was iNet i would hire a very good hacker to help them out.

That makes sense -- the government actually has hackers as well employed to work for them (some serving jail time probably -- helping the gov't as jail sentence) as they know the best..they can trace backwards. :stickout:

Passwords are hashed with salt. It would be an unprecedented event to reverse engineer our passwords.

Great so maybe it would be an unprecedented event however why make it more possible by letting the hacker know publically here what encryption method they should start trying ? Thats very similar to the news detailing movements of political figures etc so that any potential sniper knows the exact route they can place themselves on to shoot someone.

I lost around 400 posts in this so certainly not as many as others have.

Just to clarify if you're looking to restore a newer database than this one at some point are you also going to be merging new posts ? i.e. This one I am making now.
If not I guess there is no point in posting until the newer database is restored, correct ?

If not I guess there is no point in posting until the newer database is restored, correct ?

They aren't even sure if they have a newer DB, and they said there's a OK chance there isn't. :hammer:

I honestly don't see how you can say that, they had back-up servers, firewall servers, file servers, and the server that WHT vBulletin is actually on.

The hacker attacked all of them, at once, since WHT is a free website, it's not like they have a team on 24/7 stand-by for these attacks.


*EDIT: They do all this for free, **** happens, get over it. WHT will be back to normal as soon as possible, if not we'll just have to rebuild the past 6 months and try harder to prevent this from happening again.

I think one of the few reasons people cant get over it is some people paid for the premium section which they dont have now * its a cheap payment but still* and some of the people that have spent 2+ hours making a post to help out to have it deleted

To recover a Corrupted Database is a hell of a lot of work

Bull
To recover a database, you're looking at a few hours yes, but that few hours can just as easily be spent restoring it from the most recent backup, IF a proper backup setup is in place. Chances are, THAT is going to be faster, and should (theoretically) be easier to do.


RackSpace is a HUGE company, and they're really not to blame

The size of the company doesn't matter, but, you're right, RS is not to blame here. In the end, that falls on iNet for not checking up, for not making sure things are secured, for not ensuring the stability of their own backups.


nor is iNet

There, you are wrong.
Who is responsible for security of WHT servers? Not rackspace, but iNet.
Who is ultimately responsible for backup retention? Not rackspace, but iNet
Who is ultimately responsible for security updates? Not rackspace but iNet.

In the end, iNet is, very, very responsible.

Wow a lot of arm chair quarterbacking here as none of know has WHT has things setup or their budgets.

I do have one question that others have seemed to ask, why were the backup servers on the public internet in the first place if that is truly the way the intrusion happened? I'm assuming the backup servers are in the same data center, if not, that could explain it.

Why were there not appropriate firewall acls on these backup servers if they were publicly accessible?

As Ross said, why are the backup servers publicly accessible? Backups should be done over a private interface, it seems that was a very crucial mistake on WHT's part... at the very least the backup servers should have been locked down to access only from authorized IPs (even if they are accessed through the public internet). Doesn't seem like this was the case here at all.

Think they main key here is home backups.

You can have backups at every data center imaginable, but if you don't have backups in your very own hands physically, you are at risk.

Though, I'd be curious to know how the hackers managed to find the WHT backup server.

Is there a possibility it's an inside job at rackspace? I don't want to blame them or jump to conclusions though, but you never know, could be a malicious employee or something. Of course, that would be termination on the spot, so is it really worth it? Probably not.

Is there a possibility it's an inside job at rackspace? I don't want to blame them or jump to conclusions though, but you never know, could be a malicious employee or something. Of course, that would be termination on the spot, so is it really worth it? Probably not.

why not an inside job at inet? why does it have to be rackspacE?

As Ross said, why are the backup servers publicly accessible? Backups should be done over a private interface, it seems that was a very crucial mistake on WHT's part... at the very least the backup servers should have been locked down to access only from authorized IPs (even if they are accessed through the public internet). Doesn't seem like this was the case here at all.

True, the servers should not be on the public net. Access is possible, if for instance passwordless public key auth (set up for rsync or other backup method over ssh) could be accessed once the web server was compromised. But, it should be the backup server connecting to the web server, not the other way.

By no means am I saying it was this way, just saying it is possible.

Think they main key here is home backups.

You're kidding, right? 5.5 million + posts, over 200k registered users. You're not talking a small database, you're talking something that's more around the size of 50+ gig most likely (nobody but iNet knows the exact size). A "home backup" would take even the most intense download connection days to go through. On top of that, you have trust issues there as well. Who do you trust with the data? I know there's a few up there that I wouldn't trust personally.

why not an inside job at inet? why does it have to be rackspacE?
very true as well.


Even IF the backup server wasn't public, why was the DB server? It's quite possible to have a db server running on a VPN, feeding a public forum, so that the hacker then has to guess the db server as well.

Meh, we are all doing a bit of armchair quarterbacking, myself included, but the fact that this has happened multiple times in the past year is, in fact, quite disturbing, and shows just how lax security is here.

You're kidding, right? 5.5 million + posts, over 200k registered users. You're not talking a small databae, you're talking something that's more around the size of 50+ gig. A "home backup" would take even the most intense download connection days to go through. On top of that, you have trust issues there as well. Who do you trust with the data? I know there's a few up there that I wouldn't trust personally.

I agree with you here. 50+ gigs on a daily basis, the bandwith costs, ahh! Yeah, it all comes down to trust. But, RackSpace is a trusted organization for sure..I mean they make $400 million yearly, have a stock, are a reputable company for many corporations.

However as you said linux-tech, the WAYS they backed up were incorrect. VPN would be a choice.

Maybe you should help them out; you seem experienced :P.

Good lord I need to start visiting this place more often.

I disagree with it being 50gb.

I have a client who had 44 million posts, 250k users over a million threads, and it was only 25gb.

I disagree with it being 50gb.

I had a client who had 44 million posts, 250k users over a million threads, and it was only 25gb.

Yeah, WHT puts like LOADS of advertisements, banners, addons, mods, hidden trackers, and more.

SQL dump files compress fairly well being plain text. It would probably be a few GB at most. The home backup does not need to be daily, maybe weekly, but at least you know your data is in your own hands and not thousands of miles away.