A couple weeks ago a board on my dads site was hacked and pretty much demolished. Apparently it was hacked into on Tuesday/Wednsday. I didn’t find out until Thursday night when my aunt mentioned it in passing.
Friday after work I went and took a look at it real quick. I opened up the url to the board and it wasn’t functioning at all, php errors all over the place. At right about that time I got a call from Helen asking if I could pick someone up from the bus station and give them a ride down to tacoma for a board meeting they had this weekend. Faced with the decision of what was obviously going to be a long process that would stretch late into the night vs hanging out with Helen and the Circle K district board, I opted to go hang out with them.
Saturday morning rolled around. I got up extra early to work on it, I remoted into the server and started poking around. Files were moved around, odd names were all over the place. A quick check of the bash history showed they hadn’t gotten the password for the account. I changed the ftp password on the board account just to be safe. Right after that I got a phone call. Was an odd one, the number was 12 digits long. I answered it anyways. Turned out to be the man who ran the forum, he was in the UAE. I spoke to him briefly, he had a database backup from May 9, he was going to upload it so I could do a restore. I emailed him the new password (not very secure, but what the hell) and he started
to work on uploading it. I had a full site-backup from April 27 that I had made for kicks, since we didn’t have anything really backed up, it included logs, built-in mail account, site backups, databases, everything. I considered using the database backup in that one, but didn’t, since he forum admin had a more recent backup.
I continued to poke around the files, everything was in complete disarray. I restored the board files using my April 27 backup, so all that was needed was a database. I poked around a little, then bummed for a bit while I waited. I got an email from motasim, he was having trouble transferring the file up. the ftp client kept timing out. I recommended a few clients and he tried them all with no luck. Finally I suggested he compress it with winrar. He grabbed the client, compressed and uploaded the file quickly.
This is where the fun began. Everytime I tried restoring the database, I would get a duplicate line error message from mysql. I spent literally 2 whole days trying to get that database to restore. Finally Sunday night I grabbed my April 27 backup and tried that one.
Worked flawlessly.
Suddenly we had a perfectly functional forum. But motasim wanted to have the most up to date backup, so I tried merging the two databases. It sort of worked. The threads were all there but there were more php errors caused by incomplete tables and entries. Finally, I restored the April 27 database, uploaded it somewhere motasim could get it and let him play with it.
He finally settled for the April 27 backup. I posted on the forum softwares support site, contacted their support, after speaking with them it was apparent that Motasims May 9 backup was corrupt, there wasn’t anything we could do about that.
After having dealt with all of that, I started reading up on creating backups. I setup my own backup procedures overnight and implemented them. I quickly found out that bluehosts cron doesn’t like the $ $(date). I posted on their forum and someone helped me whip up a perl script. I’ll post it here in case someone stumbles upon this post devastated by their website having gone down without backups:
#!/usr/bin/perl
use strict;
use warnings;
my $fileOrFolder = $ARGV[0] or die “You forgot to specify a file or folder name!”; #The name of the file or folder provided during execution of the script
my $upper = “U$fileOrFolder”; #Convert the file or folder name to uppercase for the label
my $date = `date +%Y_%m_%d`; #Store the date in a variable
chomp($date); #Remove the new line that gets added at the end
print `tar—cvvzhlf “/foo/backups/$fileOrFolder-$date.tar.gz” /foo/$fileOrFolder—atime-preserve—label $upper`;
Simply put, I set it up to backup the individual sites based on the folder I specified when the script ran. I use another one slightly modified for full backups, and another for sql backups. There were a few hiccups with getting them running, but they’re working beautifully.
Unfortunately, one side-effect I hadn’t considered was that when these backups ran, they created gigabytes of backups. Our account was on a server that had a 50GB limitation. We had setup this account several years ago, and Bluehost had since upgraded the plan we were on with more generous limitations. The most recent being unlimited disk space. If we had wanted, we could contact bluehost and have them move us to another server.
The disadvantage of that would be downtime, so I never bothered with that. However, apparently when the backups ran, they got an alert of some sort. So without any warning, they moved us to a new server last night around midnight. All our domains went down for a good 8 hours, and when they finally came back up, php wasn’t functioning properly for another 4.
So the moral of this story? Schedule your backups, and make sure your host doesn’t decide to randomly upgrade you without warning by making sure you don’t randomly create gigabytes of files in the course of a day.