My life has pretty much been taken over by work. I get up, go to work, come home, sleep, then wake up and repeat it all again. On the weekends I hang out with Helen. Work has been going pretty well, learning new things every day, meeting people, and learning important lessons along the way. Looking back I can tell I’ve grown quite a bit just in the past two months. I was talking to a Kiwanian a month ago who runs a Technology firm in Seattle, I was talking to him about what I’ve been doing these past couple years and how I’ve been working for my current company.

He said that while you may not enjoy working for a large corporation and supporting thousands of people, it’s an important, if not vital, experience that everyone needs to have. You can tell when someone has worked for a large corporation, their work ethic, how the interact, plan, and work, is different. He didn’t really elaborate on this. However I think I get what he was talking about. The company I’m at has many different systems, lots of internal applications and services that I’ve never seen before. Working with all of them, I have to think on my feet. In order to do that, I can’t just go off the same old check X settings. Now it requires I have an actual understanding of the system itself and how it works. I need to see what’s going on, what’s different, and what I can do to fix it.

Sometimes it requires I call up other people to ask a question about a specific function of an application. I’ve run into a lot of roadblocks with this, and actually had some people complain about me asking questions. I didn’t quite grasp the concepts of going through the proper channels for asking questions. I do now, and still think it’s stupid. I’ve run into situations where I’m needing to know how to configure a citrix application, going to the man who works on that, I have to speak to another tech, then speak to the lead, then the manager, and finally the manager refers me to someone in the application support department. After that, I speak to that person, who refers me to the man who works on it.

I’ve noticed there are a lot of processes in the company that are out of date, and it requires jumping through a lot of hoops to get those changed. There’s a lot I want to do there, and I’ve tried to get the ball rolling on some things, but despite what I’ve tried, I won’t be able to do anything. There are a lot of things that are preventing me from doing so, the biggest one of all is that I’m a contracter. Despite the day to day interactions I have with everyone, the relationships I’ve built, at the end of the day I’m still a contracter. I’ve tried to get employed direct, however they have their own internal employement process which is geared more towards hiring existing employees then outside employees (which it should be).

I’ve become a bit jaded by all this, and despite my desire to keep working there, it just doesn’t seem to make sense to stay at a company that won’t give me a real career path. It is really frustrating when I come to a company looking for real employement only to be tricked into accepting. When I decided to join this company, I was told by my agent that there was a good chance of being hired direct after 3 months. But this is incredibly unlikely. The reason this company hired contractors was because several people were promoted out of the department. They needed to hire some people to help with the workload while they found replacements. I was one of the contractors hired to help with the workload. Since then they have found replacements. Right now I’m just treading water.

I’m essentially just waiting for my contract to end.

So I figured out why Bluehost did a sudden server migration on me without any warning.

I sorta forgot to add a flag and exclusion to the full backup script. I didn’t tell it to ignore symbolic links and the backup directory. On the account site, it backs up all the directories in the home directory. There are a couple links to logs, cache, and then there’s also a link called www that points to public_html.

So not only did it backup public_html twice, it also backed up the backups.

I noticed this last night when I started the weekly backup on my machine that pulls down the backups and the full backup was 27GB.

A couple weeks ago a board on my dads site was hacked and pretty much demolished. Apparently it was hacked into on Tuesday/Wednsday. I didn’t find out until Thursday night when my aunt mentioned it in passing.

Friday after work I went and took a look at it real quick. I opened up the url to the board and it wasn’t functioning at all, php errors all over the place. At right about that time I got a call from Helen asking if I could pick someone up from the bus station and give them a ride down to tacoma for a board meeting they had this weekend. Faced with the decision of what was obviously going to be a long process that would stretch late into the night vs hanging out with Helen and the Circle K district board, I opted to go hang out with them.

Saturday morning rolled around. I got up extra early to work on it, I remoted into the server and started poking around. Files were moved around, odd names were all over the place. A quick check of the bash history showed they hadn’t gotten the password for the account. I changed the ftp password on the board account just to be safe. Right after that I got a phone call. Was an odd one, the number was 12 digits long. I answered it anyways. Turned out to be the man who ran the forum, he was in the UAE. I spoke to him briefly, he had a database backup from May 9, he was going to upload it so I could do a restore. I emailed him the new password (not very secure, but what the hell) and he started
to work on uploading it. I had a full site-backup from April 27 that I had made for kicks, since we didn’t have anything really backed up, it included logs, built-in mail account, site backups, databases, everything. I considered using the database backup in that one, but didn’t, since he forum admin had a more recent backup.

I continued to poke around the files, everything was in complete disarray. I restored the board files using my April 27 backup, so all that was needed was a database. I poked around a little, then bummed for a bit while I waited. I got an email from motasim, he was having trouble transferring the file up. the ftp client kept timing out. I recommended a few clients and he tried them all with no luck. Finally I suggested he compress it with winrar. He grabbed the client, compressed and uploaded the file quickly.

This is where the fun began. Everytime I tried restoring the database, I would get a duplicate line error message from mysql. I spent literally 2 whole days trying to get that database to restore. Finally Sunday night I grabbed my April 27 backup and tried that one.

Worked flawlessly.

Suddenly we had a perfectly functional forum. But motasim wanted to have the most up to date backup, so I tried merging the two databases. It sort of worked. The threads were all there but there were more php errors caused by incomplete tables and entries. Finally, I restored the April 27 database, uploaded it somewhere motasim could get it and let him play with it.

He finally settled for the April 27 backup. I posted on the forum softwares support site, contacted their support, after speaking with them it was apparent that Motasims May 9 backup was corrupt, there wasn’t anything we could do about that.

After having dealt with all of that, I started reading up on creating backups. I setup my own backup procedures overnight and implemented them. I quickly found out that bluehosts cron doesn’t like the $ $(date). I posted on their forum and someone helped me whip up a perl script. I’ll post it here in case someone stumbles upon this post devastated by their website having gone down without backups:

#!/usr/bin/perl
use strict;
use warnings;

my $fileOrFolder = $ARGV[0] or die “You forgot to specify a file or folder name!”; #The name of the file or folder provided during execution of the script
my $upper = “U$fileOrFolder”; #Convert the file or folder name to uppercase for the label

my $date = `date +%Y_%m_%d`; #Store the date in a variable
chomp($date); #Remove the new line that gets added at the end

print `tar—cvvzhlf “/foo/backups/$fileOrFolder-$date.tar.gz” /foo/$fileOrFolder—atime-preserve—label $upper`;

Simply put, I set it up to backup the individual sites based on the folder I specified when the script ran. I use another one slightly modified for full backups, and another for sql backups. There were a few hiccups with getting them running, but they’re working beautifully.

Unfortunately, one side-effect I hadn’t considered was that when these backups ran, they created gigabytes of backups. Our account was on a server that had a 50GB limitation. We had setup this account several years ago, and Bluehost had since upgraded the plan we were on with more generous limitations. The most recent being unlimited disk space. If we had wanted, we could contact bluehost and have them move us to another server.

The disadvantage of that would be downtime, so I never bothered with that. However, apparently when the backups ran, they got an alert of some sort. So without any warning, they moved us to a new server last night around midnight. All our domains went down for a good 8 hours, and when they finally came back up, php wasn’t functioning properly for another 4.

So the moral of this story? Schedule your backups, and make sure your host doesn’t decide to randomly upgrade you without warning by making sure you don’t randomly create gigabytes of files in the course of a day.