I’m constantly amazed at the conspiracy theories that abound about my forum network, especially since our early January 2006 crash. Most people have been very supportive and understanding. There’s a small pocket of people that have been absolutely toxic about the situation, each one seeming to have their own special conspiracy spin on everything.
I refuse to constantly explain and defend the network across other people’s forums. Several people have acted as spokesman and have been no help at all, in fact they seem to encourage flaming. If you have a question you can e-mail me. When I do explain in part, those that are mean-spirited have a tendency to twist my words and/or fill in with conspiracy details that don’t exist. If you want info on blackouts, then come here to www.conradaskland.com. I’m happy to provide info here. Some people said they have contacted me via regular mail and/or phone – my mailing address has changed as well as my phone numbers since I’ve relocated up to Washington.
So for those that are interested, here’s all the details of what happened. There’s been a couple of people I have detailed everything to, and as I tell them I notice their eyes start to glaze over. It’s not highly interesting, and I admire anyone who has the fortitude to read all the details here. But here it is:
First of all, to answer a few questions that come up ALL the time:
WHY HAVE THE NETWORK?
When my company was very prosperous, I set up the forum network as a tithe back to the community at large. It was our goodwill donation. When my company scaled back I have tried to keep the forum network going because I have seen so many lives positively impacted by it. It’s fun and I’ve made a lot of good friends through this network. When my company scaled back I started incorporating some ads to offset the costs of running it. Running a couple boards doesn’t cost much, but I’m running about 100 – so yes, there’s some cost and time involved in running it.
DO YOU MAKE MONEY OFF THE NETWORK?
No, I have never made a profit off it. Making a profit was never the point. Many people have pointed to a grand master plan – but there never was one. It just is what it is. I try not to badger people about donations, etc. – but that option is available for those that feel like donating from time to time.
WHY DOES THE NETWORK KEEP CRASHING?
For those that run a couple forums and compare it to what I’m runnning, I’m sorry but there’s no comparison. Because my network at it’s conception was tied in to my other networks, there are many processes that run across multiple servers and call information from non-local databases. A couple people have offered to temporarily host certain forums, but to do that I have open IP access to our other networks to pull in data, and they would be privy to all the backend info that accesses our other servers – I’m not comfortable with that, in addition I have to deal with members of other communities wondering why THEIR forum isn’t coming up. At the time the network was created I had a full time staff of techs that would oversee it and make repairs as needed. In January 2005 my servers were completely compromised due to shoddy security work by my former techs. I got things back and running, and in mid-2005 had some new techs work primarily on disengaging our forum network from my other networks. In the last half of 2005 I started to learn how to do upgrades, etc. on our forum software and how to navigate our custom server and database setups. What I am working on now is simplifying our forums and splitting them across multiple servers so we never have a full blackout on all forums.
WHY DID WE CRASH?
Our server was compromised. The software we run, phpbb, has a long-standing problem of security exploits which are uncovered over time by hackers. That is why they release a new version about every two months – primarily for security reasons. Our server was compromised in early January and the hosting company shut down the server. It is normal for hosting companies to monitor servers and typically will shut down a server if it’s compromised to keep the rest of their networks safe. Were our sites specifically targeted? Probably not. Hackers that exploit the security vulnerabilities of phpbb typically have software that is automated to search out sites runnning phpbb. Why do they hack? Usually to use the server for sending spam. The hosting company says our server was being used to send spam. PHPBB released their version 2.19 on December 31, and I had all forums updated the next day on New Years Day. So I’m assuming the hack was from a previous version release, or maybe an exploit of a pre-2.19 – So the next question would be, why don’t I switch all forums to different software? To be clear, most all forum software has occasional security problems, just that phpbb historically has had more. Because we’re running 100 forums, it would be cost-prohibitive to switch all forums over. I did buy a different software format in December 2005 and was experimenting with that as a text for future conversions – so in time we may switch. Having 100 phpbb forums makes us a prime target for vulnerability crawls.
WHY DIDN’T YOU EMAIL EVERYONE THAT THE SERVER WAS GOING TO SHUT DOWN?
This is one comment I have seen a lot, and befuddles me a bit. Our server was compromised and terminated by the host, there was no warning. Some people seem to think our server was shut down over a money issue, and that I had warning. If it had been a money issue, then yes I would have had warning – but I’m more resourceful than that and would have found a way around it. But, that wasn’t the case, the server was compromised and shut down by the host.
YOU SAID THE SERVER WOULD BE BACK UP IN A COUPLE DAYS
Yes, I did. Originally the host gave me two options – 1) Have their techs reformat the drive and reinstall from my former hard drive. But I would have to send them a list of each file to be copied over, anything not on the list would be omitted and there was not guarantee of functionality of their work. This server, needing to pull in info from other servers, etc. I thought was far too complicated for house staff, and tech support was very negative about what the outcome would be. 2) Option two was for them to Fedex the drive to my server admin who could reinstall info remotely. I chose this option and thought it was a good opportunity to review security on the server and give the tech a chance for suggestions on streamlining the server. This tech works on several of my other servers and had done a good job up to this point. The PLAN was that the server would be with the tech in two days, within two days after that we would have all sites back up. This is where everything fell apart.
GETTING THE SERVER BACK – THE DRAMA BEGINS
First mistake, I had the hard drive sent to my tech on the East Coast, which is where I send all my payments. Turns out that’s just the head company, he lives on the West Coast and hadn’t told me until the server was enroute. We had the server rerouted – but by this time it took a week for the server to reach the tech on the West Coast. By this time he was in Florida working on a major install job. He told me no problem, he would have one of his employees mount the hard drive and he would work on it remotely from Floriday. He had a four hour block at a certain time, which was now a week after the crash. I called to verify he was ready to work, then had the server company re-open the ports to the server. The had reinstalled a fresh version of the operating system and were waiting for me to tell them to open the ports. So at this point, I’m thinking everything is cool and the tech said we’d be up within a couple days, maybe even the same night.
Three days later we’re still not up (Now 10 days after the crash). I contact the tech who says it will all be finished the following day. The following day he calls me to say he can’t get on the server. Turns out he never started working on the server four days before, he thought he would work on it later. What the tech didn’t understand is that when this particular host opens up a server – they open it up with absolutely no firewalls or security measures. They leave it wide open – so of course within this four days the server was hacked again. I called the hosting company to tell them to reformat the hard drive a second time and to wait for my call to open it up – the hosting company tech support said my account was under review because they think I’m a spammer. I had to wait three days for them to say they reviewed my account and would keep me as a customer. Meanwhile I relieved my tech of duties and had him send me the hard drive – I didn’t want him messing up again. In retrospect, this was a mistake of mine – Maybe I should have given him a second chance, but I had lost trust in him after he misled me.
At this point I’m leaving out of state for a three month job. The job is time consuming so I can’t devote lots of time to getting the forum server back. I spend another 10 days out of state trying to find a tech to assist me, had one setup and he kept flaking. Went online and found a good tech in Sweden, we finally arrange a time where we can chat online while he works on the server – it’s now the end of January – I call the hosting company to open the ports and they inform me my account has been closed and that I’m a spammer. In mid-January I told them to do a third fresh reinstall and to close all ports – that I would call when I was ready to open. Turns out they did a fresh reinstall and left it wide open again! Unbelievable. I called tech support to inform them of their blunder, and was told they wouldn’t talk to me because I was a spammer. I called customer support and was told the same thing. I think I threw the phone across the room and broke it on a wall. I was really steamed. So, there’s the first month of what happened.
I still cannot get around how a hosting company could open a server wide open with absolutely no security, and without notifying the owner that the server was open. An impossible scenario.
I’m getting kind of bored of detailing all this, pretty dry stuff. Add to this trying to get my server hard drive mounted so I can use it remotely, interviewing for other jobs in my new area, not having my computer gear with me (it was all back in California), and needless to say everything was very time consuming to deal with.
I spent about a week looking at new hosts and now have a server setup with a new company. I’m going to go to California and ship up all my computer gear. It will be a little slow getting the first forums up. I have to clean things up a lot and disengage our dependency across different servers. I’m also going to split the network up into subgroups.
And last but not least, some people seem really steamed I didn’t return their emails. I received over a THOUSAND emails on my AOL account in mid-January, most regarding the forums. I just couldn’t respond to them and it clogged my account so I had to delete them. My company emails don’t have a limit like AOL does, but I didn’t have access to them because all my passwords were back in California. Many of the processes I need to use are IP restricted, and being out of state on a rotating IP I didn’t have quick access.
Check back, I’ll post news as forums become active again.
