Skip to main content

IP-hijacking and then some

Almost a month of downtime. The servers were fine, but our IP addresses were not. Somehow, a system bug from one of the hosting companies caused our plan to include 14 other IP addresses, and we only maintained 6 as far as I am concerned.

So the hosting company eventually understood what was happening and started asking the right questions. The bug is that the system allowed network administrators to assign to new clients, the IP addresses that have already been assigned to a current client.

We spent crazy hours figuring out what happened. Since the domains were not loading our sites, and instead were redirecting to some domains we don’t own (real bad sites btw), we checked if .htaccess was modified and tested if it was working at all. I also did double checks on virtual host configurations and they all looked clean.

I started thinking that it was a routing issue. We were not using the publicly routable IPs for the virtual hosts; instead, we used internal IPs as mapped to the firewall hardware. A lookup on the domain/s that we were being redirected to reported different IP addresses, which didn’t readily confirm my suspicions, but stressed the possibility of IP-hijacking.

It’s not supposed to be taken in the context of BGP (border gateway protocol), as this is beyond me. Instead, think of your assumption over IP addresses that we’re using, to serve your websites. Sounds clear? I hope it does. I got pretty lost myself and it isn’t after deep digging at the hosting company’s end that their errors became apparent.

So in the light of this realization, I’ve already moved the domains to some other hosting. Way too much work and way uncool.

Comments

Popular posts from this blog

Uncle Bong in SG

It's been over two decades since I last saw Uncle Bong. He would usually drop by Diamond Village in Novaliches, Quezon City to visit Lola Magding and my mother's side of the family there. Back then, I was still in college, and I think he had also just started working, in the automotive industry. He has since worked in cruise ships and it was a good stroke of fortune when he told me that they would be dry-docking in Singapore. Mama's side has always been close-knit and could see the fondness in his eyes as he recalled his dealings with Lola Magding and her children, Mama included. He would then share stories of how many of our relatives emigrated off the "isla" to find better opportunities in Manila. Given the brief opportunity, I'm still blessed to have seen him again after all these years. I am also happy that he was able to meet Mae and his "apos". Until we next see each other, Uncle Bong!

Being a Systems Administrator @37signals (Podcast)

 I use Basecamp a lot. Kudos to these guys for making my job as a Systems Administrator more bearable than usual. :p Episode #12: Being a Systems Administrator at 37signals Time: 22:50 | 04/13/2010 |  Download MP3  |  Link to this episode Mark, Joshua, and John on life as a 37signals Sys Admin The Sys Admin team discusses hosting the 37signals apps, working with programmers, helping support, telecommuting, dealing with vendors, improving speeds in Europe, and more. Related links: Rackspace Dell Definitions: Redundancy Facebook Just Gets It Right via  37signals.com  
March was a good month. Not just because it's my birthday month, but I've been able to do more physical exercises as far as I'm allowed, and I've also been able to get through my first month back in the job. For the first couple of weeks, I thought I was going to do 4 hours a day at most, and turn up the heat bit by bit until I'm able to do 100% again. But I suppose having to rest over a month does have its toll. It took significant effort to try not go all in with guns blazing on the first day back at work. LOL Anyway, I'm trying something new this year. We've migrated our applications to a better platform, and we'd like to push that to the cloud this year. It'll obviously need a lot of liaising and cooperative work with other teams and domains so I'm going to do more engagement and solutioning than the usual engineering this time. Last week, we did a platform-related hackathon and a critical refresh across clusters. There's going to be a lo