You don’t do any work on Friday in IT. If it goes wrong, you’ll be there all weekend fixing it. So, in the spirit of being careful, friday afternoon drinks were a tradition. 4pm Friday was beer o’clock, and as the resident only-person-not-excited-by-Crown-Lager, responsibility for arranging the drinks fell to me. No big deal right? Except that this was the day that I finally got an unlimited account with the local liquor store that would be billed to the company automatically. I wasn’t going to waste it. I did not waste it. Our small 10-person company got rip-roaringly drunk. Like ‘arrested for being outside in this state’ drunk. There were Jack Daniels cans stacked to the ceiling. Chips had fallen liberally to the floor. Someone couldn’t find a bin and filed a chicken wing in the file cabinet, under ‘C’, for chicken. It was one of /those/ drinking sessions where everyone is just a total mess. Around 9pm, after five solid hours of Aus-Spec partying, we broke off and headed into the night. I wandered down to a nearby bar and watched some bands play for an hour, downed another jug of beer, and smiled to myself that the week had ended. Fate, it seems, is not without a sense of irony. My phone buzzed in my pocket. I ran outside, tripping up the stairs as I went, managed to steady myself against a signpost, and answered. It was the CEO. The primary and secondary route servers were down. I stood frozen in time for an instant, the same way a deer looks at the headlights of an oncoming car, and then asked him to repeat himself. CEO: YES BOTH THE ROUTE SERVERS ARE DOWN THE ENTIRE STATE IS OFFLINE GET IN THERE NOW FIX IT DO WHATEVER IT TAKES I cannot stress enough that these two servers were the most important thing our company had. They, in and of themselves, were the primary thing around which our business existed, and all other things were secondary to them. My state was by far the biggest, with some of the biggest ISPs and content providers in the country attached. And this was the first full network outage we’d ever had. And it was my problem. And I’d consumed enough alcohol that my blood could have been used as a fire accelerant. I yelled .. something, and ran off in the direction of work. It was only when I bumped into the glass front doors before they opened that I started to realise how drunk I was. When the elevator arrived at my floor, and I bumped into both sides of the hallway before making it to the door, I knew I was in trouble. That hallway was only 20 feet long. But it didn’t matter. My wallet hit the card reader. I’d made it. Habit’s a funny thing. You get so used to the noises, clicks, beeps and responses that you realise something’s wrong in an instant. There was no response from the card reader. An error, surely? Interference, something new in my wallet? I dug the card out, throwing my wallet on the ground and badged it on its own. Nothing. Not an ‘Access Denied’ six beeps, or a ‘Card Format Unrecognised’ five beeps. Nothing. The lights were on, but no-one was home. A few feet away, the keypad for the alarm was lit up like a headlight convention. All the lights were on, the screen totally blacked out. No beeps for keypresses. Just .. nothing. The blood drained from my face. The route servers were inside, suffering some unknown fate, our customers probably getting more furious by the minute, and I /could not open the door/. AGAIN. No, sod it. I wasn’t taking any more of this security system’s crap. I was getting into this datacentre, security system be damned. You all know what I’d tried before, and I knew as well, so I didn’t bother trying again. My tools, once again, were behind the locked door, and then the light went on over my head. Chhopsky: I can’t .. go through the door … I can’t .. go AROUND the door .. I can’t go .. UNDER it …. but can I go OVER it!? This is the logic of a drunk engineer; try all the dimensions! There was a chair that we left outside for people working outside the DC, so in my infinite wisdom, I dragged the chair over to the wall, and lifted a ceiling tile. Unlike the DC, where the ceiling tiles were weighed down with hundreds of heavy cables, the office was free and clear. And the wall itself stopped at the ceiling. So, pushing the tile into the cavity between the suspended ceiling and the concrete, I hoisted myself up into the ceiling. This did not work as well as I’d hoped because I was not very strong. I kicked and pushed off the wall, scrambling to push myself up onto what I now realised was a very thin wall. For those not familiar with a suspended ceiling, metal rods are drilled into the concrete block above, and a grid pattern hangs below it. Inside those grids are weak, light tiles basically made of a combination of cardboard and plaster. Looking at the predicament I’d gotten myself into, it became apparent that the only things that were going to support my weight up here were the tie-rods into the concrete. So I’d hold onto the rods with my hands, and lying prone in the ceiling, distribute the rest of my weight along the horizontal connectors. I’d drop down onto the file cabinet at the far end of the room, about 15 feet away. This plan was /flawless/. And it worked. For about 6 of the required 15 feet, upon which point my hands slipped, and I fell through the centre of the ceiling tile, towards the floor below. By some insane miracle, I landed mostly on my feet, scrambling ungracefully to regain balance, coughing up ceiling tile dust and god knows what else. Probably asbestos. When the coughing stopped, I ran over to the security panel, pulled the power, and plugged it back in. It beeped a single happy POST beep and hummed to life, making normal sounds instead of the endless buzzing it had been making before. My access restored, I quickly found the problem – a circuit breaker had tripped, and due to a wiring error on the part of an electrician at some point, both route servers had been wired into the same circuit, rather than the different feeds on different UPS’s via different distribution boards that they were supposed to. With a dustpan and brush, I set about cleaning up the nightmare my dramatic entrance had caused. It was not a small mess – ceiling tiles are about 5 feet by 2 feet, and this one had exploded. It took about an hour. After finally sweeping up all the mess, putting the ceiling tile I’d broken to get up there back together, and replacing the one I’d broken getting down, I walked my ass out the door, feeling smug that no-one would be the wiser for my ceiling entrance, and I’d have a grand story to tell. Monday morning rolled around and I was the last one in. Aaron stared at me. Aaron: What the hell did you do to my desk? Chhopsky: … wha? I walked into the office, and stared in horror. I don’t know what the hell I’d cleaned up but it looked someone had hit a bag of flour with a baseball bat. It was /everywhere/. How wasted was I? What did I spend an hour cleaning? And how in almighty crap did I diagnose an electrical circuit being miswired and split with no electrician tools of any kind? I have no idea. But what I did know, was how to break in. So I documented the procedure, and added it to the Tech Support Wiki.
