A false smoke alarm almost drowned Jane Street's only NYC datacenter

(michael.bacarella.com)

56 points | by mbac32768 42 minutes ago

4 comments

  • reticulates 9 minutes ago
    The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.

    “The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”

    https://news.ycombinator.com/item?id=4720388 (https://web.archive.org/web/20121101194314/http://status.fog...)

    https://arstechnica.com/information-technology/2012/11/how-o...

    • skinfaxi 5 minutes ago
      Diesel is ~7lbs/gallon. That is truly superhuman damn.
    • HighGoldstein 1 minute ago
      I wonder how well these "superhuman folks" were paid
  • hackermailman 23 minutes ago
    In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
  • rshnotsecure 19 minutes ago
    Surprised they even had sprinklers set up. Historically, things that avoid moisture like Halon Gas [1] have been used.

    [1] - https://en.wikipedia.org/wiki/Bromotrifluoromethane

    • dklsf 12 minutes ago
      If those go off, they end up destroying hard drives (the spinning kind), due to the sound pressure....

      So fire suppression and data centers don't mix too well

      • wolvoleo 2 minutes ago
        They don't have to blow that fast. It can be regulated.

        But yes, I've seen low-O2 installations also. These have other issues though, like no protection when humans need to be in there for maintenance. And the need to be very very careful when reducing it again, making sure everyone is out. It happened to my colleague once that this went wrong and while it's not low enough that you'll die, he wasn't exactly feeling great for the next few hours.

      • shagie 6 minutes ago
        Ahh... the Vax War (bit hard to search for it now... gotta search for the first line). VAXen, my children, just don't belong some places.

        https://www.hack.org/~mc/texts/vaxen-joke.txt

        Realizing that I'm spoiling some of it... but we're here, so some of it is inherently spoiled...

                   The fire alarm klaxon went off and the siren warning
                   of imminent halon gas release was screaming.  We
                   started to panic but the data center manager shouted
                   over the din, "Don't worry, the halon system failed
                   its acceptance test last week. It's disabled and
                   nothing will happen."
         
                   He was half right, the primary halon system indeed
                   failed to discharge. But the secondary halon system
                   observed that the primary had conked and instantly
                   did its duty, which was to deal with Dire Disasters.
                   It had twice the capacity and six times the discharge
                   rate.
    • mbac32768 14 minutes ago
      Other datacenters we built had them (which scared the shit out of me tbh). I don't remember why this one had to stay on sprinklers.
      • dklsf 11 minutes ago
        Maybe because it was in a normal office building? The gas extinguishers might be too hazardous there
    • hvb2 13 minutes ago
      Or CO2 but that's a life or death situation if there's any people still inside
      • thechao 12 minutes ago
        Halon is not exactly fun to breathe, either.
        • wolvoleo 5 minutes ago
          Especially after it's been in contact with fire, the resulting products are even worse.

          But it's only used in military and aviation these days anyway.

    • gosub100 11 minutes ago
      Probably not safe to use on a floor that has people on it.
      • wolvoleo 10 minutes ago
        We had it at work at a datacenter right beside offices and reception.

        You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.

        PS: Halon itself is of course gone but it's other less harmful gases these days.

        What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).

    • doctorpangloss 15 minutes ago
      [flagged]
  • wolvoleo 11 minutes ago
    How is it the only datacenter?

    33 Thomas St is basically one huge datacenter, that's why it doesn't have any windows. https://en.wikipedia.org/wiki/33_Thomas_Street

    • maronato 0 minutes ago
      The headline is confusing. It’s Jane Street’s only data center in NYC, not NYC’s only data center.
    • advisedwang 0 minutes ago
      It's Jane Street's only datacenter. not the only one in NYC. As they explain, they chose to specially locate their datacenter.
    • kccqzy 3 minutes ago
      Jane Street’s only NYC data center. Not the only data center in general.
    • dklsf 9 minutes ago
      This was pre-Hurricane Sandy, so before 2012. I'm guessing data centers have grown since then
    • lelandfe 9 minutes ago
      Work nearby, it looks just as evil in person.