Hey, there! Log in / Register

Shorting power supplies spark fire in Downtown Crossing data center

Boston Fire Department at Downtown Crossing Macy's

Firefighters responded to the eighth and seventh floors of 1 Summer Street in Downtown Crossing late this afternoon where rooms containing uninterruptible power supplies and large batteries at Markley Group's data center began filling with arc'ing and smoke and then water when the sprinklers went off.

The outage hit MIT, which uses the data center for many of its servers. MIT says it hopes to have a bypass in place by 10 a.m. on Saturday.

Neighborhoods: 

Ad:
Like the job UHub is doing? Consider a contribution. Thanks!

Comments

There is a data center in that building. Lots of power cords, UPSes, HVAC equipment, etc.

up
Voting closed 0

Yikes!

up
Voting closed 0

Yes - many of these DC in old bldg's still have sprinklers...insane.

Anyone have any updates?

up
Voting closed 1

That the Markley data center was knocked offline. If I find out more, will post.

up
Voting closed 1

The whole thing. Yikes. That's impressive if true, but I have services live in there and they didn't go offline, so I sorta don't think so.

up
Voting closed 0

Yes, this cut off MIT's email for most of the evening (along with several other IT services). See https://3down.mit.edu/

Update - Email service has been fully restored (with the exception of a handfull of users), and is running out of an alternate location.

Electrical crews will be working through the night to run an alternate power feed that bypasses the damaged electrical equipment. The estimated time to restore power is 10am Saturday, June 22, at the earliest. Once power is returned to the data center, IS&T staff will begin the process of restoring access to affected services.
Jun 22, 22:27 EDT

Identified - Email service remains impacted by a fire and power outage in the OC11 data center. Estimated time for full restoration of email service is now 10pm.
Jun 22, 21:13 EDT

Investigating - An issue affecting the OC11 data center has impacted multiple services, including Exchange email, TSM backups, non-production environments for MITSIS, and other services. IS&T is investigating the issue and anticipates email service will be fully restored by 7:30.
Jun 22, 18:40 EDT

up
Voting closed 0

What do you propose as an alternative? Halon, Inergon, FM-200? They all have additional issues in a large data center environment.

up
Voting closed 0

Inergen, not inergon.

Often times you don't have a choice in which agent to use, because they have different requirements in terms of agent delivery rate/quantity, how sealed the space is, and so on.

up
Voting closed 0

Suffocation hazard, yes, but non corrosive, cheap, easy to find a supply (like when you havea GC lab built in an ISO module shuttling between Johnson Island and Hawaii) and nontoxic. After all, we aren't talking about a football field - just smaller sealed rooms rented to individual companies.

N also works - this is a system used for power turbine enclosures, among other water sensitive facilities:

/factsplaining

up
Voting closed 0

That looks like a great system for use in enclosed space with no possibility of a trapped person. A container on a ship is a fine use for it.
If you want to avoid water, probably the best would be an Ansul system. Dry chemical. Messy, but they do the job. They're everywhere. Go to a restaurant , look in the kitchen. Go to a gas station and look up. Fairly inexpensive, meets codes and the insurance companies love them.
No life safety hazard. 1010 Mass Ave will be happy.

I'm a native Code Talker. A 'codesplainer' but for real answers, call 1010. They'll know.

Batteries present a problem. How do you kill the power to the source of the power?

up
Voting closed 0

.

up
Voting closed 0

Wet fire suppression has been the choice for most companies for over a decade. Dry (gaseous) suppression is too difficult and expensive both from capex and opex perspectives for most datacenter facilities and clients. Fires in datacenters are incredibly rare for a lot of different reasons, dry suppression is difficult and expensive, and the money is usually better spent elsewhere.

You only see dry suppression in specific applications where protection of physical infrastructure is paramount, or where water can't be used because it (or shutting off power, as all sprinkler systems in DCs do first) would cause critical failure.

up
Voting closed 0

They don't "still", that's what was spec'd from the very beginning.

1)The systems shut off power to the room as part of the activation process and there are pre-activation alarms to give on-site staff a chance to intervene if the fire is manageable with an extinguisher or if the alarm is accidental (ie construction for example kicking up a lot of dust.)

2)Gas suppression systems are incredibly expensive to install (both in planning, preparatory construction ie necessary sealing of the room, and actual system installation), require at least one full system test with the suppressant (which is also pretty expensive), and a lot more inspection and testing, and if the room is big enough, you could be looking at needing to purchase, install, maintain, and train people on SCBA equipment.

3)In today's world, it is more resilient and generally useful to have clustered or replicated systems because they can provide coverage for more incidents than just fire.

4)Doing gas suppression for a datacenter the size of Markley is difficult because it's massive. It has the footprint of a football field if not bigger, and numerous floors high. All full of computers, networking, and telecom equipment. That said, *some* MDC customers have gas suppression; I've seen the systems. They tend to be used in areas where it the physical equipment and wiring/fibre is itself very important.

In short: spend the money on computers elsewhere, because they can cover you for things like equipment failure, theft, power failure, malicious action IRL, and so on. Money spent on gaseous suppression is money that only protects you from fire.

Swirrrly, you're one of the most prolific commenters here and you're honestly just like a mansplaining dude....you speak authoritatively with your opinion on all sorts of things you clearly know absolutely nothing about. I dunno if it's your PhD or just your personality but you seriously need to put a sticky note on your monitor that says:

"Do I have any education or training on the topic at hand?"

up
Voting closed 1

up
Voting closed 3

...on that solid and excellent foundation...

NFPA sets guidelines.

Rule one...life safety is paramount.

Rule two...ya, property.

If there's a life safety hazard, that comes first. Burning computer cables can probably floor you in seconds. Nasty stuff, like a car fire or other burning plastic. Dumping water on it is not the worst of ideas. It's like a manhole fire. Primary concern is to stop the spread of the fire. Nothing like a heavy stream of sprinkler water to cool and knock down a fire, especially in its incipient stages. So, you lose some equipment and save the rest.

Also good point about backups...how often do you replace servers? Is it worth it to cover all of them if they're trash in eight or nine years?

This was a battery backup room. All bets are off. Think Teslas in a building.

up
Voting closed 0

You make it pretty clear that you have never been to the Markley Data Center. It isn't open rooms, it isn't the size of a football field.

You might want to read what someone described below because you have never been to this facility. Your description of a wide-open rural warehouse-style data center says a lot about the last time you set foot in such a place.

Markley is made up of self-contained sectors - rooms, really - that contain an emergency in any single area. Like bulkheads on a boat. They do offer gas fire suppression if you want to pay for it, too. You might want to do that if your equipment is specialized.

Maybe you should stick to what you know.

up
Voting closed 0

When the building is worth more than the datacenter inside it, you have sprinklers.

up
Voting closed 3

Google the address 1 Summer St and look at the Satellite view
The reason the building is worth protecting is that in addition to whatever overheated and caught fire in some rack sits under all the stuff on the roof and most likely above the floor with all the telecom stuif to connect to wherever at incredible rates

ps my e-mail which is an mit.edu was hardly affected at all

up
Voting closed 0

Boston Fire Dept is the Authority Having Jurisdiction and I believe they require a "wet" system for fire suppression in the Data Center. Most Data Center Suppression systems in Boston are double zone interlock pre-action. Dry pipe sprinklers until two smoke detectors are triggered filling the pipes with water. Sprinklers won't discharge until ceiling temps hit 165F when the fusible link in the pendant melts.

up
Voting closed 0

If anyone affected by the Markley fire needs an alternative data center, endoffice (endoffice.com) is available all night and tomorrow to install equipment at a data center a mile away on the Charlestown-Somerville line.

up
Voting closed 2

Coresite on Inner Belt Road in Somerville is the best DC around

up
Voting closed 1

Jeff and the crew there are a great bunch

Unfortunate this happened - has been a well run facility for many years

up
Voting closed 0

up
Voting closed 2

I've had a few companies who have had equipment in that data center.

That DC is the Boston Internet Exchange, which over 100 companies have cross connects in that building. Essentially its the hub for much of the internet for the northeast.

http://www.markleygroup.com/services/boston-internet-exchange

As far as that building itself.. I think Markley has 4 floors in that building (1,2 are Macy's, 3-6 are Markley.. 6 being Markley's corp offices, 3,4, 5 being DC). Depending on who you are with and what service you're there for will depend where you go. But from what I've seen are large hallways of doors. Many have familiar names (i.e. Comcast, Partners, MIT, Harvard, RCN, etc) on them. Some doors just lead to are just shared spaces.. like my former company's. We had a cage where we had a few cabinets. Some companies just rent cabinet space too.

But each one of these rooms.. it's all separated and air locked. Each space probably has its own fire suppression, alarms, heating, cooling, and electrical systems. This is by design, and is one of things that make Markley.. well Markley (this DC is highly regarded in tech circles). It's pretty impressive if you ever can go inside... it's also pretty impressive knowing that it used to be a Jordan Marsh :-)

Regardless, I am guessing only *parts* of Markley are down. I just did a few traces to customers I know who are inside that inside, and they are still online. So I am sure its restricted to what customers used that particular equipment. MIT must be one of them.

but again it's what makes Markley.. Markley. One part can go down and the rest can still go. And tbh, if they were entirely down, we'd have so many internet issues in our region right now....... (Comcast & RCN having large presences in that DC)

up
Voting closed 0

There's definitely DC space on 7.
FWIW (or speculate away..) there was a notice for UPS Static Bypass scheduled for June 28th. That same UPS that was identified, is the one that malfunctioned. Stuff happens, just glad no one got hurt.

up
Voting closed 2

Yeah I only went by what i saw (remember) in the elevator :p Maybe there was a 7. I know they have a shitton of crap on the roof.

up
Voting closed 0

I work at one of the affected companies.

floor 3 is office space and the shipping and receiving area.
4 is datacenter. 5 is the office and some more datacenter.
6, 7, and 8 are all datacenter space.

This affected floors 7 and 8, they did a full power shutdown on those floors.

up
Voting closed 3

There was water in the basement as well.

up
Voting closed 0

Why is so much of the Internet routed through one building? You'd think in this day and age, with the cloud and everything, that there would be multiple redundant sites and paths.

up
Voting closed 0

That is indeed a problem for much of New England, and also in other metro markets too. NYC is getting more diverse lately with new players providing meet-me rooms outside of Manhattan (165 Halsey, NJFX, 1025Connect, etc).

Boston now has the Massachusetts Internet Exchange (MASS-IX), which provides internet peering exchange across multiple redundant sites and paths, as opposed to housing peering exchange in just one location.

up
Voting closed 0

Anyone who knows the basics about water and electricity knows that water sprinklers are bad design. Non water fire suppressing system is what's needed.

up
Voting closed 0

Lets see how many people attack you for daring to mention that such systems exist and are even an option in this particular center - while describing them as a football field or even mentioning halogens.

Nitrogen exists and many places use it.

I think someone has things mixed up with a grow.

up
Voting closed 0

Gaseous fire suppression is incredibly expensive, and they have gone off accidentally many more times than they have put out fires. Pre action sprinkler system (i.e. dual interlock and dry pipe) are the norm in electronic environments such as data centers.

In fact, a leading industry standards group is in process of recommending the cessation of gaseous fire suppression systems in that environment. If anything thing, gas systems are the outdated method.

up
Voting closed 0

Oracle went down for a time, too.
Let's face it: there's no good time to do maintenance on power systems - what's good timing for one client is never good for another.
I just hope whomever was working on the systems are OK.
My rule of thumb: "if it's more voltage than what it takes to start a car, I don't mess with it."

up
Voting closed 0