(The following story by Rafi Guroian appeared on the Austin American Statesman website on August 27.
AUSTIN, Texas — Amtrak spokeswoman Karina Romero was nice enough to give Cox Newspapers a post-mortem followup to this past weekend’s complete shutdown of Amtrak’s reservations and ticketing systems:
Amtrak expects to identify lost revenue as a result of the system shutdown, but will be looking for a spike in sales toward the beginning of this week from people who are calling back in to book reservations they intended to book over the weekend. That said, Romero suggested that the amount of revenue lost may not have been as substantial compared to what might have happened if the shutdown occurred during peak booking hours. “We’re typically not as busy on a Saturday or a Sunday as compared to a Monday,” said Romero.
“For ARROW to go down [across] the entire nation was unprecedented,” said Romero. In cases where ARROW goes down at individual stations, the practice in the past has been for station ticket agents to call other stations to assist in looking up reservations. Because every terminal in the country was down, it appears that some station agents may have become confused as to what the standard operating procedure should be.
Romero says that the correct procedure, which was observed in New York Penn Station among other locations, was to permit passengers with pre-paid reservation computer printouts to board trains; Amtrak conductors collected those printouts, using them as tickets. Other locations, however, like Amtrak’s Baltimore-Washington International Airport station, were reportedly instructing customers with existing pre-paid reservation printouts to purchase duplicate, hand-written tickets at full price from onboard train conductors; station personnel reportedly promised passengers that the older reservation would be refunded if the passenger phoned in a request when ARROW came back online. “My fiancee was actually told to purchase a new ticket onboard the train at BWI, and when she got on the train, the Amtrak conductor instead collected her reservation print-out,” said David Johnson, Legislative Director of the National Association of Railroad Passengers.
The cause of the system shutdown was unclear until today. Initial unconfirmed reports from station personnel in Providence, Rhode Island indicated that a lightning bolt had struck a mainframe facility in Manassas, Virginia that houses many of Amtrak’s servers. A later report from AP indicated that the failure was due to a new piece of software being installed on the ARROW servers. Romero confirmed today that that initial diagnoses were incorrect, and the shutdown was actually related to a power failure at Amtrak’s Manassas mainframe facility: “There are three circuit breakers and one failed. All three must be active in order for them to work.” As a result, all connectivity with the Manassas data facility was severed.
Amtrak has also provided a company-wide memo that circulated today:
At 7:11 a.m. EDT on Saturday, Aug. 25, the Manassas Data Center building experienced a failure on one of the three major power distribution panels providing electrical power to the data center (each panel is capable of supporting twice the current utilized capacity of the data center). Key network components and many application servers were lost as a result of the power outage, however, the mainframes and other components remained available, but without network connectivity. All Revenue and Business systems were brought down and a replacement panel was located and shipped from Connecticut overnight. An additional panel was located in Texas and airshipped overnight to the data center to provide spare capacity. Revenue systems were brought back online starting at 1 p.m. EDT on Sunday. Recovery of business systems continues with identification of non-essential batch, resolution of minor outstanding issues and prioritization of remaining batch applications to recover all production business applications.
Romero stressed that despite the failure, no trains were delayed and Amtrak is currently in the process of evaluating what went wrong and how to prevent it from reoccurring in the future. Johnson adds, “One point to consider: at least when Amtrak’s reservation system goes down, the trains still run! When an airline system quits, so do the flights. It may not be perfect, but you do get to where you’re going.”