As of this writing, we are (literally) in the middle of Hurricane Sandy. The editors at Mobile Enterprise and sister publications are working remotely (so far), but the power is predicted to go out and the company’s servers are locally hosted — we are about 50 miles north of where the storm makes land fall — so we expect major disruptions to business. At times like this, technology is both the enabler of business and the downfall. Is your company ready for a natural or manmade disaster?
The center of the financial world is located in downtown Manhattan, or Zone A, which has been ordered evacuated by Mayor Michael Bloomberg. While many companies have other locations in the city, the entire transit system and tunnels have been shut down. There are no trains and few passageways into New York, nor are the subways running internally.
In this highly regulated industry, mobility is often not fully enabled, but The New York Times reported on Sunday that reps from several major firms said they were implementing “continuity plans to ensure that their operations continued relatively smoothly.”
Much of that involves letting employees work from home, and many firms plan to route trading activity through London during the storm, which is expected to be in full force for 48 hours.
From Google to Startup
Google cancelled a planned press conference where the Nexus 4 smartphone and Nexus 10 tablet were expected to launch and Nexus 7 news may have been revealed.
Nonetheless, the tech company came out with an immediate innovation in the form a “crisis map,” which provides users the ability to track the storm with a variety of features and find help, if needed.
It seems that the smaller tech startups, however, are looking to embrace the “downtime.” These smaller companies are obviously more agile due to size alone. In fact, Jason L. Baptiste, CEO of New York startup, OnSwipe, was quoted by venturebeat.com as saying, “We actually built our office to be a lot like an apartment, so myself and a few others are crashing there for a couple of days. I see the hurricane as an opportunity to be heads down and get a lot of work done.”
Preparing Your Devices
According to Sprint, wireless consumers residing in Sandy’s projected path should use the following tips to prepare:
- Keep your wireless phone and backup batteries fully charged, and be aware that an interruption of wireline and commercial power could affect wireless calls
- If possible, get extra batteries and charge them
- In times of commercial power outages, a car adapter for your wireless phone should enable you to recharge the battery
- Keep phones and necessary accessories in a sealed plastic bag to avoid water damage
- Load family and emergency numbers into your wireless phone
- Use your smartphone camera to take digital pictures or video of your property and valuables before the storm hits, so you have "before" pictures in the event of any storm damage
- Wireless networks sometimes experience heavy traffic during emergency events, so rather than call, send a text message
For its part in keeping networks up and running, Verizon suspended most non-emergency activity in the East, such as non-essential construction projects and internal training programs, to ensure resources are focused on serving customers for the duration of the storm.
Verizon wireline and wireless business units have activated national and regional command and control centers, enabling Verizon operations teams to monitor the storm's progress and company operations, including network performance.
Verizon has established communications with power and other service providers to ensure proper coordination in the event of storm damage. The company also has contacted vendors and other outside partners so that critical communications equipment and supplies can be prioritized, stocked and shipped expeditiously.
Company equipment, including poles, fiber-optic and copper cable, portable cell sites that can replace a damaged cell tower, and mobile emergency generators that can be used when local electrical power fails, is being staged in and around the mid-Atlantic and Northeast regions.
Of course, Verizon technicians may be ready to repair storm-damaged facilities, but they may have to wait for approval from local power companies, first-responders or law enforcement before beginning restoration work.
AT&T says it has an “arsenal of disaster response equipment and personnel on standby.” The company said it is the first private sector company in the nation to receive disaster preparedness certification under the Department of Homeland Security Voluntary Private Sector Preparedness Program.
The company has invested significantly in its Network Disaster Recovery (NDR) function to help ensure the flow of wireless and wireline communications during emergencies. Equipment includes more than 320 technology and equipment trailers that can be quickly deployed to respond to events such as hurricanes. The Network Disaster Recovery team works closely with local AT&T network personnel, regional Emergency Operations Centers and Local Response Centers to restore and maintain service until permanent repairs can be made.
To help maximize network reliability during a storm, AT&T conducts readiness drills and simulations throughout the year to ensure networks are prepared and personnel are ready to respond at a moment's notice. Networks are monitored 24/7.
T-Mobile says it has also made significant investments throughout the year in supplemental cell site backup generators, microwave technology equipment and cell-on-wheels (COWs), along with other tools and equipment to enhance the stability and, when necessary, the recovery of network operations.
The company, like most others, has access to additional fuel supplies for generators and company vehicles for the use in network recovery and other emergency circumstances. Backup generators and fuel tanks for regional network switch operations are being put in place and tested.
Data Centers Prep
Based on the U.S. data center map, approximately one-third of the data centers located in the U.S. are in states in Sandy’s path. Rich Miller, editor of dataknowledgecenter.com, reported that East Coast centers have emergency plans in place and will be focusing on maintaining telecom and network connectivity for response efforts.
He pointed out that most of New York’s major data center hubs are located outside Zone A. Others “carrier hotels” are in Zone B, which would only see a storm surge threat from a “moderate” Category 2 hurricane. The New York Stock Exchange data, unlike the stock exchange itself, is located Zone C, and supposedly would only flood in the event of a direct hit from a category 3 or 4 storm.
“Several other major financial data centers are in areas of northern New Jersey that local officials say should prepare for hurricane damage. The storm surge maps from the N.J. Office of Emergency Management appear to indicate that a Digital Realty Trust building in Weehawken is in an area that could be affected by a storm surge associated with a Category 1 or Category 2 hurricane. Digital Realty has done its own survey of the area’s elevation, and says its facility is above the area that would be affected by a storm surge. Equinix has several data centers in Secaucus, which is susceptible to flooding. The company’s most prominent facility, NY4, is on slightly higher ground than the surrounding area,” according to Miller.
IT Disaster Recovery Plan
An IT disaster recovery plan (IT DRP) should be developed in conjunction with the business’ continuity plan, according to ready.gov. Recovery strategies should include networks, servers, desktops, laptops, wireless devices, data and connectivity.
Priorities for IT recovery should be consistent with the priorities for recovery of business functions and processes that were developed during a business impact analysis.
IT resources required to support time-sensitive business functions and processes should also be identified. The recovery time for an IT resource should match the recovery time objective for the business function or process that depends on it.
Some business apps cannot tolerate any downtime. They utilize dual data centers capable of handling all data processing needs, which run parallel with data mirrored or synchronized between the two centers. This is a very expensive solution that only larger companies can afford. However, there are other solutions available for small to medium-sized businesses with critical business apps and data to protect.
Where to Start
A full recovery plan begins by compiling an inventory of hardware (e.g. servers, desktops, laptops and wireless devices), software apps and data. The plan should include a strategy to ensure that all critical information is backed up.
Identify critical software apps and data and the hardware required to run them. Using standardized hardware will help to replicate and reimage new hardware. Ensure that copies of program software are available to enable re-installation on replacement equipment. Prioritize hardware and software restoration.
Document the IT disaster recovery plan as part of the business continuity plan. Test the plan periodically to make sure that it works.
Developing the Data Backup Plan
Identify data on network servers, desktop computers, laptop computers and wireless devices that needs to be backed up, along with other hard copy records and information.
The plan should include regularly scheduled backups from wireless devices, laptop computers and desktop computers to a network server. Data on the server can then be backed up. Backing up hard copy vital records can be accomplished by scanning paper records into digital formats and allowing them to be backed up along with other digital data.
Options for Data Backup
A business impact analysis should evaluate the potential for lost data and define the “recovery point objective.” Data restoration times should be confirmed and compared with the IT and business function recovery time objectives.
Tapes, cartridges and large capacity USB drives with integrated data backup software are effective means for businesses to back up data. The frequency of backups, security of the backups and secure off-site storage should be addressed in the plan. Backups should be stored with the same level of security as the original data.
Data backup services include storage in the cloud, which can be a cost-effective solution for businesses with an internet connection. Software installed on client servers, computers and mobile devices is automatically backed up.
Cloud usage is clearly increasing. According to Gartner, the 2014 forecast for cloud technologies is $148.8 billion in revenue. But, what if the cloud goes down?
Last week, with no disaster looming, Amazon’s cloud was out. George L. Koroneos, Editor of Vertical Systems Reseller (VSR), a sister publication of Mobile Enterprise, reported that East Coast users of Amazon's Elastic Compute Cloud (EC2) service experienced major service slowdown and outages on Monday, Oct. 22, causing online businesses, websites, and brick and mortar companies that rely on Amazon for cloud software and backup to go dark or switch to local systems to continue doing business.
He said, “This latest Amazon outage compounds fears that small and medium sized businesses can't rely on cloud-based software and services as the sole means of delivering solutions, such as point of sale, according to the article.”
Lee Hutchinson of blog Ars Technica explained, "The takeaway for anyone considering cloud hosting for their website or application is that there is no magic pill, and just because something is ‘in the cloud’ doesn't mean it can't come crashing down to earth when there are problems."
Channelnomics' Chris Gonsalves stated in an interview with VSR, "The situation is a reminder that solution providers need to be both wary and diligent when entrusting their own and their clients’ precious IT assets to a cloud service. As we’ve said in the past, in the cloud, availability is everything. Cost savings, elasticity, scalability and speed mean nothing if you or your customers can’t access applications and data."
Costs of Downtimes
The Disaster Recovery Journal points, “Every minute spent trying to recover data or get service up and running again is time not spent on business operations. Most companies state that they can only afford downtime of four hours or less at one time.”
The article calls out stats from the book Storage Area Network for Dummies which illustrate the costs of downtime. Lost revenue per hour by industry includes energy ($2.8 million), telecom ($2 million), manufacturing ($1.6 million), finance ($1.5 million), IT ($1.4 million), insurance ($1.2 million) and retail ($1.1 million).
Clearly, the cost of a disaster recovery plan and solutions are far less than the damage to business by a catastrophe.