With Business Continuity Top of Mind, Internet Firm Shares Recovery Plan
With its corporate headquarters entrenched in a hurricane hot zone, CableOrganizer.com, an eTailer of electrical, telecom/datacom/networking, home theater, cable and wire management products whose operation is largely IT system-driven, takes its disaster recovery plans very seriously — and with good reason: the company has been seriously affected by hurricanes in the past, including experiencing frequent power shutdowns both controlled and uncontrolled by its Florida Power & Light (FPL) utility.
In preparing for the inevitable operational downtime related to hurricanes and other natural disasters as well as unforeseen brown and black outs common in its region, CableOrganizer.com has deployed both a comprehensive disaster prevention and disaster recovery plan rather than one or the other to best assure the eCommerce company, which needs to serve consumers 24/7/365, has suitable measures in place to remain up and running to the greatest extent possible when adversity strikes.
"Each year, prior to the month of June when hurricane season commences, we review our disaster prevention procedures to ensure readiness in the event that a tropical storm or a hurricane heads our way," says Nicolas Dubus, IT director for CableOrganizer.com. "But, rather than a simple checklist and seasonal preparatory efforts, we review and maintain disaster prevention and recovery plans for all levels of the operation year-round to best assure we’re truly prepared."
Fail to Plan, Plan to Fail
CableOrganizer.com has implemented multiple routine procedures, including the multi-level measures detailed below.
Surge Protector and UPS Maintenance
The company routinely tests and maintains the UPS surge protectors for all of its computers and server banks. It typically replaces the UPS units for each desktop computer every two years, and those for the servers every 3 to 5 years.
CableOrganizer.com subscribes to iBackup, an integrated online backup for PCs and servers, and has scheduled and automated daily backups of its most important documents and databases.
The company purchased a Quantum tape library with LTO 5 tapes to run hourly, daily and weekly data backups with Symantec Backup Exec software. Every evening, the tapes from the previous day are taken off-site, and then placed in a safe the following morning.
All of CableOrganizer.com’s shared folders are continuously backed up from its headquarters in Fort Lauderdale to its co-location site at Host.net, located in nearby Boca Raton.
All company emails and Outlook PST files are also backed up with the Carbonite Online Backup Pro system.
Finally, CableOrganizer.com regularly ensures that all backup solutions are fully operational and running the way they are supposed to. Every week, the company runs tests on small files to ensure the integrity of the backup files. Twice a month, it runs a complete restore of all backup data.
Backup Internet Connections
CableOrganizer.com has multiple Internet connections available, load balanced with Peplink—a Multi Wan Internet load-balancing router. In the event that one, or several, Internet connection options are down, the company always has one or more alternative connection options to fall back on, as follows:
- Metro E Fiber
- Comcast Cable
- T1 with Windstream
- Satellite with HughesNet
CableOrganizer.com has also purchased all the necessary equipment for removing and remounting the satellite on the roof of its location themself, and its IT department has been fully trained in how to carry out all emergency satellite-related procedures.
CableOrganizer.com has two satellite phones from GMPCS that allow it to maintain contact with the owners of the company when land lines and cell reception are unavailable. It chose the emergency satellite plan, which costs around $275 a year plus an additional $2.00 per minute when calls are made. When the satellite phones aren’t in use, the company stores them in a weatherproof Pelican case.
Off-Site, Disaster-Proof Data Centers
While CableOrganizer.com houses some servers in a server room at its headquarters in Fort Lauderdale, they have strategically divided Web servers among several additional data centers in Connecticut, Texas and Florida. They are taking advantage of the Metro E fiber optics to connect to the Host.net data center in Boca Raton, just 15 miles north of their location. There the company maintains a cage at the facility in which CableOrganizer.com’s critical servers, including the company’s phone system, domain controller, SQL, application and file server, are housed. This is where CableOrganizer.com continuously backs up all of its data. CableOrganizer.com also installed Citrix, along with Citrix Access Gateway, on one of the servers to allow remote users access to the different company applications from anywhere with an Internet connection.
One of the main reasons that CableOrganizer.com chose to use the Host.net data center is because it’s been built to face any major disasters, and features protective measures that include:
- Concrete Type 1 Construction
- Category 5 rated Twin-T roof (able to withstand winds of 225 mph), with 4.5" of poly-iso insulation and reflective coating to reduce heat /electricity
- Elevated foundation a minimum of 15 feet above sea level
- Service Amps (3200 Amps)
- Generator Backup (150Kw)
- Cooling (320 tons of cooling)
- 5500 gallons of on-site diesel fuel
CableOrganizer.com personnel have access to the facility 24/7, yet it's an extremely secure location requiring a pin number and biometric handprint for entrance into the data center, as well as a key to open the company’s designated cage.
Backup Customer Support Options
CableOrganizer.com's main phone system, which is vital to the everyday phone-driven aspects of its customer service, is Shoretel. The server and main T1 switch for Shoretel are hosted at the data center, where they also keep the PRI. If it goes down for any reason, the company has a set of analog lines that can be used with the same phone system as a backup solution right in its own office. For more extreme cases in which in-office backup isn't an option, they have subscribed to a disaster recovery plan with GlobalResponse in Margate, Florida, just 10 minutes from CableOrganizer.com's corporate headquarters.
GlobalResponse's building is equipped to face disaster with on-site backup generators and a diesel fuel tank, as well as a redundant network. They are connected to AT&T through an OC12 Sonnet SMARTring, effectively becoming a node on the AT&T local fiber loop. The loop is self-healing; if one pair is cut, the opposite pair continues carrying all the traffic. Global Response is a registered member of the Telephone Service Priority List maintained by the US Department of Homeland Security. In the event of a carrier failure, they serve as a vital asset to US businesses, including CableOrganizer.com.
In addition to GlobalResponse's technical backup, there is also a backup customer support group who is able to take phone calls on CableOrganizer.com's behalf 24/7. For times when its offices may be damaged or inaccessible in the wake of an emergency or disaster, the company also maintains access to several seats at their location, where they can send its own employees to take phone calls – all that’s required is the set up of a few phone-ready, Citrix-equipped computers.
This year, CableOrganizer.com added an extra layer to its disaster and recovery plan by installing 144 solar panels and 48 backup batteries that can provide power for several days. All of its mission-critical servers and computers are able to remain up and running in the event of a power outage. In addition, the company added a 1,000 gallon propane tank and 45kW generator to keep the business running for up to weeks on end, if necessary.