Bell, Telus offered to help, but Rogers determined it was not possible
OTTAWA – There were no other measures Rogers Communications could have taken to restore 911 services any quicker than it did during the July 8th network outage, the company has told the CRTC in a response to a demand for a detailed report on what happened that day and why.
In a heavily redacted document made public by the CRTC last Friday, Rogers explained it explored the possibility of shutting down its RAN (radio access network), because if customers were unable to connect to it, they would “automatically connect to the strongest signal available, even from another carrier, for the purpose of making a 9-1-1 call.”
However, this was not a path Rogers was able to take because its IP core was disabled, which meant the RAN could not be turned off – “we could have had instability issues,” the document reads.
“Additionally, restoring the RAN would have taken several hours to complete after the core network had been restored, further extending the outage. While considered many times during the day, shutting down the RAN was simply not a solution. The best and fastest way to restore 9-1-1 was to restore the network itself.”
Rogers indicated in the document some 911 calls from its customers did make it through on July 8, however, information about how many was redacted in the publicly available version of the report. Additionally, Bell and Telus confirmed to Rogers some of its customers were able to connect to their wireless networks to place 911 calls. (Information on how many is also redacted.)
The July 8 outage impacted Rogers’ customers across the nation. Wireless services including voice, text, and data, were impacted, as was home phone telephony, Internet and TV services. Some of the company’s wireless customers did have intermittent services throughout the day on its GSM and 3G networks, the company said.
The issue causing the outage occurred during a routine update, which had been in the works for months. During the sixth phase of a seven-phase process, at 4:43 a.m. EDT, coding was introduced in the company’s Distribution Routers, which two minutes later triggered the failure of Rogers’ IP core network, according to the document submitted to the CRTC.
A routing filter had been deleted, which allowed all possible routes to the Internet to pass through the routers, which “immediately began propagating abnormally high volumes of routes throughout the core network,” the document reads. “Certain network routing equipment became flooded, exceeded their capacity levels and were then unable to route traffic, causing the common core network to stop processing traffic.”
Many Rogers employees were impacted, slowing the company’s response to the outage because while some had emergency SIMs on alternate carriers, which enabled them to assist in coordination efforts, some “teams needed to travel to centralized locations where management network access was established,” the document explained.
“To complicate matters further, the loss of access to our VPN system to our core network nodes affected our timely ability to begin identifying the trouble and, hence, delayed the restoral efforts.”
Rogers told the CRTC both Bell and Telus offered assistance, however because of the nature of the issue, they “rapidly assessed and concluded that it was not possible to make the necessary network changes to enable our wireless customers to move to their wireless networks.”
This is because Rogers needed access to its own Home Location Register, Home Subscriber Server and Centralized User Database, which was not possible during the outage.
“Furthermore, given the national nature of this event, no competitor’s network would have been able to handle the extra and sudden volume of wireless customers (over 10.2M) and the related voice/data traffic surge,” Rogers explained.
“If not done carefully, such an attempt could have impeded the operations of the other carriers’ networks.” Rogers indicates this is, however, being explored further as a possibility for the formal agreement to improve network resiliency the industry minister announced he expects Canadian telecoms to reach.
The Commission is currently studying Rogers’ submission and will determine next steps from there, a CRTC spokesperson told Cartt.ca via email.
The Commission is also currently considering a request from the Public Interest Advocacy Centre (PIAC) to open an inquiry into the outage.
PIAC is now further seeking information from the CRTC on the process for challenging Rogers on the redacted information in its report to the Commission.
The consumer advocacy organization sent the CRTC a letter today asking for clarification on whether “the present set of RFIs and redacted answers is to be considered a ‘proceeding’ (we note two file numbers were assigned) and whether PIAC and others may challenge Rogers’ assertions of confidentiality and if so, what procedure should be undertaken to challenge those assertions.”
CRTC chair Ian Scott told a House of Commons standing committee earlier today (in French) that the Commission believes transparency is important and indicated the regulator would ensure Rogers shares the information Canadians need to know about the outage.
