Dialing Up a New PBX

Migrating 24,000 ports to a new PBX is an enormous challenge under the best conditions, but when the University of New Mexico took on the project last year it was hardly blessed with ideal conditions. The UNM team was faced with a university hospital expansion project that was constructed around them, the challenge of keeping a 30-year-old PBX that was no longer supported by the vendor operable long enough to move users to the new system, and a condensed timeframe that forced the team to complete a scheduled three-month project in just three weeks.

Thanks to proper planning and the flexibility to make tough decisions along the way, the project turned out successfully.

The UNM Information Technology Services (ITS) department is responsible for providing communication services and products in support of education, patient care and research for the main UNM campus, UNM Hospital and clinics, and Valencia campus, managing more than 19,000 analog and digital phone lines and voice-mail systems with more than 10,000 mailboxes.

Decision One: Move or Reinforce?

The hospital construction plans called for the expansion to be built directly on the roof of the building that housed one of UNM’s main PBXs.

So the first issue faced by the school’s ITS team was whether to relocate the PBX out of the way of the construction project or reinforce the existing location to withstand any potential construction hazards, says Mark Reynolds, associate director for ITS. The decision came down largely to price. Installing a new PBX at a different site would cost an estimated $10 million — far beyond the available funding and far more than the $400,000 price tag quoted to make the existing site construction-proof.

Before the construction, the ITS team met with architects, the project team from its telecommunications consultant, NEC Unified Solutions, and other stakeholders to forge a plan to minimize the impact of the construction on communication services and address any contingencies.

“We were a single point of failure in that building for every bit of our communications on campus,” says Paula Loendorf, director of ITS.

The early meetings focused on being prepared for whatever could go wrong. “We were constantly looking at ways of mitigation, and we planned for worst-case scenarios and asked the vendors how soon we could get a replacement PBX here and up and running [if the old PBX went down during construction],” adds Reynolds. “If that happened, we would truly be out of business for three to six weeks.”

One of the first projects involved moving all of the carrier services for the hospital to the hospital building, so the hospital telecommunication would keep working if something happened to the PBX building. Fiber and copper cabling were rerouted to the existing hospital building, and the PBX at the hospital was upgraded at the same time.

Decision Two: Service or Buy New?

Despite extensive contingency planning, the team ran into challenges right away. The project to upgrade the PBX facilities encompassed moving the administrative group out to a new location on campus, and then hardening the PBX building to ensure that the equipment, power sources and voice/data lines would be safe from the adjacent construction.

The $400,000 estimate ended up costing $2 million, and even that price tag didn’t buy enough protection as planned. Despite assurances from the architects that nothing would happen to the existing PBX, the room quickly filled with construction dust.

Reynolds says the dust that collected on the PBX was so bad that “you could run your finger across it and write your name in the dust.” That much dust was too much for a 30-year-old PBX.

That left ITS with another decision: to service the existing PBX or purchase a new unit. The first choice was to clean the existing unit one module at a time. But the vendor no longer supported the original unit, and there was no guarantee the PBX would come back to life if it was powered down.

“You have a PBX that’s been sitting there for more than 25 years, and its components have settled in,” Reynolds says. “The soldered joints and the electronics are all stable, but when you shut it down and cool it off, there’s a very good chance that it’s not going to come back.”

So with a need for a new large-scale PBX and no budget to meet its requirements, ITS went to the university administration and convinced the Board of Regents to allocate $5 million in bond money for a PBX. That brought the total cost of the project to around $7 million.

Decision Three: Three Months or Three Weeks?

UNM purchased a hybrid PBX that supports both analog Time Division Multiplexing phones as well as Voice over IP telephony (VoIP). Taking the hybrid approach gave the university a platform for future wide-scale deployment of IP-based services, while still providing a cost-effective service for the mostly analog phone customers at the school and hospital, Reynolds says.

“Strategically, that put us in a position where we could move to the future with IP products, but because we had so much analog, we felt it was a better solution to upgrade the PBX to a hybrid solution and then upgrade to IP as it seemed the appropriate thing to do because of cost,” he explains.

This added the extra step of having to validate the quality of the university network after the decision to employ VoIP. “When you move your users to VoIP, you have to make sure that your data network is robust enough to handle the traffic,” says Ken Morrison, the project manager for NEC.

The PBX building was still being fortified and refitted to eliminate the dust problems when installation of the new hybrid PBX started amidst a $200 million construction project surrounding the location.

The installation went smoothly, but as the team was preparing to test the new unit in advance of the cutover, the old PBX shut down one weekend. Many locations on campus were left without dial tones, numbers weren’t being forwarded properly and authorization codes did not work. The tech support team inspected the PBX and determined that no repairs were possible, given the age and lack of parts, and any maintenance operations would increase the risk that the unit would not return when powered on.

The old PBX did come back online, but the scare forced the team to fast-track the conversion to the new PBX. That meant a project planned for three months would have to be completed in three weeks. Extra help was brought onboard to assist with the challenge of migrating 20,000 pairs of copper cables and 11,000 dial tones between the PBXs, while keeping customer impact to a minimum.

“We went into fast track,” Loendorf says. “We cut over about 4,000 pairs of cables a day. We originally planned 200 a day. We worked day and night to cut over the cables.”

The cutover required a half-tap splice of the copper cables in the tunnel under the PBX room to move customers from the old PBX to the new hybrid. Working from 6 a.m. to 6 p.m., ITS completed the cutover in just eight days.

Success

Despite the challenges that the ITS team struggled with, the transition to the new PBX was largely transparent to most customers, Reynolds says. He adds that his group has seen an 85 to 95 percent reduction in monthly trouble tickets with the new system. The school continues to migrate more locations to IP-based systems as part of an ongoing upgrade plan that allows multiple dissimilar PBX systems to function together as a single virtual PBX with a consistent feature set across all of the connected systems. That centralizes administration features such as call accounting and billing systems that compile and manage call records, and eliminates the cost of having to deploy voice mail and other systems at each local PBX.