Pac-12 Takes Innovative Approach to Launch TV Network

Teamwork and long hours enabled Pac-12 Networks to be up and running a month before launch, say Michael Harabin and Harold Reynolds, the television network’s senior engineering staff.
Credit: Cody Pickens

Fans with an insatiable appetite for Pac-12 college sports now have their own cable TV network to watch and cheer on their favorite teams.

Pac-12 Networks, which celebrated its first anniversary in August, televised 550 live sporting events during its inaugural year. But before launching a national network and six regional channels last year, the company first had to build a technology infrastructure from scratch — and in doing so, it's taken an innovative approach to broadcasting.

Rather than rent expensive satellite time, Pac-12 uses the optical-fiber Internet2, a high-speed research and education network, to deliver content to its San Francisco headquarters, where many of the broadcasts are produced. In another cost-saving move, the announcers often work out of the San Francisco studios, where they watch games on high-definition TV and deliver play-by-play on games taking place throughout the conference.

"To launch one network is hard. To launch seven is near impossible. What should have taken two years, we did in about six months," says Scott Adametz, Pac-12 Networks' director of systems architecture and technology. "Building the broadcast data network was the main project, but we also had to build the office space and install everything from HVAC, power and lighting before we could even start bringing up the campus network."

With the launch of Pac-12 Networks, the Pac-12 Conference joins a small yet growing number of collegiate athletic conferences and institutions launching 24/7 TV sports networks — a lucrative business endeavor that increases the exposure of their athletic teams. The Pac-12, which includes West Coast colleges such as the University of California, Berkeley; Stanford; and UCLA, is currently the only one that fully owns and operates its network.

Pac-12 Networks, which was profitable in its first year, got its start in 2011 when Pac-12 Commissioner Larry Scott signed a $3 billion, 12-year contract with ESPN and Fox, allowing the two networks to air the conference's marquee football and men's basketball games. The new network allows the conference to broadcast football and men's basketball games not carried by ESPN and Fox.

It also lets the Pac-12 provide equal exposure to women's athletics and to broadcast lower profile sports seldom seen on television, such as field hockey, rowing and water polo, Adametz says.

The Race to Build a TV Network in 6 Months

The fledgling TV network, which now boasts 170 employees, is much like the other startups that have set up shop in San Francisco's hip South of Market neighborhood.

When the Pac-12 got the keys to its new headquarters in spring 2012, the technology staff had about six months to build the national network and six regional channels (Bay Area, Southern California, Arizona, Oregon, Washington and Mountain) for the intended Aug. 15 launch.

The eight-person engineering and IT staff worked at a frenetic pace, often 15 to 18 hours per day, seven days a week, to install the tech infrastructure.

Pac-12's technology staff simultaneously built a broadcast data network, a corporate network and a data center. They essentially built the networks with two of everything to ensure redundancy but kept each network separate to improve security and ensure that no corporate users affected broadcast data, Adametz adds.

Photo: Cody Pickens

"To launch one network is hard. To launch seven is near impossible. What should have taken two years, we did in about six months," says Pac-12 Networks' Scott Adametz.

"It was like a three-ring circus. Everyone was working on top of everybody else," recalls Harold Reynolds, Pac-12 Networks' senior vice president of engineering. "Each person on our team was assigned a couple of things, and we went off and did them independently of each other, but every day we coordinated. It was definitely a team effort."

CDW•G's professional services played an integral role in helping Pac-12 Networks implement the networks, as well as an IP phone system. Because of the rushed schedule, CDW expedited the delivery of products and quickly made network engineers available to help design and configure the networks and IP phone system and provide training to Pac-12's technology staff.

CDW's engineers are typically booked up to five to eight weeks in advance, but CDW re-arranged schedules and made its engineers available onsite in half the time — two to four weeks, says David Greenberg, field account executive for CDW's Advanced Technologies practice.

"We understood that the Pac-12 was under a tremendous time crunch, so we went out of our way to make sure we met their aggressive timeline," Greenberg says.

Pac-12's novel approach to eliminate satellite and other traditional transmission technologies as much as possible and use an IP network saves millions of dollars a year, according to Adametz.

Instead of renting and using satellite uplink trucks at every sports venue, the Pac-12 uses Internet2 to transport video from live sporting events to San Francisco headquarters and to Pac-12's distribution center in Denver, which delivers the video feed to cable TV companies and other affiliates.

Every Pac-12 college is connected to Internet2, and by connecting to Internet2 over fiber, the TV network is then linked to every Pac-12 college.

"By streaming over Internet2, we avoid having to buy 'occasional-use' satellite time or fiber circuits, which are scarce and billed at a very high rate," Adametz explains. "It's massive cost savings."

The technology department built the broadcast data network by installing two Cisco Nexus 7000 Series 10-Slot switches at the core. In the data center, a storage area network houses 300 terabytes of video, and 40 HP servers are dedicated to a digital asset management system — software used for logging, editing and packaging video.

To transfer video across Internet2, the Pac-12 campuses, the San Francisco office and the Denver distribution center have deployed Juniper Networks SRX firewalls and Cisco Catalyst 3750-X switches to create secure IP Security virtual private network tunnels to transfer the video.

Nacho Vega, a CDW network solution architect, came onsite to help the late Brandon Bernier, Pac-12 Networks' network engineer, design the broadcast data network. While Bernier drew up the initial design, Vega helped refine it. Another CDW engineer worked hand in hand with Bernier to configure the equipment. Bernier, along with the rest of the Pac-12 technology staff, installed the technology themselves, but during the implementation process, Vega worked with Pac-12 to validate the network, checking to make sure it was installed correctly using industry best practices.

Because security of broadcast video is critical, Vega designed the network to take advantage of Virtual Routing and Forwarding (VRF), technology that allows Pac-12 Networks to securely segment the network into three separate virtual networks.

"Pac-12 needed three separate security zones for network traffic that needed high, medium and low security, so we used VRF to segment and isolate that traffic into three separate networks," Vega says.

Internet2 heavily reduces — but does not completely eliminate — the network's use of satellite for broadcast, however. Pac-12 still uses satellite for about 20 percent of its games because sponsors don't want any potential IP hiccups during the most high-profile broadcasts, Adametz says. The Denver distribution center also uses satellite to uplink Pac-12 Networks' front haul (video feed) to cable TV affiliates and other distribution partners.

These partners installed Cisco 9800 series integrated receiver decoders, which allow them to receive and decode Pac-12 Networks' video feeds from satellite.

IP Network Breeds Innovation

Besides reducing satellite usage, an IP network permits the Pac-12 to off-load live-event production from onsite production trucks to its San Francisco headquarters.

When doing so, the company needs only a camera and audio crew at the sporting events. The camera feeds come back to a central control room in San Francisco, where a production team directs the broadcast, creates highlight reels and adds graphics and music.

Pac-12 Networks, which developed the technology last fall, produced 50 games remotely last school year and will increase that to 200 games this year. "We're just ramping this up," Adametz says.

Announcers attend some games. For the others, they broadcast from a soundproofed booth with large multiview screens. "They see every camera angle as it is happening, live," he says. "There is a 15-millisecond delay, which is imperceptible to the human eye."

This efficient and cost-effective approach to producing games allows Pac-12 to air a large number and wide range of sporting events, says Michael Harabin, Pac-12 Networks vice president of technology and engineering.

"We would not be able to justify covering events like field hockey if we didn't have low production expenses," he adds. "This allows us to provide exposure to many more sports and athletes in our conference."

Taking Care of Corporate IT Needs

Besides creating a broadcast data network, the Pac-12 Network tech staff needed to provide employees with computers, software and a phone system.

The company equipped workers with Lenovo ThinkPad T420 and T430 notebook computers, Lenovo ThinkCentre M91 and M92 enterprise tower desktops, Apple Mac Pros and 24-inch Samsung monitors on their desks.

In the data center, IT virtualized three servers using VMware vSphere. The virtual machines (VMs) run corporate file servers, accounting software and other business applications — and for developers, a testing and development environment.

When spinning up VMs, the tech staff can quickly provision data storage using two Nimble Storage arrays, which are 16TB iSCSI-attached solid-state drives.

"We've had unbelievable spin-up times," Adametz says. "Instead of a bunch of slow, spinning disks that may take up to 40 minutes to make up a 2TB volume, Nimble does it in two seconds."

The company standardized on Cisco in the corporate network, installing two Catalyst 4900M switches at the core and Catalyst 3750-X stacked switches in the wiring closets.

IT workers also deployed Cisco Meraki for Pac-12 Networks' wireless network and Cisco's Unified Communications Voice over IP phone system with IP video phones on each employee's desk.

In addition, Pac-12 built an IP television system to stream the conference's channels throughout the office. A CDW engineer helped configure the IP phone system onsite and provided training and initial support to staff, Vega says.

With the CDW engineer's help, the Pac-12 tech staff got the phone system working in late June, one week before employees moved into the new headquarters.

A Remarkable Team Effort

Pac-12 Networks executives say they could not have launched the network without CDW. "They worked really hard and were very helpful and always available," Reynolds says.

Although the network launch was sometimes chaotic and stressful, Adametz says, the company is not resting on its laurels. It plans to increase the number of live sporting events it broadcasts to 750 this coming school year and 850 the following year.

"We just divvied up the work and went off on our separate directions, and came back together when we needed to, and everything worked," Adametz says. "We did some amazing things in an unbelievably short amount of time."