Business Intelligence: Why Does It Take So Long?

Implementations of data warehouse and business intelligence solutions are necessarily complex for a variety of reasons. Unfortunately, that is not well understood by those who haven't implemented them.

The general view of business intelligence (BI) projects tends to be that a new, slick front-end analytical tool can be installed and start producing reports, cubes or dashboards within days or weeks.

What's not understood is that the front-end tool is just the tip of the iceberg. The real work — where the bulk of the effort (and time and cost) is incurred — is in gathering, ­integrating, organizing, modeling and defining the data to be used by the tool.

From an organiz­ational ­perspective, BI projects cross many lines. They touch multiple business ­processes, merge data from ­separate operational systems potentially housed in disparate geographical locations, and integrate several business concepts.

Key business rules may not be understood or documented; data elements may have multiple, ­conflicting definitions; and the quality of the data throughout the enterprise must be managed within the BI solution. Given those complexities, a ­structured ­approach to designing, developing and ­deploying a solution is an absolute requirement.

The following steps have resulted in faster, better BI launches at the University of Washington while improving data quality and ­increasing confidence in the data across the organization.

Define Leaders, Process

Streamline the consensus-driven culture through clearly ­defined ­decision-making and data ­governance structures. Effective institutional data management requires organizationwide standards, business process documentation and agreed-upon data definitions.

To get there, align business concepts with data by identifying the organization's high-level subject areas such as academics, ­financial resources, human resources and research administration. Those broad areas are made up of major business processes, or subdomains.

At the level of the subdomains, identify data stewards to help estab­lish data governance and data ­management standards. In addition to authority over ­data policies, rules and standards, data stewards also should have the final say over data definitions.

A strong commitment to institutional data management ­standards can break the cycle of ineffective change management, re­working and churning associated with the complexities of information delivery.

Free the Data

UW's governance group, the Data Management Committee, also is instrumental in safeguarding security. A Roles and Access Matrix that defines the access and use of data, in conjunction with in-house tools, effectively manages access to a large volume and data for large populations of users. Intuitive front-end interfaces empower business unit owners to administer security and access without complex and risky processes involving IT staff.

The greatest surprise for us was that, by applying more security ­access controls we are able to provide more and broader access to data faster, which was instrumental in shaping UW's shift to a culture in which decisions are informed by data. Defining row- and column-level access up front, coupled with intuitive technical implementations, eliminated the overhead access management that commonly delays projects. (For more information about UW's data governance and access controls, see washington.edu/uwit/im/dmc/index.html.)

Be Agile

Being agile requires more than just a 15-minute huddle with the team every morning. It also requires early prototyping, breaking down a project into small two- to four-week ­iterations and focusing initial ­delivery on minimal ­marketable ­features. When preparations are aligned with the complexities of the solution and the maturity of the BI team, results can be seen more quickly. Why is failing quickly a good thing? The sooner a BI team can identify project issues and risks — by showing real data to real business users in real tools — the sooner the business and BI teams can collaborate to course-correct, resolve data quality issues, redesign data models or ­engage stakeholders for quick decision-making.

Being agile is much less about regimen than it is about results. At UW, following agile practices has significantly ­reduced BI delivery time from no releases for several months to one to two monthly releases over two years.

Model Business Concepts

Whether the team favors normalized databases or adheres to conformed ­dimensions and an enterprise data bus, the data warehouse should ­represent an organization's business concepts. The team at UW is investigating Dan ­Linstedt's data vault modeling method as the best of both worlds. Increasingly popular in Europe, the technique is designed to provide long-term historical storage of data from disparate operational systems, while facilitating auditing, data lineage tracing, loading speed and resilience to change. The idea is that all data is relevant data ("single version of facts"); data interpretation, cleansing and grouping (­creating the ­"single version of the truth") happen at the point of reporting or extracting data from the data vault. Data vault modeling may lead to developing a faster and more accurate underlying architecture.

Collaborate Nonstop

Each step in the process of ­delivering a BI ­project has many activities and outcomes. Many of those require collaboration and cooperation from multiple people and systems.

Across all phases of a BI project, IT and business groups must work together. Different skill sets and communication styles within ­organizational ­cultures introduce greater complexities in BI ­engagements, and can lead to costly ­midcourse corrections, ­possibly ­derailing ­adoption. Ensure that business partners are part of the BI project team, and be clear about responsibilities and roles. For some tasks (defining requirements or mapping BI ­deliverables), business partners may operate independently. Other tasks should be shared. When successful, teams will ­discover advocates in ­partners who can represent BI's strategic value.

BI projects often serve as the catalyst that uncovers, then addresses, key ­issues in institutional data management. While they can seem to cause delays, successful projects result in better data in core systems, including the data warehouse, and ­ultimately lead to better informed and more ­reliable, data-driven ­decision-making.