PDF of this tutorialArchitecture: Enterprise versus Data Marts
As previously mentioned, the choice between a single enterprise-wide warehouse architecture and an architecture consisting of many smaller data marts is a point of some contention. The difference is not necessarily one of cost, although a well-planned enterprise solution may require a larger investment initially to account for the more extensive data analysis and modeling up front, as well as to avoid continual hardware upgrades as the warehouse grows. A data mart solution is typically more modular, thus exhibiting a more linear cost curve. Over the long term, however, costs tend to even out, and both systems can allow for expansion and growth. The choice of architecture usually depends more heavily on organizational factors. Depending on the situation, strong arguments can be made for both sides.
Pros and Cons
Decision-support systems existed long before the term data warehouse was coined, often starting in the marketing, sales, and finance departments, with each group pulling the data they required down from the legacy systems into small databases on their local servers. These solutions were often implemented independently and without assistance from information services (IS). Consequently, results tended to be inconsistent, particularly as compared to results obtained by sister organizations with similar but separate decision-support systems of their own.
Thus, early wisdom in the data warehousing market favored a single, central enterprise system that would serve all organizations’ needs. Although some compromises might be necessary in serving the needs of the many, this type of solution would at least ensure consistency of data and scope across organizations. In addition, the information was typically more reliable, as the process of acquiring and cleansing data was more deliberate and more carefully managed.
However, the enterprise data warehouse approach has a drawback. It takes time, and it takes coordination between many groups across potentially many different organizations within a company. Some pioneers of this approach ran into long delays in their warehouse implementations as well as coordination stumbling blocks that either terminated or significantly set back their deployments. In particular, companies that are very compartmentalized in their structure do not lend themselves easily to an enterprise-wide solution. In these cases, integrated data marts can offer greater customization, less contention for system resources, and greater independence for participating organizations. It is important, however, for the reasons stated earlier, that particular care be taken in planning the data acquisition, transformation, and consolidation stages for a data mart. Centralized consolidation can often help minimize data inconsistency between data marts.
A sound deployment strategy is to harness the benefits of both approaches by engaging a few key end-user groups as well as the IS organization and designing a skeleton architecture for the broad enterprise data warehouse. However, rather than begin by trying to load, cleanse, deploy, and manage all of the enterprise data, select one or two key applications to focus on first, define the data marts required to support them, and start with a manageable project. Then, as new applications are brought on-line, merge them into the enterprise warehouse architecture and implement them one at a time. This strategy is sometimes referred to as the architected data mart approach.
Critical Success Factors
Know Your Goals
Perhaps the most important success factor in the data warehouse is to keep one’s long-term goals in mind. It is easy to let a warehouse project explode beyond the initial boundaries set for it. One user group finds out that it has been excluded from initial rollout and demands to be included. The initial target user group decides it would be even better if it could do churn management and pricing analysis during phase one, rather than wait until phase two to bring the pricing analysis system on-line. The IS director decides it makes sense to define, build, and load the entire warehouse infrastructure before trying to deliver data to end users. Clearly, it is almost impossible to eliminate scope creep completely. Valid issues arise and must be dealt with. However, it is generally a bad sign when objectives change significantly after final sign-off—or, worse, after implementation has begun. Whenever possible, initial goals to which all are agreed should be maintained. Exceptions will always occur, but it is an important rule of thumb.
Performance
The data warehouse will fail if users do not get answers back in a timely fashion. This is why parallel database technology, advanced indexing, data modeling, aggregation, and sampling are so critical to the success of the data warehouse project. This is an area in which investment makes sense—not only in technology, but in experts who know how to maximize the utility of the technology.
Data Volumes: Loads and Management
One of the characteristic challenges of a telecommunications business is managing enormous volumes of data. Not only do telecom systems generate huge numbers of detailed call records (and others), but the records themselves are very large, and it is not always possible to know what, if anything, can be filtered out of the warehouse. Complicating matters is that common telecom applications, like fraud detection, require real-time data feeds for optimal effectiveness. As a result, telecom data warehouses must be able to handle tremendous insertion rates, as well as manage databases that will likely grow into the multiterabyte range. Building a system with technology that will not scale effectively as data volumes grow can be one of the most frustrating (and expensive) experiences a company can endure.
Relationship between IS and Users
In any data warehouse implementation, it is critical that IS and the user community be in agreement about the objectives of the warehouse and be in constant communication throughout the project. A project driven solely by IS runs a significant risk of slow adoption, low return on ROI, or even outright failure. A project solely driven by the user community risks a duplication of effort, a lack of integration with other critical systems across the organization, and an unmanageable enterprise environment.
Executive Sponsorship
Data warehouses are very large, expensive, complex systems. All warehouse projects will encounter tumultuous periods, whether due to technological setbacks, staffing issues, insufficient infrastructure, or poor communication. And this is not an exhaustive list. All problems are resolvable, but if there are no sponsors at the senior management level, these problems can quickly take on monumental proportions. By sponsors, we mean people who understand the business pain being addressed by the project and believe wholeheartedly that building the data warehouse is the best route to resolving that pain. A powerful executive sponsor can help knife through challenges before they impact project momentum. A lack of executive support raises constant doubts about a project and can lead to multiple starts and stops in the implementation, badly demoralizing the project team and the business community.
