The Operational Data Store (ODS) - The Next Evolutionary Step

What is an Operational Data Store?

In the early 2000s, Gartner Group coined the term ZLE, or Zero Latency Enterprise. In their words, a ZLE was "the instantaneous awareness and appropriate response to events across an entire enterprise." This response was later renamed the Real-Time Enterprise (RTE).

HP was a leader in ZLE. Its ZLE architecture centered around an Operational Data Store (ODS) that was similar to the data store used by a data warehouse. However, rather than periodically loading the ODS with massive amounts of data via an ETL facility, the ODS was trickle-fed transactions as they occurred so that it always contained the latest state of the enterprise as well as historical information (Figure 12). Using the ODS, classical data-mining engines could generate strategic information and knowledge, and real-time rules engines could make tactical decisions regarding immediate actions to take.

One special characteristic of an ODS is the requirement for it to handle mixed workloads. On the one hand, it must be able to respond to complex queries from knowledge users, data-mining facilities, and rules engines. This characteristic is the realm of OLAP (online analytical processing). The database structures suitable for OLAP are characterized by fat keys that allow rapid searching of the database to respond to complex queries.

On the other hand, the ODS must be capable of processing an extremely high transaction rate, as it is being fed transactions in real-time from many enterprise systems. This processing is the realm of OLTP (online transaction processing). The database structures suitable for OLTP are characterized by skinny keys that require a minimum of updating as data is added to the database.

The Operational Data Store

Another special characteristic of an ODS is that it is bi-directional. Unlike a data warehouse, which only accepts information from enterprise systems, an ODS both accepts information from and delivers information to the other enterprise systems. An example of this characteristic is the act of keeping databases in synchronization. A particular data item, like a customer’s address, may be stored in several databases around the enterprise. If one system changes this data item, the ODS acts as a central data repository that informs the other systems of the new data value so they update their databases.

Another example of outgoing information is the results of the rules engine. If the rules engine decides to recommend a particular immediate action, that action is communicated to the appropriate enterprise system for execution. For instance, if the rules engine for a credit card processor detects suspicious activity, it immediately alerts the authorization system to take appropriate action.

In concept, the ODS, which contains all corporate data, could become the database of record – the single version of truth – for the enterprise. However, this action generally does not happen because of regulatory requirements and other considerations. The database of record remains on the legacy systems, where it was resident for decades.

The Evolution of RTBI to ODS

Will the ODS make it into common usage?

As seen from the above description, the ODS is simply the RTBI system described previously but extended to the enterprise. However, though RTBI systems are becoming common today, full-blown ODS systems have yet to make a significant appearance due to the complexity and cost of designing and building such far-reaching systems.

It is conceivable that RTBI systems may evolve eventually to ODS systems, as shown in Figure 13. Early dedicated business intelligence systems used data warehouse or EAI technologies. As these technologies proved their worth, data replication was added to move the technologies to real-time business intelligence systems, thus expanding their reach.

However, though RTBI systems exist today in many applications, each RTBI system generally supports a single purpose such as fraud detection, instant customer promotions, or just-in-time inventory. These kinds of RTBI usage are shown in several case studies later in this paper.

Considerable effort was invested by some companies to consolidate a multitude of RTBI systems into a single ODS supporting enterprise-wide tactical and strategic decision making, shown as the final step in Figure 13. However, the cost and disruption imposed by conversion to an ODS has so far resulted in little progress in expanding RTBI systems to support both tactical and strategic decision making for any particular corporate function, much less the enterprise.

The advantages of such integration are clear:

  • The single ODS supports both tactical and strategic decision making.
  • The ODS is made highly available through redundancy, such as by using an active/active system to achieve not only high availability of the system but also to achieve disaster tolerance. Corporate functions are less affected by the failure of one of the other systems. For instance, a customer’s credit status is checked against the ODS without having to interrogate a credit-authorization system that might be down.
  • The scope of decision making is extended to many more areas across the enterprise. For instance, a drop in sales is correlated with increased accounts receivables on store-branded credit cards. Easier credit terms might help to restore sales to their previous level.
  • The various corporate IT systems are isolated. No longer do they have to interact with each other through EAI. They each will communicate only with the ODS system. New applications being added do not have to be configured to interface with multiple other applications. They need only interface with the ODS. Furthermore, other systems do not have to be modified to interface with the new system.

The Evolution of the Operational Data Store

However, so far the obstacles to achieving this goal have been daunting. For instance:

  • The ODS is an expensive system in terms of hardware acquisition, software licensing, and development. This problem is compounded by the need for a highly available redundant system as a platform for the ODS.
  • New technologies, such as data-replication engines, must be adopted. More powerful data-mining engines and rules engines may need to be incorporated.
  • The design of the ODS database is much different. It needs to support both tactical and strategic queries yet be very efficient in handling a large volume of updates. Fast update processing requires skinny keys in which a minimum of alternate indices must be updated. Efficient query processing requires fat keys providing many access paths to the data.
  • The size and depth of the ODS can be quite extensive. It typically has to store large volumes of historical and archival versions of the data, instead of just the current value of the data.
  • Applications may have to be seriously rearchitected, which is not only expensive and time consuming, but risky.
  • Third-party products that do not readily lend themselves to adapting to an ODS architecture may be involved.
  • The conversion of current decision-making processes might not only be difficult but may, in fact, be resisted by the user community.

The bottom line is that today, companies are achieving RTBI by directly integrating their systems using real-time data replication, or they are trickle-feeding data marts in real-time and using these data marts to gather information. These warehouses or application networks may or may not morph into an ODS as consolidation occurs. If no warehouse currently exists to act as the stepping stone to an ODS, companies may find it more economical to simply interconnect their systems in a mesh network using existing EAI technologies instead of following a more planned, fruitful, but expensive path to an ODS.

Real-Time Business Intelligence Pages