Shadowbase Solutions and the SOLV™ Family of Products
SOLV ETL
- Snapshot point-in-time loading utility
- Extract, transform, filter, cleanse, and load
SOLV Loader
- Initial Loading Utility
- Integrates with the Shadowbase replication engine to keep target synchronized during and after the load completes
SOLV V&V (Available in 2010)
- Verification & validation Utility
- Identifies discrepancies between source and target while source is updated
SOLV Resync (Available in 2010)
- Reconciliation utility
- Optional resolution of target discrepancies
SOLV Overview
The SOLV product is architected to provide online loading of a source database into a target database, meaning that the source and/or target databases can be open for reading or updating while the load occurs. In contrast to other loading products, when using SOLV there is no need to take either the source or target databases offline (and make them inaccessible to applications) while the load occurs.
SOLV and Shadowbase Replication
The figure above depicts a typical SOLV loading scenario. In this example, an Application is actively processing user requests and updating the Source Database. SOLV is loading the Source Database into the Target Database via the Shadowbase Replication Engine. Note that the Application has full access to the Source Database while the load progresses. The Shadowbase Replication Engine is responsible for merging the Application’s source database changes that have been collected in the Audit Trail with the data being loaded before it applies them into the target database. SOLV and Shadowbase Replication Engine then keep the target database synchronized with the changes being made to the source after the load completes.
The product can load audited and non-audited NonStop Enscribe source files and SQL tables into any target environment and database combination supported by the Shadowbase line of data replication products (e.g., NonStop Enscribe or SQL targets, or Open Server targets such as Oracle, Sybase, SQL Server, DB2, and MySQL).
SOLV ETL & Loader
By itself, the SOLV ETL & Loader can produce “snap-shot” loads of all or part of the source database into the target environment. These are often called “point-in-time” loads as they are initiated “as of” a particular point-in-time. When used in this mode, any changes made to the source database while the load runs and after the load completes are not subsequently replicated to the target environment.
During a SOLV load, note that the target database is loaded in a consistent state in the sense that the SOLV utility only reads and loads consistent source database data (i.e., unless configured to do so, it does not read through locked or otherwise dirty data). However, since the load takes time, if the source database is being updated while the load is occurring, then the target will have load-time consistent data in it. When run in this mode, to ensure that the target matches the entire source after the load completes, the source database needs to stop updating when the snap-shot load is taken.
However, note that the SOLV ETL & Loader is fully integrated with the Shadowbase database replication engine for data transport and target database I/O. However, one does not need to license the Shadowbase engine for data replication in order to use SOLV ETL & Loader.
More specifically, SOLV ETL & Loader uses the Shadowbase replication engine for certain services. If the Shadowbase engine is being used for data replication of the source file or table being loaded to the target, the SOLV ETL & Loader can integrate with it for loading the target database while database change replication is occurring. The SOLV ETL & Loader fully integrates with the Shadowbase data replication engine so that the data being replicated and the data being loaded are properly serialized when applied to the target database. Hence, while the load occurs, the Shadowbase engine properly keeps that part of the target database that has already been loaded synchronized with the changes that the application is making to the source database. This approach is particularly useful for high volume applications because there is no queue of changes applied into the target environment after the load completes. Rather, the changes made to the application database that need replication are consumed while the load occurs, and the target is always kept current with the source’s changes during the load. The target does not grow stale after the load completes because the Shadowbase data replication engine continues to keep the target database synchronized to the source database.
- The SOLV utility uses the Shadowbase replication engine’s powerful data transformation, data filtering, and data cleansing routines and scripts. Many data mappings are performed out-of-the-box by the Shadowbase data replication engine. Other (more complex) data mappings must be developed once within Shadowbase technology. Both the Shadowbase data replication engine as well as the SOLV ETL & Loader use these same functions during the load. The routines/mappings only need to be written and tested once, since they are then used for both replication and loading. They do not have to be implemented in a separate custom ETL tool, script, or program.
- When used alongside Shadowbase data replication, the SOLV utility integrates the loading data with the source application‘s database changes so that the database changes to be replicated are properly sequenced and applied to the target database at the same time as the data is being loaded. At the end of the load, the target database is current and there is no queue of database changes that accumulated during the load that needs to be replicated and applied; instead, these changes are consumed during the load.
- The SOLV utility leverages the Shadowbase technology that allows the source and target to be similar (such as like-to-like replication), or markedly different (such as loading a non-relational source file into a target SQL table or when loading a data warehouse). Once Shadowbase replication is configured, the SOLV utility automatically uses this information for performing the load.
SOLV V&V
The SOLV V&V (verification and validation) utility will soon have the ability to feed data to a new verification and validation server named VVS. This new server gives a user the ability to compare a target database file or table to its source file or table, subject to the data mapping rules in use, and report any discrepancies found. Discrepancies are either extra rows in the target that are not in the source, extra rows in the source that are not in the target, or rows that exist in both the source and the target but that have different values in them.
The V&V run can occur without taking either of the two databases (source or target) offline, meaning the SOLV V&V utility can still perform the comparison even while the source application is actively changing the database!
The result of running VVS is a report that contains a list of discrepancies found between the source and target databases. The report can be generated in a BRIEF mode (just showing counts of the discrepancies found), as well as a DETAIL mode (showing the individual columns that mismatch and the data values in each).
SOLV Resync
The SOLV Resync utility is an extension of the SOLV V&V utility. The SOLV V&V utility reports on the discrepancies found, while the SOLV Resync utility generates the SQL scripts needed to bring the target database back into synchronism with the source database.
More specifically, if a row mismatch is found, the SOLV Resync utility generates the SQL scripts to repair the target database as follows:
- If both rows exist but the row contents do not match, the target row is updated (replaced) with the source row’s data values.
- If the source row exists but the target row does not, the source row is inserted into the target database.
- If a target row exists but the source row does not, the target row is deleted.
For more information, please see the Shadowbase SOLV Datasheet.
The SOLV product uses patented technology, with additional patents pending. (See U.S. Patent Numbers 6,745,209, 7,003,531, and 7,321,904 at www.uspto.gov.)
