Coglin Mill - RODIN, Enterprise Ed.

• a fully architected Business Intelligence (BI) environment including Data Warehouses (DW), Operational Data Stores (ODS), Data Marts (DM) and Reporting Databases and their associated reports (RD), collectively referred to herein as Data Repositories,
• data integration programs required to provide one-way or two-way data integration between disparate applications (i.e. to integrate software from different vendors or different internal applications),
• data conversion programs required to move and manipulate data when converting from an old business application to a new one, and
• many other DAM applications,

All without writing a single line of program code (HLL or SQL), in a fraction of the time it would otherwise take using conventional programming methods.

Because RODIN is purpose built for full life cycle DAM applications, it provides many advanced facilities, often automatically or with a single click of the mouse via its fully graphical interface, that would otherwise take many hours, days, weeks or months to program manually, even if using the best industry 4GLs.

RODIN and its integrated components can acquire data from any business application on many hardware platforms from many databases, file systems, spreadsheets, etc., from internal (within the business) or external (e.g. syndicated, trading partners, etc) data sources. It can then extract, validate and transform the acquired data using as many input sources as required, and together with flexible and intuitive business rule definitions stored in RODIN’s in-built Meta Data Repository, load the required data into one or more target Data Repositories or application database in a single process. There is rarely a need for multiple passes of the same data even when there are multiple targets.

RODIN’s unique parallel load technology provides scalability on a massive scale. It is as happy processing and managing terabytes as it is processing and managing megabytes. Not only does RODIN support a 2 stage Extract, Transform, Load (ETL) process that allows one or more ETL steps to be executed concurrently on the same CPU for the same or separate ETL processes, it also allows a single step to be physically split so it can be executed across multiple CPUs. The equivalent of using all lanes of the highway to move freight in the shortest possible time.

Coupled with advanced parallel hardware and database technologies, RODIN can deliver staggering performance and has been tested to in excess of 625 million rows per hour (using complex ETL logic with 8 separate source tables) on a single machine. It also supports machine clusters in a Massively Paralleled (MPP) environment.

Because RODIN generates and compiles object code there are many functions it supports that are not easily achieved using SQL or SQL based “transformation engines” or “data loaders”. Even though it is possible within RODIN, it is rare that “user exits” are required to be written and executed for specific situations that are too complex to specify within the RODIN Meta Data. This is certainly not the case with all SQL and SQL based software products that profess to be ETL tools. Most of these tools require many user exits to satisfy most organization’s needs, user exists that require significant technical expertise to design, write, test, document and maintain and most of which aren’t supported by accessible Meta Data. Such tools often also require multiple passes of the data to achieve the desired result, even when there is only a single target. When there are multiple targets, multiple passes are mandatory.

The unique RODIN “Active Meta Data” technology that generates all the necessary programs from the requirements definitions, frees up often time-constrained technical staff from having to design, write, test, document and maintain any HLL programs or SQL code. And again, because its purpose built, RODIN only compiles in the required functions ensuring performance is optimum.

Data quality management is an often-overlooked aspect of the ETL process. It is imperative to be able to define business rules to specify data quality requirements, but equally important to report, manage and allow correction and re-processing of rejected data. While many tools fall far short in this task, RODIN has extremely powerful data quality management capabilities.

In summary, RODIN’s value propositions are development and execution speed and accuracy, comprehensive system functions and applications and its ability to change as the organization’s requirements change. Scalability, robustness and flexibility. And you’ll like our flexible licensing options as well as our extensive service offerings even more.

Summary of Functions

RODIN is a fully managed and automated rapid application development environment encompassing the following key areas of DAM technology, delivered as a single fully integrated tool :-

Data Acquisition

• Data propagation, extraction, transformation and enhancement
• Access data from most platforms and databases including legacy sources
• Change Data Capture – either in batch or real time from journal entries
• Business rule and processing logic definition
• Data validation with automated error and reject management
• Advanced parallel technologies for extreme performance.

Data Management

• Data Repository definition, construction and maintenance
• Aggregation (if required), loading and updating of data into target data structures
• Complete index management
• Data Model information.

Data Delivery

• Data mart definition, construction and maintenance
• Aggregation (if required), loading and updating of data into target data structures
• Output to all popular OLAP, query and desktop tools as well as any other application.

Administration

• Integrated change management, versioning and archiving
• Extract, load and status reporting and statistics
• Import source and target definitions from modelling tools or existing databases
• Security, calendar and environment (e.g. development, test, production) management.

Active Meta Data

• Automatically and manually created and maintained
• Active, visible and accessible in open database structures
• Used to generate and maintain databases and applications (object code)
• HTML publication with indexing and search capabilities
• Integrated with third party technologies
  

• Designed specifically for the creation and maintenance of a complete and sophisticated data warehouse environment and as such provides many automated functions relevant to this environment that are not available in tools that were not specifically designed for such e.g. SQL and SQL based tools and data replication, propagation and / or synchronization tools
• Automates the creation and maintenance of Data Repositories by generating native program code from the user definitions, saving weeks, months or years in application development time and ongoing application maintenance
• Generates highly efficient object code that optimizes the hardware, operating system and database functionality and performance without the need to write or modify high-level language programs or SQL code
• Easy to use maintenance functions allow a phased, iterative approach to implementation without the need to get it right first time or the need to spend the whole budget before delivering business benefit
• Visual Development Client with point and click, drag and drop ease of use
• True client / server implementation combining the power, security and scalability of the server with the ease of use and presentation capabilities of a graphical client
• Designed specifically for high performance and fully utilizes all available hardware and database SMP and MPP technologies (where appropriate)
• Highly scalable with no limits (other than hardware and database limits) to the size of the Data Repository
• Advanced date management and arithmetic capabilities
• Full international language support (for data, meta-data and user interface) including DBCS support
• Comprehensive on-line user documentation and hard copy training manuals
• Full range of consulting, education and implementation services available
• Flexible licensing options including subscription options with very low initial cost, LPAR pricing and multi-site discounts
• Optional annual maintenance agreement includes access to telephone and electronic support as well as new software releases, development of any customer required new functions and discounts on software upgrades resulting from hardware upgrades (i.e. tiered pricing)new functions and discounts on software upgrades resulting from hardware upgrades (i.e. tiered pricing)

• The RODIN development client provides a single point and click, drag and drop, user interface that can access all administrative functions.

  
  

• Support for multiple “environments”, including movement between environments, to cater for such things as separate development, testing, training and production environments, current, previous and next software releases, different companies, divisions or organizational entities where data consolidation is not required and / or is not advisable and different DAM applications
• Comprehensive application level security by individual user or group of users, as well as full integration with iSeries security
• Extensive capture and reporting of extract, load and usage statistics that can be used for performance tuning, capacity planning, cost recovery or billing applications
• Extensive reporting of the in-built Meta Data Repository (via reports or web-based HTML documents) for user and system documentation and change impact analysis

• Created automatically and / or manually and includes a limitless text capability for every object e.g. column, table, application, extract process, business rule (at the component level if required), etc, etc, to cater for the full definition and description of each object in the overall environment
• Is accessible, visible and navigable via the RODIN Graphical Integration Manager, a web-browser or hard copy reports
• Is stored in open database tables which are fully accessible by many specific database and ODBC capable tools
• All RODIN processes are automatically generated and controlled by active meta data eliminating significant development time and cost and ensuring the meta data and the physical implementation (i.e. the databases, rules, logic, security, calendars, etc) are always in sync
• In-built Change Management and versioning capabilities for full control of the development life-cycle
• Full support for SQL (alias) names as well as system names
• RODIN also integrates with many leading vendor products at the meta data level and is a member of several meta data initiatives including OMG’s Common Warehouse Meta-model (CWM) and the IBM DB2 Metadata program. This provides cross platform and cross tool integration of meta data as required

• Access remote System i/iSeries tables seamlessly in real time
• Access data from most commercial relational databases as well as many legacy (non-relational) sources directly from ETL process in real-time, potentially joining and merging data from disparate sources in a single process (non-IBM databases may require additional separately licensed middleware)
• Optionally pre-stage data from many RDBMS such as MS SQL Server, Oracle, DB2 UDB, MS Access and MySQL, avoiding the need for a real-time connection where inappropriate

Change Data Capture
In many cases, the changes to a table (inserts, updates and deletes) will need to be identified and selected for input to the ETL process. The Change Data Capture module is also designed to be implemented ‘stand-alone’ and therefore supports any application where changes are required to be captured, even if not related to building a Data Repository.

• Changed Data Capture servers may be deployed to capture changes to multiple tables using the journal logs. Extensive configuration options allow capture of detailed changes (eg, before and after images), consolidated changes (merge all changes down to a single resulting image), or replication, whereby a mirror image of the source table is maintained.
• A separate Apply process which may be requested for any or all tables creates a point-in-time result table that can be used as the source to an ETL process.
• User-defined journal entries can be used to trigger the Apply process, or to stop the server – allowing for an Apply or shutdown at a specific point in the journal, rather than being based on a set time.
• Extensive business rules allow changes to be filtered based on critical columns, values in the actual data or journal header information such as user name or program name
• Remote journals are supported
• Checkpoints are automatically created; extensive audit information is automatically generated and retained
• A separate batch change data capture process can be used to capture/apply changes at a specific time, based on the journal logs or, if the tables is not journaled, by comparison with checkpoint data from the previous run

• Automatic introduction and capture of source system metadata / data definitions (where available) allows instant access to existing source system tables, both local and remote.
• Includes integration with some popular vendor data dictionaries. e.g. JDE World and One-World dictionaries
• Ability to enhance the automatically captured meta data, including the ability to change names, descriptions, lengths, data types, etc. and to add limitless textual metadata to any table or column definition
• An existing table definition can be “refreshed” and if different to the old definition an automatic impact analysis report is immediately produced to show where the table and any changed columns are referenced
• Sample data function allows developers to quickly look at the source data from within the development client
• Ability to import source system data models from popular data modelling tools
• Tables and columns can have both global rules (which apply in all data acquisition processes where the table / column isused) and local rules (which only apply within a specific data acquisition process)
• Ability to use one or more source and reference tables (with no practical limits) to create, via easy to use wizards, an extract definition with comprehensive rules based selection, validation and transformation logic at table, column and / or processing logic sequence levels
• Full IF, THEN, ELSE, CASE, AND, OR logic can be specified for all selection, validation, creation, transformation and processing logic rules
• Rule components support all standard operations such as comparisons (equal to, less than etc), arithmetic, string operations etc, as well as table join tests (Row Found, Row not found) , Counters (at row and key change level) and many more
• Data selection (filtering) based on fixed rules or run-time parameters
• Global and local variables, with the same rules based logic capability, can be specified and used in any data acquisition processes
• Any rule can have any number of components e.g. If A is greater than B and A is less than C then D equal A multiplied by N. Nesting of conditions is supported, as well as CASE statements.
• Any number of distinct rules may be applied to a table or column.
• Rules can be applied to the source data, the target data or internal variable data
• Graphical visualization and maintenance of extract definitions including tables, columns, joins, rules, variables and processing logic via point and click, drag and drop facilities. Draw lines to represent joins and source to target mappings, significantly boosting development productivity
• Ability to map a source column to multiple target columns, with unique rules for each mapping
• Automatic conversion (casting) of data type where necessary (e.g. convert character to numeric, numeric to character etc)
• User definable justification or rounding options where applicable

  
  

• All file access methods supported including indexed, sequential and embedded SQL selection/ordering, with read, position to, read previous and read next operations on full or partial keys for joined tables to provide full control over data and processing logic, including the ability to access temporal data i.e. nearest previous date / time keyed data
• Full control over the sequence of table and column access, the application of logic rules and the initialization of derived and output data
• Automatic generation and management of “surrogate’ keys
• Support for Slowly Changing Dimensions (all types)
• Automatic detection and conversion of all legacy date and time formats to true date (*ISO) format
• Full date and time arithmetic allows calculation of days, months and years between two dates (including today), addition or subtraction of days, months or years from a date as well as age and expiry date calculations etc.
• Advanced automatic data validity checking includes the ability to test referential integrity, the contents of numeric data and the validity of dates
• Handling of invalid data, including automatic correction or substitution, or rejection, reporting and reprocessing (after correction and without the need to develop and maintain separate definitions or logic) of data that fails validation rules
• Ability to update a column or delete a row in a source table to signify the data has been “acquired” successfully into the Data Repository and / or to ensure data is not processed more than once
• Ability to define global or local External Calls (i.e. User Exits) with 2-way execution time parameter passing to cater for any non-standard cleansing, validation and manipulation of data (if required), including the ability to define rules to condition the calls
• Ability to add, change, refresh or delete a source table(s) and / or column(s) definition in an existing data acquisition definition at any time without having to touch unaffected logic or structures
• Automatic generation (from the user-defined extract definitions stored in the in-built Meta Data Repository) of source and object programs
• Optional Single-Stage or 2-stage load capabilities. In a single-stage load, the extract and load processes are combined into a single highly efficient program. In a 2-stage load, the load step is a separate (concurrent) job to the Extract/Transform step. This can be more efficient on multi-processor systems. It also allows for greater flexibility and customization in the load where necessary
• Target (i.e. load) definition is external to the data acquisition definition so a single data acquisition definition, can process all requirements regardless of how many targets there are with a single pass of the source data i.e. no more multiple passes to load detail and summary targets
• Flexible “many to many” ETL technology dynamically provides for (from active meta data or by execution time parameters) single or multiple target tables and / or columns using the same data acquisition definition, thus avoiding the need to write different definitions for initial loads, incremental updates, reject processing and additional targeting of already defined source data
• Advance parallel loading technology allows large jobs to be automatically split (based on volume of source data) into multiple parallel jobs to fully utilize all available processors
• In extreme cases, fully supports the operating system’s MPP capabilities to spread very large tables across multiple servers
• Automatic and user defined audit / hash totals generated into the Meta Data Repository and audit reports generated when an extract program is executed
• Extract definitions are secured by the “owner” who specifies what authorities other users have to the definition, who owns the program object and who has authorities to execute the program object
• Comprehensive change management functions, including check-in, check-out, versioning, change logs and comparison reports provide for full control over the development process. It is always possible to revert back to an earlier version of an extract definition
• Messages sent to user workstation or email address if excessive errors (user definable) or abnormal termination of an ETL process
• Works with all standard available system functions for database management and access including Distributed Data Management (DDM), opti-connect, object-connect and journalling (including remote journalling)
• Full (optional) commitment control logic ensures data (at the row level) is either processed or not processed, thus avoiding the possibility of a partially applied insert or update that requires a database restore or even worse, a full reload (can be turned off at execution time to facilitate initial historical loading)
• Ability to copy an existing data acquisition definition to create a new one or to move it to another environment e.g. from the development environment to the production environment

• Ability to import a target definition from a data modelling tool
• Ability to reverse engineer an existing target database (library containing any number of tables and indexes) definition to create any number of tables and associated indexes as a single step
• Ability to manually define target definitions (tables) using data elements (target columns) derived from the data acquisition process and / or defined through rules based logic within the data set
• Caters for any type or format of data supported by the database including *ISO date and time data types as well as null values
• Totally flexible definition of fully relational, de-normalized or snow-flake star schema data structures in detail, summarized and / or time summarized data sets (a data set is a Data Repository table with optional indexes, totals and security attributes automatically managed as one integrated definition and object within RODIN
• Any level of detail and / or summarization can be specified for a data set e.g. by product within invoice or region within company by day, etc., by simply defining the unique dimensions (where summarization is required)
• Populate (i.e. load, refresh or incremental update) the Data Repository at any time from historical, current operational and / or external data, with the ability to specify the data source(s), the data element(s), the data set(s) and the time span(s) to populate, to facilitate both the initial load and the ongoing maintenance of the Data Repository from the same automatically generated data load program(s)
• Allows the loading and maintenance of a single data set from multiple sources of data, that are loaded at different times if required
• Accumulate or replace option (column by column and table by table) when updating data sets allows for automatic aggregation or update of data
• Ability to create / manage data sets with no index, a single index or multiple indexes, unique or non-unique, including the ability to add, change, delete, maintain, drop and recreate indexes at any time and to reject attempts to create identical indexes
• Support for Encoded Vector Indexes (EVIs), which can dramatically improve query performance and / or reduce disk utilization when a “perfect” index is not available or desirable when accessing or extracting from the Data Repository
• User Exit programs can be executed before and after a data set load or update e.g. to drop and recreate and user defined indexes or views (RODIN defined indexes are automatically managed by RODIN)
• Add or change the definition of a data set at any time with automated conversion of existing data whenever these definitions are changed (after checking for sufficient available disk space beforehand obviously) as well as impact analysis of affected downstream processing
• Full change control over data set definitions, including check-in, check-out, versioning, change logs and comparison reports
• Sample data function allows developers to quickly look at the data set data from within the development client
• Process level journaling and commitment control can be easily and quickly turned on and off as required to facilitate fast initial loading of tables (off) along with total data integrity for subsequent incremental loading (on)
• In-built user definable totaling at all levels allows you to store data at detail and total level within the same data set. Totals are automatically updated whenever the detail level changes facilitating easy definition, maintenance, aggregation and fast “percentage of total” processing (single pass when using the Data Repository rather than double pass)
• Generation of new data from existing data with user definable conditional calculation routines and logic
• Subject Areas provide for data model information to be maintained in the RODIN metadata. Visual representations of relational or star-schema models aids in understanding
• Advanced filtering of definitions in the client based on name, partial name or subject area greatly simplifies maintenance, reducing lists of possibly hundreds of tables or thousands of columns to a relevant more manageable list
• User-definable attributes can be assigned to most entities, allowing for further organization/categorization of data sets and data elements

• develop and install your first data mart and set of outputs (which you must do)
• validate the data coming in is properly integrated and is of a consistent, known quality (which it must be)
• add the second, third, fourth, fifth, sixth and more data marts (which you will do)
• change an existing data mart to meet a changed requirement (which will happen all the time)
• introduce new data and requirements into the existing environment (which you hope happens)
• recover from a processing “glitch” caused by an operational problem (which will happen occasionally)
• understand the true meaning of the data by knowing its derivation rules (which you’d always like to happen)
• automate the processes to ensure stability, consistency, predictability and accuracy and possibly compliance with government regulations (which must be done)