Virtusa’s vDataAid solution helps organizations accelerate data modernization platform development by deploying a configurable and metadata-driven solution.
The solution covers different phases, including data ingestion, data validation, and Slowly Changing Dimensions (SCD) data processing. It combines multiple data frameworks, such as Generic Data Ingestion, Data Validation, and SCD Type 1 and Type 2, that are easily configurable, customizable, and deployable for any Microsoft Azure platform.
The vDataAid solution is developed using Azure Data Factory for data ingestion and Spark Notebook for data validation. Azure data integration pipeline is a generic pipeline used for data ingestion and validation that is completely driven by metadata.
For instance, the first step with any data source configuration is to capture the ingestion details like source and target paths, objects to be ingested, etc., in pre-configured metadata tables. Then, we use the single generic pipeline for ingestion, validation, and transformations (SCD) of all the objects without creating and maintaining multiple pipelines.
Key features
Ensuring comprehensive and accurate data with vDataAid
vDataAid is a low code solution that audits each level in the data loading process to provide in-depth insights and control the pipeline behavior. Its key features include:
Data ingestion
- Ingests data from all sources without creating multiple ingestion pipelines
- Ingests data from various data sources like RDBMS (Oracle, DB2, SAP, Teradata, etc.) and files (Txt, CSV, Excel, Json, XML, etc.)
Auditing
- Stores and manages statistics including start time, end time, batch, and job status
- Captures the source and target count, data validation count, and data validation summary
Job control and dependencies
- Pre-checks in batch/jobs before proceeding with execution and enables the ability to restart feature for failed jobs
- Maintains the metadata of the jobs and the dependencies
Job and object lineage
- Maintains the job and object lineage across multiple data layers in a data lake
Data validation framework
- Integrated, configurable, and customizable data validation framework for quality checks like not null, uniqueness, minimum, maximum, is between, length, mean and median, text matches, etc.
SCD Type 1 and Type 2 framework
- Integrated and configurable SCD Type 1 and Type 2 framework for maintaining the most up-to-date data in the data lake
Error logging and notifications
- Captures error at every activity and logs in the auditing tables
- Sends error notification emails on errors in the pipeline
Dashboards and reports for operational insights
- Use tools like Power BI to create dashboards and reports for additional insights
One-stop solution for seamless data integration
vDataAid is a configurable and extensible solution with a plug-and-play feature for Azure that enables hassle-free data integration.
- High operational efficiency with 30-40% reduced effort on pipeline development
- Reduction in development time and errors
- Faster onboarding of new data set and sources
- High code consistency
- Processing quality data across layers in the data lake
- Better insights to improve business process