success story

Paradigm shift to data integration frameworks for a leading global consumer electronics company

with Virtusa’s metadata-driven data integration framework on AWS using Talend

In recent times, the consumer electronics industry has been facing the challenge to keep up with the multiple data integrations from different sources across their products. With ever-changing customer demand, there has been a need to have a robust data integration framework that can ingest data from various data sources and transform as and when needed.  

Ensure that you open the links to enable the Next button, assuming you have read all the documents.
Okay & NextClick to Next
The Challenge

Our client, a leader in this space, had an architecture which was tightly coupled with storage and computation capabilities.

As a result, the entire system became expensive with heavy performance issues which was difficult to maintain. This led to many other challenges including slow adoption, lot of unused capacity during off-peak hours, storage capacity and disaster recovery issues. There was an acute need for increasing storage which would again lead to scaling expensive compute resource, delay in onboarding new apps and delayed data pipeline and process.

 

The Solution

Meticulous approach to data integration with metadata-driven framework by Virtusa.

In a typical/traditional ETL or data warehouse solution, you need to ingest data into your data lake from various source systems and cleanse them before they can be processed further by downstream applications. Additionally, in current scenario data migration from on-premise systems to cloud has been becoming more and more popular.

Virtusa as a strategic business partner initiated the process with migration of client’s existing data acquisition framework using Hadoop modernization techniques on AWS. In addition, we used Talend to extract, transform and load (ETL) leveraging customer’s serverless data lake framework solutions. 

Post which, we originated the idea to leverage metadata-driven data integration framework and developed the outline to ingest data from any structured data sources into any destination by adding metadata information into a metadata file/table. This framework can ingest data from any structured data source systems (RDBMS like Oracle, Local File, FTP server pulls etc.) and store data to any destination (AWS S3, Azure ADLS, RDS etc.). 

This accelerator supports schema evolution. Any change in schema of any existing feed doesn’t have any impact on the solution framework, thus reducing the need for any code change. This will save build and testing time and lot of effort by reducing the need for impact analysis of any schema changes.

 

Data Integration Frameworks Solution
The Benefit

Standardizing the process of data ingestion and integration with Virtusa metadata-driven data integration framework and data storage with AWS S3.

With Virtusa’s metadata-driven framework and data storage with AWS S3, we provided the ability to replicate and add new data sources in less turnaround time. The entire framework was at an abstraction layer where it’s easy to define/reuse mappings, ease of defining different sources and targets of where the data is supposed to be, and also effective define/reuse transformation rules in the metadata portion of the framework. We assisted the client with - 

  • Decoupled Compute and Storage scale independently
  • Persistent and transient Cluster deployment
  • Ease of use; isolation of workload; Automated
  • Elastic; build-in Auto-scaling
  • Cost Efficient
  • Workloads are moved from batch to Realtime with improved runtimes ~30 mins

In a nutshell, metadata driven ETL framework by Virtusa is an excellent approach for standardizing incoming data. It assisted simplifying a complicated process with speed development on the ETL side by providing more flexibility during the process of incorporating different data sources into a data warehouse. Customer can easily replicate the process without recreating something totally unique for each integration effort, or for each new set of data that needs to be integrated.

Analytics, Insights, Data

Modernize your data platforms and apply AI/ML to redefine, re-engineer business processes leading to superior customer experience and higher productivity.

Learn more Click to Learn more Request a demo Click to Request a demo
Related content