Data Engineer with Cloud Data Integration & Transformation

About this role

Dice is the leading career destination for tech experts at every stage of their careers. Our client, Next Gen IT Inc, is seeking the following. Apply via Dice today!


Data Engineer – Cloud Data Integration & Transformation
Location: Remote (CST/EST Horus)
Exp: 14+yrs
Note: Need 14 + years of experience. Candidate should have good communication skills.
Note Candidate must have experience of migration, integration from Legacy system (COBOL, DB2) to Azure Databricks.


About The Role
We are seeking a hands-on Data Engineer to develop and maintain scalable data pipelines and transformation routines within a modern Azure + Databricks environment. This role is focused on executing ingestion, cleansing, standardization, matching, merging, and enrichment of complex legacy datasets into a governed data lakehouse architecture.


The ideal candidate brings deep experience with Spark (PySpark), Delta Lake, Azure Data Factory, and data wrangling techniques — and is comfortable working in a structured, code-managed, team-based delivery environment.


Key Responsibilities
Pipeline Development & Maintenance
Build and maintain reusable data pipelines using Databricks, PySpark, and SQL.


Implement full and incremental loads from sources including VSAM, Db2 (LUW and z/OS), SQL Server, and flat files.


Use Delta Lake on ADLS Gen2 to support ACID transactions, scalable upserts/merges, and time travel.


Leverage Azure Data Factory for orchestration and triggering of Delta Live Tables and Databricks Jobs as part of nightly pipeline execution.


Data Cleansing & Transformation
Apply cleansing logic for deduplication, parsing, standardization, and enrichment based on business rule definitions.


Use Spark-Cobol Library to parse EBCDIC/COBOL-formatted VSAM files into structured DataFrames.


Maintain ''bronze → silver → gold'' structured layers and ensure quality during data transformations.


Support classificat...