Data Engineer

About this role

Job Summary

We are seeking an experienced Azure Data Engineer to design, build, and optimize scalable data pipelines and lake house architectures within the Azure ecosystem. This role is responsible for developing end-to-end data ingestion, transformation, and governance frameworks using Azure Data Factory, Databricks, and ADLS Gen2 to support enterprise analytics and reporting needs.

The ideal candidate will have hands-on experience managing batch and near real-time data ingestion from diverse structured and unstructured sources, implementing Spark-based transformation processes across Bronze, Silver, and Gold data layers, and establishing robust data quality, security, and monitoring frameworks. This position also requires expertise in data governance, automation, and cost optimization to ensure high-performing, secure, and reliable data platforms aligned with organizational SLAs.

Key Responsibilities

Azure Data EngineeringLakehouse Architecture

• Design and manage scalable data ingestion pipelines using Azure Data Factory and Azure Service Bus for batch and near real-time processing
• Implement Bronze, Silver, and Gold data layer transformations using Databricks (PySpark, SparkSQL) within a lakehouse architecture
• Integrate structured, semi-structured, and unstructured data sources using formats such as Parquet, Avro, and JSON into ADLS Gen2

Data Governance, SecurityQuality Management

• Establish data quality frameworks and validation processes using Databricks and Azure Data Factory
• Configure RBAC and ACL-based access controls to secure sensitive datasets and ensure compliance
• Manage credential security with Azure Key Vault and enforce governance standards across data platforms.

Automation, MonitoringCloud Optimization

• Automate pipeline orchestration using Databricks Workflows and CI/CD tools such as Azure DevOps, GitHub Actions, and Terraform
• Monitor pipeline health and performance using Azure Monitor, logging, and alerting...