Lead Data Engineer

About this role

Job Summary:
We are seeking a skilled Senior Data Engineer with strong expertise in Databricks, Python, PySpark, AWS, and Medallion Architecture (Bronze, Silver, Gold layers) to design, build, and optimize scalable data pipelines and modern data platforms. The ideal candidate will have experience developing cloud-based data solutions, implementing data lakehouse architectures, and transforming large volumes of structured and unstructured data into reliable business-ready datasets.

Key Responsibilities
Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
Build and manage data lakehouse solutions utilizing Medallion Architecture (Bronze, Silver, Gold layers).
Develop ETL/ELT processes to ingest, transform, and load data from various sources.
Optimize data processing workflows for performance, scalability, and cost efficiency.
Collaborate with Data Analysts, Data Scientists, and Business Stakeholders to understand data requirements.
Implement data quality, validation, monitoring, and governance processes.
Create and maintain technical documentation, data models, and architecture diagrams.
Troubleshoot and resolve data pipeline issues and production incidents.
Follow best practices for security, compliance, and data management.

Required Qualifications
Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
10+ years of experience in Data Engineering.
Strong hands-on experience with Databricks.
Proficiency in Python and PySpark.
Experience implementing Medallion Architecture (Bronze, Silver, Gold layers).
Experience building and optimizing large-scale ETL/ELT pipelines.
Understanding of Data Lake, Data Warehouse, and Lakehouse concepts.
Experience with SQL and performance tuning.
Strong problem-solving and analytical skills.

Pay: $65.00 - $75.00 per hour

Work Location: Remote