Databricks Data Engineer and Scientist

About this role

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Databricks Data Engineer and Scientist based in United States.
In this role, you will design, develop, and optimize enterprise-scale data pipelines within a Databricks and AWS-based Lakehouse environment. You will transform complex data into reliable, analytics-ready assets that support informed business decisions and advanced analytical initiatives. Working across data engineering, data modeling, and AI-assisted development, you will modernize legacy systems while ensuring data accuracy, performance, and scalability. You will contribute to a Bronze, Silver, and Gold medallion architecture and build robust ETL processes with comprehensive logging and error handling. The role combines hands-on technical delivery with collaboration across cross-functional teams in support of a critical federal data environment. This is a full-time, remote opportunity for an experienced professional who enjoys solving complex data challenges and improving enterprise data infrastructure.


Accountabilities

• Design, build, deploy, and maintain end-to-end data pipelines for an enterprise Lakehouse using Databricks and AWS.
• Develop and manage reliable ETL processes with comprehensive logging, error handling, troubleshooting, and issue resolution.
• Design and maintain data models within a Bronze, Silver, and Gold medallion architecture, selecting appropriate dimensional or relational modeling approaches.
• Analyze and decompose complex legacy SQL, rebuild pipelines and data models, and validate results for accuracy and performance.
• Optimize data pipeline performance, scalability, and query execution using partitioning, bucketing, caching, and other relevant techniques.
• Use AI-assisted coding tools to accelerate pipeline analysis, data engineering, debugging, and data modeling tasks.
• Support data security and compliance requirements while ...