Senior Software Engineer - Airflow

About this role

Business Area:

Engineering

Seniority Level:

Mid-Senior level

Job Description:

At Cloudera, we empower people to transform complex data into clear and actionable insights. With as much data under management as the hyperscalers, we're the preferred data partner for the top companies in almost every industry. Powered by the relentless innovation of the open source community, Cloudera advances digital transformation for the world’s largest enterprises.

As a Senior Airflow Engineer at Cloudera, you will be a key architect and builder of the orchestration layer for the Cloudera Data Platform (CDP). This isn't just about writing DAGs; you will be optimizing the Airflow infrastructure that powers mission-critical data pipelines for the world's largest enterprises.

You will bridge the gap between core platform engineering and data workflow automation, ensuring scalability, security, and high availability across hybrid and multi-cloud environments.

## As a Senior Airflow Engineer you will:

• Engineer the Platform: Design, deploy, and manage large-scale Airflow environments on-prem and on Kubernetes (K8s) using YARN or Kubernetes executors.
• Develop Custom Providers: Build bespoke Airflow operators, hooks, and sensors to integrate seamlessly with CDP services like Ozone, Hive, and Impala.
• Optimize & Scale: Tune Airflow schedulers and workers to handle thousands of concurrent tasks with minimal latency.
• Secure the Workflow: Implement robust IAM roles, Kerberos integration, and secret management to meet rigorous enterprise security standards.
• Drive CI/CD & DevOps: Develop automated testing frameworks for DAGs and maintain Infrastructure-as-Code (IaC) for Airflow deployments.
• Mentor & Lead: Conduct high-impact code reviews and provide technical guidance on best practices for complex workflow orchestration.

## We’re excited about you if you have:

• Deep Orchestration Expertise: Advanced knowledge of Apache Airflow, including XComs, Task Grou...