Senior Data Engineer

About this role

No third party recruiters please

Must be able to work W2

Job Summary:

We are seeking an experienced Data Engineer with 6–10 years of expertise in designing, building, and maintaining scalable, high-performance data pipelines and processing frameworks. The ideal candidate will have strong hands-on experience with orchestration using Apache Airflow and distributed data processing with Apache Spark (EMR). This role requires a solid understanding of big data architecture, data engineering best practices, and a commitment to delivering efficient, reliable, and maintainable data solutions that align with both business and technical requirements.

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Roles and Responsibilities:

• Design, build, and manage scalable and reliable data pipelines using Apache Airflow.

• Develop and optimize large-scale data processing workflows using Apache Spark, including both batch and Structured Streaming.

• Collaborate with data architects, analysts, and business stakeholders to translate data requirements into efficient engineering solutions.

• Ensure the quality, performance, and security of data processes and systems.

• Monitor, troubleshoot, and optimize data workflows and job executions.

• Document solutions, workflows, and technical designs.

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Qualifications Required:

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field.

• 6–10 years of experience in data engineering or related roles.

• Strong experience with Apache Airflow for data orchestration and workflow management.

• Proven expertise in building and tuning distributed data processing applications using Apache Spark (PySpark), including both Structured Streaming and Batch.

• Experience with AWS cloud-based data ecosystems, particularly Athena and Redshift.

• Proficient in SQL (Athena version) and expe...