Data Automation Engineer

About this role

Dice is the leading career destination for tech experts at every stage of their careers. Our client, Robert Half, is seeking the following. Apply via Dice today!


Description
Position Overview


We are seeking a delivery-focused Data Automation Engineer to design and implement innovative automation solutions across a Microsoft Azure-based data analytics platform. This role partners closely with engineering teams and stakeholders to translate business requirements into scalable data engineering and AI-enabled solutions.


The ideal candidate is hands-on with Azure Data Factory, Synapse Pipelines, Apache Spark, Python, and SQL, and brings experience building reliable ETL pipelines across SQL and NoSQL environments. This role emphasizes performance optimization, automation, and proactive data quality within Agile DevOps delivery models.


Key Responsibilities


Data Engineering & Automation

• Develop high-performance data pipelines using Azure Data Factory, Synapse Pipelines, Spark Notebooks, Python, and SQL.
• Design ETL workflows supporting advanced analytics, reporting, and AI/ML use cases.
• Implement data migration, integrity, quality, metadata, and security controls across pipelines.
• Monitor, troubleshoot, and optimize pipelines for availability, scalability, and performance.

Performance Testing & Optimization

• Execute ETL performance testing and validate load performance against benchmarks.
• Analyze pipeline runtime, throughput, latency, and resource utilization.
• Support tuning activities (e.g., query optimization, partitioning, indexing).
• Validate data completeness and consistency after high-volume processing.

Platform Collaboration & DevOps Support

• Collaborate with DevOps and infrastructure teams to optimize compute, memory, and scaling.
• Maintain versioning and configuration control across environments.
• Support production, testing, development, and integration environments.
• Actively participate in...