Senior Data Engineer

About this role

Job title: Senior Data Engineer

Location: Denver CO/ Remote

Only W2 (Need Visa Independent Candidates)

Data Engineer IV designs, builds, and maintains the ETL pipelines and data infrastructure that feed the IIA Data Lake, anomaly detection models, and AI agents. This role focuses on constructing robust, scalable data pipelines using Spark/Scala, ensuring data quality and availability across a growing portfolio of network data sources, and enabling downstream consumers (data scientists, agents, dashboards) to access reliable, well-structured data.

MAJOR DUTIES AND RESPONSIBILITIES

· Design, develop, and maintain scalable ETL pipelines using
Apache Spark (Scala)
to ingest, transform, and load network data into the IIA Data Lake.

· Onboard new data sources (network telemetry, syslogs, SNMP traps, device configuration data, ticketing systems) by building ingestion pipelines from raw source to query-ready format.

· Implement monitoring and alerting solutions to ensure data pipeline reliability and performance.

· Develop and manage
deployment pipelines to facilitate continuous integration
and delivery of data engineering solutions.

· Manage and optimize data storage solutions, including distributed file systems, relational databases, flat files, and external sources accessed via API.

· Implement data quality checks, validation rules, and automated testing to ensure pipeline reliability and data integrity.

· Optimize pipeline performance for large-scale data processing (billions of events per day) across batch and mini-batch processing patterns.

· Manage and evolve data schemas, partitioning strategies, and storage formats to support efficient querying and downstream consumption.

· Support data backfills and recovery when upstream issues or schema changes require reprocessing.

· Collaborate across teams to ensure data solutions align with existing production architectures and business requirements.

· Work with data scientists and agen...