AI Automation Engineer - Virtual

About this role

Job Description
AI Automation Engineer - Virtual


Alight Solutions - Lincolnshire, IL

• On-site, Remote

Machine Learning & Artificial Intelligence


Full-time


Medical, Life, Retirement, PTO


Posted 5 hours ago


Job Description
As the AI Automation Engineer in the AICoE, you will be instrumental in standing up theinitialautomation framework for AI Quality.You'llwork closely with the AI Quality Lead and developers to build, run, and scale automated test suites for AI models and agents.


Responsibilities

• Develop andmaintainautomated test frameworks for LLM, RAG, agentic, and classic ML systems
• Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use)
• Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments
• Author and validateOpenAPI/JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines
• Instrument observability metrics (latency, success rate, cost, drift) usingOpenTelemetryand export to dashboards
• Support data QA, lineage tracking, and drift monitoring (Evidently,OpenLineage)
• Collaborate with AI developers, SDETs, and SRE to troubleshoot,optimize, and document automation processes
• Contribute to red-team playbooks and securitytestbooksmapped to OWASP/ATLAS risks

Requirements

• Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but notrequired)
• Proficiencyin Python, JavaScript/Node.js, or similar programming languages
• Familiarity with CI/CD, version control, and cloud-based deployment environments
• Experience with test automation tools (promptfoo, Ragas, Postman)
• Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
• Basic knowledge of fairness, safety, and security testing for AI systems
• Experience with AI infrastructure, orchestration, data integ...