Software Engineer - AI Evaluation & Automation

About this role

IDR is seeking a Software Engineer - AI Evaluation & Automation
to join one of our top clients for a remote opportunity. This role is within the technology sector, focusing on developing tools to measure the performance of AI-powered software development solutions. The position offers the chance to work on innovative projects that ensure the reliability and reproducibility of AI evaluations.

Position Overview for the Software Engineer - AI Evaluation & Automation:

• Help build and scale tooling used to measure the performance of AI-powered software development tools.
• Develop evaluation harnesses, automate benchmark runs, and ensure results are reproducible and trustworthy.
• Focus on validating measurement approaches against human judgment and analyzing benchmarking results.
• Collaborate with engineering and data teams to improve tooling and document evaluation processes.
• Support execution-based benchmarking across quality, productivity, and efficiency measures, including cost and latency.

Requirements for the Software Engineer - AI Evaluation & Automation:

• Strong software engineering background, with experience building automation, developer tooling, or validation systems.
• Proficient in Python, and comfortable with Java, JavaScript, or similar languages.
• Solid working knowledge of Git, including branch management, history, and worktrees, and containerization with Docker.
• Experience with APIs, development environments, CI/CD pipelines, and engineering workflows.
• Some familiarity with AI, LLM, or agent evaluation processes, including understanding common pitfalls.

What's in it for you?

• Competitive compensation package
• Full Benefits; Medical, Vision, Dental, and more!
• Opportunity to get in with an industry leading organization.

Why IDR?

• 25+ Years of Proven Industry Experience in 4 major markets
• Employee Stock Ownership Program
• Dedicated Engagement Manager who is committed to you and your success.
• Medical, Dental...