Senior Staff Software Engineer (AI Platform)

About this role

• As a Senior Staff Software Engineer on the AI Platform team, you will shape and scale Pinterest’s AI/ML evaluation platform into a unified, reliable system adopted company-wide
• You will identify modernization opportunities, elevate the roadmap, and execute technical strategy of a durable evaluation system for GenAI and ML workloads at Pinterest, partnering across engineering, research, product, and safety to improve quality, launch decisions, and trust in AI-powered experiences
• Identify modernization opportunities and scale the roadmap for a company-wide evaluation system in partnership with technical leadership
• Define cross-team technical strategy, align partner teams, and drive broad adoption of shared evaluation capabilities
• Make durable architectural decisions that improve the user experience and reliability of Pinterest’s AI/ML evaluation platform
• Establish evaluation methodologies and standards that balance automated metrics, expert judgment, human feedback, safety, and business outcomes
• Lead complex cross-team workstreams, resolve technical tradeoffs and risks, and deliver impact

• Deep familiarity with GenAI and RecSys evaluation libraries and frameworks, AutoEvals, OpenEvals, Ragas, etc
• 3+ years of deep, hands-on experience designing and operating evaluation systems for GenAI and complex ML systems, including benchmark design, golden datasets, automated metrics, human evaluation, model-based grading, and failure analysis
• Strong programming skills in Python coupled with a solid grasp of distributed systems principles, and comfortable iterating with AI tools on coding and design
• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent experience
• Strong judgment around GenAI risks, including robustness, bias, privacy, security, and the limitations of automated evaluation signals
• 5+ years of experience building scalable platforms for distributed experimentation, data and model versioning, orchestration, obse...