Senior Principal Software Engineer

About this role

The Data And AI team is a highly focused effort to lead digital-first execution and transformation at Red Hat leveraging data & AI strategically for our customers, partners, and associates. The engineering team is focused on building and delivering strategic AI Factory platform for building Skills and Agents, designed to augment human capabilities, accelerate business workflows, and scale operations across the enterprise.

In this role, you'll take ownership of end-to-end AI/ML systems, champion best practices, and deliver impactful, production-grade models. You’ll work autonomously, mentor others, and collaborate with data, engineering, and product teams to bring AI features into production.

What you will do?

• Design, build, and evolve ML pipelines that cover data ingestion, preprocessing, feature engineering, training, validation, deployment, and monitoring.
• Design, build, and evolve MCP server, Skills and Agents that enable and empower Red Hatters to do business efficiently.
• Translate research prototypes and models into production-quality code, ensuring robustness, scalability, and maintainability.
• Select appropriate algorithms and modeling techniques, perform hyperparameter tuning, and conduct comparative experimentation.
• Evaluate and validate model performance using advanced metrics (e.g. ROC-AUC, precision/recall curves, calibration, fairness, drift) and set up continuous validation/regression checks.
• Collaborate with software engineers, data engineers, and SRE/DevOps to integrate ML services into broader systems.
• Instrument models and systems with monitoring, logging, alerting, and automated healing or scaling mechanisms.
• Troubleshoot and resolve production incidents, root-cause errors, data drifts, performance regressions, or infrastructure issues.
• Mentor more junior engineers, lead code reviews, and help establish ML lifecycle and quality standards.
• Stay current with emerging ML research, frameworks, and tooling, and proacti...