Data Scientist III

Responsibilities

You will design, develop, and improve AI/ML solutions, including RAG pipelines and agentic workflows, to address enterprise business challenges. Additionally, you will collaborate with engineering teams on production readiness, model evaluation, and the implementation of MLOps/LLMOps practices.

Requirements

Candidates must have 5-9 years of relevant experience in Data Science or Machine Learning and a bachelor's degree in a related field. Strong proficiency in Python, statistical analysis, and practical experience with LLMs and MLOps are required.

Skills

• Python
• Machine Learning
• Statistical Modeling
• Generative AI
• RAG
• LLMOps
• MLOps
• SQL
• PyTorch
• TensorFlow
• Pandas
• NumPy
• Data Science
• Agentic AI
• Prompt Engineering
• Cloud Computing

About this role

BASIC INFORMATION

Job Name
Data Scientist III - USA (B)
Country
United States
State
California
Date Published
05-Nov-2026
Job ID
47815
Travel
You may occasionally be required to travel for business
Additional Locations
San Francisco - Los Angeles - San Diego - Sacramento - Anaheim - San Francisco
This role can be based remotely in United States 
Looking for more details about our benefits? You can also learn all about them by clicking HERE

DESCRIPTION AND REQUIREMENTS

BMC empowers nearly 80% of the Forbes Global 100 to accelerate business value, faster than humanly possible. Our industry-leading portfolio unlocks human and machine potential to drive business growth, innovation, and sustainable success. BMC does this in a simple and optimized way by connecting people, systems, and data that power the world’s largest organizations so they can seize a competitive advantage.
We're looking for an experienced Data Scientist to join our AMI team. This hands-on role combines Machine Learning, statistical modeling, Generative AI, Retrieval-Augmented Generation (RAG), model evaluation, and applied AI experimentation for enterprise AI applications. AMI supports commercial AI providers and on-premises models, with agents, MCP tools, and A2A interoperability delivered in partnership with AI Engineering. 

About the Role:
You will design, develop, evaluate, and improve AI/ML solutions that address enterprise business and technical challenges. You will apply statistical rigor, reproducible experimentation, and MLOps/LLMOps practices to improve model quality, RAG effectiveness, and AI application performance. 
You will take ownership of defined Data Science initiatives, lead moderately complex or ambiguous modeling and evaluation tasks, collaborate with engineering teams on production readiness, and mentor less experienced Data Scientists. 

Key Responsibilities:
Machine Learning & Applied Data Science 
* Apply classification, regression, ranking, forecasting, clustering, and anomaly detection to enterprise AI and agentic use cases. 
* Develop and evaluate intent classification, semantic matching, agent/skill/tool ranking, and model-routing techniques using traditional ML, embeddings, and LLM-based approaches. 
* Apply statistical modeling to analyze execution latency, token usage, inference costs, workload trends, and resource requirements. 
* Use clustering and anomaly detection to identify recurring intents, unusual agent behavior, capability gaps, and execution regressions. 
* Conduct exploratory analysis, dataset preparation, feature engineering, hypothesis testing, and reproducible experiments. 

Foundation Models, Prompting & Context Engineering 
* Evaluate commercial and self-hosted open-weight models for accuracy, instruction adherence, structured outputs, latency, and cost. 
* Develop and test prompt templates, few-shot examples, tool-calling prompts, and context assembly approaches. 
* Compare model alternatives for defined enterprise use cases and document quality, performance, and cost tradeoffs. 
* Collaborate with Senior Data Scientists and AI Engineering on model performance analysis and inference optimization.

Retrieval-Augmented Generation (RAG) 
* Develop and evaluate document chunking, metadata, embedding, retrieval, reranking, and context selection strategies. 
* Compare vector, keyword/BM25, hybrid, and semantic retrieval approaches. 
* Measure retrieval effectiveness using Recall@K, Precision@K, nDCG, groundedness, faithfulness, and answer quality. 
* Investigate retrieval and generation failures and recommend improvements based on evaluation results. 
* Benchmark embedding and reranking models using representative enterprise datasets.

Model Fine-Tuning & Adaptation 
* Support evaluation of prompting, RAG, supervised instruction fine-tuning, and parameter-efficient model adaptation approaches. 
* Prepare and validate instruction datasets, training examples, and holdout evaluation datasets. 
* Participate in SFT, LoRA, QLoRA, or comparable fine-tuning experiments under appropriate technical guidance. 
* Compare adapted models against baseline models for accuracy, generalization, regressions, and operational cost. 
* Collaborate on validation and production handoff of fine-tuned model artifacts. 

Agentic AI, MCP & A2A 
* Apply modeling and evaluation techniques to intent classification, agent/skill discovery, tool selection, and routing effectiveness. 
* Evaluate task completion, tool-call correctness, parameter extraction, delegation quality, and recovery behavior. 
* Understand agentic AI concepts, including planning, tool calling, MCP tools, A2A Agent Cards, skills, and task delegation. 
* Collaborate with AI Engineering to investigate model-related issues affecting agentic workflows. 
* Contribute model-based decision signals and evaluation approaches without owning agent runtime, MCP server, or A2A protocol implementation. 

AI Evaluation & Experimentation 
* Develop evaluation datasets, golden test cases, holdout sets, and automated evaluation workflows. 
* Evaluate ML models, foundation models, prompts, RAG pipelines, fine-tuned models, and agentic AI behaviors. 
* Apply statistical analysis, human evaluation, and LLM-as-judge approaches with appropriate calibration. 
* Track accuracy, relevance, groundedness, instruction following, task completion, latency, and cost. 
* Investigate evaluation failures and recommend improvements based on experimental evidence. 
* Support automated regression testing and production readiness validation. 

MLOps, LLMOps & Model Lifecycle 
* Apply experiment tracking and versioning across datasets, code, model configurations, prompts, retrieval settings, metrics, and artifacts. 
* Develop and maintain reproducible model training, benchmarking, and evaluation workflows. 
* Use model registries, experiment tracking tools, and automated evaluation pipelines. 
* Contribute to model quality gates, regression te...

About BMC Software

IT Services and IT Consulting · 5,001-10,000 employees · headquartered in Houston, Texas · founded 1980