AI Applied Scientist, AI Evaluation Science

AppleSeattle, WashingtonOn-siteFull-timeMid level, 2–5 yearsListed 50 minutes ago

Apply now

About this role

AI systems are only as trustworthy as the methods used to evaluate them. At Apple, where AI powers experiences for billions of people, getting evaluation right is not a support function, it is a foundational science. Our team, part of Apple Services Engineering, is building that scientific foundation: rigorous, scalable evaluation methodology for LLMs, agentic systems, and human-AI interaction. Our goal is evidence that teams can act on: what to measure, whether a change helped, why a system failed, and whether it is ready to ship.

What makes this team unusual is its interdisciplinary core. You will work alongside measurement scientists (psychometrics, validity theory), ML researchers, and platform engineers, bringing together ML research, statistical rigor, and production engineering.

We are looking for an Applied Scientist who works across the full arc of evaluation science: turning research into tools engineers can use on their own, and turning what those engineers run into new research. You care whether a method holds up on real data, at real scale, for a real decision, and you are curious enough to go after the open question underneath when it doesn't.

Our team is building self-service evaluation: tools, SDKs, and AI-powered capabilities that let engineers across Apple evaluate their AI features on their own, quickly and correctly, so they can build and ship those features faster. You will develop the methods that make this possible: raising the quality of the evaluations engineers run, and helping them understand what their results mean and what to do next. Often the method a problem needs doesn't exist yet. When that happens, you will figure it out, whether that means extending a recent paper or developing something new, and share what you learn through publications and internal research reports. We are looking for individuals across a range of experience levels.

With research and measurement scientists, you will co-develop new methods, make sure the tools give engineers valid, trustworthy signals, and bring back the hard problems that should shape the team's research agenda. With platform and research engineers, you will define what each method needs to work reliably without an expert in the loop, including its assumptions, failure modes, and validation tests, so it can ship in the evaluation SDKs and APIs used across Apple.

The successful candidate combines scientific depth with practical judgment: you know which methods fit a problem and why, can design experiments that show whether they actually work, can diagnose why they break down on messy real-world data, and can explain clearly what an evaluation result does and does not support.

Minimum Qualifications

Ph.D. in Computer Science, Machine Learning, Statistics, or a closely related field, or a Master's degree in one of these fields with 3+ years of applied research experience.
Deep expertise in evaluating AI systems, gained through research on evaluation methods, substantial hands-on experience evaluating LLMs or agentic systems in real products, or both.
Strong proficiency in Python and the modern ML ecosystem, with demonstrated ability to implement complex methods from recent papers and run large-scale experiments.
Hands-on experience with large language models, whether fine-tuning, inference, prompting pipelines, or building LLM-powered applications.
Demonstrated research ability through publications at peer-reviewed venues, preprints, or equivalent research output.
Track record of applying research to practical problems: methods, prototypes, or tools adopted by other teams or used to inform real product decisions.
Solid software engineering fundamentals for research code: version control, testing, reproducibility, and code others can build on.
Excellent written and verbal communication skills, including the ability to explain complex results, and their limitations, to diverse audiences.
Comfort with ambiguity and new problem spaces; you thrive when building something that doesn't yet have a playbook.

Preferred Qualifications

First-author publications at top-tier venues (NeurIPS, ICML, ICLR, ACL, EMNLP), especially on evaluation methodology: papers about how to evaluate, not just papers that use evaluation to demonstrate model improvements.
Specific experience in one or more of: LLM-as-judge approaches (calibration, rubric design, bias mitigation); benchmark design and validation (IRT, contamination detection); human evaluation methodology (protocol design, quality control); experimentation and comparisons between system versions (A/B testing, uncertainty quantification); or agentic and multi-agent system evaluation, including failure diagnosis and automated red-teaming.
Familiarity with theoretical foundations of evaluation: measurement theory and validity frameworks, statistical learning theory (calibration, reliability, decision theory), causal inference, or preference aggregation.
Experience with the practical challenges of evaluating generative AI and agentic systems: non-deterministic outputs, token economics, multi-turn agent trajectories, and tool use.
Experience building methods or tools used directly by non-experts, such as developer-facing ML tooling or self-service analytics.
Familiarity with modern evaluation tools and frameworks (e.g., DeepEval, Ragas, TruLens, LangSmith, or agent tracing and observability tools).
Experience working with distributed computing or cloud-native tooling (e.g., Ray, Spark, Docker/Kubernetes) to run evaluations at scale.
Experience with cross-disciplinary work, such as collaboration with social scientists, psychometricians, or domain experts.
Experience in fast-moving, early-stage teams where you helped define technical direction from the ground up.