About this role
Our AIML Data Operations team helps deliver high-quality human-annotated and human-evaluated data for AI and machine learning. The Quality Assurance team develops the methods, tools, and metrics that help us measure quality, catch issues early, and improve how data is produced and evaluated at scale.
We’re looking for a hands-on software engineer to build the tools and data systems behind that work. You’ll automate QA workflows, build pipelines that turn annotation and review events into reliable metrics, and create dashboards and alerts that help teams act on those metrics. You’ll also analyze quality data to understand where errors occur and whether changes to guidelines, tools, or workflows improve outcomes.
You’ll own projects from problem definition through deployment and ongoing operation. A project might start with a question such as “Where are annotation errors increasing?” or “Is our audit sample giving us enough coverage?” You’ll work with QA, Data Science, Engineering, and Operations to define the measurement, build the workflow or pipeline, validate the results, and put them in the hands of the people who can act on them.
Minimum Qualifications
Bachelor's degree in Computer Science, Data Science, Engineering, or related field.
Experience designing, building, and maintaining production software, internal tools, or data systems used by other teams.
Strong programming skills in Python and strong SQL skills for data transformation, validation, and analysis.
Experience building automated workflows or data pipelines that integrate multiple data sources or services.
Experience presenting data through dashboards, analytical applications, or reporting interfaces that support operational decisions.
Ability to investigate data discrepancies and translate ambiguous quality or operational questions into measurable outcomes.
Ability to work across technical and operational teams and explain findings and tradeoffs clearly.
Preferred Qualifications
Experience with data annotation, human evaluation, ML data pipelines, or quality engineering for data-intensive products.
Familiarity with statistical sampling, confidence intervals, inter-annotator agreement, or measurement design for subjective tasks.
Experience evaluating rule-based or model-assisted QA against human judgments, including precision, recall, and error analysis.
Experience building systems for global operations or workflows with multiple sites and review teams.