About this role
About the Role
Own complex data and evaluation programs for AI research teams and internal partners at an early-stage AI data infrastructure startup. You will bring structure to ambiguous technical requests and build reliable processes that help improve LLM and agent evaluations.
What You'll Do
- Lead data and evaluation programs from scoping and requirements gathering through production, quality assurance, delivery, and retrospectives.
- Turn evolving technical requests into clear specifications with milestones, owners, dependencies, and acceptance criteria.
- Assess and improve data quality using quantitative metrics and qualitative review, including task coverage, difficulty, and reward distributions.
- Manage external vendors across the delivery lifecycle and scale delivery pipelines.
- Improve workflows, documentation, automation, and cross-functional handoffs for data production.
- Work with research, platform, and marketplace teams to turn operational learnings into product improvements.
What We're Looking For
- At least 3 years of experience owning complex technical programs in AI/ML, data operations, support engineering, applied research, forward-deployed engineering, or a similar cross-functional area.
- Experience as a Technical Program Manager or Strategic Program Lead at a startup, including supporting AI research teams, technical enterprise customers, or other partners with urgent, changing needs.
- Strong quantitative and qualitative judgment, with the ability to assess both aggregate dataset metrics and individual examples.
- Experience managing vendors and improving data quality processes, workflows, and delivery pipelines.
- Familiarity with RLHF, reinforcement-learning environments, agent evaluations, human-in-the-loop data pipelines, annotation, or expert data collection is valuable.
Compensation & Benefits
Visa sponsorship and relocation support are available for strong full-time candidates.
Location
On-site in San Francisco, California. Remote candidates may be considered if they can maintain substantial time zone overlap with San Francisco or Singapore.