About this role
Job Purpose
As an AI Research Engineer in the TrainAI research team, you will bridge applied research and engineering execution. Your core focus will be designing, building, and scaling continuous LLM evaluation pipelines, experiment orchestration tools, and automated pre-annotation workflows. Working directly with the Head of Innovation and data engineering teams, you will turn cutting-edge multilingual evaluation methodologies into reliable, automated research and production tools.
Job Overview
Key Responsibilities
- Evaluation Pipelines & Benchmarking: Design, build, and maintain automated execution pipelines for continuous LLM benchmarking (such as TrainAI Bench) across diverse languages, tasks, and models.
- Pre-Annotation & QA Automation: Develop and roll out LLM-assisted pre-annotation and quality assurance pipelines to automate production data annotation, lowering costs and improving data consistency.
- LLM-as-a-Judge Frameworks: Implement and refine automated evaluation methodologies, including LLM-as-a-judge pipelines, custom evaluation metrics, and scoring algorithms.
- Experiment Orchestration & Research Support: Build experiment infrastructure, datasets, and execution pipelines to support flagship multilingual LLM research studies and thought leadership reports.
- Research-to-Engineering Bridge: Translate applied NLP research into scalable prototypes and work alongside the core data engineering team to integrate research tools into production operations.
- Thought Leadership & Publication: Collaborate with the research team on writing up research findings, benchmark reports, and thought leadership pieces for internal and public release.
- Emerging AI Methods Exploration: Prototype novel capabilities in emerging areas such as agentic AI evaluation, synthetic data quality certification, and domain-specific LLM assessment.
- Documentation & Best Practices: Maintain comprehensive documentation, code bases, and technical guides for TrainAI evaluation frameworks and tools.
Skills & Experience
- Education & Experience: Master’s degree (MSc or equivalent) in Computer Science, Artificial Intelligence, Computational Linguistics, or a related technical field, combined with 3–5+ years of hands-on experience in NLP or ML engineering.
- Programming & ML Stack: Advanced proficiency in Python and deep learning/NLP frameworks (PyTorch or TensorFlow, Hugging Face transformers).
- LLM Tooling & Frameworks: Proven experience with LLM orchestration frameworks (e.g., LangChain, LlamaIndex) and automated evaluation techniques (e.g., LLM-as-a-judge, benchmarking harnesses).
- Data & Analytics Libraries: Expertise in data manipulation, structured evaluation data processing, and analysis using pandas, NumPy, or related tools.
- Cloud Infrastructure: Practical experience deploying and managing ML workflows on cloud platforms (Azure, AWS, or Vertex AI).
- System Design: Experience building scalable data or experiment orchestration pipelines.
- Familiarity with multilingual NLP, machine translation, or computational linguistics (e.g., cross-linguistic variations, syntax, morphology).
- Knowledge of localization workflows, annotation platforms, or human-in-the-loop data pipelines.
- Track record of contributing to open-source NLP projects, evaluation frameworks, or published technical benchmarks.
Life at RWS
Life at RWS - If you like the idea of working with smart people who are passionate about growing the value of ideas, data and content by making sure organizations are understood, then you’ll love life at RWS.
Our purpose is to unlock global understanding. This means our work fundamentally recognizes the value of every language and culture. So, we celebrate difference, we are inclusive and believe that diversity makes us strong. We want every employee to grow as an individual and excel in their career.
In return, we expect all our people to live by the values that unite us: to partner , putting clients fist and winning together , to pioneer , innovating fearlessly and leading with vision and courage, to progress , aiming high and growing through actions and to deliver , owning the outcome and building trust with our colleagues and clients.
RWS embraces DEI and promotes equal opportunity, we are an Equal Opportunity Employer and prohibit discrimination and harassment of any kind. RWS is committed to the principle of equal employment opportunity for all employees and to providing employees with a work environment free of discrimination and harassment. All employment decisions at RWS are based on business needs, job requirements and individual qualifications, without regard to race, religion, nationality, ethnicity, sex, age, disability, or sexual orientation. RWS will not tolerate discrimination based on any of these characteristics.
RWS Values
Get the 3Ps right – Partner, Pioneer, Progress – and we´ll Deliver together as RWS.
Recruitment Agencies: RWS Holdings PLC does not accept agency resumes. Please do not forward any unsolicited resumes to any RWS employees. Any unsolicited resume received will be treated as the property of RWS and Terms & Conditions associated with the use of such resume will be considered null and void.