About this role
AI Evaluation & Quality EngineerWeQuel is looking for an AI Evaluation & Quality Engineer for an assignment with one of our clients. You will make sure agentic workflows are measured honestly before and after they reach production – so the organisation knows what works, what does not, and why.About the assignmentWorkload: 100%Location: HybridLanguage: EnglishYour responsibilitiesDesign evaluation strategies and quality metrics for LLM agents (accuracy, task success, safety, cost, latency)Build and maintain evaluation datasets, test cases and automated regression testsSet up LLM-as-judge and human review processes where neededRun red-teaming and robustness tests, including prompt injection scenariosMonitor agent quality in production and feed findings back to development teamsReport results clearly to product owners and stakeholdersRequirementsAt least 3 years of experience in test automation, QA engineering or data scienceHands-on experience evaluating LLM or ML solutionsStrong Python skillsUnderstanding of statistics and experiment designFluent English, written and spokenMeritoriousExperience with evaluation and observability tools (e.g. Langfuse, LangSmith, Ragas, DeepEval, promptfoo)Experience with CI/CD pipelines for ML or LLM applicationsKnowledge of AI safety and the EU AI ActBackground in industrial or manufacturing environmentsSwedishWho you areYou are analytical, sceptical in a constructive way and thorough. You enjoy finding where things break, explain results in plain language, and help teams make better decisions based on evidence.Applications are reviewed continuously – apply today!