About this role
Accountabilities
- Design, develop, and maintain automated test suites and frameworks for distributed, cloud-based systems.
- Maintain and extend agentic test generation and analysis capabilities within core testing frameworks.
- Build AI-powered workflows and agent skills that reduce manual testing effort, including test data setup, failure triage, and release validation.
- Maintain evaluation suites, golden datasets, and scoring pipelines to assess AI agents across correctness, tool usage, execution trajectories, faithfulness, safety, latency, and cost.
- Implement CI evaluation gates to measure changes to prompts, models, tools, and orchestration before release, incorporating production failures into future evaluations.
- Identify and mitigate unreliable AI outputs, including hallucinated assertions, flaky or redundant tests, inaccurate failure analysis, and evaluation bias.
- Design and execute performance, load, and stress tests to identify bottlenecks and validate the scalability of distributed services.
- Test integrations across platform systems, services, data, applications, and customer-facing features.
- Review product, system, and agent designs to assess testability, evaluation coverage, and opportunities for improvement.
- Share technical knowledge and contribute to team-wide quality and evaluation standards.
Requirements:
- Degree in Computer Science or equivalent practical experience, with a solid understanding of object-oriented programming, data structures, algorithms, and distributed systems.
- At least 5 years of experience in software quality, including 3+ years of hands-on SDET or test automation development.
- Strong programming skills in at least one of Java, Node.js, or Python; Python is preferred for AI evaluation and agent-related work.
- Demonstrated experience designing, developing, and maintaining test frameworks integrated into CI/CD pipelines such as Jenkins or GitLab CI/CD.
- Experience testing web services and APIs, including REST, gRPC, and JSON.
- Hands-on performance and load testing experience with tools such as Locust or k6, including workload design and interpretation of latency percentiles, throughput, and error rates.
- A skeptical, verification-first approach to AI-generated output, with the ability to distinguish when deterministic assertions are appropriate and when probabilistic evaluation is needed.
- Experience applying AI technologies to improve decision-making, streamline workflows, increase efficiency, and deliver business outcomes.
- Strong analytical, problem-solving, and collaboration skills, with the ability to communicate technical findings and contribute to shared quality standards.
Nice to have:
- End-to-end eval-driven development experience, including release gating, trajectory and tool-use evaluations, and calibrating LLM-as-judge systems against human review.
- Experience with prompt engineering and designing agent context or tools.
- Experience integrating performance tests into CI/CD pipelines and using observability tools such as Grafana, Prometheus, or CloudWatch.
- Background in cybersecurity, threat intelligence, malware analysis, or sandboxing and detonation environments.
- Familiarity with technologies such as MongoDB, MinIO, RabbitMQ, Kafka, AWS services (EC2, S3, DynamoDB), Docker, Nomad, Terraform, Unix/Linux, Git, Jenkins, GitLab CI/CD, Jira, and Bitbucket.
- Experience with LLM APIs, agent frameworks, and LLM evaluation tooling.
Benefits:
- Annual base salary ranging from CA$120,000 to CA$170,000 , plus variable or incentive compensation, equity, and benefits.
- Market-leading compensation and equity award opportunities.
- Comprehensive physical and mental wellness programs.
- Competitive vacation allowance and holidays.
- Paid parental and adoption leave.
- Professional development opportunities available to employees at every level.
- Employee networks, local community groups, and volunteer opportunities.
- Remote work opportunities across eligible Canadian location
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1