SDET 3

The Role

We are seeking a Senior SDET to take ownership of quality across our customer-facing products, including web dashboards, APIs, notifications, and GenAI solutions such as chatbots, voice IVR, and AI agents. In this role, you will define testing methodologies for AI agents, develop automated evaluations, and establish quality criteria for releases.

You will be responsible for the test strategy in your area and continually build and enhance UI, API, and end-to-end automation and regression coverage. This is a hands-on engineering position working closely with GenAI engineering, CX engineering, and product teams.
 
Key Responsibilities

● Own the test strategy and release quality criteria for your product area.
● Build and maintain UI automation (using Selenium or Playwright), API automation, and end-to-end regression test suites.
● Design evaluation strategies for AI agents, utilizing LLM-as-judge, rubrics, and other assessment techniques.
● Create evaluation datasets and automated evaluation suites in Python, integrated with CI/CD pipelines.
● Test AI agents for accuracy, consistency, tool usage, safety, and data handling.
● Debug agent failures using LangSmith or similar AI observability tools.
● Analyze quality across releases to identify regressions.
● Review product requirements to pinpoint gaps in requirements and test coverage.
● Review test code, define testing standards, and mentor other SDETs.
● Communicate findings and recommendations with engineering, product, and leadership teams.

Requirements

● 6–9 years of software testing experience, including hands-on SDET roles.
● Proficiency in both Python and Java.
● Experience owning test strategies and enhancing automation and release quality.
● Experience with UI automation tools such as Selenium or Playwright.
● Experience developing API automation and end-to-end frameworks within CI/CD.
● Ability to analyze product requirements and spot testing gaps.
● Previous experience testing chatbots, IVRs, and GenAI agents.
● Experience designing evaluations, rubrics, and LLM-as-judge assessments.
● Working knowledge of LangChain, LangGraph, LangSmith, or similar tools.
● Experience creating evaluation frameworks or tools for AI agents.
● Strong analytical and communication skills.
 
Tech You Will Work With

Python, Java, Selenium, Playwright, REST API testing, Postman, TestNG, PyTest, LangChain, LangGraph, LangSmith, LLM evaluation, LLM-as-judge, RAG testing, AI agent testing, end-to-end testing, regression testing, CI/CD, GitHub Actions, Jenkins, AWS.