EIS QA Engineer

Job Title: EIS QA Engineer / AI Quality Analyst Job Description
The EIS QA Engineer / AI Quality Analyst plays a critical role in ensuring the performance, safety, and reliability of advanced AI and machine learning models. In this position, you sit at the forefront of the development lifecycle, designing and executing comprehensive evaluation strategies that uncover model weaknesses, potential biases, and critical edge cases. You do more than identify defects—you provide clear, actionable insights that drive model improvements and guide ongoing research and development efforts. This role requires strong analytical rigor, solid technical aptitude, and a deep curiosity about how AI models behave in real-world scenarios.
Responsibilities
- Design, develop, and maintain a comprehensive suite of test cases and evaluation benchmarks for AI/ML models, ensuring broad coverage of typical use cases and critical edge cases.
- Proactively identify potential model failure points, including edge cases, adversarial inputs, and sources of bias, and incorporate them into evaluation strategies.
- Conduct systematic error analysis to categorize model failures, identify underlying patterns, and assess their impact on performance and safety.
- Triage defects by prioritizing them based on severity, user impact, and business risk, and collaborate with development teams to drive timely resolution.
- Source, curate, and manage high-quality datasets for model evaluation and testing, ensuring relevance, diversity, and completeness.
- Perform data annotation and validation to maintain the integrity and accuracy of ground-truth data used in training and evaluation.
- Execute exploratory and unscripted testing to discover unexpected model behaviors and uncover hidden failure modes.
- Participate in adversarial and red-teaming exercises to intentionally challenge AI models and identify potential safety, security, and reliability vulnerabilities.
- Set up, maintain, and troubleshoot testing and demonstration environments to ensure a stable, reliable, and repeatable evaluation pipeline.
- Apply test automation tools and frameworks, such as Selenium, Playwright, Cypress, and REST-assured, to build and maintain automated test suites for web, mobile, and API-based applications.
- Perform comprehensive testing across different platforms, including mobile Android and iOS applications and web applications, to validate end-to-end user experiences.
- Use bug tracking systems, such as Jira, and test case management tools to document issues, track progress, and maintain clear traceability between requirements, test cases, and defects.
- Leverage scripting skills, particularly in Python, to automate test execution, manipulate data sets, and support custom evaluation workflows.
- Analyze and synthesize test results into clear, actionable reports tailored for both technical and non-technical stakeholders.
- Translate complex technical findings and model behaviors into concrete recommendations that improve model performance, safety, and user experience.
- Actively participate in post-hoc evaluation reviews to assess testing effectiveness and identify opportunities for process refinement.
- Contribute to the continuous improvement of testing methodologies, tools, and quality assurance processes across the AI/ML development lifecycle.
- Collaborate closely with cross-functional teams in AI domains, sharing insights and aligning on quality goals for generative AI and frontline solutions.
Essential Skills
- Proven experience in a quality assurance, software testing, or data analysis role, preferably within the AI or machine learning domain.
- Deep understanding of the machine learning lifecycle, including data preparation, training, evaluation, deployment, and monitoring.
- Knowledge of common failure modes of AI models, such as bias, hallucinations, robustness issues, and performance degradation in edge cases.
- Hands-on experience with data annotation and data validation, ensuring high-quality ground-truth data for model training and testing.
- Experience managing large datasets for evaluation and experimentation, including organizing, versioning, and tracking data changes.
- Meticulous attention to detail and a methodical, structured approach to problem-solving and test design.
- Strong analytical skills with the ability to identify patterns in data, interpret model outputs, and draw meaningful conclusions.
- Expertise with industry-standard test automation tools and libraries, including Selenium, Playwright, Cypress, and REST-assured.
- Experience conducting comprehensive testing of mobile applications on Android and iOS platforms, as well as web applications.
- Familiarity with bug tracking systems such as Jira and test case management tools for organizing test suites and defect workflows.
- Scripting skills in Python for building test automation scripts, processing datasets, and integrating evaluation tools.
- Excellent written and verbal communication skills, with the ability to clearly document bugs, articulate complex technical issues, and present findings to diverse audiences.
Additional Skills & Qualifications
- Preferred experience in testing AI systems, including evaluating agentic responses, assessing model performance metrics, and verifying data integrity.
- Familiarity with computer vision or other specialized AI domains relevant to generative AI and frontline solutions.
- Experience working with AI/ML evaluation frameworks or tools to measure accuracy, robustness, fairness, and safety.
- Background in designing adversarial and red-teaming tests to probe AI model vulnerabilities and resilience.
- Exposure to RESTful APIs and service-oriented architectures in the context of testing AI-enabled applications.
- Ability to collaborate effectively with data scientists, machine learning engineers, and product teams in a fast-paced environment.
- Strong curiosity about emerging AI technologies and a passion for improving the safety and reliability of AI systems.
Work Environment
This role operates within a Gen AI Frontline Solutions team focused on building and validating cutting-edge AI capabilities. You work closely with AI/ML engineers, data scientists, and product stakeholders in a collaborative, technically driven environment. The position is based onsite in Lincolnshire, IL, providing direct access to test environments, demonstration setups, and cross-functional partners. You will regularly use modern test automation tools such as Selenium, Playwright, Cypress, and REST-assured, along with scripting in Python and standard QA platforms like Jira and test case management systems. The environment emphasizes rigorous evaluation, continuous improvement, and open communication, offering opportunities to influence the quality standards and safety practices of next-generation AI solutions.
Job Type & Location
This is a Contract position based out of Lincolnshire, IL.
Pay and Benefits
The pay range for this position is $45.00 - $55.00/hr. Individual compensation offered for this position within this range will depend on many factors, including qualifications, skills, relevant experience, job knowledge, geographic location, internal equity, and other pertinent job-related factors. Eligibility requirements apply to some benefits and may depend on your job
classification and length of employment. Benefits are subject to change and may be
subject to specific elections, plan, or program terms. If eligible, the benefits
available for this temporary role may include the following:

• Medical, dental & vision
• Critical Illness, Accident, and Hospital
• 401(k) Retirement Plan – Pre-tax and Roth post-tax contributions available
• Life Insurance (Voluntary Life & AD&D for the employee and dependents)
• Short and long-term disability
• Health Spending Account (HSA)
• Transportation benefits
• Employee Assistance Program
• Time Off/Leave (PTO, Vacation or Sick Leave)
Workplace Type
This is a fully onsite position in Lincolnshire,IL.
Application Deadline
This position is anticipated to close on Sep 28, 2026. About Actalent

Actalent is a global leader in engineering and sciences services and talent solutions. We help visionary companies advance their engineering and science initiatives through access to specialized experts who drive scale, innovation and speed to market. With a network of almost 20,000 consultants and 5,000 clients across the U.S., Canada, Asia and Europe, Actalent serves many of the Fortune 500. We are proud to be an Engineering News-Record (ENR) Top 500 Design Firm for our engineering design services and a ClearlyRated Best of Staffing® winner for both client and talent service.

The company is an equal opportunity employer and will consider all applications without
regard to race, sex, age, color, religion, national origin, veteran status, disability,
sexual orientation, gender identity, genetic information or any characteristic protected
by law.

If you would like to request a reasonable accommodation, such as the modification or
adjustment of the job application process or interviewing process due to a disability,
please email [email protected] for other accommodation options.

San Francisco Fair Chance Ordinance:
Pursuant to the San Francisco Fair Chance Ordinance, for all positions located in the
city and county of San Francisco, we will consider for employment qualified applicants
with arrest and conviction records.

Massachusetts Lie Detector:
It is unlawful in Massachusetts to require or administer a lie detector test as a
condition of employment or continued employment. An employer who violates this law
shall be subject to criminal penalties and civil liability.

Use of Artificial Intelligence (AI):
We may use Artificial Intelligence (AI) to support parts of our hiring process,
including sourcing, screening, and evaluating candidates. AI helps assess applications
and qualifications, but final decisions are made by our hiring team. By applying, you
acknowledge and agree that your application may be reviewed using AI tools.