About this role
You will own quality for data pipelines, microservices, APIs, GUIs, and LLM/ML integrations running on Databricks and Microsoft Azure. The role focuses on test strategy, test case design and execution, automation, ticket management, and agentic QA for LLM workflows. Minimum experience: 5 years
Key Responsibilities
Test Strategy and Planning
Define and maintain QA strategy for data, API, GUI, microservices, and LLM/ML features; prioritize test coverage based on risk and business impact.
Test Case Design and Execution
Create clear, reusable test cases for functional, integration, regression, performance, and pricing scenarios; execute manual and automated tests and maintain test artifacts.
Data and Database Testing
Validate ETL/ELT pipelines on Databricks and Delta Lake; write and run T‑SQL validation queries; verify data lineage, completeness, and correctness.
API Testing
Design and run contract, integration, and negative tests for RESTful APIs; use Postman, OpenAPI/Swagger, and automated API test suites.
GUI Testing
Create and execute UI test plans for web applications; perform exploratory testing and maintain automated UI tests using Selenium, Playwright, or equivalent.
Microservices Validation
Test service contracts, message flows, resilience, and backward compatibility across containerized microservices; validate observability and error handling.
LLM and ML QA
Design evaluation metrics and test harnesses for ML models and LLM integrations; validate inference correctness, hallucination rates, latency, and cost tradeoffs.
Prompt Engineering and Agentic QA
Create minimalistic, cost‑aware prompts and prompt templates; implement agentic QA processes for autonomous agents including safety checks, guardrails, and evaluation pipelines.
Pricing and Business Logic Testing
Validate pricing engines, discount rules, billing flows, and edge cases; create scenario matrices and automated checks for pricing correctness.
Automation and CI/CD
Integrate tests into CI/CD pipelines; author automated test suites, test runners, and validation scripts; collaborate with SRE/DevOps on test environments.
Ticketing and Incident Management
Raise, triage, and manage tickets in Azure DevOps or Jira; document reproduction steps, root cause analysis, and remediation.
Documentation and Collaboration
Produce test plans, runbooks, release checklists, and postmortems; mentor junior QA engineers and participate in cross‑functional reviews.
Required Skills and Experience
Experience: Minimum 5 years in QA, software testing, or related roles.
Databricks and Data Testing: Hands‑on experience validating Databricks pipelines, Delta Lake, and Spark outputs.
Microsoft SQL Server: Medium proficiency in T‑SQL for validation queries and database testing.
Azure: Practical experience with Azure Data Factory, ADLS/Blob Storage, Azure DevOps, and Azure Functions.
API Testing: Strong experience with REST APIs, Postman, OpenAPI/Swagger, and automated API testing.
GUI Testing: Experience with Selenium, Playwright, or equivalent UI automation tools and exploratory testing.
Microservices: Familiarity testing containerized services, message queues, and service orchestration.
LLM and ML Fundamentals: Understanding of ML lifecycle, model evaluation, prompt engineering basics, and LLM failure modes.
Prompt Engineering: Ability to craft minimalistic prompts and maintain prompt libraries for reproducible tests.
Agentic QA Awareness: Knowledge of agentic QA concepts, safety checks, and evaluation metrics for autonomous agents.
Testing Tools and Automation: Experience with test frameworks, CI/CD integration, and test data management.
Ticketing Tools: Experience using Azure DevOps, Jira, or similar.
Soft Skills: Strong analytical thinking, clear written communication, and cross‑team collaboration.
Nice to Have
Certifications: ISTQB, Azure Data Engineer, or Databricks certifications.
Programming: Python for test automation and data validation; familiarity with Scala or Java.
Observability: Experience with Application Insights, Datadog, Prometheus, or Grafana.
Performance Testing: Experience with JMeter, k6, or similar tools.
LLM Ops: Experience with prompt versioning, cost monitoring, and model evaluation frameworks.
Infrastructure as Code: Familiarity with Terraform or ARM templates for test environment provisioning.
Interview and Evaluation Guide
Technical screen: SQL validation problem and API testing scenarios.
Practical exercise: Create test cases and automated checks for a sample Databricks ETL and an API that returns pricing.
LLM prompt exercise: Provide minimal prompts for a given LLM task and design evaluation metrics for correctness and safety.
System design: Design an agentic QA pipeline for autonomous agents including test harness, safety gates, and monitoring.
Behavioral: Examples of ticket triage, incident postmortems, and cross‑team collaboration.
Success Criteria at 6 Months
Test Coverage: Comprehensive, automated test suites for data pipelines, APIs, and core GUI flows.
LLM Readiness: Reproducible prompt templates, evaluation metrics, and agentic QA checks integrated into CI.
Reliability: Reduced production incidents and faster mean time to resolution through clear runbooks and test automation.
Pricing Confidence: End‑to‑end validated pricing flows with scenario tests and regression guards.
Collaboration: Trusted QA partner to engineering and product teams with documented processes and mentoring outcomes.