Senior QA Automation Engineer

About the Role

We are seeking a highly skilled QA Automation Engineer to safeguard the quality, reliability, performance and safety of our AI-powered data products. In this role, you will architect intelligent, test automation frameworks, design rigorous evaluation systems for ML/LLM outputs and build quality infrastructure that scales alongside rapidly evolving AI capabilities.

You'll be a key quality gatekeeper for AI products used in production, ensuring models behave safely, consistently, and as intended — before, during, and after release.

Roles & Responsibilities

Test Automation & Framework Engineering

• Architect and build scalable, modular automation frameworks using Python, Pytest, Playwright, Selenium, or other custom-built tooling.
• Automate testing across UI, APIs, microservices, and AI models to enable fast, high-confidence releases.
• Build internal bots or agents that auto-generate test cases and run intelligent, AI-assisted exploratory testing.
• Champion best practices in test architecture, design patterns and maintainable, reusable test code.

AI/ML & LLM Evaluation

• Own end-to-end test strategy and execution for AI/ML models, GenAI, LLMs, and multi-agent systems, spanning functional, adversarial, safety, and security testing.
• Design evaluation frameworks to measure hallucination rate, bias, toxicity, and prompt-response consistency.
• Curate and maintain golden datasets; apply both deterministic rule-based checks and LLM-as-a judge techniques to evaluate output quality at scale.

Data Validation

• Validate data pipelines, training/inference data quality, feature stores, schema contracts to prevent data leakage or drift.
• Verify embedding quality, RAG retrieval relevance and the accuracy of agentic tool- and function calling.

CI/CD & Production Reliability

• Embed automated test suites into CI/CD pipelines, defining data-driven release gates based on AI quality metrics.
• Establish real-time production monitoring and dashboards to detect model drift, concept drift and anomalies.
• Continuously triage failures and tune performance to reduce pipeline execution time.

Stakeholder Collaboration

• Partner with Data Scientists, ML Engineers, and Product Managers to define measurable acceptance criteria.
• Translate testing and quality metrics into actionable outcomes for both technical and business stakeholders.

Who we are looking for

Work Qualifications

• At least 5 years of hands-on experience in QA, Test Automation, or SDET roles, including some experience in testing AI/ML, LLM or agentic systems.

Technical Experience

Programming & Frameworks

• Strong proficiency in Python with solid software engineering fundamentals
• Hands-on experience with Pytest, unittest, or equivalent test frameworks; comfortable writing maintainable, well-documented test code.
• Proficient in SQL for querying, validating and profiling test data.
• Comfortable working with structured/semi-structured data formats and large-scale datasets

Automation Expertise

• Proven experience with API and UI test automation
• Experience designing end-to-end, data-driven and keyword-driven automation frameworks.

Testing Proficiency

• Strong grasp of test design techniques
• Experience with risk-based testing and defining measurable quality/exit criteria

Functional Skills

• Excellent communication skills with the ability to translate complex technical concepts for both technical and non-technical stakeholders
• Comfortable working in cross-functional, fast-moving environments where priorities may evolve.