AI Reliability & Test Automation Engineer

About the role

We are building real-time conversational AI systems powered by LLMs, ASR, and TTS. Our products evolve quickly, with new services, workflows, models, and customer-facing features constantly being developed. We are looking for an AI Reliability & Test Automation Engineer to work directly within the LLM AI team to build our automated test capability, helping our systems ship safely and confidently.

This is an embedded engineering role focused on the craft of software testing in complex AI systems. You will design and maintain automated test suites, regression checks, mocks, CI/CD test gates, and release validation processes across LLM-powered workflows, APIs, RAG systems, and conversational journeys.

You will work closely with Machine Learning Engineers, Data Scientists, Backend Engineers, and external testing teams to ensure new features are testable from the start and changes can be evaluated quickly across the wider platform.

What you’ll do

• Build and maintain automated unit, integration, API, and end-to-end test suites for LLM-powered services.
• Create regression test coverage for conversational workflows, RAG pipelines, prompt changes, model updates, and service-level behaviour.
• Develop test harnesses, mocks, fixtures, and tools that make it easier for engineers to validate new features and services.
• Integrate automated tests into CI/CD pipelines to provide fast, useful feedback during development and release.
• Work with MLEs, Data Scientists, and backend engineers to identify failure modes, edge cases, and high-risk product behaviours.
• Collaborate with external testing teams to connect manual/exploratory validation with automated engineering checks.
• Help improve testability across Python services by supporting better interfaces, dependency isolation, and observability.
• Support release confidence by creating clear automated checks for critical user journeys and system behaviours.
• Design load and stress testing frameworks for high-concurrency conversational AI systems.
• Use production telemetry and failure analysis to continuously evolve automated test coverage.

What we're looking for

• Experience in software test automation or SDET environments
• Experience building automated API or backend testing frameworks
• Strong Python, JavaScript, or Typescript programming skills
• Experience integrating automated tests into CI/CD pipelines
• Ability to think critically about system behaviour, edge cases, and failure modes
• Strong debugging and problem-solving skills

Nice to have

• Understanding of modern AI systems, LLMs, or conversational AI products
• Experience testing distributed systems, APIs, or production platforms
• Experience in fast-paced, high scale SaaS environments and building tests for rapid development cycles
• Understanding of how to test distributed, asynchronous, or non-deterministic systems
• Comfortable working collaboratively, cross-functionally with research and engineering teams

Backgrounds that fit well

• SDET or QA Automation Engineers with strong Python and backend testing experience
• Backend Software Engineers who have a strong preference for building internal tooling, CI/CD, and testing frameworks.
• Automation engineers from fast-paced, high-scale SaaS environments who know how to build test automation that keeps up with rapid development cycles.