QA Engineer(AI)

We're looking for a senior QA Engineer with deep AI and LLM testing expertise to design and execute test strategies for AI-powered systems and securitization platforms at a major Canadian bank. The ideal candidate brings hands-on experience with RAG pipelines, LangChain, and prompt engineering, combined with strong automation skills and the ability to lead quality initiatives across cross-functional teams.

What You'll Do

• Design and execute comprehensive test strategies for AI systems including prompt engineering, output evaluation, hallucination detection, and bias and safety testing
• Develop deep understanding of LLM behavior including tokenization, embeddings, attention mechanisms, and inference to anticipate failure modes
• Test AI systems integrated with RAG pipelines and knowledge bases, validating retrieval accuracy, embedding drift, and data quality
• Leverage LangChain and LangGraph frameworks to understand chain and graph construction, identify failure points, and write test harnesses
• Validate integration points using Model Context Protocols, testing tool availability and error handling
• Define and maintain automated test suites to accelerate release cycles and improve coverage
• Validate end-to-end securitization deal workflows including setup, structuring, processing, and distributions
• Ensure data integrity across upstream and downstream systems through reconciliation and reporting
• Coordinate regression testing for platform releases and infrastructure changes
• Lead defect triage sessions, prioritize issues based on business impact, and maintain documentation through to closure
• Define and monitor quality indicators including defect density, test coverage, and automation rates
• Mentor junior QA team members and foster a culture of quality across the team
• Provide production support during critical processing windows and coordinate root cause analysis

What You Bring

• 7+ years of quality assurance or quality engineering experience with at least 3 years in a lead or senior capacity
• Hands-on expertise in AI/LLM testing including prompt engineering, hallucination detection, bias evaluation, and safety testing
• Experience testing RAG pipelines, vector databases, and embedding-based systems
• Working knowledge of LangChain and/or LangGraph frameworks
• Familiarity with Model Context Protocols and agentic solution testing
• Strong automation experience with tools such as Selenium, Robot Framework, or Playwright
• Proficiency in Python and Java with demonstrated framework implementation
• Hands-on API automation and backend system validation experience
• Experience with CI/CD pipelines and DevOps practices including Jenkins and GitHub
• Proficiency in database query development, data validation, and reconciliation testing
• Strong domain knowledge in securitization, capital markets, or similar asset classes
• Eligibility for a mandatory background check

Nice to have

• Experience with red-team testing methodologies for AI systems
• Familiarity with sparse versus dense embedding testing and near-duplicate document edge cases
• Prior experience mentoring QA teams in an AI or ML testing context

Work setup

• Onsite in Montreal, QC, 4 days per week
• In-person interview required
• 6+ month contract
• English proficiency required; French an asset given the location