Hiring QA Gen AI Automation Engineer for Jersey City, NJ

Hiring QA Gen AI Automation Engineer for Jersey City, NJ

Role: QA Gen AI Automation Engineer

Location: Jersey City, NJ

Duration: Long Term

Key Responsibilities

Automation Testing

Develop and maintain automated regression tests using tools like Selenium WebDriver/Playwright and pytest.

Conduct API testing using tools such as Postman.

Generative AI Testing

Design and execute test cases for Generative AI models and applications.

Evaluate the performance and quality of Generative AI outputs, ensuring they meet functional requirements and quality standards.

Validate AI systems such as Chatbots, RAG applications, and LLM-based workflows.

Perform LLM evaluation using frameworks such as DeepEval and RAGAS.

Programming & Scripting

Write and maintain automated test scripts in Python.

Work with XML, JSON, and other scripting languages to create efficient testing environments for AI systems.

Testing Tools & Frameworks

Utilize JMeter for performance testing and load testing.

Implement and manage Continuous Integration (CI) pipelines using tools such as Jenkins and Bamboo.

Database & SQL

Hands on experience with Panda Data frames

Execute SQL queries for database testing (inserts, updates, joins, etc.).

Test non-GUI applications (e.g., SQL, flat files, XML).

Work with Vector Databases such as OpenSearch or similar vector DB technologies.

Agile & DevOps

Collaborate within an Agile (Scrum) framework for efficient testing and development.

Utilize Atlassian JIRA and Confluence for project tracking and documentation.

Product Evaluation & Troubleshooting

Identify and troubleshoot product issues, ensuring quality and stability.

Evaluate product performance and provide feedback to development teams.

Web Application Testing

Conduct testing for web applications and web services (REST, SOAP).

Required Skills & Experience

Programming

Proficient in Python for writing test scripts and automating processes.

Automation Testing

Experience with Selenium WebDriver/Playwright and pytest for automation testing.

Hands-on experience in API testing using tools like Postman.

Generative AI & AI Testing

Understanding of AI systems and AI Testing concepts.

Knowledge of Large Language Models (LLMs), RAG, and Chatbot AI systems.

Experience designing and testing Generative AI systems and applications.

Knowledge of LLM evaluation techniques and frameworks such as DeepEval and RAGAS.

Ability to assess the output quality, coherence, and accuracy of Generative AI models.

Database & Search Technologies

Strong understanding of SQL for testing databases (including joins, inserts, and updates).

Understanding of Vector Databases such as OpenSearch or similar vector DB technologies.

Testing Tools

Experience with JMeter for performance and load testing.

Familiarity with CI/CD tools like Jenkins or Bamboo.

Agile Methodology

Experience working in an Agile (Scrum) environment.

Familiar with JIRA and Confluence for issue tracking and project management.

Web & API Testing

Experience testing web applications and web services.

Product Performance

Capable of evaluating and troubleshooting performance-related issues.

Nice to Have

Agentic AI & Observability

Understanding of Agents, MCP, and Agentic AI systems.

Familiarity with observability tools such as Langfuse.

Cloud Experience

Familiarity with AWS services such as AWS Bedrock for cloud-based AI environments.

Development Experience

A background in software development is a plus.

Industry Knowledge

Understanding of the Property and Casualty (P&C) insurance industry is beneficial.