Senior AI QA Engineer for Sora Union

Talent is everywhere, opportunity is not. Sora Union changes that.

Sora Union is a professional services company specializing in brand design, engineering services. The Sora Union team is globally-distributed and made up of diverse and talented professionals impacted by or at risk of displacement due to conflict or climate change.

We’re looking for an experienced Senior AI QA Engineer to join us to help expand our team of diverse and experienced professionals.  

As a AI QA Engineer working on projects in partnership with our clients, you’ll work with products and companies both big and small, primarily responsible for ensuring they deliver the highest quality experience possible for their users. You’ll work with software developers, product managers, designers, and customer service teams to increase confidence in our clients’ releases, including testing applications, investigating issues, and reporting on results. 
You’ll be successful in this role if you enjoy working towards becoming one of the most knowledgeable individuals about a product and how it gets to consumers.  You should be an effective communicator, comfortable exchanging information in a clear and concise way with both technical and non-technical stakeholders.

You’ll be responsible for: 

• AI and LLM Testing: Design and execute tests to evaluate LLM-generated responses for accuracy, relevance, consistency, completeness, instruction adherence, and potential failure cases.

• AI Agent Development: Build, maintain, and improve cost-effective AI-powered testing agents and test harnesses to support software development and delivery.

• MCP Testing: Test Model Context Protocol (MCP) tools and integrations, validating tool execution, input/output handling, expected behavior, and error scenarios.

• AI-Driven Automation: Use Claude, ChatGPT, Playwright MCP, and related tools to streamline test creation, execution, analysis, and reporting.

• Test Strategy and Execution: Develop and maintain comprehensive test plans, test cases, regression suites, and manual testing checklists.

• Automation and Regression Testing: Design, execute, and maintain automated tests for web applications, APIs, and end-to-end workflows.

• Functional and Exploratory Testing: Investigate product behavior, identify edge cases, and uncover functional, usability, integration, and data-related issues.

• API and Data Validation: Test backend APIs, verify data integrity, work with SQL queries, and validate system integrations and data flows.

• Defect Management: Reproduce, document, triage, prioritize, and track defects, providing clear evidence and actionable information to development teams.

• Cross-Functional Collaboration: Partner with engineering, product, and other teams to clarify requirements, assess risks, and improve release confidence.

• Continuous Improvement: Identify opportunities to improve QA processes, expand test coverage, reduce repetitive manual work, and introduce practical AI-driven testing solutions.

• Product Expertise: Develop a deep understanding of the product, its users, and its underlying workflows to ensure testing reflects real-world usage

Ideally, you’ll have:

• A degree in computer science or mathematics, coupled with a minimum of 7 years of experience in a professional QA role.
• Hands-on experience with AI/LLM testing, including evaluating AI-generated responses, validating expected behavior, and identifying inconsistencies or failure scenarios.

• Practical experience using AI tools such as Claude and ChatGPT to support testing, test generation, debugging, or QA workflow automation.

• Experience with Playwright and MCP-based automation, including using Playwright MCP or similar approaches to automate and validate application workflows.

• Experience building or maintaining AI-powered testing agents, test harnesses, or automated evaluation workflows.

• Strong understanding of manual, exploratory, functional, smoke, sanity, regression, integration, and end-to-end testing.

• Experience testing web applications and backend APIs, including validating data flows and system integrations.

• Familiarity with test automation frameworks such as Playwright, Selenium, or Cypress.

• Experience with API testing tools such as Postman and a working knowledge of REST APIs and, ideally, GraphQL.

• Working knowledge of SQL and databases, including the ability to write and execute queries for data verification.

• Familiarity with defect tracking and test management tools such as Jira, TestRail, Azure DevOps, or GitHub.

• Understanding of Agile/Scrum methodologies and experience working in cross-functional software development teams.

• Strong analytical and problem-solving skills, with exceptional attention to detail and the ability to investigate complex issues.

• Ability to work independently, manage priorities, and adapt to evolving requirements and emerging technologies.

• Excellent written and verbal communication skills, with the ability to explain technical findings clearly to both technical and non-technical stakeholders.

• English - Upper-intermediate or higher

Sora Union is committed to creating and fostering a diverse team. We encourage people from underrepresented backgrounds and all walks of life to apply. We are committed to providing reasonable accommodations to all applicants throughout the application process.