Back to jobs
New

QA Analyst (AI Systems)

Latin America

About CodeRoad

CodeRoad provides end-to-end software development services, helping businesses scale with ideal infrastructure solutions. From staff augmentation to dedicated IT teams and general software engineering, our nearshore technology services empower businesses to thrive in an ever-evolving digital landscape.

About the Role

We’re looking for a QA Analyst specialized in modern AI systems. You’ll be responsible for validating not only APIs and performance, but also the behavior and reliability of AI-driven outputs. This role goes beyond traditional QA — you’ll test intelligent systems, not just deterministic ones.

Key Responsibilities

  • Design and execute test strategies for AI-powered APIs and workflows
  • Validate agent inputs/outputs and ensure consistency and reliability
  • Perform API testing, exploratory testing, and automated testing
  • Run performance, load, and bulk testing scenarios
  • Collaborate with engineers to identify edge cases in AI behavior
  • Contribute to testing frameworks for LLM-based systems

Requirements

  • 3+ years of experience in QA (Automation + Manual).
  • Strong API Testing: Expert level with Postman, Insomnia, or RestAssured.
  • Automation Specialist: Proficient in Playwright or Selenium for end-to-end flows.
  • Performance & Scale: Experience with Load/Bulk testing using Locust or JMeter.
  • AI/LLM Validation: Familiarity with testing Agentic workflows, validating AI payloads, and using tools like LangSmith.
  • Observability: Exposure to OpenTelemetry for tracing and system monitoring.
  • Analytical Rigor: Ability to design test cases for non-linear, agentic AI tasks.
  • Advanced English: Fluent for documentation and global collaboration.

Nice to Have

  • Experience testing AI/ML systems
  • Familiarity with LangSmith or similar tools
  • Experience with MLFlow or model evaluation frameworks

What You’ll Love

  • 100% Remote
  • Holidays off
  • Paid Time Off
  • Health insurance assistance program
  • Competitive USD compensation
  • Strong team culture and collaborative environment
  • Ongoing training and growth opportunities

Apply for this job

*

indicates a required field

Resume/CV*

Accepted file types: pdf, doc, docx, txt, rtf

Cover Letter

Accepted file types: pdf, doc, docx, txt, rtf

Select...

This role is 100% remote and LATAM-only.

Select...
Select...
Select...
Select...