Queue Manager / Annotation Lead with Trust & Safety Background (AI Data Ops)

Queue Manager / Annotation Lead with Trust & Safety Background (AI Data Ops)

 

Start date: Immediate / TBC

End Date: TBC

Location: Remote

Type of contract: Freelance

Languages required: English

Hourly Rate: Include the rate you are comfortable with on your CV.

 

About Prolific

Prolific is not just another player in the AI space. We are building the biggest pool of quality human data in the world. Over 35,000 AI developers, researchers, and organizations use Prolific to gather data from paid study participants with a wide variety of experiences, knowledge, and skills.

 

The Role

We are seeking a Queue Manager / Annotation Lead to join a high-priority initiative in partnership with a leading AI company. You will own day-to-day execution for one or more annotation queues, keeping work flowing, quality on target, and stakeholders unblocked - balancing throughput, SLAs, and accuracy in a live human-data pipeline.

 

In short, the role involves owning day-to-day execution of one or more annotation queues - managing intake, prioritization, and SLAs, running QA and calibration to keep quality on target, training and supporting annotators, and tracking key metrics like throughput and rework rates.

 

For this particular opening, we're specifically looking for candidates with a trust & safety / integrity background - someone who can tell when responses are shallow, low-effort, or phoned-in, and who can keep queues healthy on quality, not just catch bad actors. Think signal-quality analyst rather than a general queue lead.

 

What you'll be doing

Queue Operations & Task Management

• Manage work queue: intake, triage, prioritization, task assignment, and SLA delivery
• Monitor task queue and backlog control
• Identify and troubleshoot bottlenecks (unclear policy, tooling issues, supply gaps)

Quality Assurance & Performance Management

• Perform QA: make corrections, provide annotator feedback, flag systematic confusion to Client Delivery Lead
• Monitor quality at project and individual annotator levels
• Drive guideline adherence, run calibration sessions, reduce rework
• Manage annotator performance: unassign underperformers, identify high-performers

Training & Support

• Train new annotators on project guidelines
• Conduct live or recorded webinars as participants join a project
• Host "office hours” for participants
• Oversee dedicated Slack channels: answer questions, provide support, escalate complex issues to Client Delivery Lead

Metrics & Continuous Improvement

• Track key metrics: throughput, disagreement rates, defect/rework rates
• Provide feedback for process improvements

 

What you'll bring

• 3+ years in high-throughput ops (data labeling, trust & safety, BPO, content ops, CX ops, or similar)

Trust & safety / integrity background required

• Able to identify shallow, low-effort, or phoned-in responses.
• Proven queue management under ambiguity (prioritization + execution)
• Strong quality instincts; can spot systematic errors and close the loop with training/QA
• Comfortable with metrics and tooling (spreadsheets required; SQL/BI a plus)

 

Bonus

• Experience with ML data / LLM eval workflows (ranking, rubric-based judging, gold sets, calibration)
• Distributed team coordination across time zones; vendor/contract workforce experience

 

Success looks like

Predictable SLA delivery, a stable backlog, and improving quality with declining rework.

Why Prolific is a great platform to join as an expert participant

Joining our platform as a participant will give you the chance to influence the AI models of the future. Once you pass our assessment, you can join Prolific in just 15 minutes and start enjoying competitive pay rates, flexible hours, and the ability to work from home. 

We've built a unique platform that connects researchers and companies with a global pool of participants, enabling the collection of high-quality, ethically sourced human behavioral data and feedback. This data is the cornerstone of developing more accurate, nuanced, and aligned AI systems.

We believe that the next leap in AI capabilities won't come solely from scaling existing models, but from integrating diverse human perspectives and behaviors into AI development. By providing this crucial human data infrastructure, Prolific is positioning itself at the forefront of the next wave of AI innovation – one that reflects the breadth and the best of humanity.

Links to more information on Prolific

Website 

Youtube 

Privacy Statement

By submitting your application, you agree that Prolific may collect your personal data for recruiting and global organization planning. Prolific's Candidate Privacy Notice explains what personal information Prolific may process, where Prolific may process your personal information, its purposes for processing your personal information, and the rights you can exercise over Prolific's use of your personal information.