Job Title: LLM Annotator – Execution (Tier 2 & Tier 3)

Function: AI Operations / Data Annotation

Location: Philippines

Employment Type: Full-Time

Experience Level: Tier 2 / Tier 3

Experience Requirement: At least 1 year of LLM-related work experience preferred

Job Summary

The LLM Annotator – Execution (Tier 2 & Tier 3) is responsible for performing advanced annotation, evaluation, and quality assurance tasks that support the training, tuning, and validation of Large Language Models (LLMs). This role executes structured annotation workflows, applies detailed labeling guidelines, and ensures high-quality outputs that meet accuracy, consistency, and compliance standards.

Tier 2 and Tier 3 annotators are expected to demonstrate higher judgment, complexity handling, and quality ownership compared to entry-level roles.

Key Responsibilities

• Execute complex text annotation, labeling, and classification tasks following detailed project guidelines.
• Perform LLM output evaluation, including relevance, accuracy, safety, and instruction adherence.
• Handle edge cases, ambiguous inputs, and nuanced language scenarios with sound judgment.
• Apply consistent annotation decisions across large datasets with minimal supervision.
• Identify and flag quality issues, guideline gaps, or unclear instructions for escalation.
• Support quality assurance (QA) activities such as peer reviews, spot checks, and error analysis.
• Maintain productivity, accuracy, and quality metrics aligned with project standards.
• Document annotation decisions and rationales when required.
• Adhere to data privacy, confidentiality, and security requirements.

Tier-Specific Expectations

Tier 2

• Executes moderately complex annotation tasks independently.
• Demonstrates strong guideline comprehension and consistency.
• Meets productivity and quality benchmarks with minimal rework.
• Escalates unclear cases appropriately.

Tier 3

• Handles highly complex or sensitive annotation tasks.
• Applies advanced judgment to nuanced language, intent, and context.
• Provides informal guidance or calibration support to Tier 1 and Tier 2 annotators.
• Contributes to quality improvement by identifying recurring issues and suggesting refinements.

Qualifications

• Bachelor’s degree in Linguistics, Communications, Computer Science, Data Science, Psychology, or a related field (preferred).
• At least 1 year of LLM-related work experience is preferred, including annotation, evaluation, prompt review, or AI operations.
• Strong English comprehension, writing, and analytical skills.
• Ability to follow detailed guidelines and maintain high accuracy.
• Experience working with annotation tools, internal platforms, or AI evaluation systems (preferred).
• High attention to detail and ability to work on repetitive yet complex tasks.

Preferred Skills

• Prior experience with LLM annotation, model evaluation, RLHF, or content moderation.
• Familiarity with NLP concepts, prompt-response evaluation, or AI safety guidelines.
• Ability to work independently in a structured, metrics-driven environment.
• Experience supporting global AI or data operations teams.