AI LABSRLHF. Model Evaluation. Agent Testing. Safety Review.

Great AI training data
starts with great people. Verified. Calibrated. Global.

Etech builds managed rater and evaluator teams for AI labs, model builders and companies deploying AI. Our teams in the United States, Jamaica and India rank model outputs, score responses against your rubric, test AI agents end to end and probe for safety issues. Every rater is verified and matched to the task. Every week, their work is audited, calibrated and reported, using the same quality discipline we have run in contact centers since 2003.

<5% monthly

attrition across Etech agent teams.
The same raters on your program, week after week.

QA audit on every rater, every week

3 countries. 7 sites.

United States, Jamaica and India

The AI Labs practice of Etech Global Services LLC provides human feedback and AI evaluation services to AI labs, model builders, conversational AI vendors and enterprises deploying AI agents. Etech provides managed rater and evaluator teams in the United States, Jamaica and India for RLHF preference ranking, model evaluation and regression testing, AI agent and voice testing, red teaming and policy review, customer service and regulated-industry review, and demographic and multilingual testing, with identity verification, gold-set qualification, weekly QA audits, calibration and inter-rater agreement reporting, supported by SOC 2 Type II attestation and ISO 27001:2022 alignment.

What AI teams are dealing with

Open crowds were built for volume.
Post-training needs judgment.

Models now improve on human preference, human evaluation and real-world testing. That work only helps if the people doing it are who they say they are, follow the rubric, and do the thinking themselves. A contact center solves this every day: hire to a profile, train to a rubric, audit every week, coach, and report.

Unvetted contributors Reported period

2023 to 2024

Spam and AI-written work have been reported in training programs.

Inc. reported that one open-platform program received gibberish, incorrect answers and GPT-written reasoning, and that contributors were not background checked. The platform disputed the reporting.

Inc., June 2025, as summarized by Futurism (1)

AI grading AI Reported removals

2026

Raters who use AI to do the rating are being removed from lab programs.

Contract reviewers working on an AI lab's projects were reportedly removed after using AI tools on tasks meant for human judgment. The staffing vendor involved confirmed that it removes contributors when such use is confirmed.

404 Media, September 2026 (2)

Accuracy gap Managed team accuracy

75 to 85%

Managed teams outscored the crowd on a classification task.

In one 2019 study, a managed team reached 75 to 85 percent accuracy on a product-recall classification task, against 50 to 60 percent for crowd workers. The study was sponsored by a managed workforce provider.

Hivemind study, published as a sponsored article on VentureBeat (3)

Who we work with

Built for teams that train, test and deploy AI.

The same operating model serves four kinds of buyers. What changes is the rater profile, the rubric and the cadence. What does not change is verification, weekly QA and one accountable team.

Frontier and research labs

Preference data, human evaluations and red teaming on a steady weekly cadence inside training loops.

RLHF · EVALS · SAFETY

Model builders and fine-tuners

Release-over-release regression testing and rubric scoring for domain-specific and open-weight models.

EVALS · REGRESSION

Conversational AI vendors

Voice and chat agents tested end to end by people who handle real customer conversations every day.

AGENT TESTING

Enterprises deploying AI agents

Pre-launch and ongoing review of customer-facing AI in healthcare, insurance, banking and telecom.

DOMAIN REVIEW

What our rater teams run

Six human feedback disciplines. One managed team.

Each program gets a named program lead, dedicated team leads, a QA analyst and a workforce planner. You get one scorecard and one weekly report, whatever mix of work you run.

RLHF

Preference Ranking and Rating

Pairwise and multi-way comparisons of model outputs, plus Likert ratings, each with a written reason. Rationale quality is audited weekly, and the tester guide sets tie and conflict rules so choices stay consistent.

EVALUATION

Model Evaluation and Regression

Rubric scoring on helpfulness, accuracy and instruction following, with side-by-side comparisons between model versions. Regression sets are rerun on every release so you see exactly what changed.

AGENTIC AI

AI Agent and Voice Testing

Multi-step tasks that check whether an agent finishes the job: tool use, handoffs, recovery from errors, accents and noise on voice. Scored the way QEval™ scores live contact center conversations.

SAFETY

Red Teaming and Policy Review

Adversarial prompts, misuse attempts and policy labeling written to your safety guidelines. Severity tagging and escalation paths are agreed at kickoff, with content boundaries and wellbeing check-ins for raters.

DOMAIN EXPERTISE

Customer Service and Regulated-Industry Review

Reviewers with contact center experience in healthcare, insurance, banking and telecom judge whether an AI answer would hold up with a real customer, including disclosure language, tone and resolution.

REAL-WORLD TESTING

Demographic and Multilingual Testing

Raters matched to age, region, life stage and occupation, with English and Spanish review and global teams for products used across markets. Differences between groups are reported as findings.

How a program runs

Run the way we run
every quality program.
Reported every week.

Human feedback work fails in the same places contact centers fail: the wrong people, a rubric nobody reads the same way, and quality checked too late. Etech runs four stages with an owner and an output you can see at each one. Pilots start from people already on our teams, so training can begin soon after kickoff while recruiting for harder profiles runs in parallel.

1 business day

To answer a rater question and add it to the shared FAQ

Every week

QA audit on every rater, not a sample of raters

2 weeks

Indicative time from kickoff to first weekly report

1 report

Volume, quality, agreement and roster in one place

Timelines are indicative and confirmed in your program plan.

  1. 1

    Recruit and verify

    The right people, confirmed before day one

    • Candidates matched to the domain, language, location or demographic profile each task needs. Existing Etech teams first, then targeted recruiting in the US, Jamaica and India.
    • Identity and address check, skills test, live onboarding call and a short writing sample.
    • Confidentiality agreement and a no-AI-assistance rule signed before any access is granted.
  2. 2

    Train and qualify

    Your rubric, turned into a working guide

    • Your training module, plus a tester guide with a strong, weak and borderline example for every rubric dimension.
    • Each rater scores a gold (known-answer) set and must clear the bar agreed with you before any live work.
    • Pass rate reported by cohort, so you see how many qualified and why the rest did not.
  3. 3

    Calibrate and go live

    Shared judgment before volume

    • The cohort scores the same tasks, compares, and talks through every difference. Weekly in month one, then every two weeks.
    • Live work starts with qualified raters only. Scheduled hours are tracked daily against your weekly volume.
    • A trained backup bench covers absences, so delivery does not dip when someone is out.
  4. 4

    Audit, coach and report

    Quality checked every week, per rater

    • QA analysts review a sample of every rater's work for accuracy, specific reasons, real effort and signs of AI-written text.
    • Below the bar means one-on-one coaching that week. Repeat misses mean replacement from the bench.
    • Weekly report: volume against plan, gold accuracy, inter-rater agreement, roster changes and open questions.

The quality system

Not a crowd score at the end.
A quality record for every rater, every week.

These are the practices human feedback programs depend on, run through the same QA, calibration and coaching structure Etech uses on every contact center program, and measured with the same scorecard logic as QEval™.

Weekly quality report

Sample layout, not client data

98%

tasks delivered on SLA

93%

gold accuracy

0.74

inter-rater agreement

Volume delivered vs. plan, by work stream

Preference ranking 98%
Response evaluation 101%
Agent testing 96%
Safety review 100%
Multilingual review 99%
Raters: 2 added, 1 coached, 0 replaced 3 guideline questions closed

Disagreement is not always error

In preference and demographic studies, rater groups are meant to see things differently. Toggle to see why we measure agreement within each group as well as across the pool.

All raters pooled 0.46

Pooled together, agreement falls in the moderate range. It can read as a quality problem, but much of the gap is real differences in how each group experiences the answer.

Illustrative values for explanation only.

Worked examples

Your rubric becomes a rater guide with a strong, weak and borderline example per dimension, and a rule for which dimension wins in a conflict.

What you see

Guide version and changes

Qualification first

A gold set and a pass bar agreed with you before any live work.

What you see

Pass rate by cohort

Gold checks in live work

With your approval, items with agreed answers are mixed into live work, so accuracy is tracked all the time.

What you see

Accuracy per rater

Inter-rater agreement

A sample of tasks is scored by two or more raters and measured with Cohen's kappa (for rater pairs) or Krippendorff's alpha (for two or more raters). Values of 0.61 to 0.80 are commonly read as substantial agreement (4).

What you see

Agreement by group and task type

Path for hard calls

Unclear cases go to a team lead, then to your team. Every answer joins a shared FAQ within one business day.

What you see

Questions and answers log

The difference buyers feel

Volume is easy to buy.
Consistent judgment is not.

Open platforms are built to reach many contributors quickly, and they work well for high-volume, low-ambiguity tasks. Judgment-heavy work such as preference ranking, safety review and domain evaluation depends on the same people applying the same rubric over time. That is an operating problem, and operating is what Etech does.

Etech

A managed rater team

Who the raters are
Verified, matched to the task
Continuity
Same team each week, backup bench
Quality control
Weekly per-rater audits, calibration
Accountability
Named program lead, weekly report
Guideline changes
Every rater within one business day
Employment model
Direct-hire Etech team members under local labor rules
Typical open platform

A contributor marketplace

Who the raters are
Varies; often profile data plus automated checks
Continuity
Contributors come and go with volume
Quality control
Platform filters; individual coaching varies
Accountability
Varies; often an account team or queue
Guideline changes
Updated instructions, uneven uptake
Employment model
Often independent contractor arrangements

The operating record behind every program

The same numbers that run our contact centers run your rater team.

These are company-wide figures from Etech's contact center operations. They describe the people, management and measurement system your rater program runs on.

<5%

Monthly attrition

against an industry average of 30%+

6.3 yrs

Average tenure

against an industry average under 12 months

2.5B+

Interactions analyzed

cumulative platform figure, scored by QEval™

95%

Client retention, 2025

clients retained year over year

250+

ETS Labs engineers

tooling, integration and reporting in-house

What your security team checks

Prompts, rubrics and model outputs are your IP. We handle them that way.

SOC 2 Type II is attested. Etech also holds PCI DSS Level 1 certification and HIPAA/HITECH compliance. Everything below is operated to the evidence bar your security team checks: who can see what, where the work happens, who owns the output, and what happens to the data at the end. Live attestations and the current subprocessor list are public on trust.etechgs.com.

SOC 2 Type II attested

Security, availability and confidentiality attested across our delivery centers. The report is available on trust.etechgs.com, with no NDA required for standard evidence packages.

ISO 27001:2022 aligned

Information security management operated to ISO 27001 principles, mapped to the evidence bar of enterprise and AI-lab security questionnaires.

Your data stays yours

Prompts, rubrics, model outputs and every rating, ranking and written reason belong to you. Etech does not use client data to train models and does not share or resell it.

Your tools only

Raters work only inside the platforms you approve. Each rater sees only the items assigned to them, and data is returned or deleted at program end on your instructions.

No AI assistance

Raters may not use AI tools on tasks. Weekly QA audits check for signs of AI-written text, and repeat issues lead to removal from the program.

On site or remote

Programs run on controlled production floors in our delivery centers, or remotely on Etech-managed devices, depending on your data rules.

Zero compliance breaches since 2003. Live attestations and subprocessor list are public. Your security team does not have to wait for a sales call to verify.

Live attestations on trust.etechgs.com

Run. See. Build. Applied to AI labs.

We run the operation, built the platform that measures it, and own the AI lab. No competitor does all three.

Human data vendors usually offer a workforce or a platform. Etech brings both, plus an engineering team. Quality findings from QEval™ flow into coaching, coaching flows into better data, and ETS Labs automates the reporting so your team spends time on the model, not on vendor management.

RUN

We operate the rater team

Recruiting, verification, training, scheduling, coaching and replacement, run by the same operations teams that staff our contact centers in the US, Jamaica and India. Servant-leadership management. Dedicated teams, not rotating contributors.

SEE

We measure every rater against your rubric

QEval™ scores every interaction against client scorecards, and the same platform that manages agent performance manages Voice AI performance. We apply that scorecard logic to gold accuracy, audit scores and agreement for every rater.

BUILD

We engineer the tooling inside the contract

ETS Labs. 250+ engineers. Connectors to your annotation tool's API, automated weekly reporting, and the workflow automation that keeps a program running cleanly. The tooling built for your program belongs to your program.

Global rater teams

Three countries. One operating model. Rater location by program.

Some programs need raters from one country, for demographic fit or data residency. Others need follow-the-sun coverage and scale across markets. Etech staffs from one location or blends all three under the same verification, QA and reporting.

Etech delivery center in Texas

United States

Nacogdoches, Lufkin, San Antonio and Dallas, Texas

US rater pools for programs that need US demographic fit, US English and Spanish, or US data residency. Anchor location for program leadership and QA.

Etech delivery center in Montego Bay, Jamaica

Nearshore

Montego Bay, Jamaica

English-first rater teams working in US time zones. Scale for evaluation, preference ranking and agent testing where your program allows non-US raters.

Etech delivery center in India

Offshore and AI lab

Gandhinagar and Vadodara, India

Gujarat-based rater teams with 24/7 coverage, review for products used across markets, back-office program operations and the ETS Labs engineering center.

Questions AI teams ask

Straight answers before the first call.

If your question is not here, a program lead will answer it on a 30-minute review.

What is a managed human feedback team?

A dedicated group of verified raters who rank model outputs, score responses, test AI agents and write feedback against your rubric. The same people work your program week after week, under a named program lead, with QA audits, calibration and a weekly report.

Where are Etech's raters located?

In our delivery centers in the United States, Jamaica and India. Programs can be staffed from one country, for example when US demographic fit or US data residency is required, or blended across countries for coverage and scale.

How does Etech verify that raters are real and fit the task?

Every rater passes an identity and address check, a skills test, a live onboarding call and a short writing sample. A structured intake records domain experience, languages, location and demographics, and each rater is matched to the profile your task needs.

How do you stop rushed or AI-written feedback?

Raters agree to a no-AI-assistance rule, work only in the tools you approve, and are audited every week. QA analysts check a sample of each rater's tasks for specific reasoning, real effort and signs of AI-written text. Gold items mixed into live work show whether accuracy holds up over time.

How do you measure feedback quality?

Through qualification tests before live work, gold checks, weekly QA audits, calibration sessions and inter-rater agreement (IRR) measured with Cohen's kappa or Krippendorff's alpha. Targets are agreed with you at kickoff.

Which annotation tools do you work in?

Your in-house platform, or a commercial annotation tool you license. Raters work only in tools you approve. ETS Labs can connect to your tool's API to bring daily output and quality data into the weekly report.

Who owns the data and the feedback?

You do. Prompts, rubrics, model outputs and all ratings, rankings and written reasons belong to you. Etech does not use client data to train models, and data is returned or deleted at program end on your instructions.

How is a program priced?

On delivered hours or on a single all-inclusive monthly rate that covers raters, team leads, QA, workforce management and reporting. There are no separate line items for supervision or quality review.

Speak with a program lead

Human feedback program review: see the full plan in 30 minutes.

Bring your rubric and a sample task. We will walk through rater profiles and locations, verification, the quality system, a sample weekly report and a pilot plan. A program lead, not a sales coordinator, will send a concrete point of view inside one business day.

An Etech program team in discussion

Sources: (1) Inc., reporting on Scale AI's contributor programs, June 2025, as summarized by Futurism. (2) 404 Media, "People Training OpenAI's AI Fired for Using AI to Train the AI," September 22, 2026. (3) VentureBeat, sponsored article presented by CloudFactory, reporting a 2019 Hivemind study. (4) Landis and Koch, "The Measurement of Observer Agreement for Categorical Data," Biometrics, 33(1), 159 to 174, 1977.