Mechanical Turk closes September 30: replacing penny-task labeling in your eval pipeline

Amazon will close AWS Mechanical Turk on September 30, 2026, 21 years after it launched. New registrations already stopped on July 30, alongside SageMaker Ground Truth and Augmented AI. For most of that time MTurk was the crowdsourcing layer under the AI stack: HITs for labeling, transcription and moderation, distributed to more than 500,000 workers. If any dataset you evaluate against was built on it, the shutdown is a five-week migration problem and a longer reproducibility problem.

Why it was already dying

The market split around it. High-value annotation, the kind that needs domain expertise and quality controls, moved to Scale AI, Mercor and Prolific. Cheap classification, the kind that used to be a penny a HIT, moved to LLMs. MTurk sat in the middle with neither story, and the shutdown makes official what demand already decided.

That split is also the map for replacing it. There is no single successor. You need to sort every MTurk-dependent task into one of two buckets.

Bucket one: tasks a model can label

Sentiment, topic, language, toxicity, near-duplicate detection, simple relevance: these are the tasks where a current LLM matches or beats an untrained crowd worker, and where the crowd's only remaining advantage was that it was a different population from your model. Move them to a model, but do it deliberately:

• Use a labeler model that is not the model under test. Labeling your eval set with the system you are evaluating is circular.
• Prefer an open-weight labeler you can pin and rerun. IBM's Granite 4.2, released August 26 under Apache 2.0 in 3B, 8B and 30B sizes with a 128K context, is the kind of thing that fits here and runs locally through Ollama.
• Freeze the labeler version, prompt and temperature, and store all three with the dataset. That is what reproducibility means now.
• Keep a human-labeled slice, a few hundred items, as the reference you calibrate the model labeler against, and report agreement.

A crowd label was reproducible because you could re-post the HIT. A model label is reproducible only if you froze the model, and most teams have not.

Bucket two: tasks that still need people

Nuanced judgement, expert domains, anything where the label is the product's definition of correct: these go to a managed vendor or a panel you recruit yourself. The cost per item goes up by an order of magnitude compared with MTurk, so the design work is deciding what deserves it. Usually that is the gold set for your most important metric and nothing else. Pair it with output validation in production so the human-labeled set is checking the validators rather than every response.

The five-week checklist

• Inventory every dataset, eval harness and moderation queue that touches an MTurk HIT type or a Ground Truth job.
• Export raw HIT results and worker-level metadata now, before the console goes away; you will want inter-annotator agreement later.
• Sort tasks into the two buckets and pick a labeler model or vendor for each.
• Re-label a sample with the replacement and measure agreement against the original MTurk labels. Disagreement must be explained, not assumed to be error.
• Version the dataset: tag the MTurk-era snapshot, tag the migrated one, and never overwrite the first.
• Record labeling provenance in the eval report, so a score from October is not silently compared with one from July.

What this does to old benchmarks

Any public eval set built on MTurk labels is now unreproducible in the strict sense: you can rerun the model, but you cannot re-collect the labels the same way. Their labels are frozen artefacts, and a drift in your model's behaviour relative to a frozen crowd from years ago is not the same thing as a regression. Treat those sets as historical anchors and build your own current ones.

The honest gap

Nothing above comes with cost numbers, because the announcement has none and vendor pricing varies with volume and domain. The claim that LLMs already handle cheap classification is the industry's reading of where demand went, not a measured accuracy comparison across task types. Before you move a task to a model labeler, run the agreement test on your own data; the shutdown is a deadline, not evidence that the model is good enough.