I worked on a harmful-content detection system at a nonprofit. Text classifiers and a YOLO image detector scored incoming posts, and high-scoring posts were flagged for human reviewers. Most of the training data came from silver labels, meaning scores from third-party APIs. The reviewers were looking at the models' mistakes every day, and none of that fed back into training.

Over several months I built pieces of a loop to change that: reviewers mark a flagged post as a false positive (or confirm it), those labels get exported as training rows, a model is retrained and approved, and new posts get scored by the new model. Some of it shipped and some was still on branches when I left. This post describes both, and which was which.

Four labels, all model-relative

The labeling UI was a menu on each post card with four checkboxes, one set for image classification and one for text: Confirmed, False Positive, Misclassified and Needs Review. The image labels went in first and the text labels followed. Ticking a box saved it right away and unticking removed it. Later I fixed the optimistic update so the saved state looked the same everywhere the post appeared.

The most important design decision was that the labels are model-relative. "Confirmed" means the model was right, whether it flagged the post or not. It does not mean the content is harmful. "False Positive" means the model flagged something that is fine. "Misclassified" means the content is a problem, but the model put it in the wrong category. "Needs Review" means the reviewer is unsure.

I wrote this into the runbook for whoever would do the labeling, because the natural reading of "Confirmed" is "this is bad". A trainer that reads it that way learns the wrong thing from every correct negative.

What a label row does not record

Each label is one row in a generic tagging table: post ID, tag ID, tagger, whether it was automatic, and a timestamp. When I looked closely, two things were missing.

First, the tagger ID was always null. The write path never resolved the logged-in reviewer. On the relabeling branch I resolved it from the authenticated user, so each label would record who applied it.

Second, the row stores no score and no model version. A model-relative label only makes sense next to the model's output, and the label does not record which output the reviewer was looking at. The exporter therefore joins to the score currently stored for that post and recomputes "was it flagged" at a threshold. If a newer model has rescored the post since it was labeled, that join gives a different answer than the one the reviewer saw. I documented this limit but did not fix it. The fix is to write the score, threshold and model version the reviewer actually saw onto the label row when it is created.

The branch also added a free-text note per post and per classifier family. Category labels cannot say which part of an image was wrong, or which category a misclassified text belonged in, and the note is where that goes.

Exporting text labels

The text exporter, written on the relabeling branch, turns labels into text,label rows. A post can collect more than one label, and labels stay until someone unticks them, so the exporter resolves conflicts first. A stale Needs Review keeps a post out of the export until a reviewer clears it:

LABEL_PRIORITY = {"Misclassified": 3, "False Positive": 2, "Confirmed": 1}

def resolve_labels(labels: set[str]) -> str | None:
    if "Needs Review" in labels:
        return None  # someone was unsure, so skip the post
    candidates = [l for l in labels if l in LABEL_PRIORITY]
    return max(candidates, key=LABEL_PRIORITY.get) if candidates else None

def derive_label(review_label, model_score, threshold=0.5, include_misclassified=False):
    flagged = model_score is not None and model_score >= threshold
    if review_label == "False Positive":
        return 0
    if review_label == "Confirmed":
        return 1 if flagged else 0  # the model's own call was right
    if review_label == "Misclassified":
        return 1 if include_misclassified else None
    return None

"False Positive" always gives a negative. "Confirmed" gives whatever the model predicted, because the reviewer agreed with that prediction. "Misclassified" is opt-in. It says the content is harmful, but for a binary classifier trained on one category that is only sometimes the right target, which is why it is off by default. There is one bug in this I would fix. When a post has no stored score, flagged is false, so a Confirmed label silently becomes a negative. Those rows should be skipped.

After labeling, the exporter drops texts under a minimum length, dedups on text, and does a stratified train/eval split (15% eval, fixed seed). If the result is small, it prints a warning: do not train on this alone, mix it with silver labels. Human label volume was very small compared with the silver data, which stayed the primary training source.

What reached main was groundwork: I cleaned up the existing training-data export script so it read the database password from the environment instead of a hardcoded value, could upload to the training bucket with --upload-s3, and had tests. A converter from the CSV output to the JSONL format one of the trainers expected was pushed on a branch but not merged.

Image labels without boxes

The image side hit a harder limit. Review labels are per post, and a YOLO detector trains on bounding boxes. "False Positive on this post" does not say which of the detected boxes was wrong, or where a missed object was.

So the image exporter does not produce training data. It writes a JSONL curation manifest: one record per labeled image, with its review labels and every detection the model made on it (class, confidence, box). An engineer uses that file to build corrected YOLO labels by hand. The exporter warns when the manifest is small.

{"content_id": "...", "media_id": "...", "review_labels": ["False Positive"],
 "detections": [{"class": "...", "confidence": 0.41, "bbox": [...]}]}

Closing the loop, partly

After export, the loop went through the existing training pipeline. A Step Functions execution runs a SageMaker job, the model registers as pending approval, an engineer approves it, and a rolling deploy puts it in front of new posts.

The step that did not exist was checking whether the new model was better on live traffic. The runbook's honest answer was to look at the score badges on recently scored posts. For text, each stored score records which model version produced it, so you could at least confirm which model was doing the scoring.

I started on that gap. Earlier I had added a daily table of flag rates per dimension and threshold, comparing the incumbent scores with the shadow in-house scores. I built an analytics page on top of it that charts baseline and candidate flag rates over time, with a delta summary. A later fix on the same branch corrected a swapped baseline/candidate mapping, which had flipped the sign of every delta. Shortly before I left I added an upload flow to the page. You drop a dataset zip, it goes to S3 through a presigned PUT, the server expands it (with a size bound), and it starts a training execution. That branch was pushed but not merged when I left. Flag rates only show whether a new model behaves differently. They do not show that it is more accurate, which still needs labels.

What was unfinished

Where things stood when I left:

• Shipped: the four labels for images and text, consistent optimistic saving, the cleaned-up export script with S3 upload and tests, flag-rate storage, and the training pipeline with manual approval.
• On branches, not merged: tagger IDs on labels, reviewer notes, the review-label text exporter, the image curation manifest, the CSV-to-JSONL converter, the comparison page and the upload-and-train flow.
• Not built: storing the score and model version on each label, an automated path from image labels to boxes, and a dedicated rescoring workflow. For images, rescoring meant resetting the processed flag on media rows in batches by hand so the worker picked them up again. For text, the team's lifecycle design proposed re-running the shadow pipeline scoped to a model version, but that did not exist yet. Rollback was manual: revert the service to the previous task definition.

If I were starting again, the first thing I would change is recording what the reviewer saw on the label row. Without it, the exporter has to guess what a model-relative label was relative to.