## 1. Framing the Problem Every social platform has to decide what to do about harmful content: nudity, violence, hate, terrorism, trafficking. Doing nothing is illegal in many places and degrades the user experience. Over-removing legitimate posts chills expression and pushes users away. Our job is to design a system that scores every new post fast enough to act before it spreads, accurately enough to remove harmful content automatically when we are confident, and humble enough to route the rest to a small team of human moderators. ### What we are deciding and who consumes it The primary surface is every newly created post on a Twitter-scale platform: roughly `1B` posts per day, an average of `12k` posts per second, with peak factors of three to five times. Posts in v1 are text plus image. Video and live content are out of scope. The business objective is minimizing user exposure to harmful content while preserving free expression. Wrongful removal of one legitimate post is more costly per item than missing one harmful post at the auto-remove tier, but unbounded misses degrade the platform. The ML task is multi-modal multi-label classification. Given a post's text, image, and light author and reach context at publish time, we predict a calibrated probability per harmful category. The downstream consumer is the post moderation policy service, which maps that score into one of four actions: allow, demote, route to human review, or remove. The interviewer told us not to remove anything we are not at least `95%` confident about. I would clarify upfront that this means precision at least `95%` at the auto-remove threshold, not raw probability above `0.95`. That single sentence reframes calibration as a load-bearing layer, not a model implementation detail. A new post enters a moderation service that runs a multi-modal model, applies per-category calibration, and asks the policy service which action to take. Allowed posts publish, demoted posts publish but lose ranking weight, review-routed posts hold or publish based on category and enter a capacity-bounded moderator queue, and removed posts are blocked. Post events, scores, moderator decisions, and appeal outcomes flow through Kafka into nearline aggregation and an offline retraining loop that also recomputes per-category calibration and thresholds. ## 2. Requirements ### Clarifying Questions ::::qa-item :::you[You] "Are we scoring posts at publish time before they reach users, or is this more of an async cleanup system that catches harmful content after the fact?" ::: :::interviewer[Interviewer] "At publish time, before the post reaches a meaningful audience. The target is 1 to 3 seconds p99 end-to-end, fast enough that harmful content does not spread, but generous enough for real multi-modal inference." ::: :::takeaway[Takeaway] Publish-time scoring means every post must be judged before distribution, leaving no room to skip inference, and the 1-3 second budget is shaped by multi-modal model latency rather than a sub-millisecond real-time constraint. ::: :::: ::::qa-item :::you[You] "When the model flags a post, does it always get removed, or is there a graduated set of actions?" ::: :::interviewer[Interviewer] "Four actions: allow, demote, route to human review, or remove. Auto-remove only fires when we hit at least 95% precision per category. Below that, borderline posts route to a moderator queue that is capped at a fixed daily staffing budget." ::: :::takeaway[Takeaway] The 95% precision floor at auto-remove forces a conservative threshold, and the moderator queue cap turns daily staffing capacity into an explicit system constraint that bounds how many borderline posts the model can defer. ::: :::: ::::qa-item :::you[You] "Where do the training labels come from, and how do we avoid reinforcing whatever the current model already flags?" ::: :::interviewer[Interviewer] "Labels come from moderator decisions on the queue, user reports, appeal-overturned removals, and a small random fraction of borderline posts we route explicitly to break model bias. Each source has different reliability." ::: :::takeaway[Takeaway] Weighted multi-source labels with deliberate exploration slots in the review queue are needed from day one, since a model that only labels what it catches will reinforce its blind spots on every retraining cycle. ::: :::: ::::qa-item :::you[You] "What is the peak post creation rate we need to handle, and does every single post require the same full inference pass?" ::: :::interviewer[Interviewer] "Peak is roughly `12k` posts per second with burst factors of three to five. Every post must receive a score before distribution. There is no skipping inference, but inference time is budgeted inside the `1-3 s` end-to-end window." ::: :::takeaway[Takeaway] The pre-filter tier offloads most of the throughput, so the expensive multi-modal inference path only handles the fraction that reaches it, keeping inference cost tractable at peak load. ::: :::: ::::qa-item :::you[You] "Do we use the same score threshold across the board, or do we tune them per category?" ::: :::interviewer[Interviewer] "Definitely per category, because the cost of making a mistake is totally different for hate speech versus extreme violence. Just make sure that whatever decision we make can be audited and traced back to the exact version of the model and thresholds we used." ::: :::takeaway[Takeaway] Per-category calibration and versioned threshold governance are first-class system components, required for appeals and regulatory compliance rather than optional audit features. ::: :::: ### Functional requirements - Score every new post (text plus image) at publish time and return a calibrated probability per harmful category to the moderation policy service. - Map the calibrated score to one of four actions through per-category thresholds: allow, demote, route to review, or remove. - Route a bounded subset of posts to a human moderator queue, prioritized by harmful probability, reach estimate, and model uncertainty, sized to fit a fixed daily moderator-hour budget. - Match incoming images against known-harmful image hash banks using perceptual hashing such as *PDQ (perceptual hashing for visual content)* as a sub-millisecond pre-filter before model inference. - Ingest moderator decisions, user reports, and appeal outcomes back into the training and calibration pipeline as multi-source labels with explicit per-source weighting. - Persist a moderation decision record per post (action, per-category calibrated probability, model version, calibration version, threshold version, reviewer if any) for appeals and audit. ### Non-functional requirements - Auto-remove decisions hit precision at least `95%` per category, measured on a held-out unbiased slice using calibrated probabilities, not raw model logits. - End-to-end publish-time decision latency on the order of `1-3 s` p99, tight enough that posts are judged before they reach a meaningful audience but generous enough for real multi-modal inference. - Throughput of roughly `12k` posts per second on average and `40-60k` per second at peak, with inference as the dominant cost. - Production prevalence of harmful posts is `<1%`. Sampling, loss weighting, calibration, and evaluation must reflect that, not the `50%` balance of the seed set. - Cold-start with only `50k` labelled posts (`50%` harmful, `50%` benign). The system must produce a usable v1 model from this seed and improve as moderator and appeal labels accumulate. - Roughly `10M` likely-harmful posts per day (`<1%` of `1B`). The review queue must fit a fixed daily moderator-hour budget, and everything else is auto-handled. - Auditability: each removal or demotion must be reproducible from `(model_version, calibration_version, threshold_version)` plus the post features. :::collapse[Out of scope]{variant="warning"} - Video, live broadcasts, and stories. - Comment-level classification as a target. Comments may appear as features in the asynchronous re-score path only. - Direct messages and other private surfaces. - Account-level enforcement (repeat-offender bans, network takedowns). The system signals into this layer but does not own it. - Multilingual or locale-specific policy variants in v1. - Coordinated-actor or network detection. - Ads brand-safety classification. - Moderator tooling and appeals UX. ::: ## 3. Data & Feature Engineering Two questions get conflated here. Where do labels come from, and what features can we trust at publish time? Treating them as one question hides the part that actually breaks under load: removed posts never accrue social signal, and the moderator queue itself is model-chosen, so naive labels reinforce the model's existing blind spots. ### Training data construction and label sources The `50k` seed (`50%` harmful, `50%` benign) is enough to fine-tune a pretrained backbone but not to train a strong multi-modal model from scratch. Production labels arrive from four other sources, each with different precision and coverage: - Moderator decisions on the review queue: high precision, biased coverage (the queue is model-chosen). This is the bulk of training labels going forward. - User reports: high coverage, noisy precision. We denoise by report velocity and reporter-trust signals, then treat them as soft labels in training and as candidates for review. - Appeal-overturned removals: small volume, very high value. These are confirmed false positives, so they correct the model's most damaging mistakes. - Exploration slots in the review queue: a small random fraction of borderline posts routed to moderators specifically to debias the label distribution. We weight these sources explicitly when assembling training data. Treating moderator labels and user reports as equal would let report noise drown the moderator signal. Joining everything with point-in-time correctness matters too: features must reflect the post's state at publish time, not at label time, otherwise we leak future moderator decisions into training. ### Feature tiers and what each one trusts Features split into three freshness tiers. Mixing them up is a common training-serving skew bug. - Real-time, computed at publish time (`<1 s`): the post's text, the post's image bytes (we assume the post-create flow already has them), `account_age`, `prior_enforcement_actions`, `follower_count` as a reach proxy, `locale`, and `posting_client`. These are the model's direct inputs. - Nearline, refreshed in seconds to minutes via Flink: rolling per-author harmful rate over the last hour, per-image-hash duplicate burst (PDQ perceptual hashing), per-author posting velocity. Stored in Redis with *TTL (time-to-live)*. These feed both the model and the review-queue priority. Comment-burst on the post is also a nearline signal but applies only to the asynchronous re-score path, since a fresh post has no comments yet. - Batch, refreshed hourly to daily: long-window per-author harmful rate, repeat-offender flags, matches against known-harmful image hash banks (terrorism, *CSAM (child sexual abuse material)*), and historical moderator-disagreement signals. Stored in the warehouse as the authoritative training source and exported to Redis for serving. > [!IMPORTANT] > Always say which tier a feature lives in. Calling a feature "in the feature store" hides whether it is the rebuildable hot tier in Redis, the authoritative warehouse tier, or both. The rollback story for each tier is different, and so is the staleness budget. ### Class imbalance, sampling, and leakage Production prevalence is `<1%`. The seed is `50/50`. If we train on the seed distribution and threshold at fixed precision, the model looks great offline and falls over in production. We handle this with class-imbalance-aware sampling (over-sample positives during training, but compensate in the loss) and with calibration on a held-out unbiased slice that reflects production prevalence. Without that step, no precision guarantee is real. Leakage is subtle here. Moderator labels arrive after the post is created, often after it is demoted or held. If we join features at label time, we leak the demotion decision and the post's exposure trajectory into training. Point-in-time joins solve this, but only if the offline pipeline replays features as they were at publish time, including stale Flink aggregates if those were what the online model actually saw. ## 4. Modeling Pick any moderation paper and the discussion is about the model. In a real interview the harder question is the layer we add on top of the model: calibration. The model returns a probability per category. The policy ladder maps that to an action through per-category thresholds. Without calibration, the precision target is fiction. ### Cold-start, then production The tempting first move is to train a binary classifier on the `50k` seed and threshold at `0.95`. That answer is fast to give and obviously wrong. The seed is `50/50` while production is `<1%`, raw probability is not calibrated, one global threshold ignores per-category cost differences, and there is no path for the model to ever learn what it is missing today. The right cold-start lever is pretraining. We start from a pretrained vision-language backbone (a *CLIP (contrastive language-image pretraining)*-style joint encoder is the canonical example) and fine-tune on the `50k` seed. Pretraining gives both the text and image encoders strong general representations and an early-fusion-like joint signal, so we get usable cross-modal classification with limited labels. Calibration on a held-out unbiased slice happens immediately after fine-tuning, because the seed's `50/50` distribution does not match production's prevalence. ::img[Multi-Modal Harmful Content Classifier Topology]{src="https://assets.hack2hire.com/post/69f67ef230f41dfd539fbf8a/2a070edb-dabd-4b01-a675-b27e4a0587ad_20260508.svg"} Once the moderator-labeled dataset grows beyond the seed, the production model becomes a dedicated multi-modal architecture: a text encoder, an image encoder, a cross-attention fusion layer, and a per-category multi-label head. Each branch encodes its modality, the fusion layer mixes them, and the head emits one logit per category (nudity, violence, hate, terrorism, trafficking). The reason the architecture matters here is cross-modal interaction. A benign image of a child becomes hateful when paired with a hateful caption. A benign caption becomes harmful when overlaid on a violent image. A late-fusion ensemble of independent text and image classifiers cannot represent these interactions, no matter how strong each branch is on its own. :::collapse[Bad: Late fusion]{variant="bad"} Two independent classifiers, one per modality, with a learned weight or rule combining their outputs. The architecture is simple to build and easy to debug per modality. This loses the interactions that drive a meaningful share of harmful posts. Each model sees only half the post, so the system cannot tell that the same image is benign in one caption and harmful in another. With limited labels, a late-fusion ensemble also wastes the strongest cold-start asset we have, which is pretrained joint encoders. ::: :::collapse[Good: Pretrained vision-language backbone]{variant="good" open} A single joint encoder, pretrained on a large image-text corpus, fine-tuned on the `50k` seed. This is the practical cold-start choice and the one I would lead with in the interview. It gives early-fusion-quality cross-modal signal cheaply at cold-start. The backbone has already learned to align the two modalities, so fine-tuning teaches the harmful-vs-benign decision boundary instead of the basics of language and vision. The cost is that calibration and per-category heads still need to be added on top. ::: :::collapse[Great: Custom early/cross-attention multi-modal model]{variant="great"} Separate text and image encoders feeding a cross-attention fusion layer, with a per-category head. Trained on the growing moderator-labeled dataset with class-imbalance-aware sampling. This captures the strongest cross-modal signal at scale, including subtle interactions a frozen backbone cannot easily learn. It is only justified once labels grow well past the seed and the calibration discipline is in place. Without that discipline, a stronger model just produces sharper miscalibrated scores. ::: ### Calibration as a first-class layer The model emits a per-category probability per harmful class. The policy ladder needs precision per category, not probability per category. We add a calibration layer that maps each raw probability to a calibrated probability, fit on a held-out unbiased slice that reflects production prevalence. The auto-remove threshold for each category is then chosen so that ```KaTeX \Pr\big(y_c = 1 \mid \hat{p}_c \ge \tau_c\big) \ge 0.95 ``` Read this as: among posts whose calibrated probability for category `c` exceeds the threshold, at least `95%` are actually harmful. That is the operational meaning of "95% confident." For most categories I would use isotonic regression because it is non-parametric and handles non-monotonic miscalibration. For low-data categories, Platt scaling is the safer fallback. We refit calibration on every model promotion and we recompute thresholds at the same time, because changing one without the other silently breaks the precision contract. ## 5. High-Level Design The goal of this section is to walk one post through the system, then back out into the loops that keep the model and calibration current. The architecture has to make three things obviously true: publish-time scoring is fast enough to act before the post spreads, the offline loop refreshes both the model and its calibration together, and the human review queue is bounded by real moderator capacity rather than infinite hope. ::img[Harmful Content Detection System Architecture]{src="https://assets.hack2hire.com/post/69f67ef230f41dfd539fbf8a/15c21467-b83f-403b-9c18-37a4685c78ab_20260502.svg"} The reader should look at the anchor diagram for two things: where the publish-time path crosses into asynchronous logging, and how moderator decisions, user reports, and appeal outcomes loop back through Kafka into the offline retraining and recalibration jobs. ### One post, end to end ::img[Publish-Time Scoring Flow]{src="https://assets.hack2hire.com/post/69f67ef230f41dfd539fbf8a/e905abb3-ac25-4f8b-b8dd-0d258e8517e5_20260502.svg"} A user submits a post containing text and an image. We assume the post-create flow already has the image bytes (uploaded directly or fetched once at create time), so the moderation service does not have to round-trip an external URL on the hot path. The Post Service issues a synchronous `ScorePost` call to the Moderation Service over gRPC. The service first checks the image's perceptual hash (PDQ) against known-harmful image banks. A match short-circuits the model with a high-confidence action, which is both faster and more defensible than a model score for content that is already known. Otherwise the service fetches nearline features from Redis, runs the multi-modal model on accelerator-backed inference, applies the per-category calibration layer, and asks the Policy service to map the calibrated probabilities through per-category thresholds to one of allow, demote, route_to_review, or remove. The Post Service then applies the action: publish for allow, publish-with-demotion for demote, hold-or-publish-with-pending-review for route_to_review (depending on category severity), and block for remove. Every decision, including the underlying calibrated probabilities and the `(model_version, calibration_version, threshold_version)` tuple, is written asynchronously to Kafka. ### The nearline and offline loops ::img[Offline Pipeline: Events to (Model, Calibration, Thresholds)]{src="https://assets.hack2hire.com/post/69f67ef230f41dfd539fbf8a/3024ef66-585a-47f2-92ed-59daad02df58_20260502.svg"} Three event streams feed Kafka: post create events with model scores, moderator decisions from the review queue, and user reports plus appeal outcomes. Flink reads these to maintain rolling aggregates per author, per image hash, and per category, written to Redis with TTL. These are model features and review-queue priority signals at the same time. Spark runs the offline path: point-in-time joins to assemble training rows that reflect the post's state at publish time, multi-source label assembly with explicit per-source weighting, training of the multi-modal model with class-imbalance-aware sampling, and recompute of per-category isotonic calibration on the held-out unbiased slice. Per-category thresholds are recomputed against the new calibration in the same job so that `(model_version, calibration_version, threshold_version)` is published as one atomic unit. A canary rollout (`1% -> 10% -> 50% -> 100%`) checks for appeal-overturn-rate or false-positive-rate spikes before full promotion. For the highest-risk categories (terrorism, *CSAM*) we keep an additional always-on PDQ hash bank populated by both internal moderator decisions and external industry hash exchanges. This lets the system catch known content even when the model itself is rolled back. ### The review queue, sized by capacity > [!NOTE] > Roughly `1B` posts per day at `<1%` true prevalence implies about `10M` likely-harmful posts per day. A small global moderator team cannot review all of them. The review queue must therefore prioritize, not collect. Posts above a low triage threshold but below the per-category auto-remove threshold enter the review queue with priority ```KaTeX \text{priority} = \hat{p}_c \cdot \text{reach\_estimate} \cdot \text{uncertainty} ``` where the calibrated probability is for the most-likely category, reach_estimate is a follower-and-velocity-derived projection, and uncertainty is the model's entropy across categories. The queue is sized to fit the daily moderator-hour budget `D`. As a sense-check: at `D = 5000` moderator-hours/day and `30 s` per post, the queue absorbs about `600k` items, around `6%` of the daily harmful pool. Items below the cutoff stay either demoted or allowed depending on the band. A small random exploration slice of borderline allowed posts is also routed into the queue regardless of priority, to debias future labels (we return to this in deep dives). ### What happens when something fails If the Moderation Service is slow or unavailable, the Post Service applies a degraded policy: allow with mandatory async re-score for typical posts (fail open, because false-banning legitimate users is the costlier mistake), but block-and-route-to-review for posts that match known-harmful image hashes (fail closed for the highest-risk known content). A model rollback automatically returns to the previous matched `(model_version, calibration_version, threshold_version)` tuple so calibration and thresholds never drift relative to the model. ## 6. Inference & Evaluation Holding the latency contract on a quiet hour is easy. The real job is holding it through a peak that triples post volume while proving the precision floor still holds and the model has not silently started enforcing harder against real users. ### Latency budget under peak The publish-time decision targets `1-3 s` p99 end-to-end. On a typical post the budget breaks down roughly as: feature fetch from Redis around `10 ms`, image preprocessing plus multi-modal inference on accelerator-backed servers around `200-800 ms` depending on model size, calibration and threshold lookups well under `1 ms`, and network plus serialization in the low hundreds of milliseconds. The headroom relative to a single second exists for two reasons: queueing under peak, and the occasional case where the post-create flow has not yet fully landed the image bytes. PDQ image-hash matching against the known-harmful banks is a sub-millisecond pre-filter. It catches republished known-harmful images and reduces model load. At average load this might short-circuit `5-10%` of inference calls. Under coordinated waves it can short-circuit much more, exactly when accelerator capacity matters most. Throughput-wise, average load is around `12k` posts per second and peaks reach `40-60k`. Inference dominates cost, so the accelerator pool is sized for peak with pre-warmed batching and request-coalescing. Cost-wise, this is the largest line item in the system. Right-sizing it against the diurnal curve is a deliberate Other-Considerations topic, not an afterthought. ### Offline evaluation that actually predicts production The contract is precision per category at the auto-remove threshold, not aggregate area-under-curve metrics. Concretely we track: - **Calibrated precision at threshold**: for each category, precision among posts with calibrated probability above the per-category threshold, evaluated on the held-out unbiased slice. This is the contract. - **Recall on the unbiased slice**: recall computed on a random sample of all posts including unmoderated, to expose what the production queue selection misses. - **Calibration error**: expected calibration error and per-category reliability diagrams on the unbiased slice. Drives recalibration triggers. - **Per-category breakdown**: every metric reported per category. A global average hides regressions in low-prevalence categories like terrorism. - **Exposure-weighted miss rate**: estimated harmful exposures per `1k` user impressions on the unbiased slice, weighted by the actual reach of each post. This is the closest offline proxy to the business objective. The reason exposure-weighting matters: a missed harmful post seen by `5` people is not the same as a missed harmful post seen by `5M`. Vanilla recall counts them equally and lets the model trade impactful misses for cheap ones. ### Online experimentation under bounded risk Shadow scoring is the default for new model candidates. We run the candidate in parallel, log calibrated probabilities and predicted actions, and compare against production without applying anything. This lets us measure would-be precision and would-be queue load before any real-user impact. Promotion uses gradual rollout (`1% -> 10% -> 50% -> 100%`) with auto-rollback on appeal-overturn-rate spikes or false-positive-rate spikes against the legitimate-account golden set. We also run counterfactual evaluation on logged events: replay through candidate model + candidate calibration + candidate thresholds and estimate would-be precision and recall on the unbiased slice. Counterfactual eval is the only way to tell whether a candidate would do something different on a population we never asked it about. > [!IMPORTANT] > Offline accuracy is structurally biased because removed posts never accrue social signal and the moderator queue is model-chosen. The held-out unbiased slice, the exploration slots in the review queue, and the appeals-driven negatives exist specifically to break that bias. Without all three, every offline number drifts away from production reality as the model retrains on its own outputs. That structural bias is exactly what the next section's deep dives are about. ## 7. Deep Dives These two deep dives address the parts of the system that most candidates skip and most interviewers probe. The first is the layer that turns a model output into an action under a real precision contract. The second is the structural reason offline accuracy quietly diverges from production reality and what to do about it. ### Calibration, per-category thresholds, and the 95% precision contract ::img[Calibrated Score to Action Ladder]{src="https://assets.hack2hire.com/post/69f67ef230f41dfd539fbf8a/948357d1-14a6-49bf-91e4-aac9d8282fb5_20260508.svg"} The score-to-action ladder is the spine of the entire system. The reader should look at the ladder diagram for one thing: how a single calibrated probability becomes one of four actions through per-category thresholds, with the precision floor annotated on the high band. A common move at this point is to ship a sigmoid output and threshold at `0.95`. It feels right because the interviewer said "95% confident." It is wrong for three compounding reasons. Raw probability is not calibrated. The seed's `50/50` distribution does not match production's `<1%`. And one global threshold ignores the fact that the cost of a false-positive removal differs across categories. A wrongful nudity removal is annoying. A wrongful terrorism removal is reputation-damaging. ::img[Per-Category Threshold Governance and Recalibration Cycle]{src="https://assets.hack2hire.com/post/69f67ef230f41dfd539fbf8a/ce8ebbd8-cef1-4840-933c-6270a7f65317_20260508.svg"} The threshold governance diagram shows why this is not a one-time decision. Each panel shows a calibrated probability distribution for one category, with its threshold `tau_c` sitting at a different position: `tau_nudity` sits higher because a 95% precision target requires less of the distribution to its right, while `tau_terrorism` sits lower because a higher precision target (and higher wrongful-removal cost) requires a tighter remove band with more mass captured. The row beneath both panels shows what happens after every retrain: calibration error on the unbiased slice triggers a refit of isotonic regression per category, new thresholds are recomputed, and all three: model, calibration, and thresholds are promoted as one atomic tuple. Rolling back only the model without rolling back its calibration and thresholds is the silent way to break the precision contract. :::collapse[Bad: Sigmoid at 0.95 from a single binary classifier]{variant="bad"} Sigmoid output of a single binary classifier, threshold at `0.95`. The model returns a probability and we remove posts above the cutoff. Easy to ship, easy to defend in a junior answer. Three problems compound. Raw model probabilities are not calibrated, especially after training on a `50/50` seed when production is `<1%`, so `0.95` rarely corresponds to `95%` precision. The squashed score distribution hides the miscalibration. And one global cutoff treats every category the same when the cost of a wrongful removal differs across nudity, hate, and terrorism. ::: :::collapse[Good: Calibrated probabilities with per-category precision thresholds]{variant="warning" open} Per-category isotonic / Platt calibration, thresholds picked at fixed precision. Train the multi-modal model, fit per-category isotonic regression on a held-out unbiased slice, and pick each per-category threshold to hit `precision >= 95%` on that same slice. Refit calibration and recompute thresholds on every model promotion as one atomic unit. This makes the precision contract real. The threshold for nudity might be `0.62` after calibration. For terrorism it might be `0.34` if we chose a higher precision target there. The model emits probabilities, the calibration layer reshapes the distribution, and the threshold picks the operating point. As long as the unbiased slice is representative, the production precision actually matches the offline number. ::: :::collapse[Great: Drift-triggered recalibration with atomic rollback]{variant="good"} Drift-triggered recalibration with atomic `(model, calibration, thresholds)` rollback. Stream the unbiased exploration slice continuously, monitor per-category calibration error, and trigger recalibration when error exceeds a category-specific tolerance. Roll back the entire `(model_version, calibration_version, threshold_version)` tuple together when an experiment regresses, never one piece at a time. This keeps the precision contract honest under adversarial drift. Rolling back only the model leaves behind a calibration table that no longer matches its underlying score distribution, which is silent breakage. Atomic rollback eliminates a whole class of incidents that would otherwise look like model regressions but are actually calibration drift. ::: Borderline scores between demote and auto-remove can publish-then-rescore later when nearline signals change, trading a small exposure window for a stricter auto-remove threshold. ### Label feedback loop and counterfactual evaluation The subtler problem is that the model's own decisions determine which posts ever get reviewed and labeled. Removed posts never accrue any social signal at all. The next model trained on this stream gets better at catching what it already catches, but its blind spots persist forever. Naive retraining widens the gap rather than closing it. ::img[Label Feedback Loop and Bias Correction]{src="https://assets.hack2hire.com/post/69f67ef230f41dfd539fbf8a/0acf5d4c-f54e-46e0-a40a-03d56fc33662_20260502.svg"} The feedback-loop diagram makes the bias visible. The reader should look for two specific arrows: where exploration slots enter the queue regardless of priority, and where appeal-overturned removals re-enter training as confirmed negatives. Those are the two interventions that actually break the loop. :::collapse[Bad Solution: retrain on moderator-confirmed labels only]{variant="bad"} Use only the moderator-confirmed labels from the production review queue as training data. This is the path of least resistance because the labels are already there and they look authoritative. The moderator queue is selected by the model. Posts the current model would have allowed are never reviewed and never become training data. Removed posts have no engagement, no comments, and no user reports, so the next model has even less signal about content like them. Each refresh tightens the model around its existing decision surface and silently blinds itself to whole regions of harm. ::: :::collapse[Good Solution: label-source weighting plus active sampling]{variant="warning"} Weight labels by source quality (operator-confirmed highest, then appeals-overturned, then user reports, then weak rule-triggered labels). Randomly route a small fraction of borderline posts into the review queue regardless of priority, so the model sees what it would otherwise hide from itself. This breaks the worst version of the loop. Active sampling injects coverage of borderline content into the labeled set, and weighting prevents noisy report labels from drowning the higher-quality moderator signal. The remaining limitation is that random sampling burns moderator hours on content that is often legitimate, and the sampling rate becomes a tradeoff between labeling cost and feedback-loop correction. ::: :::collapse[Great Solution: held-out exploration slice plus appeals-driven retraining]{variant="good" open} Carve out a small fraction of allowed posts as a continuous unbiased slice with shadow review for offline calibration. Feed appeal-overturned removals back into training as high-value confirmed negatives. Replace vanilla recall with an exposure-weighted miss-rate metric on the unbiased slice as the long-term steering signal. The unbiased slice gives an honest answer to "what would this candidate have done on the population we never asked it about." Appeals-overturned negatives directly correct the model's most damaging false positives. Exposure-weighting makes the metric reflect actual user harm, not raw item counts. The cost is real: the unbiased slice consumes moderator hours, and exploration accepts a small known increase in user exposure to harmful content. ::: ## 8. Other Considerations A few topics did not earn their own deep dive but would absolutely come up in the second half of an interview. Each one ties back to a specific constraint from earlier sections, which is what keeps this chapter moderation-shaped instead of a generic "production checklist." ### Cold start under 50k seed labels The most binding short-term constraint is that the seed is small for a multi-modal model. Three levers help. First, lean hard on a pretrained vision-language backbone so fine-tuning teaches the decision boundary, not the basics of language and vision. Second, use weak supervision from policy heuristics (rule-flagged content, image-hash matches against industry hash banks) as low-confidence training signal until moderator labels accumulate. Third, run active learning on the production stream: route the model's most uncertain posts to moderators preferentially, so each moderator hour produces the most informative label. This is also where the exploration slice from the deep dive lives, and the two should share queue infrastructure rather than competing for moderator capacity. ### Adversarial drift and refresh cadence Harmful content evolves. Bad actors swap image overlays, paraphrase known-violating text, and recycle old material with new framing. The system has to detect drift before precision silently degrades. The signals worth watching are calibration error on streaming exploration slots per category, image-hash novelty rate (the share of incoming posts whose perceptual hash has never been seen), and per-category positive rate compared against a slow-moving baseline. Any of these crossing a threshold triggers either a recalibration pass on a fresh unbiased slice or a full retrain. Refresh cadence in steady state is roughly weekly. Under active drift it can collapse to daily, which is one reason the offline pipeline has to be cheap to rerun rather than a quarterly event. ### Moderator-capacity governance Roughly `10M` likely-harmful posts per day cannot be reviewed by hand. The queue prioritization formula (calibrated probability, reach estimate, model uncertainty) is what makes review tractable, but it is also a piece of the system that needs explicit governance. Three things to nail down: who can change the priority weights, who audits the queue's category mix to make sure the team is not unintentionally over-indexing on one harm type, and how the daily moderator-hour budget `D` is sized against the cost of false-negative exposure. Treating priority weights as a hard-coded configuration is fine in v1, but they will be tuned, and the audit trail matters as much as the formula. ### Appeals and reversal-driven retraining Appeals are not just a user-experience feature, they are a high-value training signal. An appeal that overturns a removal is a confirmed false positive: the strongest possible negative example for the model. The pipeline should automatically reinstate overturned content, feed those examples back into training with high weight, and monitor systemic false-positive bias by author segment (account age cohort, locale, follower band). When false-positive rates skew across segments, that is signal of a feature or label bias that no global precision metric will surface, and it usually shows up first in the appeal stream. ### Multi-category threshold policy One global `95%` precision is the right place to start in an interview answer, but the production reality is per-category thresholds with category-specific cost asymmetry. A wrongful nudity removal annoys a single user. A wrongful terrorism-flagging causes account-level enforcement and can become a legal incident. The policy team usually wants higher precision targets on the highest-stakes categories and tolerates slightly lower precision on lower-stakes ones to gain recall. The system design supports this directly because thresholds are already per-category, but the governance question (who signs off on each threshold, how often they are reviewed) is what turns it from an ML knob into an operational policy. The answer to "how do you keep `95%` honest" eventually becomes "we publish a precision *SLO (service level objective)* per category and recompute it on every release." ## 9. Interviewer Expectations ### Junior A junior candidate is expected to land the basic shape of the system: a model that scores posts, a threshold that triggers removal, a queue that routes to human moderators, and a logging path that lets the model improve over time. They should be able to describe text-plus-image as multi-modal and pick a reasonable model family with prompting. They will usually treat "95% confident" as "model probability above `0.95`" and will not yet separate calibration from raw score. If you can sketch the publish-time path, the review queue, and the offline retraining loop on a whiteboard without prompting, you are in good shape for this level. ### Intermediate An intermediate candidate adds a layer of realism: they recognize that production prevalence is `<1%` and the seed is `50/50`, so sampling and evaluation must reflect that. They will name calibration as a missing piece if asked, and they can argue for late fusion vs. a pretrained backbone with a real tradeoff. They will discuss multi-source labels (moderator decisions, user reports, appeals) and acknowledge that not all labels are equal. They typically still treat the feedback loop as a monitoring problem rather than a structural design constraint. The clearest signal at this level is being able to explain why the naive `0.95` threshold breaks and what the next move is, without being walked there. ### Senior A senior candidate frames the entire system around calibration and the score-to-action ladder before discussing model architecture. They name precision-at-fixed-threshold as the contract, propose isotonic or Platt calibration on a held-out unbiased slice, and treat per-category thresholds as a real design variable rather than an implementation detail. They surface the label feedback loop early, propose exploration slots and appeals-driven retraining as first-class interventions, and tie the moderator queue to a fixed daily capacity budget. They distinguish staff-level moves (atomic `(model, calibration, thresholds)` rollback, exposure-weighted miss rate, drift-triggered recalibration) from senior-level moves and only escalate when the interviewer signals they want depth there. If you can defend why a strong-senior answer that ships an uncalibrated `0.95` cutoff would silently miss its precision target in production, you are at this level.