An LLM Moderation Plugin, End to End (Part 1)

Multiforum runs plugins in response to events like new comments and discussions. One plugin sends a comment and the forum's rules to a large language model and asks whether the comment breaks a rule. This post walks through running that plugin end to end on the live site, and what I found.

I wrote about the plugin system itself in an earlier post. This one is about a single plugin and the model behind it.

What the plugin does

The plugin listens for content events: comment.created, comment.edited, and the discussion equivalents. When one fires, it collects the submitted content and the forum's rules and sends them in one request to a separate moderation service.

The service is a FastAPI app on Cloud Run. It holds the provider credentials, limits its own request rate, retries transient failures, and asks the model for a fixed response shape rather than free text: which rules were violated, a recommendation (edit, remove, or none), a confidence score, and a short rationale. If the response is valid and the confidence is above the forum's threshold, the plugin files one report for the forum's moderators, authored by a bot account.

Nothing is removed automatically. The model flags and explains; a moderator decides.

The test forum

I tested this in a forum called Bad Advice, whose only rule is inverted:

Good advice is prohibited.

It is a forum for deliberately bad suggestions, so the moderator's job is inverted: catch anything genuinely helpful. The model has to judge whether advice is actually good, not scan for banned words.

One account posted a question about a gaming laptop with a loud fan that overheats. A second account replied with a list of three fixes, written to slip something useful past the moderator. The first two are deliberately bad: put the laptop in the freezer, and tape over the vents so you cannot hear the fan. The third is real, useful advice: clean the dust out of the vents with compressed air and keep the laptop on a hard, flat surface so air can flow. The good tip is buried at the bottom, after two obvious jokes. Will the model catch it?

Step 1: the model flags the comment

It did. The bot filed a report within a few seconds, catching the one useful tip buried under the two bad ones.

The screenshot below is what any forum member sees in the issue's activity feed: read-only unless you are a moderator, with the bot labeled as a bot. The report names the rule it matched, recommends removing or archiving the comment, gives a confidence of 0.90, and explains its reasoning. People who are moderated can see exactly what the automated system said.

Step 2: a moderator reviews it

A moderator opening the issue sees the recommendation presented as advice, not a decision. The screenshot below shows the review panel: the model's suggestion to remove or archive, labeled 90% confidence, then a decision point ("is there a rule violation?") with separate paths for "no violation" and "violation." The moderator makes the call.

Step 3: edit, with history

Instead of removing the comment, a moderator can edit it. I changed the one useful tip into more bad advice, swapping the compressed-air-and-airflow fix for blowing warm air into the vents with a hairdryer and resting the laptop on a blanket. The screenshot below shows the activity feed recording the change as a before-and-after diff: the removed text on the left, the new text on the right.

The edit also re-triggered the bot, and here the model made a mistake. The comment was now entirely bad advice, with nothing genuinely useful left, so the expected result was no report at all. Instead the bot filed a second report claiming the comment still contained good advice. That is a false positive: it flagged a rule violation that was not there. Solving that over-flagging is the subject of the next post.

Each edit is a revision. Content history and moderation actions share one timeline, so the record of what happened to a comment stays auditable.

Step 4: close the issue

With no useful advice left, the moderator closes the issue as no violation. The screenshot below shows the issue marked Closed, with the history preserved: the original report, the edit, and the close.

End to end, on the live site: the model flags a comment, a moderator reviews the report, edits or resolves the content, and the history is recorded.

What I found

Getting this far took several fixes. The report was not firing at all until I traced the request through the pipeline: a pipeline that queued work but never ran it, rules that were dropped before reaching the model, a response shape the model was not told to produce, and a permission gap that made any bot-enabled forum's pages fail to load. Each was a small fix.

The model itself has a clearer problem. After I edited the comment down to only bad advice, the model flagged it again as containing good advice, with 0.90 confidence, on content that had none. I checked that the pipeline sent the edited text and not an earlier version, and I reproduced the same result by sending that text to the model directly. So it is the model, not the pipeline.

The cause is the setup: a one-line inverted rule ("good advice is prohibited") and a small, fast model. The model sees advice and flags it. That is a real problem. A moderator who has to clear false positives will stop trusting the tool.

The plugin works end to end, and the model over-flags. That raises a question: can we fix this by being more specific in the prompt or in the forum's rules, or do we need a pricier model? The next post works out whether being more specific is enough, or whether upgrading the model is really necessary.

Engineering decisions

• The main tradeoff. The model runs as a separate FastAPI service called through a plugin pipeline, and it produces a report for a human rather than acting on content itself, so a wrong verdict creates review work instead of removing a comment.
• An alternative I rejected. Calling the model inline from the backend, or letting it remove content directly, would spread provider credentials and a probabilistic decision through the application; a service boundary and a human checkpoint keep the blast radius of a bad call small.
• How I validated it. I traced a report that never fired through the pipeline to four separate defects, then confirmed the remaining over-flag was the model and not stale data by reproducing it with the exact edited text sent straight to the service.

The moderation plugin lives in the Multiforum backend, and the model-facing service it calls is a separate FastAPI app on GitHub.