Trust & safety

2 Workflows2 Response Formats3 Backend Steps

Content Moderation Classifier

Reviews user-generated content against your policy and records a clear decision - approve, flag, or block.

Works with

Primary
Backup

Pick your providers and models. The download is customized to your selections.

Best for

  • UGC platforms
  • Community managers
  • No-code app builders
Download JSONSee the integration guide

How the data flows

Requests move between your backend and the AI workflows automatically. Here's the exact path a request takes.

Customer request arrives

AI workflow

content_moderation_workflow

Reviews content and starts the decision pipeline

Backend event

content_submitted

apply_decision fires, then starts moderation_decision_workflow

Backend event

decision_recorded

AI workflow

moderation_decision_workflow

Records the final moderation action

Customer receives the response

What you get

Every template is built from two kinds of pieces: Workflows and Response Formats.

Workflow

content_moderation_workflow

Reviews content and starts the decision pipeline

Workflow

moderation_decision_workflow

Records the final moderation action

Response format

moderation_triage_response

Initial moderation verdict on the submitted content

verdictrisk_categoryexplanation

Response format

moderation_decision_response

Final decision and action taken

final_decisionaction_takennote

Details

The full specification of each workflow, with plain-English explanations. Everything below is set in the downloaded file and can be changed after import.

content_moderation_workflow

Reviews content and starts the decision pipeline

Model & backupsLearn more
deepseek · deepseek-v4-flashBackup 1: anthropic · claude-haiku-4-5-20251001

The primary model plus automatic fallback models if the primary is unavailable.

Response formatLearn more
moderation_triage_responseReturns a structured object instead of free-form text.

The structured JSON this workflow returns. Fields are defined by you and enforced by ModelRiver.

Customer data it usesLearn more
content_textcontent_typesubmitter_id

Fields you include in each request so the workflow can personalise its answer.

Memory windowLearn more
5 minutes

Identical requests are answered from memory within this window, which is faster and cheaper.

Automated pipelineLearn more
content_submittedapply_decisionstarts moderation_decision_workflowdecision_recorded
How multi-step pipelines work

Steps that run automatically around this workflow, in order.

ModeLearn more
Production mode

Runs live against your connected AI provider.

moderation_decision_workflow

Records the final moderation action

Model & backupsLearn more
anthropic · claude-haiku-4-5-20251001Backup 1: deepseek · deepseek-v4-flash

The primary model plus automatic fallback models if the primary is unavailable.

Response formatLearn more
moderation_decision_responseReturns a structured object instead of free-form text.

The structured JSON this workflow returns. Fields are defined by you and enforced by ModelRiver.

Customer data it usesLearn more
content_textcontent_typesubmitter_id

Fields you include in each request so the workflow can personalise its answer.

ModeLearn more
Production mode

Runs live against your connected AI provider.

How to use it

  1. 1

    Download the template

    The file is a portable JSON bundle - safe to share and easy to read.

  2. 2

    Open a project in ModelRiver

    Sign in, pick a project, and make sure the AI providers used by this template are connected. You can connect them from the Providers section.

  3. 3

    Import it

    Open the Import button in your project, paste or upload the file, review the preview, and confirm. Everything is created atomically, so nothing is overwritten. How import & export works

Template file

Technical users can inspect the raw bundle below.

View full template JSON
modelriver-content-moderation-template-v1.json3.5 KB
{
  "version": 1,
  "structures": [
    {
      "name": "moderation_triage_response",
      "description": "Initial moderation verdict on the submitted content",
      "example": {
        "verdict": "flag",
        "risk_category": "spam",
        "explanation": "Repeated promotional links with no original content."
      },
      "schema": {
        "type": "object",
        "properties": {
          "verdict": {
            "type": "string",
            "enum": [
              "approve",
              "flag",
              "block"
            ]
          },
          "risk_category": {
            "type": "string"
          },
          "explanation": {
            "type": "string"
          }
        },
        "required": [
          "verdict",
          "risk_category",
          "explanation"
        ],
        "additionalProperties": false
      }
    },
    {
      "name": "moderation_decision_response",
      "description": "Final decision and action taken",
      "example": {
        "final_decision": "flag",
        "action_taken": "hidden pending human review",
        "note": "Referred to a moderator for confirmation."
      },
      "schema": {
        "type": "object",
        "properties": {
          "final_decision": {
            "type": "string",
            "enum": [
              "approve",
              "flag",
              "block"
            ]
          },
          "action_taken": {
            "type": "string"
          },
          "note": {
            "type": "string"
          }
        },
        "required": [
          "final_decision",
          "action_taken",
          "note"
        ],
        "additionalProperties": false
      }
    }
  ],
  "workflows": [
    {
      "name": "content_moderation_workflow",
      "description": "Reviews content and starts the decision pipeline",
      "provider": "deepseek",
      "model": "deepseek-v4-flash",
      "backups": [
        {
          "provider": "anthropic",
          "model": "claude-haiku-4-5-20251001"
        }
      ],
      "request_type": "chat",
      "system_instructions": "Review the submitted content against a typical community policy: approve content that is safe and relevant, flag content that needs a human check, and block clear policy violations. Explain each verdict briefly.",
      "structure": "moderation_triage_response",
      "customer_fields": [
        "content_text",
        "content_type",
        "submitter_id"
      ],
      "test_mode": false,
      "cache_window_seconds": 300,
      "backend_pipeline": {
        "enabled": true,
        "events": [
          {
            "name": "content_submitted"
          },
          {
            "name": "apply_decision",
            "target_workflow": "moderation_decision_workflow"
          },
          {
            "name": "decision_recorded"
          }
        ]
      }
    },
    {
      "name": "moderation_decision_workflow",
      "description": "Records the final moderation action",
      "provider": "deepseek",
      "model": "deepseek-v4-flash",
      "backups": [
        {
          "provider": "anthropic",
          "model": "claude-haiku-4-5-20251001"
        }
      ],
      "request_type": "chat",
      "system_instructions": "Confirm the final moderation decision and describe the action taken, such as approving, hiding pending review, or blocking the content. Keep the note concise for moderators.",
      "structure": "moderation_decision_response",
      "customer_fields": [
        "content_text",
        "content_type",
        "submitter_id"
      ],
      "test_mode": false
    }
  ]
}

Looking for something else?

More templates are on the way. In the meantime, you can build your own workflows from scratch in the ModelRiver console.

Browse all templatesRead the workflow guide