Validate workflows before production

Overview

ModelRiver provides multiple testing environments—test mode, production playground, and test mode playground—each tagged with unique seed_batch identifiers. Request Logs let you filter and review requests from each environment independently, ensuring test traffic doesn't obscure production monitoring while still giving you full observability into test results.

Testing environments

Test mode

What it is: Workflows configured in test mode use sample data from structured outputs instead of calling actual AI providers.

Seed batch format: "test_mode:{channel_id}"

When to use:

• Validating workflow configuration without consuming provider credits
• Running integration tests in CI/CD pipelines
• Development environment testing
• Verifying structured output schemas return correct sample data

What's captured in logs:

• Complete request/response data (using sample data)
• Token counts (simulated)
• Duration (processing time without actual provider call)
• No actual provider charges incurred

Playground (Production)

What it is: The console's playground feature using production workflows, which makes actual calls to AI providers.

Seed batch format: "pg:{channel_id}"

When to use:

• Validating workflow changes with real provider responses
• Ad-hoc testing of specific prompts or configurations
• Verifying structured outputs parse correctly with real AI responses
• Pre-deployment validation

What's captured in logs:

• Real request/response data from actual provider calls
• Actual token usage and duration
• Real provider charges apply
• Webhook deliveries and callbacks (if applicable)

Playground (Test mode)

What it is: The console's playground feature using test mode workflows with sample data.

Seed batch format: "pg_test_mode:{channel_id}"

When to use:

• Quick workflow validation without provider charges
• Testing structured output schemas
• Demonstrating workflows without consuming credits

What's captured in logs:

• Simulated request/response data
• No actual provider charges

Callback logs

Seed batch formats: "callback:{channel_id}" or "pg_callback:{channel_id}"

Backend callback log entries for backend pipeline workflows. These are automatically created when your backend calls the callback URL.

Filtering test vs. production logs

Available filters

The Source filter group in Request Logs separates these environments:

With no filters selected, you see every request. Selecting several chips in one group shows requests that match any of them, so playground runs of a test mode workflow appear under both Playground and Test. Endpoint and Revision filters are also available; see Using filters.

Why filtering matters

• Production monitoring: Filter Source to Production to monitor traffic from your applications without playground noise
• Test validation: Filter Source to Test to review CI/CD results
• Pre-deployment review: Filter Source to Playground to validate changes
• Complete picture: Clear all filters when you need a comprehensive view

Testing workflow validation

Validating workflow changes

Before deploying workflow changes to production:

1. Test in playground (test mode) – Verify the workflow structure is correct using sample data
2. Test in playground (production) – Make a real provider call to verify the response
3. Review logs for both tests – Filter Source to Playground
4. Check request bodies – Verify the correct prompt, system message, and parameters are sent
5. Check response bodies – Verify the AI response meets expectations
6. Check structured output parsing – If using structured outputs, verify the response matches the schema

Validating structured outputs

Structured outputs require testing both schema validation and actual provider compliance:

1. Test mode – Verify the sample data matches your schema
2. Production playground – Verify the real provider returns responses matching your schema
3. Review response bodies – Check that all required fields are present and correctly typed
4. Test edge cases – Try different prompts to ensure the schema handles various inputs

Validating async workflows

For async and backend pipeline workflows:

1. Make a playground request with the async workflow
2. Check the timeline for:
   • Main request completed successfully
   • Webhook delivery succeeded
   • Backend callback was received (for backend pipeline)
3. Inspect payloads – Verify webhook payload format and callback response data
4. Check simulated indicators – In playground mode, callbacks may be simulated

Testing best practices

Keep environments clean

• Use test mode for routine testing
• Reserve playground (production) for pre-deployment validation
• Always filter to the right environment when reviewing results

Test before every deployment

• Run playground tests for any workflow changes
• Review request/response bodies in the logs
• Verify structured output compliance
• Test async webhook delivery if applicable

Use seed_batch for correlation

• Each test creates a unique channel_id in the seed_batch
• Use this to find all related logs for a specific test
• Channel IDs link requests, webhooks, and callbacks together

Compare test vs. production

• After deploying, compare playground test results with initial production requests
• Verify that production requests match the expected patterns from testing
• Catch any environment-specific differences early

Next steps

• Debugging – Investigate test failures
• Compliance – Separating test and production for audits
• Cost Analysis – Understanding test vs. production costs
• Back to Observability – Return to the overview