Documentation

Early access

Testing hosted tools

Check the endpoint with a signed test call, check the model's choices with mock results, then read the real thing in the Timeline.

There are two things to test, and a tool for each:

  • Does my endpoint work? Use Send test call. It makes one real, signed call to your endpoint.
  • Does the model use my tools well? Use the Playground. It runs the whole loop with mock results and never calls your endpoints.

Send test call

On the workflow's Tools tab, open a tool's ⋯ menu and select Send test call.

  1. Choose where it goes: Endpoint URL, or My CLI to call your local server through a running modelriver tools session. My CLI is chosen for you while you have a session connected.
  2. Edit the Arguments if you like. They start from an example built from the tool's schema.
  3. Set End user id to a user that exists in your test data. It is sent as end_user.id.
  4. Select Send test call.

ModelRiver calls the endpoint exactly as a request would: signed with the project's secret, with the same address checks, timeout and size limit. A tool that only reads data is retried once after a 5xx or a dropped connection, as in a request; one that changes something never is. The body has "mode": "test" in request, so your endpoint can tell a test from live traffic.

You see whether it worked, the HTTP status and duration, the result the model would read, and the headers and body that were sent. The signature value itself is hidden: together with the body it would be a request your endpoint accepts for five minutes. To check your verifier against a known signature, use the test value.

Your production endpoint should refuse calls with "mode": "test" unless you expect them, so a test call can never act on live data.

Things to know:

  • The tool is taken from the form, so you can test a tool, or a change to one, before saving.
  • The arguments are checked against the tool's schema first, as they are for the model, and can be up to 16 KB.
  • The project needs a signing secret.
  • Organization owners and admins can send test calls, up to 20 a minute each and 300 an hour per organization. Each one is recorded in the audit log, with the endpoint's host.
  • A test call is not a request on your plan.
  • With My CLI, the call goes to your own connected session only (never a teammate's), at the same path as the tool's endpoint URL on your local server, and is never retried. Its host is never called. See Testing a local endpoint.
  • For a tool that changes something, a test call doesn't ask for approval, so you can try the endpoint before writing the approval prompt. It really calls the endpoint, once, and is never retried, so point it at test data.

Playground

Select the workflow in the Playground and turn on Async mode. A run then goes through the tool loop: the model sees your tools, chooses, gets results and answers.

The Playground never calls the endpoints saved on your tools. Each tool returns a mock result instead, unless you send the calls to your own machine with the CLI:

  • By default, a tool echoes the arguments the model sent, so you can see what it asked for.
  • To say what a tool returns, add tool_mocks to the payload, keyed by tool name:
JSON
1{
2 "messages": [
3 { "role": "user", "content": "How did we do this week, and what sold best?" }
4 ],
5 "tool_mocks": {
6 "get_sales_summary": { "period": "this_week", "revenue": 48200, "orders": 912, "change_vs_previous": 0.12 },
7 "get_top_products": { "products": [{ "name": "Desk lamp", "units": 310 }] }
8 }
9}

The notice under the workflow has a link that adds a tool_mocks block with one entry per tool.

This is the quickest way to tune tool descriptions: change a description, run the same question, and see whether the model picks the right tool with the right arguments. The model's arguments are still checked against each tool's schema.

Things to know:

  • Async mode only. Without it, the Playground runs the model without your tools, and says so.
  • Playground runs don't need end_user, and are not counted as requests.
  • The Run choice (Auto, Draft, Live or Historical) decides which revision's tools run, so you can try a draft's tools before publishing.
  • A workflow in test mode returns its sample data and runs no tools at all.

Approvals in the Playground

A call to a tool that changes something waits for approval in the Playground too, as it would for the end user. The run pauses, and a card under it shows the tool's approval prompt and the arguments ticked under Show to the user, with Approve and Decline buttons and the time the wait ends.

  • Approve: once every waiting call of the round is decided, the call gets its mock result (from tool_mocks, or the echoed arguments), and the run carries on. If the run sends its tool calls to your CLI, the approved call is sent to your local endpoint instead, once, never retried.
  • Decline: the call is not run, and the model is told the user declined it.
  • No answer: after the workflow's Approval wait, the call expires and the model is told nobody answered.

Use it to check what the end user will see: whether the prompt reads well, and whether the shown arguments are enough to decide. Things to know:

  • Playground calls can be decided only in the console, by people who can run the Playground on the project. The public approve and reject endpoints answer 404 for them.
  • A Playground run sends no tool.approval_required webhook.
  • If you reload the page while a call waits, the card comes back.
  • In Request Logs, the decision shows as Approved in the Playground or Declined in the Playground.

Testing a local endpoint

ModelRiver calls tool endpoints from its own servers, so a saved Endpoint URL has to be a public HTTPS address. http://localhost:3000 is refused when you save the tool.

To try an endpoint running on your machine, let the ModelRiver CLI carry the calls to it:

Bash
modelriver tools --forward-to http://localhost:3000

The CLI prints a session id. While it runs:

  • In the Playground, turn on Send tool calls to my CLI under the workflow (Async mode).
  • From your own backend, add the session to the request:
JSON
1{
2 "workflow": "sales_chat",
3 "messages": [{ "role": "user", "content": "How did we do this week?" }],
4 "end_user": { "id": "user_812" },
5 "tool_route": { "cli_session": "cts_…" }
6}

Each tool call is built and signed exactly as it would be for the real endpoint, and the CLI posts it to the same path on your machine: a tool saved as https://api.example.com/tools/sales is called at http://localhost:3000/tools/sales. Your endpoint's answer goes back to the model. The body has "via": "cli" in request.

Things to know:

  • The CLI only forwards to your own machine (localhost, 127.0.0.1 or ::1).
  • A session lasts 24 hours and belongs to the project of the API key the CLI uses, and to that key's owner. A request can only name a session of its own project. The Playground offers the sessions opened with API keys you created, so share keys with care.
  • Starting modelriver tools again with the same API key and --forward-to rejoins the session it had, so a request that names it keeps working after a restart. Up to 5 sessions can be connected at once per person and project; opening another reuses the room of ones that are no longer connected.
  • If the CLI isn't running, a request that names its session is refused (409 cli_session_not_connected). If it stops part-way, a call already sent to it times out and the next ones fail at once as unavailable. Calls are never sent to the saved endpoint instead.
  • A run through the CLI waits on your machine, so it counts toward the project's limit on hosted-tools requests running at once, from the Playground too.
  • Requests that use the CLI are normal requests: they count toward your plan, and the response's meta.tool_loop has "via": "cli". In the Timeline, each call is marked via CLI.
  • Remove tool_route before you deploy. A production request should never name a CLI session, and your production endpoint should refuse calls with "via": "cli" (see Security).

If you'd rather expose the endpoint itself, a tunnel such as cloudflared tunnel --url http://localhost:3000 or ngrok http 3000 gives it a public HTTPS address to save on the tool. Keep signature verification switched on: the tunnel address is public while it is up.

A separate project for development, with its own signing secret and tools that point at a staging backend, keeps test traffic away from production.

Reading a request in the Timeline

Open Request Logs and select a request. For a request with hosted tools, the Timeline shows:

  • Each model round, labelled with what the model did: asked for tools, or answered.
  • Under each round, every tool call: the tool, the result (ok, error, timeout, or unknown for a call to a tool that changes something), the HTTP status, the duration and the number of attempts.
  • For a call to a tool that changes something, its approval and when: approved by the end user, declined, no answer in time, or not run because the request was stopped. While the request waits, the round that paused shows on its own with Waiting for approval, until the time the wait ends.
  • For a failed call, the reason: for example that the arguments didn't match the schema, or the endpoint returned 503.
  • With Save full request and response text on for the project, each call's arguments and result, under Arguments and result.
  • Calls answered through the CLI, marked via CLI.

The request list shows how many tool rounds each request took, and the cost panel adds up the rounds. Calls a Playground run answered with a mock result are labelled Mock.

A checklist before going live

  • The endpoint rejects a call with a wrong signature and one with an old timestamp.
  • The endpoint reads who is asking from end_user, and returns nothing for a user it doesn't know.
  • Every tool's schema has "additionalProperties": false.
  • Results are small: only the fields the model needs.
  • The endpoint answers well inside its timeout under load.
  • The frontend shows progress, handles error.type, and has a Stop button.
  • For each tool that changes something: the approval prompt says what will happen, the shown arguments are enough to decide, your approve route takes end_user.id from the session, and the endpoint handles a repeated call id without doing the action twice.

Next steps