CLI and API

Everything the dashboard does with evaluations is also available from the rlmesh CLI, the Python client, and the HTTP API. Install the CLI and sign in first; see Your First Evaluation.

Submit from the CLI

An evaluation request is a JSON file. It names one or more models and one or more tasks. Each task names an environment, a task id, and an episode count.

Save this as request.json. It runs the prebuilt SmolVLA model on one LIBERO scenario:

{
  "models": [{ "model": "prebuilt/smolvla/libero" }],
  "tasks": [{ "taskId": "libero_goal/0", "env": "prebuilt/libero", "episodes": 5 }],
  "tags": { "team": "robotics" }
}

Check it without launching, then submit and wait for the result:

rlmesh eval submit request.json --preview
rlmesh eval submit request.json --wait

--preview validates the request and prints its size and warnings. --wait blocks until the evaluation finishes and exits nonzero unless it completed. - instead of a file name reads the request from stdin.

Your own images

Reference a pushed image by its repository path in your namespace. Without a tag, the request uses the repository’s current version, the one your last :latest push selected. Add :v1 or @sha256:... to pin a version:

{
  "models": [
    {
      "model": "your-namespace/my-model:v1",
      "envVars": { "MY_SETTING": "1" }
    }
  ],
  "tasks": [{ "taskId": "pendulum", "env": "your-namespace/pendulum", "episodes": 10, "seed": 0 }]
}

• taskId is free-form for an environment without scenarios. For an environment with scenarios, such as prebuilt/libero, it names one as <group>/<scenario>, for example libero_goal/0.
• envVars on a model are set on its container. They are stored with the evaluation and shown to everyone in your organization under Raw request.
• The image must have passed its probe. An image that is still probing, or failed, is refused.

Every field is listed in the API reference under EvaluationRequest. The Configuration tab of any evaluation in the dashboard shows its Raw request, which you can save as a starting point.

Watch and read

rlmesh eval list
rlmesh eval get eval_...
rlmesh eval wait eval_...
rlmesh eval cancel eval_...

rlmesh eval get prints the evaluation as JSON, including its status, progress and request.

Submit from Python

rlmesh.platform.Client uses your CLI sign-in, or RLMESH_API_KEY when it is set. It is experimental and may change.

from rlmesh.platform import Client

client = Client()
request = {
    "models": [{"model": "prebuilt/smolvla/libero"}],
    "tasks": [{"taskId": "libero_goal/0", "env": "prebuilt/libero", "episodes": 5}],
}

print(client.preview(request)["estimates"])
evaluation = client.submit(request)
client.wait(evaluation["id"])
print(client.scores(evaluation["id"]))

scores() groups outcomes by task by default. Pass group_by="scenario", "benchmark", "perturbation" or "model" for other rollups.

HTTP API

The API is at https://api.rlmesh.dev/v1. Send an API key or the output of rlmesh token as a bearer token:

curl -H "Authorization: Bearer $(rlmesh token)" https://api.rlmesh.dev/v1/evaluations

The full reference is at api.rlmesh.dev/docs.

CLI commands

Every command takes --help. Commands that talk to the platform take --profile NAME; many take --json for machine-readable output.