Navigation

Your First Evaluation on the Managed Platform

From your invitation to a finished evaluation, then on to your own images.

This page takes you from an invitation to a finished evaluation. You run a prebuilt model against a prebuilt environment first, then bring your own images.

Before you start

You need an invitation to your organization. The invitation gives you a role:

  • Member: runs evaluations, reads results, and pushes images to your organization’s registry namespace.
  • Admin: everything a member can do, plus organization API keys, inviting members, and editing image metadata.

Your evaluations run on the RLMesh-hosted cluster. You do not run any infrastructure. Run pods can reach the internet only at Hugging Face, PyPI and GitHub; see Model Weights and Checkpoints.

1. Sign in

Open the link in your invitation, or go to app.rlmesh.dev and sign in with the same email address. The first page is your organization’s evaluation list. It is empty until you launch something.

2. Create an API key

You need an API key only for scripts, CI, or an MCP client. The dashboard and the CLI sign-in below do not need one.

Open Settings → Your account → Personal API keys, choose Create API key, and pick a preset:

  • Run evaluations: create evaluations and read results.
  • Read only: read evaluations, results and the catalog.

Copy the key when it is shown. Use it from a shell like this:

export RLMESH_API_KEY='<paste your key>'
export RLMESH_PLATFORM_URL='https://api.rlmesh.dev'

To push images, sign in with the CLI and run rlmesh registry login (step 4). Where a browser sign-in is awkward, a personal key with the Run evaluations preset also works as the docker login password, for your organization’s namespace and within your current role. For shared CI pushes, an admin creates an organization API key under Settings → Organization → API keys with the same preset. Docker uses either key as the password (paste it in place of the placeholder); the username is not checked:

echo '<paste the key>' | docker login registry.rlmesh.dev -u rlmesh --password-stdin

3. Install the CLI

The rlmesh Python package contains the CLI. Python 3.10 or newer is required.

pip install rlmesh

If you install into a virtual environment, Docker can only push while that environment is active, because Docker calls the docker-credential-rlmesh helper from your PATH. A tool install avoids that:

pipx install rlmesh
# or
uv tool install rlmesh

4. Sign in from the CLI

rlmesh login
rlmesh whoami

rlmesh login opens a browser to approve the sign-in. rlmesh whoami shows your organization and exits nonzero if you are not signed in. If you belong to more than one organization, list them with rlmesh org list and pick one with rlmesh org switch org_....

To push images, connect Docker to the registry:

rlmesh registry login

The output ends with your push target, for example registry.rlmesh.dev/your-namespace. The same namespace is shown on the Registry page.

5. Run a prebuilt pair

Start with a model and environment that are already in the catalog. This checks your access before you build anything.

  1. Open New evaluation and choose Start from scratch.
  2. Environments: pick libero, then one scenario, for example from libero_goal.
  3. Models: pick smolvla-libero.
  4. Perturbations: skip this step.
  5. Review & launch: open the run settings and set Episodes per task to a small number, such as 5. Then choose Launch evaluation.

The model needs a GPU. If no GPU node is free, the cluster starts one, and the evaluation can wait a few minutes in a queued or provisioning state before episodes begin. This is normal.

6. Read the results

The evaluation page has five tabs:

  • Overview: status, progress, and the per-scenario results table with the Success rate.
  • Episodes: one row per episode with its outcome. Open an episode to watch its recording and use Download trajectory. Recording is on by default.
  • Logs: the evaluation’s log lines, searchable. Start here when an evaluation fails.
  • Telemetry: throughput and timing.
  • Configuration: the settings the evaluation ran with, including Raw request, the exact request body.

Each episode’s files (recordings, trajectories, and other artifacts) are listed in its files dialog. To share an evaluation, send its URL to another member of your organization.

From the CLI, rlmesh eval list lists your evaluations and rlmesh eval get eval_... prints one as JSON. See CLI and API.

7. Bring your own images

Once the prebuilt pair works, package your own model or environment:

  1. Follow Upload a Model or Environment to build, check and push an image. RLMesh probes it after the push.
  2. Read Model Weights and Checkpoints before you package a model. Baking the weights into the image is the most reliable option.
  3. When the probe is ready, pick the image in New evaluation, or submit a request from the CLI.

If anything on this page does not match what you see, contact us at rlmesh.dev/contact.