Compute Variants

One version of a model or environment can ship several container images, called builds or variants: a CUDA build and a CPU fallback, a build for older drivers and one for newer GPUs. RLMesh picks the build that fits the hardware an evaluation lands on.

Who needs what:

• You push a single image. Nothing new. Push it as in Upload a Model or Environment; the platform reads what it needs from the image’s CUDA or ROCm markers, or from its requires.
• You run evaluations. Nothing. Selection is automatic, and a request can pin a build if you want one.
• You publish two or more builds of one version. Read the six concepts below. Everything after them is Advanced and optional.

The six concepts

1. Build key. Each build is a [variant.KEY] table in rlmesh.toml (cuda12, cpu).
2. Requirements. Each build says what hardware it needs with accel.* keys.
3. Constraint syntax. Versions are constraints such as ">=12.4"; VRAM is a quantity such as "16Gi".
4. One version is one OCI index. rlmesh registry publish builds each variant, then publishes them together as one image index under one tag.
5. A default build with fallback. The first build is the default. The others are fallbacks for hardware the default cannot run on.
6. Check, publish, promote. Dry-run the publish, publish the version, then move latest onto it.

This page publishes pi0:v3 with two builds: cuda12, the default, for NVIDIA GPUs with at least 16Gi of VRAM, and cpu, which runs anywhere. The whole declaration is one file next to your Dockerfiles, which stay as they are:

# rlmesh.toml
name = "pi0"

[variant.cuda12]
dockerfile = "Dockerfile.cuda"
accel.vendor = "nvidia"
accel.cuda = ">=12.4"
accel.vram = "16Gi"

[variant.cpu]
dockerfile = "Dockerfile.cpu"

1. Build key

Each [variant.KEY] table is one build, and its name is the build’s key. A key is 1 to 32 lowercase letters, digits, or dashes, starting with a letter or digit, and unique within the version. Reports and pins name the build by it.

2. Requirements

accel.* keys list what the hardware must offer. A build without them runs anywhere, unless the image’s CUDA or ROCm markers say otherwise. They use the same names as the dashboard and the API:

[variant.cuda12]
accel.vendor = "nvidia"
accel.compute = ">=8.0"
accel.cuda = ">=12.4"
accel.vram = "16Gi"

3. Constraint syntax

A version constraint is one or more comma-joined clauses that must all hold. Each clause is an optional >=, >, <=, <, ==, or = followed by a dotted version of up to three parts. A bare version is a minimum, so "12.4" means ">=12.4", and ">=8.0,<10.0" is a range. Versions compare numerically part by part, so 12 equals 12.0.

accel.vram is a quantity, written the way Kubernetes writes memory: a number with an optional suffix, Ki, Mi, Gi, Ti (powers of 1024) or K, M, G, T (powers of 1000). "16Gi", "24G", and "1.5Gi" are quantities. It is always a minimum, so it takes no comparator (">=24Gi" fails), and it must be a string: 24, "24GB", and "24 Gi" fail. A value the platform cannot read costs the build its whole variant declaration, which is why publish checks every value before building.

4. One version is one OCI index

IMAGE=registry.rlmesh.dev/your-namespace/pi0
rlmesh registry publish $IMAGE:v3

With no images named, publish reads rlmesh.toml in the current directory and builds each variant for linux/amd64 from its dockerfile. It pushes each build as pi0:v3-<key> with the declaration baked into its dev.rlmesh.package label, then assembles the builds into one OCI image index, the version, tagged pi0:v3. Its children are the builds; the per-build tags are only how they are handed over.

5. A default build with fallback

The first [variant.*] table is the version’s default. Put the build you want wherever it can run first, and add the others for hardware it cannot run on. Here an NVIDIA node with 16Gi of VRAM or more runs cuda12, and a CPU-only node, or a GPU with less VRAM, falls back to cpu. Where both fit, RLMesh prefers the default, so a fallback runs only where the default cannot.

6. Check, publish, promote

Check the declaration before building anything. --dry-run validates rlmesh.toml by the platform’s rules, reporting every problem at once, and prints each build, its requirements, and the build commands it would run:

rlmesh registry publish --dry-run $IMAGE:v3

Publish the version. Run rlmesh registry login first, as for docker push. A v3 that already holds a version is refused before anything is built; publish under a new tag. After building, each image is checked the way rlmesh check-image checks it, with a warning where the declaration contradicts the image, such as an accel.cuda older than the image’s CUDA runtime.

RLMesh probes the version’s default build on each kind of hardware it fits. The other builds are probed when you ask: press Probe on the image page’s compatibility card.

Promote the version once its probes look right, by publishing the same builds, named by their per-build tags, with --channel latest:

rlmesh registry publish --channel latest $IMAGE:v3 $IMAGE:v3-cuda12 $IMAGE:v3-cpu

pi0:v3 already holds this index, so it is left as is and only latest moves. As with a single image, latest selects the version the dashboard and an untagged request use. The summary prints the digest latest moved from, which is what you point it back at to roll back. Passing --channel latest on the first publish promotes immediately.

How RLMesh picks a build

RLMesh probes builds on the hardware classes they fit and records where each one passed. For an evaluation and the hardware it lands on, it picks a build in this order:

1. An explicit pin wins. A request that names a build gets that one.
2. Drop incompatible and failed builds: those whose requires the hardware does not meet, and those whose probe failed on it.
3. Prefer verified builds, those whose probe passed on this hardware, over ones not yet probed there.
4. Prefer the default build.
5. Fall back to a stable order, so the same request on the same hardware always gets the same build.

Most evaluations never pin. To compare builds or reproduce a run on one, pin a model’s build with models[].variant or an environment’s with tasks[].variant:

{
  "models": [{ "model": "your-namespace/pi0:v3", "variant": "cpu" }],
  "tasks": [{ "taskId": "libero_goal/0", "env": "prebuilt/libero", "episodes": 5 }]
}

A pin that the cluster cannot run is refused with the reasons, never swapped for another build. The review step of New evaluation, the evaluation’s Configuration tab, and the resolutions in rlmesh eval submit --preview --json show which build each model and environment resolved to, and why.

GPU rendering runs only when you ask for it. An environment that renders with osmesa on CPU by default and with EGL on a GPU runs osmesa unless a task pins "variant": "egl", on a cluster that offers GPU rendering.

Advanced

Nothing here is needed for the two-build path above. The full rlmesh.toml and label reference is Publishing compute variants in the rlmesh repository.

More of rlmesh.toml

The top level of rlmesh.toml is the image’s dev.rlmesh.package label written in TOML, under the label’s own key names (name, description, [[checkpoints]], envVars, resources, …). Three build keys sit beside them, and a variant may set its own: dockerfile (default Dockerfile), context (default the file’s directory), and build_args.NAME, which a variant adds to key by key, so one Dockerfile can serve every build:

dockerfile = "Dockerfile"

[[checkpoints]]
name = "base"
uri = "hf://model/acme/pi0"
default = true

[variant.cuda12]
build_args.TORCH_INDEX = "https://download.pytorch.org/whl/cu126"
accel.vendor = "nvidia"
accel.compute = ">=8.0,<10.0"

[variant.cuda12-sm100]
build_args.TORCH_INDEX = "https://download.pytorch.org/whl/cu129"
accel.vendor = "nvidia"
accel.compute = ">=10.0"

A [variant.KEY.package] table overrides package keys for one build, such as tags or resources. A file with no [variant.*] table is a single build, with its accel.* keys at the top level. rlmesh registry publish --dry-run --json prints every build’s label exactly as it will be pushed.

The label by hand

rlmesh.toml is one way to write the label; the label is what RLMesh reads. A build pipeline of your own can set it directly, as JSON, and hand publish the pushed builds:

ARG RLMESH_PACKAGE
LABEL dev.rlmesh.package=${RLMESH_PACKAGE}

docker buildx build --platform linux/amd64 --push -t $IMAGE:v3-cuda12 \
  --build-arg RLMESH_PACKAGE="$(jq -c . variants/cuda12.json)" -f Dockerfile.cuda .
docker buildx build --platform linux/amd64 --push -t $IMAGE:v3-cpu \
  --build-arg RLMESH_PACKAGE="$(jq -c . variants/cpu.json)" -f Dockerfile.cpu .
rlmesh registry publish $IMAGE:v3 $IMAGE:v3-cuda12 $IMAGE:v3-cpu

Each label carries a variant block, {"key": "cuda12", "requires": {"accel.vendor": "nvidia", ...}}, and the first build named is the default. Check each build with rlmesh check-image after building it with --load.

Facets

facets says what a build is built on: framework (torch, jax, …), accel (cpu, cuda, or rocm), and render (osmesa, egl, or none). A checkpoint whose requires names a facet, such as {"framework": "torch"}, runs only on builds that have it. facets.accel must agree with accel.vendor.

Priority

A variant’s priority is an integer from -1000 to 1000, default 0. It only orders builds the default leaves tied: among compatible builds that are not the default, a higher priority is preferred. A default plus fallbacks never needs it.

AMD and ROCm

An AMD build sets accel.vendor = "amd" and lists its GPU targets in accel.gfx instead of the NVIDIA keys:

[variant.rocm6]
accel.vendor = "amd"
accel.gfx = ["gfx942", "gfx90a"]

Rendering profiles

One image can run several ways that differ only in environment variables, GPU count, and requirements. Each is a [profile.KEY] table, shared by every build:

[variant.mujoco]
facets.accel = "cpu"

[profile.osmesa]
default = true
facets.render = "osmesa"
envVars.MUJOCO_GL = "osmesa"

[profile.egl]
facets.render = "egl"
envVars.MUJOCO_GL = "egl"
gpu.count = 1
accel.vendor = "nvidia"

A pin names a build and profile together: mujoco-egl here, or the bare egl for an image with profiles but no variant block. Setting MUJOCO_GL=egl in a task’s envVars does not request a GPU; the preview warns and names the profile to pin.

Placement preferences

managed.accelerator prefers a vendor (nvidia, amd) or a hardware class (nvidia/l40s) when ranking builds; it is a preference and never refuses one. managed.requireVerified: true admits only builds whose probe passed on the target cluster’s hardware.

Version-level package data

Data that applies to every build, such as name, description, and checkpoints, is also set on the version’s index, as its dev.rlmesh.package annotation, and wins over any build’s label. Publishing from rlmesh.toml sets it from the file’s top level. Publishing builds you pushed yourself, rlmesh registry publish --index-package version.json sets it; that needs OCI image sources, which docker buildx build --push produces.

Other publish flags

--tag TAG adds another fixed tag for the version, --force overwrites a version tag that already points at a different index, and --config FILE builds from another manifest than ./rlmesh.toml. --channel tags move without --force.