Navigation

Bring your own cluster

Connect your Kubernetes cluster to RLMesh.

Evaluate on Kubernetes you operate. The agent connects outward; no kubeconfig, exposed Kubernetes API, or inbound ports are needed. The chart is public at registry.rlmesh.dev; a one-time join token gets the cluster its own credentials for the runtime images.

Prerequisites

  • Kubernetes 1.34+, Linux amd64 nodes, and enough CPU and memory for your models and environments. Cluster-admin rights to create CRDs, roles, and namespaces. One runtime installation per cluster.
  • Helm with OCI registry support.
  • A CNI that enforces Kubernetes NetworkPolicy, such as Cilium or Calico.
  • A default dynamic StorageClass for a 100 GiB ReadWriteOnce cache and 10 GiB agent spool. For multiple nodes, set storage.className to an RWX class and artifactCache.storage.sharedPvc.accessModes: [ReadWriteMany].
  • For GPUs, install drivers and the vendor device plugin or operator. NVIDIA nodes need nvidia.com/gpu; AMD nodes need amd.com/gpu and AMD hardware labels. CPU-only clusters can disable gpuFeatureDiscovery.enabled.
  • Outbound TCP 443 to api.rlmesh.dev and registry.rlmesh.dev, DNS, and access to your Kubernetes API. /v1/info must advertise gRPC on the API host over TLS/443.

The egress proxy defaults to CoreDNS (kube-system, k8s-app=kube-dns). Set network.dnsPeers to your resolver’s NetworkPolicy peers before installing. For GKE NodeLocal DNSCache, use the resolver IP from a pod’s /etc/resolv.conf (replace the example IP):

network:
  dnsPeers:
    - ipBlock: { cidr: 169.254.20.10/32 }

For OpenShift DNS, select its namespace and pods and allow its backend port (Red Hat DNS troubleshooting):

network:
  dnsPeers:
    - namespaceSelector:
        matchLabels: { kubernetes.io/metadata.name: openshift-dns }
      podSelector:
        matchLabels: { dns.operator.openshift.io/daemonset-dns: default }
  dnsPorts: [53, 5353]

Check your cluster’s actual resolver labels, IP, and post-DNAT port. Only network.dnsPorts (default UDP/TCP 53) are allowed to these peers; private and link-local HTTP(S) destinations remain blocked.

Allow checkpoint and artifact endpoints through your firewall; the egress proxy cannot bypass it. With only API and registry access, use evaluations without external checkpoints and arrange artifact access through the control plane. Never put platform bucket keys in your values file.

Install

With clusters:write, open Clusters → Connect your cluster (or New install command on an existing cluster). It shows one command with a join token (rlm_jt_..., single use, valid for an hour):

helm install rlmesh oci://registry.rlmesh.dev/rlmesh/charts/rlmesh-runtime \
  --version <v> -n rlmesh-system --create-namespace --set joinToken=<rlm_jt_>

The agent exchanges the token for the cluster’s control and pull credentials and writes its own Secrets, so nothing secret goes into a values file. The chart’s defaults are a production cell: the API, the registry and the platform’s output hosts are built in. The dashboard also offers a variant that reads the token at a hidden prompt, keeping it out of shell history.

Add hosts evaluations may reach (your input buckets, model hubs) without restating the defaults:

egress:
  extraDomains:
    - my-inputs.s3.eu-west-1.amazonaws.com

GCS uses storage.googleapis.com, which the proxy cannot restrict to one bucket. The public proxy rejects private addresses; local MinIO needs an in-cluster route.

Bring your own Secrets

With GitOps or a secret manager (ESO, Vault, Sealed Secrets), claim the token out of band and deliver the Secrets yourself:

rlmesh clusters join --token <rlm_jt_> --format k8s-secret --out rlmesh-secrets.yaml

Then install without joinToken, naming them:

credentials:
  controlSecret: rlmesh-secrets # keys control and cluster-id, in rlmesh-system
  pullSecret: rlmesh-registry-pull # dockerconfigjson in rlmesh-system and rlmesh-runs

The chart’s full value reference is in its values.yaml and README.

Verify

kubectl -n rlmesh-system rollout status deployment/rlmesh-cluster-agent
kubectl -n rlmesh-system rollout status deployment/rlmesh-cluster-controller
kubectl get rlmeshcaches
kubectl -n rlmesh-runs get pods,pvc

Check that the dashboard shows Connected and lastSeenAt advances in GET /v1/clusters. Submit a small evaluation on this cluster; check its pods in rlmesh-runs and results in the dashboard.

  • Pending PVC: check the default StorageClass and access modes.
  • ImagePullBackOff: check registry access; with your own Secrets, the pull Secret in both namespaces.
  • Agent authentication error: check the agent’s log; a used or expired token needs a new install command.

For optional observability, save this as observability-values.yaml, replacing <cluster id> with the registered public ID. Grafana is disabled here; to turn it on, supply grafana.adminPassword or grafana.adminPasswordSecretName. The runtime installation supplies both Secrets in rlmesh-system (install it with --set observability.enabled=true so the join also claims the relay credential).

imagePullSecrets:
  - name: rlmesh-registry-pull
grafana:
  enabled: false
prometheus:
  clusterLabel: <cluster id>
  relayCredentialSecretName: rlmesh-cluster-relay-secret
  remoteWrite:
    - url: https://api.rlmesh.dev/v1/cells/<cluster id>/metrics
      authorization:
        type: Bearer
        credentials_file: /etc/rlmesh/relay/relay-secret

Use the same chart version as the runtime. Staging substitutes its API and registry hosts. Prometheus rereads the relay credential file on each request.

helm install rlmesh-observability oci://registry.rlmesh.dev/rlmesh/charts/rlmesh-observability --version <v> -n rlmesh-system -f observability-values.yaml

Rotate credentials and upgrade

Upgrade with helm upgrade rlmesh oci://registry.rlmesh.dev/rlmesh/charts/rlmesh-runtime --version <v> -n rlmesh-system --reuse-values; the agent keeps its credentials. To replace them, issue New install command and run it as helm upgrade --install: the agent rejoins with the new token. With your own Secrets, rotate a purpose in the cluster’s Settings and write the new value into your Secret within the 24-hour overlap; the agent follows its control Secret without a restart. Upgrades reapply CRDs through a pinned kubectl hook.

Uninstall

Drain the cluster, wait for evaluations to finish, then remove the runtime:

helm uninstall rlmesh -n rlmesh-system

Archive the registration to revoke all credentials. Uninstall retains rlmesh-runs and its artifact-cache PVC; delete them only when their data is no longer needed. It removes the agent spool PVC. CRDs remain; deleting them deletes their custom resources, so check they are unused first. Uninstall observability separately.

After uninstall, remove the retained upgrade-hook pull Secret:

kubectl -n rlmesh-system delete secret rlmesh-registry-join-crd-upgrade --ignore-not-found