C++

rlmesh.hpp is a header-only C++ layer over the C API. Environments and policies are ordinary classes (or lambdas) that RLMesh inspects at compile time, handles clean up after themselves, options are structs with defaults and chainable setters, and failures are exceptions. It needs C++17 or newer with exceptions enabled (the header stops with an #error otherwise) and is tested at C++17 and C++20.

Install

rlmesh.hpp sits next to rlmesh.h and links the same library, so installation is the same as for C. In short:

git clone https://github.com/ArenaX-Labs/rlmesh && cd rlmesh
RLMESH_RELEASE_BUILD=1 cargo build --release -p rlmesh-capi

c++ -std=c++17 -I crates/rlmesh-capi/include app.cpp \
  -L target/release -lrlmesh_capi -lpthread -ldl -lm -o app

With the CMake package, link the same target as C:

set(CMAKE_CXX_STANDARD 17)
find_package(rlmesh 5 CONFIG REQUIRED)
target_link_libraries(my_sim PRIVATE rlmesh::rlmesh)

See C: Install for the relocatable package, FetchContent, pkg-config, the ABI check, and why RLMESH_RELEASE_BUILD=1 matters when your program talks to other builds. The header has the same shape at every supported standard, so translation units built at C++17 and C++20 can share RLMesh types in one binary.

Quickstart

Two programs: one serves an environment, the other connects a model to it and runs three episodes. They talk over gRPC, so either side could be replaced by a Python, Rust, or C program.

Serve an environment

Any class with observation_space(), action_space(), reset, and step is an environment. Pass it to an EnvServer:

#include <cstdio>
#include <rlmesh.hpp>

// A three-step counter: the observation is the step index, every action
// earns 1.0, and the episode terminates after the third step.
class CounterEnv {
  int64_t step_ = 0;

 public:
  rlmesh::spaces::Space observation_space() const { return rlmesh::spaces::Discrete(4); }
  rlmesh::spaces::Space action_space() const { return rlmesh::spaces::Discrete(2); }
  std::optional<std::string> id() const { return "CounterEnv-v0"; }

  rlmesh::ResetResult reset(std::optional<int64_t> /*seed*/,
                            const std::optional<rlmesh::InfoDict>& /*options*/) {
    step_ = 0;
    return {rlmesh::Value(step_), {}};
  }

  rlmesh::StepResult step(std::optional<rlmesh::Value> /*action*/) {
    ++step_;
    return {rlmesh::Value(step_), 1.0, step_ >= 3, false, {}};
  }
};

int main() try {
  rlmesh::EnvServer server(CounterEnv{},
                           rlmesh::EnvServerOptions{}.with_address("tcp://127.0.0.1:5555"));
  std::printf("serving on %s\n", server.address().c_str());
  server.serve();  // blocks until Ctrl-C or a shutdown, then drains and closes the env
  return 0;
} catch (const rlmesh::ext::CancelledError&) {
  return 0;
} catch (const std::exception& error) {
  std::fprintf(stderr, "%s\n", error.what());
  return 1;
}

ResetResult is std::tuple<Value, InfoDict> and StepResult is std::tuple<Value, double, bool, bool, InfoDict>. serve() runs lane zero on the calling thread; start() serves on library threads, followed by wait() and shutdown().

Run a model against it

Any class with a predict method is a policy. Wrap it in a Model and call run:

#include <cstdio>
#include <rlmesh.hpp>

// Any class with a predict method is a policy.
struct ConstantPolicy {
  rlmesh::Value predict(const rlmesh::Value& /*observation*/) { return rlmesh::Value(0); }
};

int main() try {
  rlmesh::Model model(ConstantPolicy{});
  auto result = model.run("tcp://127.0.0.1:5555",
                          rlmesh::RunOptions{}.with_episodes(3).with_seeds({0, 1, 2}));
  std::printf("%zu episodes, %lld steps, mean reward %.2f\n", result.num_episodes(),
              static_cast<long long>(result.total_steps()), result.mean_reward());
  return 0;
} catch (const std::exception& error) {
  std::fprintf(stderr, "%s\n", error.what());
  return 1;
}

3 episodes, 9 steps, mean reward 3.00

The repository’s crates/rlmesh-capi/examples/ directory has fuller programs: cpp_model.cpp (a model against any environment, using the per-episode context) and cpp_env.cpp (a camera-and-pose environment with adapter tags).

Conventions

Detection instead of base classes. RLMesh reads your environment and policy types at compile time. An environment needs the two space getters, reset, and step; render, close, id, render_mode, and metadata are picked up when present. A policy needs at least one of predict, predict_chunk, predict_batch, or predict_chunk_batch, each with or without a trailing const rlmesh::PredictContext& parameter; load, on_episode_end, close, and static declarations (spec, load keys, workflow edition, chunk length) are optional. A lambda works as a policy too.

Options are structs with setters. Every options type is a struct whose member initializers hold the defaults, so you set only what you need. The portable spelling chains with_<field> setters, each returning the options so they work on temporaries and named objects alike; assigning members directly works too:

auto options = rlmesh::RunOptions{}.with_episodes(3).with_max_episode_steps(500);
options.close_env = true;
auto result = model.run(env, options);

With C++20 you can also write the options inline with designated initializers:

auto result = model.run(env, {.episodes = 3, .max_episode_steps = 500});

Optional values use std::optional. std::nullopt means “not set”, which is distinct from zero. Seed lists use OptionalList<int64_t>, which keeps “no list” separate from an empty list.

Durations are std::chrono. Timeouts and limits are std::chrono::duration<double> (aliased as rlmesh::Duration), so 5s or std::chrono::milliseconds(250) work directly, and their names drop the _seconds suffix the C fields carry: connect_timeout, request_timeout, max_episode.

Borrowed views are rlmesh::Span. Byte buffers, batch observations, and batch contexts are an rlmesh::Span<T> (data(), size(), empty(), operator[], begin() / end()), the same type at C++17 and C++20. It accepts a pointer and a count or any contiguous container: a std::vector, a std::array, a C array, or, in C++20, a std::span, which it also converts back to. A span borrows its storage, so keep that alive while the span is in use.

Results are tuples. reset and step return tuples, so structured bindings unpack them:

auto [observation, info] = env.reset(42);
auto [next, reward, terminated, truncated, step_info] = env.step(rlmesh::Value(1));

Lifetimes are RAII. Handles release their resources in their destructors. Value, Tensor, spaces, Model, RemoteModel, and adapters are cheap to copy: copies share the underlying object (Tensor::copy() copies the data). Environments, servers, remote environments, sessions, and recorders are move-only. EnvServer owns the environment you pass in; pass a std::shared_ptr to keep access to the simulation yourself. Destructors swallow cleanup errors, so call close() or shutdown() explicitly when you want to see them.

Language extras live in rlmesh::ext. ext::CancelToken (copies share one cancellation source), ext::EnvServerOptions and ext::ModelServeOptions (bearer tokens, describe publishing, predict concurrency), ext::env_server_describe_json, ext::register_entrypoint, ext::DlpackPtr, and ext::box_elementwise.

Using RLMesh from Unreal Engine and other hosts

rlmesh.hpp is built to sit inside large host codebases such as game engines and simulators:

• Macro-safe. The header saves and restores common Unreal (check, verify, ensure, …), X11 (None, Bool, Status, …), and Windows (min, max, ERROR, DELETE, …) macros around its own code, so it can be included after the host’s headers. If the host defines an object-like macro such as None or Bool, #undef it in your own file before spelling the matching enumerator (rlmesh::ValueKind::None, rlmesh::DType::Bool).
• No RTTI. It compiles with RTTI disabled, and compiles warning-free under -Wall -Wextra -Wshadow -Wconversion.
• Exceptions per module. The header requires exceptions. In Unreal, enable them for the module that includes it with bEnableExceptions = true; in its .Build.cs; other modules can keep them off.
• Mixed standards. One binary can combine modules built at C++17 and C++20, because the header declares the same types at both. In code that must build at C++17 (MSVC rejects designated initializers there), use the with_* setters.

Environments

• Throw from reset or step to fail a request. Throwing rlmesh::RecoverableEnvironmentException fails just that request and keeps the client’s session; other exceptions end it. An exception thrown by your code is rethrown unchanged when the failure surfaces in the same process.
• Batched simulators provide single_observation_space(), single_action_space(), num_envs(), and vector-shaped reset / step (returning VectorStepResult) and are served the same way. To serve several independent environments as one vector endpoint, pass a std::vector<rlmesh::Env>.
• rlmesh::EnvFactory<F> (a CRTP base) serves families of environments with tag parameters and variant catalogs.
• EnvServerOptions holds address (or host / port / path / transport), adapter tags, options (a ServeOptions with timeouts, workflow_edition, and remote shutdown), close_env_on_shutdown, and ext.

See Serve an Environment for addresses, authentication, and supervision.

Spaces and values

rlmesh::spaces provides Box, Discrete, and Dict, plus experimental MultiBinary, MultiDiscrete, Text, and Tuple. All of them are a Space you can seed, sample, and test with contains.

A Value holds one of ten kinds: None, Bool, Int, Float, Text, Bytes, Tensor, List, Tuple, or Dict. It converts implicitly from bool, integers, floating-point numbers, strings, Tensor, and Metadata (an insertion-ordered map, aliased as InfoDict and Kwargs). Value::list and Value::tuple build sequences. The as_* readers throw rlmesh::TypeError for the wrong kind.

using namespace rlmesh::spaces;
Dict observation({{"position", Box(-1.0, 1.0, {3})},
                  {"mode", Discrete(4, DiscreteOptions{}.with_start(1))}});
observation.seed(7);
auto sample = observation.sample();

std::vector<float> xyz{0.0f, 0.5f, -0.5f};
rlmesh::Value value = rlmesh::Metadata{
    {"position", rlmesh::Tensor(xyz.data(), xyz.size() * sizeof(float), {3},
                                rlmesh::DType::Float32)},
    {"mode", 2},
};
bool inside = observation.contains(value);
int64_t mode = value.as_dict().at("mode").as_int();

Values sent across the boundary must match their space’s structure: a Dict value with a missing or extra key throws rlmesh::ValueError before it is sent ($.position: missing key 'x' names the path), and a received value with one is rejected before it reaches your code. Tensor copies the buffer it is built from. Tensor::to_dlpack and Tensor::from_dlpack exchange CPU tensors with other frameworks. See Spaces and Values.

Models and policies

A policy method that takes a PredictContext receives the episode id, its seed, the predict index, a per-predict seed for reproducible sampling, and an EpisodeState store that is dropped when the episode ends:

// Counts its predicts per episode in that episode's state.
struct Counter {
  rlmesh::Value predict(const rlmesh::Value&, const rlmesh::PredictContext& context) {
    ++context.state.get_or_emplace<uint64_t>("predicts");
    return rlmesh::Value(0);
  }
};

rlmesh::Model model(Counter{});
model.serve("tcp://127.0.0.1:50051");  // blocks until cancelled

ModelOptions sets the adapter spec (or rlmesh::NO_ADAPTER), on_episode_end and on_close hooks, and trust_entrypoints. Model::from_config<Policy>(kwargs) default-constructs the policy and calls its load. rlmesh::describe<Policy>() and describe_json<Policy>() read a policy type’s declarations without constructing it.

A batched policy receives its observations as a rlmesh::Span<const rlmesh::Value> and returns one action per observation:

struct Batched {
  std::vector<rlmesh::Value> predict_batch(rlmesh::Span<const rlmesh::Value> observations) {
    return std::vector<rlmesh::Value>(observations.size(), rlmesh::Value(0));
  }
};

See Build a Model.

Runs and sessions

rlmesh::run (or Model::run) plays whole episodes and returns a RunResult with one EpisodeResult per episode plus telemetry. The environment can be an address string, an EnvServer, a RemoteEnv, a local Env, or a factory; the model can be a Model, a RemoteModel, or rlmesh::RANDOM_SAMPLE. A run never closes the model and closes the environment only with close_env = true.

RunOptions has episodes, seeds, max_episode_steps, max_episode, hooks, instruction, close_env, execution_horizon, view, workflow_edition, and cancel. With neither episodes nor seeds, a run plays one episode.

To observe a run, derive from RunHooks and override what you need:

class Progress : public rlmesh::RunHooks {
 public:
  uint64_t steps = 0;
  void on_step(const rlmesh::StepEvent&) override { ++steps; }
  void on_episode_end(const rlmesh::EpisodeResult& result) override {
    std::printf("episode %lld: reward %.1f\n", static_cast<long long>(result.index), result.reward);
  }
};

Progress progress;
model.run("tcp://127.0.0.1:5555", rlmesh::RunOptions{}.with_episodes(2).with_hooks(&progress));

A Session lets you drive the loop:

rlmesh::RemoteEnv env(rlmesh::RemoteEnvOptions{}.with_address("tcp://127.0.0.1:5555"));
auto action_space = env.action_space();
rlmesh::Model model([action_space](const rlmesh::Value&) { return action_space.sample(); });

auto session = model.session(env);
auto [observation, info] = session.reset(0);
for (;;) {
  auto [next, reward, terminated, truncated, step_info] =
      session.step(session.predict(observation));
  observation = next;
  if (terminated || truncated) break;
}
session.close();

Session::reset takes an optional seed and trial index. Keep the model and environment alive while the session exists. A session connects on first use and also offers run, contract, readers, and frame sources. See Run an Evaluation.

Remote clients

RemoteEnv and RemoteVectorEnv connect and handshake in their constructors, so an unreachable address throws right away. RemoteModel only validates its options and dials when a session needs it.

using namespace std::chrono_literals;

rlmesh::RemoteEnv env(rlmesh::RemoteEnvOptions{}
                          .with_address("tcp://127.0.0.1:5555")
                          .with_connect_timeout(5s)
                          .with_request_timeout(30s));

rlmesh::RemoteModel remote(rlmesh::RemoteModelOptions{}.with_address("tcp://127.0.0.1:50051"));
auto result = rlmesh::run(remote, env);

A timeout of zero means no timeout. I/O after close() throws rlmesh::RuntimeError. See Connect a Client.

Errors

Failures throw. The exception classes form one hierarchy rooted in the standard library, so a generic catch (const std::exception&) catches all of them except cancellation:

try {
  rlmesh::RemoteEnv env(rlmesh::RemoteEnvOptions{}.with_address("tcp://127.0.0.1:5999"));
} catch (const rlmesh::ConnectionError& error) {
  std::fprintf(stderr, "endpoint unreachable: %s\n", error.what());
} catch (const rlmesh::EnvironmentException& error) {
  std::fprintf(stderr, "env failed (recoverable: %d): %s\n", error.is_recoverable(), error.what());
} catch (const rlmesh::ext::CancelledError&) {
  std::fprintf(stderr, "cancelled\n");
}

ext::CancelledError deliberately has no std::exception base, so a cancelled run is not mistaken for a failure; catch it explicitly. Errors from a remote peer keep their class, code, message, and recoverability.

Adapters

rlmesh::adapters (also reachable as rlmesh::adapt) builds the two sides of an adapter from plain structs: EnvTags on the environment and ModelSpec on the model. Runs and sessions resolve them when they connect.

using namespace rlmesh::adapters;
ImageTag camera;
camera.role = IMAGE_PRIMARY;
StateTag eef_pos;
eef_pos.role = EEF_POS;
Actuator delta_pos;
delta_pos.role = ACTION_DELTA_POS;
delta_pos.dim = 3;
Action action;
action.components = {delta_pos};
EnvTags tags(ObsNode::dict({{"camera", camera}, {"eef_pos", eef_pos}}), action);

Image image;
image.role = IMAGE_PRIMARY;
State state;
state.role = EEF_POS;
ModelSpec spec(InputNode::dict({{"image", image}, {"state", state}}), action);

rlmesh::EnvServer server(
    MyEnv{}, rlmesh::EnvServerOptions{}.with_address("tcp://127.0.0.1:5555").with_tags(tags));
rlmesh::Model model(MyPolicy{}, rlmesh::ModelOptions{}.with_spec(spec));

With C++20, designated initializers shorten the tag structs, as in ImageTag{.role = IMAGE_PRIMARY} and Action{.components = {{.role = ACTION_DELTA_POS, .dim = 3}}}.

resolve, Adapter, and custom AdapterBase subclasses (held by std::shared_ptr) expose the transforms directly. The suggested part names and rotation widths come from adapters::PARTS() and adapters::ROTATION_DIMS(). See the Adapters guide.

View and record

View streams frames live to a terminal or an HTTP page; set it on SessionOptions::view, or on RunOptions::view with a remote model. View::parse("http:8008") accepts term, http:PORT, and both.

Recorder collects results into a portable bundle:

rlmesh::Recorder recorder;
recorder.add(result, "constant", "CounterEnv-v0");
auto path = recorder.export_("results");
recorder.close();

recorder.capture(model_name, env_name) returns hooks that also record video during a run.

Reference

• C++ API reference: every class, function, and constant in rlmesh.hpp.
• The C API underneath: each C++ handle exposes its C handle through raw().
• Native Languages compares the Rust, C, and C++ APIs.