NVIDIA-NeMo / Gym

NeMo Gym **Requirements** • **Quick Start** • **Available Environments** • **Documentation & Resources** • **Community & Support** • **Citations** NeMo Gym is a library for building reinforcement learning (RL) training environments for large language models (LLMs). It provides infrastructure to develop environments, scale rollout collection, and integrate seamlessly with your preferred training framework. 🏆 Why NeMo Gym? • Scaffolding and patterns to accelerate environment development: multi-step, multi-turn, and user modeling scenarios • Contribute environments without expert knowledge of the entire RL training loop • Test environments and throughput end-to-end, independent of the RL training loop • Interoperable with existing environments, systems, and RL training frameworks • Growing collection of training environments and datasets for Reinforcement Learning from Verifiable Reward (RLVR) > [!IMPORTANT] > NeMo Gym is currently in early development. You should expect evolving APIs, incomplete documentation, and occasional bugs. We welcome contributions and feedback - for any changes, please open an issue first to kick off discussion! 🔗 Ecosystem NeMo Gym is part of NVIDIA NeMo, NVIDIA's GPU-accelerated platform for building and training generative AI models. NeMo Gym integrates with a growing number of RL training frameworks and environment libraries; see the Ecosystem page for full details and tutorials. **Training Frameworks:** NeMo RL • OpenRLHF • Unsloth • more → **Environment Libraries:** Reasoning Gym • Aviary • more → 📋 Requirements NeMo Gym is designed to run on standard development machines: | Hardware Requirements | Software Requirements | | --------------------- | --------------------- | | **GPU**: Not required for NeMo Gym library operation • GPU may be needed for specific resources servers or model inference (see individual server documentation) | **Operating System**: • Linux (Ubuntu 20.04+, or equivalent) • macOS (11.0+ for x86_64, 12.0+ for Apple Silicon) • Windows (via WSL2) | | **CPU**: Any modern x86_64 or ARM64 processor (e.g., Intel, AMD, Apple Silicon) | **Python**: 3.12 or higher | | **RAM**: Minimum 8 GB (16 GB+ recommended for larger environments) | **Git**: For cloning the repository | | **Storage**: Minimum 5 GB free disk space for installation and basic usage | **Internet Connection**: Required for downloading dependencies and API access | **Additional Requirements** • **API Keys**: OpenAI API key with available credits (for the quickstart examples) • Other model providers supported (Azure OpenAI, self-hosted models via vLLM) • **Ray**: Automatically installed as a dependency (no separate setup required) 🚀 Quick Start Install NeMo Gym, start the servers, and collect your first verified rollouts for RL training. Setup Configure Your API Key Create an file that contains your OpenAI API key and the policy model you want to use. Replace with your actual key. This file helps keep your secrets out of version control while still making them available to NeMo Gym. > [!NOTE] > We use GPT-4.1 in this quickstart because it provides low latency (no reasoning step) and works reliably out-of-the-box. NeMo Gym is **not limited to OpenAI models**—you can use self-hosted models via vLLM or any OpenAI-compatible inference server. See the documentation for details. Start Servers **Terminal 1 (start servers)**: **Terminal 2 (interact with agent)**: Collect Rollouts **Terminal 2** (keep servers running in Terminal 1): This generates training data with verification scores! Clean Up Servers **Terminal 1** with the running servers: Ctrl+C to stop the ng_run process. Next Steps Now that you can generate rollouts, choose your path: • **Start training** — Train models using NeMo Gym with your preferred RL framework. See the Training Tutorials. • **Use an existing environment** — Browse the Available Environments below to find an environment that matches your goals. • **Build a custom environment** — Implement or integrate existing tools and define task verification logic. Get started with the Creating a Training Environment tutorial. 📦 Available Environments NeMo Gym includes a curated collection of environments for training and evaluation across multiple domains: Example Environment Patterns Purpose: Demonstrate NeMo Gym patterns and concepts. | Name | Demonstrates | Config | README | | ------------------ | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | Multi Step | Multi-step tool calling | example_multi_step.yaml | README | | Session State Mgmt | Session state management (in-memory) | example_session_state_mgmt.yaml | README | | Single Tool Call | Basic single-step tool calling | example_single_tool_call.yaml | README | Environments for Training & Evaluation Purpose: Training-ready environments with curated datasets. > [!TIP] > Each resources server includes example data, configuration files, and tests. See each server's README for details. | Resources Server | Domain | Description | Value | Train | Validation | License | Config | Dataset | | --------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ----- | ---------- | --------------------------------------------------------- | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------…