rerun-io / rerun

Time-aware multimodal data stack and visualizations Rerun is building the multimodal data stack to model, ingest, store, query and view robotics-style data. It's used in areas like robotics, spatial and embodied AI, generative media, industrial processing, simulation, security, and health. Rerun is easy to use! Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use. You can also query the logged data through our dataframe API. Get started in minutes – no account needed. • Run the Rerun Viewer in your browser • Read about what Rerun is and who it is for A short taste Getting started • **C++** • **Python**: or on • **Rust**: Installing the Rerun Viewer binary To stream log data over the network or load our data files you also need the binary. It can be installed with or with (see note below). Note that only the Python SDK comes bundled with the Viewer whereas C++ & Rust always rely on a separate install. **Note**: the Cargo feature requires the CLI to be installed and available in your path. Alternatively, you may skip enabling this feature, but this may result in inferior video decoding performance. You should now be able to run in any terminal. Documentation • 📚 High-level docs • ⏃ Loggable Types • ⚙️ Examples • 📖 Code snippets • 🌊 C++ API docs • 🐍 Python API docs • 🦀 Rust API docs • ⁉️ Troubleshooting Status We are in active development. There are many features we want to add, and the API is still evolving. _Expect breaking changes!_ Some shortcomings: • The viewer slows down when there are too many entities • The data you want to visualize must fit in RAM • See for how to bound memory use. • We plan on having a disk-based data store some time in the future. • Multi-million point clouds can be slow What is Rerun for? Rerun is built to help you understand and improve complex processes that include rich multimodal data, like 2D, 3D, text, time series, tensors, etc. It is used in many industries, including robotics, simulation, computer vision, or anything that involves a lot of sensors or other signals that evolve over time. Example use case Say you're building a vacuum cleaning robot and it keeps running into walls. Why is it doing that? You need some tool to debug it, but a normal debugger isn't gonna be helpful. Similarly, just logging text won't be very helpful either. The robot may log "Going through doorway" but that won't explain why it thinks the wall is a door. What you need is a visual and temporal debugger, that can log all the different representations of the world the robots holds in its little head, such as: • RGB camera feed • depth images • lidar scan • segmentation image (how the robot interprets what it sees) • its 3D map of the apartment • all the objects the robot has detected (or thinks it has detected), as 3D shapes in the 3D map • its confidence in its prediction • etc You also want to see how all these streams of data evolve over time so you can go back and pinpoint exactly what went wrong, when and why. Maybe it turns out that a glare from the sun hit one of the sensors in the wrong way, confusing the segmentation network leading to bad object detection. Or maybe it was a bug in the lidar scanning code. Or maybe the robot thought it was somewhere else in the apartment, because its odometry was broken. Or it could be one of a thousand other things. Rerun will help you find out! But seeing the world from the point of the view of the robot is not just for debugging - it will also give you ideas on how to improve the algorithms, new test cases to set up, or datasets to collect. It will also let you explain the brains of the robot to your colleagues, boss, and customers. And so on. Seeing is believing, and an image is worth a thousand words, and multimodal temporal logging is worth a thousand images :) While seeing and understanding your data is core to making progress in robotics, there is one more thing: You can also use the data you collected for visualization to create new datasets for training and evaluating the models and algorithms that run on your robot. Rerun provides query APIs to make it easy to extract clean datasets from your recording for exactly that purpose. Of course, Rerun is useful for much more than just robots. Any time you have any form of sensors, or 2D or 3D state evolving over time, Rerun is a great tool. Rerun vs. Rviz When coming from pure visualization tools like RViz, you might be used to seeing the latest data only. Rerun is more than a pure visualization solution, it provides a platform for multimodal data with a powerful visualizer, storage model and query engine (see also: *"What is Rerun?"*). In robotics, you can use Rerun e.g. to record test runs, manage and query training data, visually debug live streams or recordings (also from third-party formats like MCAP) and much more. So while Rerun makes your data streams visualizable in the viewer, integrating Rerun logging into your robotics applications also opens up the door for leveraging Rerun's broader capabilities. If you are only interested in visualization, the Rerun viewer has powerful features like the ability to go back in time thanks to its time-aware in-memory database. You can adjust the size of this buffer to your needs (see here), e.g. to a smaller size if you want to use Rerun as an RViz replacement in long-running or memory-constrained applications. Business model Rerun uses an open-core model. Everything in this repository will stay open source and free (both as in beer and as in freedom). We are also building a commercial data platform. Right now that is only available for a few select design partners. Click here if you're interested. The Rerun open source project targets the needs of individual developers. The commercial product targets the needs specific to teams that build and run computer vision…