latitude-dev / latitude-llm

Open-Source AI Engineering Platform Observability and evaluations first, then an eval-driven reliability loop to continuously improve prompts. Docs   ·   Slack   ·   X > [!WARNING] > You are viewing the **Latitude v2 Alpha** branch ( ). This branch is in **active development** and is **not ready for production use**. APIs, features, and data formats may change without notice. > > For production use, please use the stable release on the branch. 🌈 Why Latitude? Latitude is an open-source platform for building and operating LLM features in production. Most teams adopt Latitude in stages: start by instrumenting your existing LLM calls to get observability and evaluation coverage, then move into a reliability loop that turns production failures into repeatable fixes. **Start with observability + evaluations:** • **Observability** → capture prompts, inputs/outputs, tool calls, and latency/token usage/cost from real traffic • **Prompt playground** → reproduce runs, iterate with real inputs, version changes, and publish to the AI Gateway • **Datasets** → curate real examples for batch testing and regression suites • **Evaluations** → built-in evals, LLM-as-judge, and human-in-the-loop scoring • **Experiments** → compare models/providers and prompt versions with measurable results **Grow into the reliability loop:** • **Annotations** → turn human judgment into a signal you can track and optimize • **Issue discovery** → cluster failures into recurring issues and failure modes • **Automatic evals** → convert issues into continuous tests that guard releases • **Prompt optimizer (GEPA)** → search prompt variations against your eval suite and reduce recurring failures Latitude Telemetry works with most model providers and frameworks out of the box, and can be extended for custom integrations. See the full integration list (including OTLP ingest). 📚 Table Of Contents • Getting Started • Evaluations • Datasets & Testing • Prompt Manager • Custom AI Agents • Integrations & Deployment • Self-Hosting • Advanced: PromptL • Contributing • License ⚡ Quick start Latitude is available as a managed cloud product or as a self-hosted deployment: • **Latitude Cloud**: fully managed. • **Latitude Self-Hosted**: run the open-source distribution on your own infrastructure. Choose the option that best fits your needs and follow the corresponding instructions below. Latitude Cloud To get started with Latitude, follow these steps: • **Sign up** → Create an account at latitude.so and create a project. • **Instrument** → Add the telemetry SDK (recommended) or export OTLP traces to compatible backend. • **Evaluate** → Create datasets and evals to measure quality and catch regressions. • **Manage + ship** → Version prompts/agents, publish changes, and deploy via the gateway. • **Optimize** → Use eval-driven optimization to reduce recurring failures. For more details on each step, see our documentation or join the community. Latitude Self-Hosted Follow the instructions in the self-hosted guide to get started with Latitude Self-Hosted. After setting up Latitude Self-Hosted, you can follow the same steps as in the Latitude Cloud guide to create, test, evaluate, and deploy your prompts. 👥 Community The Latitude community is on Slack, where you can ask questions, share feedback, and show what you're building. 🤝 Contributing Contributions are welcome. For an overview of the repo and its architecture, see the contributor guide. If you want to help, join the Slack community, open an issue, or submit a pull request. 📄 License Latitude is licensed under the LGPL-3.0. Alternatively, we offer a more permissive commercial license for those who need it. Please contact us at licensing@latitude.so for more information. 🔗 Links • Home page • Documentation • Slack community • X / Twitter Made with love by the Latitude Team