llm4s / llm4s

LLM4S - Large Language Models for Scala ⭐ Stars Over Time Project Momentum Overview LLM4S provides a simple, robust, and scalable framework for building LLM applications in Scala. While most LLM work is done in Python, we believe that Scala offers a fundamentally better foundation for building reliable, maintainable AI-powered applications. > **Note:** This is a work in progress project and is likely to change significantly over time. Why Scala for LLMs? • **Type Safety**: Catch errors at compile time, not in production. • **Functional Programming**: Immutable data and pure functions for predictable, maintainable systems. • **JVM Ecosystem**: Access to mature, production-grade libraries and tooling. • **Concurrency**: Advanced models for safe, efficient parallelism. • **Performance**: JVM speed with functional elegance. • **Ecosystem Interoperability**: Seamless integration with enterprise JVM systems and cloud-native tooling. Features • **Multi-Provider Support**: Connect seamlessly to multiple LLM providers (OpenAI, Anthropic, Google Gemini, Azure, Ollama, DeepSeek). • **Execution Environments**: Run LLM-driven operations in secure, containerized or non-containerized setups. • **Error Handling**: Robust mechanisms to catch, log, and recover from failures gracefully. • **MCP Support**: Integration with Model Context Protocol for richer context management. • **Agent Framework**: Build single or multi-agent workflows with standardized interfaces. • **Multimodal Generation**: Support for text, image, voice, and other LLM modalities. • **RAG (Retrieval-Augmented Generation)**: Built-in tools for search, embedding, retrieval workflows, and RAGAS evaluation with benchmarking harness. • **Observability**: Detailed trace logging, monitoring, and analytics for debugging and performance insights. Architecture Project Structure • **modules/core**: Core LLM4S framework • **modules/workspace**: Workspace runner/client/shared • **modules/samples**: Usage examples • **docs**: Documentation site and references • **hooks**: Pre-commit hook installer Getting Started To get started with the LLM4S project, check out this teaser talk presented by **Kannupriya Kalra** at the Bay Area Scala Conference. This recording is essential for understanding where we’re headed: 🎥 **Teaser Talk:** https://www.youtube.com/watch?v=SXybj2P3_DE&ab_channel=SalarRahmanian LLM4S was officially introduced at the Bay Area Scala Conference in San Francisco on February 25, 2025. Setting Up Pre-commit Hooks To ensure code quality, we use a Git pre-commit hook that automatically checks code formatting and runs tests before allowing commits: Prerequisites • JDK 21+ • SBT • Docker Verify Java installation Windows Set and update your . macOS (Homebrew) Building the Project Setup your IDE > **See:** docs/reference/contributing.md#IDE-Setup Setup your LLM Environment You will need an API key for either OpenAI (https://platform.openai.com/) or Anthropic (https://console.anthropic.com/) other LLMS may be supported in the future (see the backlog). Set the environment variables: or Anthropic: or OpenRouter: or Z.ai: or DeepSeek: > **Migration Note:** The case has been added to the sealed ADT. If you have exhaustive pattern matches on , add a handler, or use a wildcard to gracefully handle future providers. Or Cohere: This will allow you to run the non-containerized examples. Running the Examples Run containerized demo Scala Version LLM4S now targets Scala 3.7.1 only. The build keeps compatibility-oriented aliases for Scala 3 verification: • - Common code for all versions • - Scala 3 specific code (add when needed) We've kept convenient build aliases: Roadmap Our goal is to implement Scala equivalents of popular Python LLM frameworks, with **multi-provider, multimodal, and observability-first design** as core principles. 📋 Detailed Roadmap **For the full roadmap including core framework features and agent phases, see the LLM4S Roadmap** The roadmap covers: • **Core Framework Features**: Multi-provider LLM, image generation, speech, embeddings, tools, MCP • **Agent Framework Phases**: Conversations, guardrails, handoffs, memory, streaming, built-in tools • **Production Pillars**: Testing, API Stability, Performance, Security, Documentation, Observability • **Path to v1.0.0**: Structured path to production release High-Level Goals • [ ] Single API access to multiple LLM providers (like LiteLLM) - **llmconnect** ✅ *Complete* • [ ] Comprehensive toolchain for building LLM apps (LangChain/LangGraph equivalent) • [x] Tool calling ✅ *Complete* • [x] RAG search & retrieval ✅ *Complete* (vector memory, embeddings, document Q&A) • [x] RAG evaluation & benchmarking ✅ *Complete* (RAGAS metrics, systematic comparison) • [x] Logging, tracking, and monitoring ✅ *Complete* • [ ] Agentic framework (like PydanticAI, CrewAI) • [x] Single-agent workflows ✅ *Complete* • [x] Multi-agent handoffs ✅ *Complete* • [x] Memory system (in-memory, SQLite, vector) ✅ *Complete* • [x] Streaming events ✅ *Complete* • [x] Built-in tools module ✅ *Complete* • [ ] DAG-based orchestration 🚧 *In Progress* • [ ] Tokenization utilities (Scala port of tiktoken) ✅ *Complete* • [ ] Examples for all supported modalities and workflows ✅ *Complete* • [ ] Stable platform with extensive test coverage 🚧 *In Progress* • [ ] Scala Coding SWE Agent - perform SWE Bench–type tasks on Scala codebases • [ ] Code maps, code generation, and library templates Tool Calling Tool calling is a critical integration - designed to work seamlessly with **multi-provider support** and **agent frameworks**. We use ScalaMeta to auto-generate tool definitions, support dynamic mapping, and run in **secure execution environments**. Tools can run: • In **containerized sandboxes** for isolation and safety. • In **multi-modal pipelines** where LLMs interact with text, images, and voice. • With **observability hooks** for trace analysis. Tool Signature Generation Using…