raga-ai-hub / RagaAI-Catalyst

RagaAI Catalyst  RagaAI Catalyst is a comprehensive platform designed to enhance the management and optimization of LLM projects. It offers a wide range of features, including project management, dataset management, evaluation management, trace management, prompt management, synthetic data generation, and guardrail management. These functionalities enable you to efficiently evaluate, and safeguard your LLM applications. Table of Contents • RagaAI Catalyst • Installation • Configuration • Usage • Project Management • Dataset Management • Evaluation Management • Trace Management • Agentic Tracing • Prompt Management • Synthetic Data Generation • Guardrail Management • Red-teaming Installation To install RagaAI Catalyst, you can use pip: Configuration Before using RagaAI Catalyst, you need to set up your credentials. You can do this by setting environment variables or passing them directly to the class: you'll need to generate authentication credentials: • Navigate to your profile settings • Select "Authenticate" • Click "Generate New Key" to create your access and secret keys **Note**: Authetication to RagaAICatalyst is necessary to perform any operations below. Usage Project Management Create and manage projects using RagaAI Catalyst: Dataset Management Manage datasets efficiently for your projects: For more detailed information on Dataset Management, including CSV schema handling and advanced usage, please refer to the Dataset Management documentation. Evaluation Create and manage metric evaluation of your RAG application: Trace Management Record and analyze traces of your RAG application: There are two ways to start a trace recording 1- with tracer(): 2- tracer.start() For more detailed information on Trace Management, please refer to the Trace Management documentation. Agentic Tracing The Agentic Tracing module provides comprehensive monitoring and analysis capabilities for AI agent systems. It helps track various aspects of agent behavior including: • LLM interactions and token usage • Tool utilization and execution patterns • Network activities and API calls • User interactions and feedback • Agent decision-making processes The module includes utilities for cost tracking, performance monitoring, and debugging agent behavior. This helps in understanding and optimizing AI agent performance while maintaining transparency in agent operations. Tracer initialization Initialize the tracer with project_name and dataset_name For more detailed information on Trace Management, please refer to the Agentic Tracing Management documentation. Prompt Management Manage and use prompts efficiently in your projects: For more detailed information on Prompt Management, please refer to the Prompt Management documentation. Synthetic Data Generation Guardrail Management Red-teaming The Red-teaming module provides comprehensive scans to detect model vulnerabilities, biases and misusage. Key Features • Support for multiple LLM providers (OpenAI, XAI, ..) • Built-in and custom detectors • Automatic test case generation • Allow users to add their own test cases • Flexible evaluation scenarios • Detailed reporting and analysis Initialization Usage Examples • Basic Usage with String Examples: • Advanced Usage with Specific Test Cases: • Mixed Detector Types (Built-in and Custom): Auto-generated Test Cases If no examples are provided, the module can automatically generate test cases: Upload Results (Optional)