# Introducing the Otoroshi LLM Extension by Cloud APIM

## 🔍 What Is the Otoroshi LLM Extension?

The **Otoroshi LLM Extension** by [**Cloud APIM**](https://www.cloud-apim.com) is a groundbreaking module that enhances the capabilities of the [open-source API Gateway **Otoroshi**,](https://github.com/MAIF/otoroshi) turning it into a powerful **AI Gateway**.

It enables a complete integration with leading **large language model (LLM)** providers such as [**OpenAI, Mistral, Anthropic, Azure, Hugging Face**](https://cloud-apim.github.io/otoroshi-llm-extension/docs/llm-gateway/providers), and much more providers : all through a unified API.

This innovation brings **AI-native API management** to the forefront, letting companies integrate conversational AI, generative models, and intelligent services directly into their infrastructure.

---

## Best features from the Otoroshi LLM Extension

### Multi-Provider AI Compatibility

Easily switch or combine multiple LLMs like [GPT from OpenAI, Claude, Mistral](https://cloud-apim.github.io/otoroshi-llm-extension/docs/llm-gateway/providers), your own internal company models or open-source models via a **single standardized API**.

This reduces vendor lock-in and increases flexibility for **AI workflows**.

### Advanced Prompt Engineering & Controls

Create dynamic prompts using templates, inject real-time context, and enforce **"prompt guardrails"** to sanitize inputs/outputs. Ideal for **data privacy, compliance**, and **response reliability**.

### AI Governance & Security

* Per-service and per-user LLM token quotas
    
* Role-based access control and moderation
    
* Full auditing and request tracing
    
* Integration with existing [**Otoroshi**](https://www.otoroshi.io/) rules for fine-grained **API-level governance**
    

### 📊 Cost Optimization & Performance

* Track [token usage](https://cloud-apim.github.io/otoroshi-llm-extension/docs/cost-optimizations/quotas) and cost per LLM request
    
* Use [**semantic caching**](https://cloud-apim.github.io/otoroshi-llm-extension/docs/cost-optimizations/semantic-cache) to avoid redundant calls
    
* Apply retry policies and load balancing to maintain stability
    

---

## Easy to install with Cloud APIM and Clever Cloud

The Otoroshi LLM Extension is fully integrated into [**Cloud APIM’s managed Otoroshi platform**,](https://www.cloud-apim.com/otoroshi-managed) available as a [**serverless AI Gateway**](https://www.cloud-apim.com/serverless).

Since **December 2024**, it can also be deployed via [**Clever Cloud**](https://www.clever-cloud.com/blog/features/2024/12/17/otoroshi-with-llm-simplify-your-api-and-ai-service-management-on-clever-cloud/) in just minutes with the “Otoroshi LLM extension” add-on.

## Use Cases

* **AI-Powered API Workflows**: Centralize prompt routing, response formatting, and LLM usage across multiple APIs.
    
* **Secure Chatbots & Agents**: Deploy moderated, auditable conversational agents for support or business operations.
    
* **Smart Routing**: Offload routine tasks to Mistral/Ollama, reserve GPT-4 for critical operations.
    
* **Compliance & Audit-Ready AI APIs**: Build trustworthy AI features for regulated environments (finance, healthcare, etc.).
    

---

## Why It Matters

In a world where **AI and APIs** are becoming inseparable, the [Otoroshi LLM Extension](https://cloud-apim.github.io/otoroshi-llm-extension/) offers a **secure, scalable, and efficient** foundation for next-generation applications.

You can build intelligent microservices, craft interactive user experiences, and automate backend operations

🔗 Learn more: [Otoroshi LLM Extension Documentation](https://cloud-apim.github.io/otoroshi-llm-extension/docs/overview)

### **📡 Stay Connected**

Follow our blog for the latest updates, tips, and best practices for our products.

### **🏢 About Cloud APIM**

[Cloud APIM](https://www.cloud-apim.com/) provides [cutting-edge, managed solutions for API management](https://console.cloud-apim.com/deployments), enabling businesses to leverage the full power of their APIs with ease and efficiency.

Our commitment to innovation and excellence drives us to offer [the most advanced tools](https://console.cloud-apim.com/serverless/projects) and [services](https://console.cloud-apim.com/wasmo_deployments) to our customers, empowering them to achieve their digital transformation goals.

### Cloud APIM Products

[**Otoroshi Managed Instances**](https://www.cloud-apim.com/) **:** Fully managed [Otoroshi](https://maif.github.io/otoroshi/manual/about.html) clusters, perfectly configured and optimized, ready in seconds

[**Serverless**](https://www.cloud-apim.com/serverless) enables scalable deployments without infrastructure management.

[**Authify**](https://www.cloud-apim.com/authify) simplifies authentication with quick and secure integration.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1717164622893/4c55fd90-812f-4369-9d80-03faeeb5158f.png align="center")