Engines

Engines are the means by which kani interact with language models. As you’ve seen, kani comes with a few engines included:

Model Name	Extra	Capabilities	Engine
All OpenAI Models	openai	🛠️ 🖼	kani.engines.openai.OpenAIEngine
All Anthropic Models	anthropic	🛠️ 🖼	kani.engines.anthropic.AnthropicEngine
All Google AI Models	google	🛠️ 🖼	kani.engines.google.GoogleAIEngine
🤗 transformers[3]	huggingface[1]	(model-specific)	kani.engines.huggingface.HuggingEngine
llama.cpp[2]	cpp	(model-specific)	kani.engines.llamacpp.LlamaCppEngine
vLLM[2]	vllm	(model-specific)	kani.ext.vllm.VLLMEngine, VLLMServerEngine, or VLLMOpenAIEngine

Additional models using the classes above are also supported - see the model zoo for a more comprehensive list of models!

Legend

• 🛠️: Supports function calling.

• 🖼: Supports multimodal inputs.

[1]

You will also need to install PyTorch manually.

[2] (1,2)

This is a model runtime that can support multiple models using a PromptPipeline.

[3]

The HuggingEngine can run most models directly from HuggingFace using Chat Templates. For more fine-grained control over prompting, see PromptPipeline.

In this section, we’ll cover engine-specific details and features you might want to be aware of when using certain engines.

We’ll also discuss how to implement your own engine to use any language model or API you can think of.

Tip

Built an engine for a model kani doesn’t support yet? kani is OSS and ❤️ PRs with engine implementations for the latest models - see Contributing.

• Implementing an Engine
  • Optional Methods
  • Adding Function Calling
• AnthropicEngine
  • Reference
  • Recipes
• GoogleAIEngine
  • Reference
  • Notes
  • Recipes
• HuggingEngine
  • Multimodal Support
  • Quantization With BitsAndBytes
  • Reference
• LlamaCppEngine
• OpenAIEngine
  • Reference
  • Recipes
• vLLM Engines