Engines

Engines are the means by which kani interact with language models. As you’ve seen, kani comes with a few engines included:

Model Name	Extra	Capabilities	Engine
GPT-3.5-turbo, GPT-4	openai	🛠️ 📡	kani.engines.openai.OpenAIEngine
Claude, Claude Instant	anthropic	🛠️ 📡	kani.engines.anthropic.AnthropicEngine
🤗 transformers[4]	huggingface[2]	(runtime)	kani.engines.huggingface.HuggingEngine
🤗 🦙 LLaMA 3	huggingface, llama[2]	🔓 🖥 🚀	kani.engines.huggingface.HuggingEngine[1]
🤗 Mistral, Mixtral	huggingface[2]	🛠️ 🔓 🖥 🚀	kani.engines.huggingface.HuggingEngine[1]
🤗 Command R, Command R+	huggingface[2]	🛠️ 🔓 🖥 🚀	kani.engines.huggingface.cohere.CommandREngine
🤗 🦙 LLaMA v2	huggingface, llama[2]	🔓 🖥 🚀	kani.engines.huggingface.llama2.LlamaEngine
🤗 🦙 Vicuna v1.3	huggingface, llama[2]	🔓 🖥 🚀	kani.engines.huggingface.vicuna.VicunaEngine
llama.cpp[4]	cpp	(runtime)	kani.engines.llamacpp.LlamaCppEngine
🦙 LLaMA v2 (GGUF)	cpp	🔓 🖥 🚀	kani.engines.llamacpp.LlamaCppEngine

Additional models using the classes above are also supported - see the model zoo for a more comprehensive list of models!

Legend

• 🛠️: Supports function calling.

• 🔓: Open source model.

• 🖥: Runs locally on CPU.

• 🚀: Runs locally on GPU.

• 📡: Hosted API.

[1] (1,2)

See the model zoo for a code sample to initialize this model with the given engine.

[2] (1,2,3,4,5,6)

You will also need to install PyTorch manually.

[3]

This is an abstract class of models; kani includes a couple concrete implementations for reference.

[4] (1,2)

This is a model runtime that can support multiple models using a PromptPipeline.

In this section, we’ll discuss how to implement your own engine to use any language model or API you can think of.

Tip

Built an engine for a model kani doesn’t support yet? kani is OSS and ❤️ PRs with engine implementations for the latest models - see Contributing.

• Implementing an Engine
  • Optional Methods
  • Adding Function Calling
• HTTP Client
• HuggingFace
  • 4-bit Quantization (🤗)
• llama.cpp