Installation

TL;DR: kani requires Python 3.10+. You can install kani though pip:

$ pip install kani

Engines

Since kani is designed to be model-agnostic, kani uses Engines to interface with different language model backends. These engines usually have their own dependencies beyond kani’s, which can be quite large (e.g. transformers).

To install these dependencies along with kani, you should specify one or more extras with your installation command (e.g. pip install "kani[openai]"). The table below lists the engines that are included with kani and the extra you should use to install its dependencies.

Model Name	Extra	Capabilities	Engine
GPT-3.5-turbo, GPT-4	openai	🛠️ 📡	kani.engines.openai.OpenAIEngine
Claude, Claude Instant	anthropic	🛠️ 📡	kani.engines.anthropic.AnthropicEngine
🤗 transformers[4]	huggingface[2]	(runtime)	kani.engines.huggingface.HuggingEngine
🤗 🦙 LLaMA 3	huggingface, llama[2]	🔓 🖥 🚀	kani.engines.huggingface.HuggingEngine[1]
🤗 Mistral, Mixtral	huggingface[2]	🛠️ 🔓 🖥 🚀	kani.engines.huggingface.HuggingEngine[1]
🤗 Command R, Command R+	huggingface[2]	🛠️ 🔓 🖥 🚀	kani.engines.huggingface.cohere.CommandREngine
🤗 🦙 LLaMA v2	huggingface, llama[2]	🔓 🖥 🚀	kani.engines.huggingface.llama2.LlamaEngine
🤗 🦙 Vicuna v1.3	huggingface, llama[2]	🔓 🖥 🚀	kani.engines.huggingface.vicuna.VicunaEngine
llama.cpp[4]	cpp	(runtime)	kani.engines.llamacpp.LlamaCppEngine
🦙 LLaMA v2 (GGUF)	cpp	🔓 🖥 🚀	kani.engines.llamacpp.LlamaCppEngine

Additional models using the classes above are also supported - see the model zoo for a more comprehensive list of models!

Legend

• 🛠️: Supports function calling.

• 🔓: Open source model.

• 🖥: Runs locally on CPU.

• 🚀: Runs locally on GPU.

• 📡: Hosted API.

[1] (1,2)

See the model zoo for a code sample to initialize this model with the given engine.

[2] (1,2,3,4,5,6)

You will also need to install PyTorch manually.

[3]

This is an abstract class of models; kani includes a couple concrete implementations for reference.

[4] (1,2)

This is a model runtime that can support multiple models using a PromptPipeline.

See also

Extensions provide additional engines you can use to extend kani with additional functionality like support for vision-language models and ReAct prompting. See Extensions for a list of available extensions!

See also

Want to use a different model with kani? Check out Engines for details on how to implement the common Engine interface and use kani with any LM.

Installing on Conda

You may need to install pip in your conda environment first:

$ conda install pip

Then, follow the instructions for installing with pip above.

Caution

In certain cases when using a conda venv, the pip binary may reference a different environment, and kani may appear to be uninstalled even if pip assures that it is. Using python -m pip install in place of pip install may mitigate this issue.

Virtual Environment

If you’re not using conda, we recommend using a virtual environment to manage your project dependencies. This will help you prevent polluting your global Python installation with all sorts of packages.

macOS/Linux

$ python -m venv ./venv
$ ./venv/bin/activate
$ pip install "kani[...]"

Windows

$ python -m venv venv
$ venv\Scripts\activate.bat
$ pip install "kani[...]"

Development Version

If you’d like to install the development version of kani, you can install it from GitHub directly:

$ pip install 'kani @ git+https://github.com/zhudotexe/kani.git@main'

This will install the latest version of kani.

Note

You may need to use pip install --upgrade --no-deps --force-reinstall ... to force pip to re-fetch the latest kani from GitHub.

To install an engine’s extras, use pip install 'kani[...] @ git+https://github.com/zhudotexe/kani.git@main'.

Caution

Development versions of kani may be unstable! Do not use development kani in production or in final research experiments; pin a released version of kani instead.

Requirements File

If you’re running experiments using kani, we recommend pinning the version of kani to ensure your runs are reproducible. To do this, we recommend storing all your Python requirements in a requirements.txt file.

kani[...]==x.y.z
# ... other dependencies

You can automatically generate this file too, by running pip freeze > requirements.txt.

Later, anyone else running your code can install the same dependency versions by simply running pip install -r requirements.txt.

Next, we’ll take a look at basic usage of kani.