Instructions to use NexaAI/Octopus-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NexaAI/Octopus-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NexaAI/Octopus-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("NexaAI/Octopus-v2")
model = AutoModelForMultimodalLM.from_pretrained("NexaAI/Octopus-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use NexaAI/Octopus-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NexaAI/Octopus-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NexaAI/Octopus-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/NexaAI/Octopus-v2

SGLang

How to use NexaAI/Octopus-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NexaAI/Octopus-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NexaAI/Octopus-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NexaAI/Octopus-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NexaAI/Octopus-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use NexaAI/Octopus-v2 with Docker Model Runner:
```
docker model run hf.co/NexaAI/Octopus-v2
```

Octopus-v2

Commit History

Update benchmark with GPT-4O

e997f4c
verified

zackli4ai commited on May 21, 2024

update benchmark for GPT-4O

15b7686
verified

zackli4ai commited on May 21, 2024

Update added_tokens.json

63b697e
verified

zackli4ai commited on May 9, 2024

Update README.md

524fc80
verified

zackli4ai commited on May 5, 2024

Update README.md

f8d8ba9
verified

zackli4ai commited on May 5, 2024

add benchmark with openELM and Phi-3

10bd713

Zack Zhiyuan Li commited on Apr 30, 2024

Add Microsoft Phi-3 benchmark

3e77051
verified

nexa4ai commited on Apr 30, 2024

Update README.md

8d122c4
verified

nexa4ai commited on Apr 22, 2024

Update android function readme

d2d36e6
verified

nexa4ai commited on Apr 19, 2024

add car function definitions

7d7e8ff
verified

nexa4ai commited on Apr 19, 2024

Add v3 logo

acbc882
verified

nexa4ai commited on Apr 18, 2024

Upload octopus-v3.jpeg

b0c20dc
verified

nexa4ai commited on Apr 18, 2024

Update README.md

77bd3f5
verified

nexa4ai commited on Apr 18, 2024

Update README.md

0f44101
verified

alexchen4ai commited on Apr 18, 2024

update

eb242cb
verified

alexchen4ai commited on Apr 13, 2024

update

468ae1a
verified

alexchen4ai commited on Apr 12, 2024

Update README.md

0e82a0e
verified

alexchen4ai commited on Apr 12, 2024

Update README.md

5bfa5fe
verified

alexchen4ai commited on Apr 12, 2024

Update README.md

b8b5434
verified

alexchen4ai commited on Apr 12, 2024

Update README.md

103d19a
verified

alexchen4ai commited on Apr 12, 2024

update

0bef7a2
verified

alexchen4ai commited on Apr 12, 2024

polish

a26d9d4
verified

alexchen4ai commited on Apr 6, 2024

update

942a5e6
verified

alexchen4ai commited on Apr 6, 2024

Update README.md

a291655
verified

alexchen4ai commited on Apr 6, 2024

Update README.md

c2ec018
verified

alexchen4ai commited on Apr 6, 2024

Upload android_benchmark.xlsx

74bc7c1
verified

nexa4ai commited on Apr 6, 2024

Fix model ID (#1)

d65d651
verified

alexchen4ai

osanseviero commited on Apr 6, 2024

GemmaForCausalLM import error fix (#3)

9572755
verified

alexchen4ai

Tonic commited on Apr 6, 2024