Llama 3.1 8B Instruct

Model Overview

The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.

• Model Architecture: Llama 3.1 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.
• Model Release Date: July 23, 2024.
• Repository: llama-models/models/llama3_1
• Model Source: meta-llama/Meta-Llama-3.1-8B-Instruct
• License: A custom commercial license
• Supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

QPC Configurations

Precision	SoCs / Tensor slicing	NSP-Cores (per SoC)	Full Batch Size	Chunking Prompt Length	Context Length (CL)	QPC URL	QPC Size	QPC Download	Onnx URL	Onnx Download	Generation Date
MXFP6	2	16	1	128	8192	https://dc00tk1pxen80.cloudfront.net/SDK1.21.2/meta-llama/Meta-Llama-3.1-8B-Instruct/meta-llama_Llama-3.1-8B-Instruct_qpc_16cores_128pl_8192cl_1fbs_2devices_mxfp6_mxint8_ccl.tar.gz	9.6GB	Download	https://dc00tk1pxen80.cloudfront.net/SDK1.21.2/meta-llama/Meta-Llama-3.1-8B-Instruct/meta-llama_Llama-3.1-8B-Instruct_ONNX.tar.gz	Download	17-Mar-2026
MXFP6	4	16	1	128	8192	https://dc00tk1pxen80.cloudfront.net/SDK1.21.2/meta-llama/Meta-Llama-3.1-8B-Instruct/meta-llama_Llama-3.1-8B-Instruct_qpc_16cores_128pl_8192cl_1fbs_4devices_mxfp6_mxint8.tar.gz	14GB	Download	https://dc00tk1pxen80.cloudfront.net/SDK1.21.2/meta-llama/Meta-Llama-3.1-8B-Instruct/meta-llama_Llama-3.1-8B-Instruct_ONNX.tar.gz	Download	25-Mar-2026

Run This Model

Download QPCs

mkdir -p meta-llama/Meta-Llama-3.1-8B-Instruct
cd meta-llama/Meta-Llama-3.1-8B-Instruct

# Download QPC
wget <QPC_Download_URL>
tar xzvf <qpc_filename.tar.gz>

Run QPC

Replace QPC_PATH with actual extracted QPC directories.

python3 -m vllm.entrypoints.openai.api_server \
  --port <PORT> \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --device-group <DEVICE_IDS> \
  --max-model-len <CTX_LEN> \
  --max-seq-len-to-capture <PREFILL_SEQ_LEN> \
  --max-num-seqs <MAX_NUM_SEQS> \
  --quantization mxfp6 \
  --kv-cache-dtype mxint8 \
  --override-qaic-config "num_cores:[num_cores] qpc_path:[qpc_path] ccl_enabled:True comp_ctx_lengths_prefill=[1024,2048,4096,8192] comp_ctx_lengths_decode=[1024,2048,4096,8192]"

Run Inference

Once the server is running, send a request to the OpenAI-compatible endpoint:

curl http://localhost:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "<PROMPT>"}
    ],
    "max_tokens": 200
  }'

Useful Links

• Environment Setup