Llama 3.3 70b instruct awq

Model Overview

This is the AWQ version of the Llama 3.3 70B Instruct model. The Meta Llama 3.3 multilingual large language model (LLM) is an instruction tuned generative model in 70B (text in/text out). The Llama 3.3 instruction tuned text only model is optimized for multilingual dialogue use cases and outperforms many of the available open source and closed chat models on common industry benchmarks.

• Model Architecture: Llama 3.3 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.
• Model Release Date: December 6, 2024.
• Model Source: casperhansen/llama-3.3-70b-instruct-awq
• Model Repository: casper-hansen/AutoAWQ
• License: llama3.3
• Supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

QPC Configurations

Precision	SoCs / Tensor slicing	NSP-Cores (per SoC)	Full Batch Size	Chunking Prompt Length	Context Length (CL)	QPC URL	QPC Size	QPC Download	Onnx URL	Onnx Download	Generation Date
MXFP6	16	16	1	128	8192	https://qualcom-qpc-models.s3-accelerate.amazonaws.com/SDK1.21.2/casperhansen/llama-3.3-70b-instruct-awq/casperhansen_llama-3.3-70b-instruct-awq_qpc_16cores_128pl_8192cl_1fbs_16devices_mxfp6_mxint8.tar.gz	100GB	Download	https://dc00tk1pxen80.cloudfront.net/SDK1.21.2/casperhansen/llama-3.3-70b-instruct-awq/casperhansen_llama-3.3-70b-instruct_ONNX.tar.gz	Download	27-Mar-2026

Run This Model

Download QPCs

mkdir -p casperhansen/llama-3.3-70b-instruct-awq
cd casperhansen/llama-3.3-70b-instruct-awq

# Download QPC
wget <QPC_Download_URL>
tar xzvf <qpc_filename.tar.gz>

Run QPC

Replace QPC_PATH with actual extracted QPC directories.

python3 -m vllm.entrypoints.openai.api_server \
  --port <PORT> \
  --model casperhansen/llama-3.3-70b-instruct-awq \
  --device-group <DEVICE_IDS> \
  --max-model-len <CTX_LEN> \
  --max-seq-len-to-capture <PREFILL_SEQ_LEN> \
  --max-num-seqs <MAX_NUM_SEQS> \
  --quantization mxfp6 \
  --kv-cache-dtype mxint8 \
  --override-qaic-config "num_cores:[num_cores] qpc_path:[qpc_path] ccl_enabled:True comp_ctx_lengths_prefill=[1024,2048,4096,8192] comp_ctx_lengths_decode=[1024,2048,4096,8192]"

Run Inference

Once the server is running, send a request to the OpenAI-compatible endpoint:

curl http://localhost:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "casperhansen/llama-3.3-70b-instruct-awq",
    "messages": [
      {"role": "user", "content": "<PROMPT>"}
    ],
    "max_tokens": 200
  }'

Useful Links

• Environment Setup