Skip to content

Nvidia Llama 3.1 Nemotron 70B Instruct HF AWQ INT4

Model Overview

This repository is an AWQ 4-bit quantized version of the nvidia/Llama-3.1-Nemotron-70B-Instruct-HF model, which is an NVIDIA customized version of meta-llama/Meta-Llama-3.1-70B-Instruct, originally released by Meta AI.

This model was quantized using AutoAWQ from FP16 down to INT4 using GEMM kernels, with zero-point quantization and a group size of 128.

QPC Configurations

Precision SoCs / Tensor slicing NSP-Cores (per SoC) Full Batch Size Chunking Prompt Length Context Length (CL) QPC URL QPC Size QPC Download Onnx URL Onnx Download Generation Date
MXFP6 2 16 1 128 4096 https://dc00tk1pxen80.cloudfront.net/SDK1.21.2/ibnzterrell/Nvidia-Llama-3.1-Nemotron-70B-Instruct-HF-AWQ-INT4/ibnzterrell_Nvidia-Llama-3.1-Nemotron-70B-Instruct-HF-AWQ-INT4_qpc_16cores_128pl_4096cl_1fbs_2devices_mxfp6_mxint8.tar.gz 43GB Download https://dc00tk1pxen80.cloudfront.net/SDK1.21.2/ibnzterrell/Nvidia-Llama-3.1-Nemotron-70B-Instruct-HF-AWQ-INT4/ibnzterrell_Nvidia-Llama-3.1-Nemotron-70B-Instruct-HF-AWQ-INT4_ONNX.tar.gz Download 18-Mar-2026

Run This Model

Download QPCs

mkdir -p ibnzterrell/Nvidia-Llama-3.1-Nemotron-70B-Instruct-HF-AWQ-INT4
cd ibnzterrell/Nvidia-Llama-3.1-Nemotron-70B-Instruct-HF-AWQ-INT4

# Download QPC
wget <QPC_Download_URL>
tar xzvf <qpc_filename.tar.gz>

Run QPC

Replace QPC_PATH with actual extracted QPC directories.

python3 -m vllm.entrypoints.openai.api_server \
  --port <PORT> \
  --model ibnzterrell/Nvidia-Llama-3.1-Nemotron-70B-Instruct-HF-AWQ-INT4 \
  --device-group <DEVICE_IDS> \
  --max-model-len <CTX_LEN> \
  --max-seq-len-to-capture <PREFILL_SEQ_LEN> \
  --max-num-seqs <MAX_NUM_SEQS> \
  --quantization mxfp6 \
  --kv-cache-dtype mxint8 \
  --override-qaic-config "num_cores:[num_cores] qpc_path:[qpc_path] ccl_enabled:True comp_ctx_lengths_prefill=[1024,2048,4096] comp_ctx_lengths_decode=[1024,2048,4096]"

Run Inference

Once the server is running, send a request to the OpenAI-compatible endpoint:

curl http://localhost:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ibnzterrell/Nvidia-Llama-3.1-Nemotron-70B-Instruct-HF-AWQ-INT4",
    "messages": [
      {"role": "user", "content": "<PROMPT>"}
    ],
    "max_tokens": 200
  }'