DeepSeek R1 Distill Qwen 32B AWQ

Model Overview

This quantized model was created using AutoAWQ version 3.2.7.post3 with quant_config: { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

DeepSeek-R1 and its distilled models represent a significant advancement in reasoning capabilities for LLMs by combining RL, SFT, and distillation. The DeepSeek-R1-Distill-Qwen-32B is a distilled version of the Qwen-32B large language model (LLM), optimized for efficient performance while retaining high-quality generative capabilities and is particularly suited for scenarios where computational efficiency is critical.

• Model Architecture: DeepSeek-R1-Distill-Qwen-32B is based on a transformer architecture, distilled from the larger Qwen-32B model to reduce computational requirements while maintaining competitive performance. The distillation process ensures that the model retains the core capabilities of the original model, making it suitable for a wide range of text generation tasks.
• Model Source: Valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ
• License: apache-2.0

QPC Configurations

Precision	SoCs / Tensor slicing	NSP-Cores (per SoC)	Full Batch Size	Chunking Prompt Length	Context Length (CL)	QPC URL	QPC Size	QPC Download	Onnx URL	Onnx Download	Generation Date
MXFP6	2	16	1	128	4096	https://dc00tk1pxen80.cloudfront.net/SDK1.21.2/Valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ/Valdemardi_DeepSeek-R1-Distill-Qwen-32B-AWQ_qpc_16cores_128pl_4096cl_1fbs_2devices_mxfp6_mxint8.tar.gz	22GB	Download	https://dc00tk1pxen80.cloudfront.net/SDK1.21.2/Valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ/Valdemardi_DeepSeek-R1-Distill-Qwen-32B-AWQ_ONNX.tar.gz	Download	18-Mar-2026

Run This Model

Download QPCs

mkdir -p valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ
cd valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ

# Download QPC
wget <QPC_Download_URL>
tar xzvf <qpc_filename.tar.gz>

Run QPC

Replace QPC_PATH with actual extracted QPC directories.

python3 -m vllm.entrypoints.openai.api_server \
  --port <PORT> \
  --model valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ \
  --device-group <DEVICE_IDS> \
  --max-model-len <CTX_LEN> \
  --max-seq-len-to-capture <PREFILL_SEQ_LEN> \
  --max-num-seqs <MAX_NUM_SEQS> \
  --quantization mxfp6 \
  --kv-cache-dtype mxint8 \
  --override-qaic-config "num_cores:[num_cores] qpc_path:[qpc_path] ccl_enabled:True comp_ctx_lengths_prefill=[1024,2048,4096] comp_ctx_lengths_decode=[1024,2048,4096]"

Run Inference

Once the server is running, send a request to the OpenAI-compatible endpoint:

curl http://localhost:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ",
    "messages": [
      {"role": "user", "content": "<PROMPT>"}
    ],
    "max_tokens": 200
  }'

Useful Links

• Environment Setup