DeepSeek R1 Distill Qwen 32B AWQ

Model Overview¶

This quantized model was created using AutoAWQ version 3.2.7.post3 with quant_config: { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

DeepSeek-R1 and its distilled models represent a significant advancement in reasoning capabilities for LLMs by combining RL, SFT, and distillation. The DeepSeek-R1-Distill-Qwen-32B is a distilled version of the Qwen-32B large language model (LLM), optimized for efficient performance while retaining high-quality generative capabilities and is particularly suited for scenarios where computational efficiency is critical.

Model Architecture: DeepSeek-R1-Distill-Qwen-32B is based on a transformer architecture, distilled from the larger Qwen-32B model to reduce computational requirements while maintaining competitive performance. The distillation process ensures that the model retains the core capabilities of the original model, making it suitable for a wide range of text generation tasks.
Model Source: Valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ
License: apache-2.0

QPC Configurations¶

Precision	SoCs / Tensor slicing	NSP-Cores (per SoC)	Full Batch Size	Chunking Prompt Length	Context Length (CL)	Generated URL	Download	Generation Date
MXFP6	4	16	1	128	8192	https://dc00tk1pxen80.cloudfront.net/SDK1.20.4/Valdemardi/DeepSeek-R1-Distill-Qwen-32B-A/qpc_16cores_128pl_8192cl_1fbs_4devices_mxfp6_mxint8.tar.gz	Download
MXFP6	2	16	1	128	4096	https://dc00tk1pxen80.cloudfront.net/SDK1.20.4/Valdemardi/DeepSeek-R1-Distill-Qwen-32B-A/DeepSeek-R1-Distill-Qwen-32B-A_qpc_16cores_128pl_4096cl_1fbs_2devices_mxfp6_mxint8.tar.gz	Download	19-Jan-2026

Run This Model¶

# Download QPC
mkdir -p Valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ
cd Valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ
wget <Download URL>
tar xzvf <downloaded filename.tar.gz>

# Run QPC
python3 -m QEfficient.cloud.execute --model_name Valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ --qpc_path <path/to/qpc> --prompt "# shortest path algorithm\n" --generation_len 128

API Endpoint¶

# Start REST endpoint with vLLM
VLLM_QAIC_MAX_CPU_THREADS=8 VLLM_QAIC_QPC_PATH=/path/to/qpc python3 -m vllm.entrypoints.openai.api_server \
  --host 0.0.0.0 \
  --port 8000 \
  --model Valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ \
  --max-model-len <Context Length> \
  --max-num-seq <Full Batch Size>  \
  --max-seq_len-to-capture <Chunking Prompt Length>  \
  --device qaic \
  --block-size 32

Useful Links¶

Environment Setup