Help with DeepSeek deployment

[limit_lt]

RTX 4090 (48GB VRAM), i9 processor, 128GB RAM, ASUS ROG Strix Z790-E. Now deploying a large language model. Using vLLM approach, not Ollama, solely for serving DeepSeek (deploying DeepSeek, plus an additional specialized small model) as an inference server. A separate OpenWebUI server is already deployed and will connect to this server. To maximize GPU performance, please provide a detailed deployment guide for the Euler OS.

[SdoHao (Shendong Hao)]

Based on your high-end hardware configuration (RTX 4090 48G / i9 / 128G RAM) and openEuler system environment, here is a maximized GPU performance deployment plan for the vLLM inference server. This setup is optimized for DeepSeek series models and supports concurrent multi-model serving.

Important Note: Consumer-grade RTX 4090 has only 24GB VRAM. The 48GB VRAM version should be the RTX 6000 Ada Generation (workstation-grade). The following guide assumes 48GB VRAM. If you’re using 24GB, adjust your quantization strategy accordingly.

---

1. System Environment Preparation (openEuler 22.03 SP4)

1. Install NVIDIA Driver & CUDA (Critical Step)

# Disable nouveau
cat > /etc/modprobe.d/blacklist-nouveau.conf <<EOF
blacklist nouveau
options nouveau modeset=0
EOF
dracut --force
reboot

# Install dependencies
dnf install -y gcc make kernel-devel-$(uname -r) elfutils-libelf-devel \
  pciutils libglvnd-opengl libglvnd-devel libglvnd-glx

# Download NVIDIA driver from official site (recommended: version 550+)
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.54.15/NVIDIA-Linux-x86_64-550.54.15.run
chmod +x NVIDIA-Linux-x86_64-*.run
./NVIDIA-Linux-x86_64-*.run --no-opengl-files --no-x-check

# Verify driver
nvidia-smi  # Should show GPU model and 48GB VRAM

# Install CUDA 12.3 (officially recommended by vLLM)
wget https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-keyring-1.1-1.noarch.rpm
rpm -ivh cuda-keyring-1.1-1.noarch.rpm
dnf install -y cuda-toolkit-12-3
echo 'export PATH=/usr/local/cuda-12.3/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.3/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

2. Install PyTorch and Dependencies

# Install Python 3.10+ (openEuler 22.03 default is 3.9, needs upgrade)
dnf install -y python3.10 python3.10-pip python3.10-devel

# Create virtual environment
python3.10 -m venv vllm-env
source vllm-env/bin/activate

# Install PyTorch 2.2+ (compatible with CUDA 12.1, works with 12.3)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Install base dependencies
pip install ninja packaging

---

2. vLLM Deployment & Performance Optimization

1. Build from Source (Recommended, Best Performance)

# Install build dependencies
dnf install -y cmake gcc-c++ git

# Clone vLLM repository (use latest stable version)
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.4.3  # Recommended stable version

# Enable FP8 & PagedAttention optimizations (Ada architecture-specific)
export VLLM_TARGET_DEVICE=cuda
export MAX_JOBS=$(nproc)  # Fully utilize i9 multi-core for compilation

pip install -e ".[tensorizer]"  # Enable tensorizer for faster loading

2. Key Performance Configuration (for 48GB VRAM)

# Example startup script: deepseek-v3-deploy.py
from vllm import LLM, SamplingParams

# DeepSeek-V3 (128K context, recommended quantization)
llm = LLM(
    model="deepseek-ai/DeepSeek-V3",
    tensor_parallel_size=1,           # Single GPU, no TP needed
    gpu_memory_utilization=0.95,      # Maximize VRAM usage
    max_model_len=32768,              # Adjust based on context needs
    dtype="bfloat16",                 # Native BF16 support on Ada architecture
    enforce_eager=False,              # Enable CUDA Graph acceleration
    enable_prefix_caching=True,       # Boost performance for repeated prompts
    quantization="fp8",               # FP8 quantization (requires model support)
    max_num_seqs=256,                 # Increase concurrency
    max_num_batched_tokens=32768      # Optimize throughput
)

# Specialized small models (e.g., medical/legal domain)
small_llm = LLM(
    model="your-specialized-model",   # e.g., Qwen/Qwen2.5-7B-Med
    gpu_memory_utilization=0.3,       # Reserve 70% VRAM for main model
    dtype="auto",
    max_model_len=8192
)

Quantization Recommendations:

• 48GB VRAM: Use BF16 (no quantization) for DeepSeek-V3 (~40GB)
• 24GB VRAM: Must use AWQ 4-bit or FP8 quantization
• Specialized small models: Use AWQ 4-bit (7B model ~4GB)

---

3. Multi-Model Service Architecture (Two Options)

Option A: Independent Processes + Nginx Routing (Recommended, High Isolation)

# Start DeepSeek service
python -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-V3 \
  --port 8000 \
  --gpu-memory-utilization 0.95 \
  --dtype bfloat16 \
  --max-model-len 32768

# Start specialized small model service
python -m vllm.entrypoints.openai.api_server \
  --model your-specialized-model \
  --port 8001 \
  --gpu-memory-utilization 0.3 \
  --dtype auto

Nginx Routing Configuration (/etc/nginx/conf.d/vllm.conf)

upstream deepseek_backend {
    server 127.0.0.1:8000;
}
upstream specialist_backend {
    server 127.0.0.1:8001;
}

server {
    listen 8002;
    location /v1/chat/completions {
        proxy_pass http://deepseek_backend;
    }
    location /v1/specialist/completions {
        proxy_pass http://specialist_backend;
    }
}

Option B: Single Process with Multiple LoRA Adapters (Ideal for Same Base Model)

# Suitable for DeepSeek-Coder + domain-specific fine-tuning
llm = LLM(
    model="deepseek-ai/deepseek-coder-33b-instruct",
    enable_lora=True,
    max_loras=4
)
llm.add_lora("medical", "path/to/medical-lora")
llm.add_lora("legal", "path/to/legal-lora")

---

4. Integration with OpenWebUI

1. Configure Model Endpoints in OpenWebUI

• Main Model: http://<vllm-server-ip>:8002/v1
• Specialized Model: http://<vllm-server-ip>:8002/v1/specialist

2. OpenWebUI Configuration Example (Settings → Model Providers)

# Main DeepSeek Model
Provider: OpenAI Compatible
Base URL: http://vllm-server:8002/v1
API Key: not-needed  # vLLM doesn't require API key by default
Model Name: deepseek-v3

# Specialized Small Model
Provider: OpenAI Compatible
Base URL: http://vllm-server:8002/v1/specialist
Model Name: specialist-model

---

5. Ultimate Performance Tuning Checklist

Optimization	Recommended Setting	Performance Gain
CUDA Graph	enforce_eager=False	+15~25% throughput
PagedAttention	Enabled by default	Supports 128K context
BF16 Inference	dtype="bfloat16"	Native Ada architecture acceleration
FP8 Quantization	quantization="fp8"	VRAM ↓30%, speed ↑10%
Prefix Caching	enable_prefix_caching=True	Repeat prompt speed ↑3x
Tensor Parallel	Set to 1 (single GPU)	Avoid communication overhead
Max Sequences	max_num_seqs=256	Boost concurrent throughput
Kernel Parameter	echo 2048 > /proc/sys/vm/max_map_count	Prevent mmap errors

Monitoring Script (Real-time GPU Utilization)

watch -n 0.5 'nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv'

---

6. Troubleshooting & Notes

1. openEuler Compatibility Issues:

   • If compilation fails, try: export CUDACXX=/usr/local/cuda-12.3/bin/nvcc
   • glibc version must be ≥ 2.34 (openEuler 22.03 SP4 meets this)

2. Out of VRAM:

   # Check actual VRAM
   nvidia-smi --query-gpu=memory.total --format=csv
   # If 24GB: DeepSeek-V3 must use AWQ 4-bit quantization
   

3. Accelerate Model Download:

   # Use hf-transfer for faster Hugging Face downloads
   pip install hf-transfer
   export HF_HUB_ENABLE_HF_TRANSFER=1
   

4. Ensure Service Stability:

   # Use systemd to manage service
   cat > /etc/systemd/system/vllm-deepseek.service <<EOF
   [Unit]
   Description=vLLM DeepSeek Service
   After=network.target
   
   [Service]
   User=root
   WorkingDirectory=/opt/vllm
   ExecStart=/opt/vllm/vllm-env/bin/python -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-V3 --port 8000 --gpu-memory-utilization 0.95
   Restart=always
   Environment="CUDA_VISIBLE_DEVICES=0"
   
   [Install]
   WantedBy=multi-user.target
   EOF
   systemctl daemon-reload && systemctl enable --now vllm-deepseek
   

---

7. Expected Performance Metrics (RTX 6000 Ada 48GB)

Model	Quantization	Context	Throughput (tokens/s)	First Token Latency
DeepSeek-V3	BF16	4K	180~220	<80ms
DeepSeek-V3	FP8	32K	140~170	<120ms
7B Specialized Model	AWQ 4-bit	8K	300~350	<40ms

Tip: Adjust --max-num-seqs and --max-num-batched-tokens to further boost throughput—ideal for high-concurrency scenarios.

---

Following this deployment plan, your 48GB GPU will achieve maximum performance, enabling efficient concurrent serving of both DeepSeek main models and specialized small models, with seamless integration into OpenWebUI frontend. For tailored model selection advice based on specific domains, feel free to reach out.