Enterprise Deployment Guide: Multi-Node Cluster Deployment of Qwen Series Models with SGLang

Objective

While Ollama offers convenience, high concurrency is sometimes more crucial. This article demonstrates how to deploy SGLang on two computers (dual nodes) to run the Qwen2.5-7B-Instruct model, maximizing local resource utilization. Additional nodes can be added if available.

Hardware Requirements

• Node 0: IP 192.168.0.12, 1 NVIDIA GPU
• Node 1: IP 192.168.0.13, 1 NVIDIA GPU
• Total: 2 GPUs

Model Specifications

Qwen2.5-7B-Instruct requires approximately 14GB VRAM in FP16. With --tp 2, each GPU needs about 7GB (weights) + 2-3GB (KV cache).

Network Configuration

Nodes communicate via Ethernet (TCP), using the eno1 network interface.

Note: Check your actual interface using ip addr command

Precision

Using FP16 precision to maintain maximum accuracy, resulting in higher VRAM usage that requires optimization.

2. Prerequisites

Ensure the following requirements are met before installation and deployment:

Operating System

• Recommended: Ubuntu 20.04/22.04 or other Linux distributions (Windows not recommended, requires WSL2)
• Consistent environments across nodes preferred, though OS can differ if Python environments match

Network Connectivity

• Node 0 (192.168.0.12) and Node 1 (192.168.0.13) must be able to ping each other:

ping 192.168.0.12  # from Node 1
ping 192.168.0.13  # from Node 0

• Ports 50000 (distributed initialization) and 30000 (HTTP server) must not be blocked by firewall:

sudo ufw allow 50000
sudo ufw allow 30000

• Verify network interface eno1:

# Adjust interface name as needed
ip addr show eno1 

If eno1 doesn’t exist, use your actual interface (e.g., eth0 or enp0s3).

GPU Drivers and CUDA

• Install NVIDIA drivers (version ≥ 470) and CUDA Toolkit (12.x recommended):

nvidia-smi  # verify driver and CUDA version

Output should show NVIDIA and CUDA versions (e.g., 12.4).

If not installed, refer to NVIDIA’s official website for installation.

Python Environment

• Python 3.9+ (3.10 recommended)
• Consistent Python versions across nodes:

python3 --version

Disk Space

• Qwen2.5-7B-Instruct model requires approximately 15GB disk space
• Ensure sufficient space in /opt/models/Qwen/Qwen2.5-7B-Instruct path

3. Installing SGLang

Install SGLang and dependencies on both nodes. Execute the following steps on each computer.

3.1 Create Virtual Environment (conda)

conda create -n sglang_env python=3.10
conda activate sglang_env

3.2 Install SGLang

Note: Installation will automatically include GPU-related dependencies like torch, transformers, flashinfer

pip install --upgrade pip
pip install uv
uv pip install "sglang[all]>=0.4.5" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

Verify installation:

python -m sglang.launch_server --help

Should display SGLang’s command-line parameter help information.

3.3 Download Qwen2.5-7B-Instruct Model

Use huggingface internationally, modelscope within China

Download the model to the same path on both nodes (e.g., /opt/models/Qwen/Qwen2.5-7B-Instruct):

pip install modelscope
modelscope download Qwen/Qwen2.5-7B-Instruct --local-dir /opt/models/Qwen/Qwen2.5-7B-Instruct

Alternatively, manually download from Hugging Face or modelscope and extract to the specified path. Ensure model files are identical across nodes.

4. Configuring Dual-Node Deployment

Use tensor parallelism (—tp 2) to distribute the model across 2 GPUs (one per node). Below are the detailed deployment steps and commands.

4.1 Deployment Commands

• Node 0 (IP: 192.168.0.12):

NCCL_IB_DISABLE=1 NCCL_P2P_DISABLE=1 GLOO_SOCKET_IFNAME=eno1 NCCL_SOCKET_IFNAME=eno1 python3 -m sglang.launch_server \
  --model-path /opt/models/Qwen/Qwen2.5-7B-Instruct \
  --tp 2 \
  --nnodes 2 \
  --node-rank 0 \
  --dist-init-addr 192.168.0.12:50000 \
  --disable-cuda-graph \
  --host 0.0.0.0 \
  --port 30000 \
  --mem-fraction-static 0.7

• Node 1 (IP: 192.168.0.13):

NCCL_IB_DISABLE=1 NCCL_P2P_DISABLE=1 GLOO_SOCKET_IFNAME=eno1 NCCL_SOCKET_IFNAME=eno1 python3 -m sglang.launch_server \
  --model-path /opt/models/Qwen/Qwen2.5-7B-Instruct \
  --tp 2 \
  --nnodes 2 \
  --node-rank 1 \
  --dist-init-addr 192.168.0.12:50000 \
  --disable-cuda-graph \
  --host 0.0.0.0 \
  --port 30000 \
  --mem-fraction-static 0.7

Note: If OOM occurs, adjust the --mem-fraction-static parameter from the default 0.9 to 0.7. This change reduces VRAM usage by about 2GB for the current 7B model. CUDA Graph allocates additional VRAM (typically hundreds of MB) to store computation graphs. If VRAM is near capacity, enabling CUDA Graph may trigger OOM errors.

Additional Parameters and Information

Original Article