DEV Community

#vllm

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Six Serving Engines on One Cloud TPU v5e Chip: Gemma 4 E2B

SGLang-JAX wins the speed test
xbill profile

Six Serving Engines on One Cloud TPU v5e Chip: Gemma 4 E2B

#googlecloud #tpu #vllm #jax
14
Comments 4
12 min read
Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050
xbill profile

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

#gemma #vllm #llamacpp #nvidia
16
Comments 1
12 min read
vLLM 0.33, Plainly: CUDA 13, Model Runner V2, and What Breaks on Upgrade
danielsamfdo profile

vLLM 0.33, Plainly: CUDA 13, Model Runner V2, and What Breaks on Upgrade

#vllm #llm #googlecloud #machinelearning
Comments
9 min read
Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack
wolfsea2357 profile

Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack

#ai #opensource #llm #vllm
1
Comments 1
15 min read
Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4
xbill profile

Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4

#aws #ec2 #vllm #nvidia
3
Comments
13 min read
verl GRPO + vLLM Rollouts: GPU Memory Settings for Single-Node Deployment
jangwook_kim_e31e7291ad98 profile

verl GRPO + vLLM Rollouts: GPU Memory Settings for Single-Node Deployment

#rlposttraining #grpo #vllm #gpumemory
Comments
12 min read
vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe
wolfsea2357 profile

vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe

#ai #machinelearning #llm #vllm
Comments
15 min read
DIY Self-Hosting 35B+ Reasoning LLMs with NVFP4 Quantization and vLLM
sharathjapa profile

DIY Self-Hosting 35B+ Reasoning LLMs with NVFP4 Quantization and vLLM

#diy #selfhosting #vllm #nvfp4
Comments
3 min read
Managing Agent GPU Kernels: vLLM's Kimi K3 Fast-Track RFC
rammehta1899 profile

Managing Agent GPU Kernels: vLLM's Kimi K3 Fast-Track RFC

#vllm #gpukernels #cutedsl #codereview
Comments
1 min read
vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap
wolfsea2357 profile

vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap

#ai #python #llm #vllm
Comments 1
13 min read
vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap
wolfsea2357 profile

vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap

#ai #machinelearning #llm #vllm
Comments
14 min read
The Local vLLM Speculative Decoding Blueprint: How I Quadrupled DeepSeek-R1 Inference to 48 Tokens/Sec on Consumer GPUs with...
lifehacknexus profile

The Local vLLM Speculative Decoding Blueprint: How I Quadrupled DeepSeek-R1 Inference to 48 Tokens/Sec on Consumer GPUs with...

#aitoolsproductivity #aitools #deepseekr1 #vllm
Comments
14 min read
Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data
wolfsea2357 profile

Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data

#ai #machinelearning #vllm #llm
Comments 1
13 min read
FP8 Cut Our TPOT by 25%. At High Concurrency, It Also Changed the Answers.
huilin_ye_f804826bf674576 profile

FP8 Cut Our TPOT by 25%. At High Concurrency, It Also Changed the Answers.

#llm #vllm #quantization #benchmarking
Comments
3 min read
Architecting On-Premise Mistral Large 4 for Enterprise SLAs: A Systems Blueprint
burra_harish_d574a4c7327a profile

Architecting On-Premise Mistral Large 4 for Enterprise SLAs: A Systems Blueprint

#architecture #ai #vllm #infrastructure
Comments
4 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.