Sensible benchmarks exhibiting quicker inter-token latency when deploying Qwen3 fashions with vLLM, Kubernetes, and AWS AI Chips.
Speculative decoding on AWS Trainium can speed up token technology by as much as 3x for decode-heavy workloads, serving to scale back the fee per output token and bettering throughput with out sacrificing output high quality. If you happen to construct AI writing assistants, coding brokers, or different generative AI functions, your workloads possible produce way more tokens than they eat, making the decode stage the dominant value of inference. Throughout autoregressive decoding, tokens are generated sequentially, leaving {hardware} accelerators memory-bandwidth-bound and underutilized. This drives up the fee per generated token. Speculative decoding addresses this bottleneck by letting a small draft mannequin suggest a number of tokens directly, which the goal mannequin verifies in a single ahead go. Fewer serial decode steps means decrease latency and better {hardware} utilization, serving to to cut back your inference prices.
On this submit, you’ll study:
- How speculative decoding works and why it helps scale back value per generated token on AWS Trainium2
- Tips on how to allow speculative decoding with vLLM on Trainium
- The benchmarking methodology we used to guage efficiency
- Tips on how to tune draft mannequin choice and the speculative token window dimension in your workloads
- Step-by-step directions to breed the outcomes utilizing Qwen3
What’s speculative decoding?
Speculative decoding accelerates autoregressive technology by utilizing two fashions:
- A draft mannequin proposes n candidate tokens shortly.
- A goal mannequin verifies them in a single ahead go.
For a deeper have a look at the underlying mechanics, together with token acceptance and rejection, EAGLE-based hypothesis, and normal speculative decoding ideas, see weblog submit Inferentia2this SageMaker EAGLE walkthrough on AWS Inferentia2, this SageMaker EAGLE walkthrough, and this primer. Right here, we deal with the 2 knobs you management in apply: the draft mannequin and num_speculative_tokens.
The draft and goal fashions should share the identical tokenizer and vocabulary, as a result of speculative decoding operates on token IDs verified straight by the goal mannequin. We advocate selecting fashions from the identical architectural household as a result of their next-token predictions agree extra typically. You may pair fashions with totally different architectures in the event that they share a tokenizer, however decrease settlement between the draft and goal fashions reduces acceptance charges and removes a lot of the efficiency acquire.
When the goal mannequin accepts the draft tokens, they’re dedicated with out incurring the total value of sequential decode steps. The first parameter you management is num_speculative_tokens, which units what number of tokens the draft mannequin proposes directly. Growing this worth allows you to skip extra serial decode steps per verification go, straight decreasing inter-token latency when acceptance charges are excessive.
The efficiency acquire comes from two results. First, speculative decoding reduces the variety of target-model decode steps, which lowers the variety of KV-cache reminiscence spherical journeys. (The KV cache shops beforehand computed key and worth tensors so the mannequin doesn’t recompute consideration for previous tokens. Every decode step reads the total cache from reminiscence, making decode memory-bandwidth-bound.) Second, speculative decoding improves {hardware} utilization throughout decoding. In commonplace autoregressive decoding, every decode step produces solely a single new token: the accelerator launches costly matrix-multiply kernels to supply only one token of labor, leaving the processing-element engine largely underutilized. Throughout verification, the goal mannequin as an alternative processes n tokens directly, amortizing reminiscence entry and turning a sequence of small, inefficient single-token computations right into a extra compute-dense workload. Setting num_speculative_tokens too low limits pace beneficial properties.
Setting it too excessive will increase the probability of early rejections, losing draft compute and elevating target-model verification value. You tune this worth by balancing draft compute towards verification value based mostly in your noticed acceptance charge.
Determine 1 Speculative decoding config tradeoffs
For instance these tradeoffs, we in contrast Qwen3-0.6B and Qwen3-1.7B draft fashions. The smaller 0.6B mannequin was quicker to run, however its acceptance charge was roughly 60% decrease, sufficient to cancel out the compute financial savings. Qwen3-1.7B struck a greater stability between pace and acceptance.
For num_speculative_tokens, we evaluated values from 5 to fifteen. Smaller settings (for instance, 5) provided restricted speedup. Bigger home windows (for instance, 15) elevated rejections and degraded efficiency. The very best configuration depended closely on immediate construction. We examined each structured prompts (similar to repetition, numeric sequences, and easy code) and open-ended pure language. The very best stability got here from Qwen3-1.7B with 7 speculative tokens. See the Classes discovered part for full tuning particulars.
What NeuronX Distributed Inference (NxD Inference) helps
AWS Neuron is the SDK for AWS AI chips. NeuronX Distributed Inference (NxDI) is its library for scalable, high-performance LLM inference on Trainium and Inferentia. NxDI supplies native assist for speculative decoding on Trainium throughout 4 modes:
- Vanilla speculative decoding — Separate draft and goal fashions compiled independently. The best method to get began.
- Fused hypothesis — Draft and goal fashions compiled collectively for improved efficiency. That is the mode we use on this submit.
- EAGLE hypothesis — The draft mannequin leverages hidden-state context from the goal mannequin to enhance acceptance charges.
- Medusa hypothesis — A number of small prediction heads run in parallel to suggest tokens, decreasing draft-model overhead.
For full documentation, see the Speculative Decoding information and the EAGLE Speculative Decoding information. This submit makes use of fused hypothesis, the place the draft mannequin (Qwen3-1.7B) and goal mannequin (Qwen3-32B) are compiled along with enable_fused_speculation=true for optimum efficiency on Neuron.
Getting began with speculative decoding on AWS Trainium
We deploy two vLLM inference companies on Trainium cases in the identical Amazon Elastic Kubernetes Service (Amazon EKS) cluster, maintaining all the pieces equivalent besides the decoding methodology to isolate the efficiency affect. The baseline service (qwen-vllm) serves Qwen3-32B with commonplace decoding. The speculative service (qwen-sd-vllm) serves the identical Qwen3-32B goal mannequin, including a Qwen3-1.7B draft mannequin with num_speculative_tokens=7.
Each companies run equivalent configurations on Trn2 (trn2.48xlarge), the identical accelerator allocation, tensor parallelism (which distributes mannequin weights throughout a number of NeuronCores to suit giant fashions), sequence size, batching limits, and Neuron DLC picture. The one distinction is the addition of the Qwen3-1.7B draft mannequin and num_speculative_tokens=7 for the speculative service. See Determine 2 for full setup particulars.
To check the 2 configurations below equivalent load, we used llmperf to generate the identical site visitors patterns towards each endpoints. We captured infrastructure telemetry with CloudWatch Container Insights and printed request-level customized metrics (TTFT, inter-token latency, and end-to-end latency) to CloudWatch dashboards for side-by-side evaluation.
Determine 2 System structure
Benchmarking setup
We used LLMPerf to run structured, decode-heavy check circumstances towards each the baseline and speculative decoding deployments. The benchmarks ran inside a Kubernetes pod, qwen-llmperf-pod.yaml, issuing concurrent requests to each endpoints and logging token-level latency metrics. Our check circumstances ranged from extremely structured prompts (repetitive sequences, numeric continuations, easy code patterns) to open-ended pure language completions, overlaying each best-case and worst-case conduct for speculative decoding. The complete immediate set is out there within the samples repository.
For readability, we focus the evaluation on two consultant immediate varieties: a extremely structured, deterministic immediate (repetitive textual content technology) and an open-ended immediate. These two circumstances illustrate each the best-case and worst-case conduct of speculative decoding.
The pod ran llmperf with managed enter and output lengths and temperature=0.0 to emphasize deterministic decoding paths. We logged and printed metrics together with inter-token latency, TTFT, throughput, and end-to-end latency to CloudWatch.
Outcomes
Determine 3 Speculative decoding E2E latency
Speculative decoding reduces latency selectively: its effectiveness relies upon strongly on immediate construction, and this dependency seems constantly throughout the measured metrics. Here’s what you possibly can anticipate for every immediate sort:
- Structured prompts (for instance, “Repeat the next line precisely 50 instances”). Speculative decoding delivers a measurable discount in end-to-end latency. When the draft mannequin reliably predicts what the goal mannequin would generate, the system skips a considerable fraction of target-model decode steps. In our exams, inter-token latency dropped to roughly 15 ms per token (in comparison with roughly 45 ms for open-ended prompts), and the speculative decoding curve remained constantly under the baseline all through the run.
- Open-ended prompts (for instance, “I imagine the which means of life is”). Speculative decoding supplies no constant profit. The draft mannequin regularly diverges from the goal mannequin, inflicting token rejections that negate the potential beneficial properties. The speculative and baseline end-to-end latency curves largely overlap, and inter-token latency stays close to 45 ms per token for each configurations.
Determine 4 Speculative decoding inter-token latency (Decode)
TTFT (Time to First Token) stays successfully unchanged throughout the configurations (Determine 5). TTFT is dominated by the prefill section, the place the mannequin encodes the enter context. Speculative decoding doesn’t alter this stage, so prefill latency is neither improved nor degraded.
Determine 5 Speculative decoding TTFT (Prefill)
Taken collectively, these outcomes present that speculative decoding improves complete latency by decreasing the variety of target-model decode steps executed, not by accelerating the decode step itself or the prefill stage. This explains why beneficial properties seem in end-to-end latency for structured prompts, however are absent in inter-token latency and TTFT, and why speculative decoding returns to baseline conduct for open-ended technology.
Reproducing the outcomes
We offer end-to-end code samples and Kubernetes configurations within the AWS Neuron EKS samples repository. The repository consists of:
- Kubernetes manifests for deploying baseline vLLM and speculative decoding vLLM companies on Trn2
- Instance vLLM configuration flags for enabling fused speculative decoding
- Pattern llmperf benchmarking scripts used to generate load and acquire metrics
- Directions for mounting mannequin checkpoints and compiled artifacts by the S3 CSI Driver
- Steering on configuring Neuron DRA, tensor parallelism, and NeuronCore placement
These samples allow you to recreate the identical experimental setup used on this submit, from mannequin deployment by benchmarking and metrics assortment.
Conclusion
Decode-heavy LLM workloads are constrained by the sequential nature of autoregressive technology. Speculative decoding breaks this bottleneck on AWS Trainium2 by decreasing the variety of target-model decode steps wanted to supply the total output, successfully rising the tokens generated per ahead go. For workloads the place the output house is predictable, similar to code technology, structured information extraction, templated report technology, or configuration file synthesis, this could translate on to decrease value per output token and better throughput, with out sacrificing high quality. Speculative decoding just isn’t a common optimization. Its effectiveness relies on immediate construction, draft-model high quality, and speculative parameter tuning. When utilized to the proper workloads, it delivers significant latency and price enhancements on Trainium-based inference methods.
Subsequent steps
To get began with speculative decoding on AWS Trainium, discover these sources:
Concerning the authors
Yahav Biran is a Principal Architect at Amazon, specializing in large-scale AI workloads. He contributes to open-source initiatives and publishes in AWS blogs and tutorial journals, together with the AWS compute and AI blogs and the Journal of Methods Engineering. He regularly delivers technical shows and collaborates with prospects to design Cloud functions. Yahav holds a Ph.D. in Methods Engineering from Colorado State College.
Truong Pham is a software program engineer at Annapurna Labs, Amazon. He makes a speciality of optimizing giant language mannequin inference efficiency on AWS AI accelerators similar to AWS Inferentia and Trainium — and designing developer-friendly APIs for the AWS Neuron software program stack. Truong holds a Ph.D. in Chemical Engineering from the College of Minnesota.
