DNeo: Your Parallel Draft Model Needs Better Target Feature Encoding
Abstract
A lightweight feature-encoding method with rejection memory and architectural enhancements enables faster large language model inference by improving speculative-diffusion draft verification.
Speculative-diffusion decoding uses a small parallel drafter model to generate b draft tokens in one forward pass, which then undergo parallel verification via a larger target model. Faster inference can be achieved when multiple draft tokens pass verification simultaneously. Recent parallel draft architectures (DFlash, DSpark) improve draft quality by conditioning on rich features computed by the target, through a mechanism known as feature encoding. However, standard feature encoding mechanisms have three limitations: (1) they are lossy, applying encoding projections that are destructive to the underlying target feature-space via a discovered effect known as residual mixing, (2) they are memory-less, with no record of previously processed proposals, and (3) they utilize features sampled uniformly from across the target layers, omitting important hidden states. To address these issues, we propose Mixture-of-Features Attention (MoFA), a lightweight feature-encoding approach equipped with a working memory that encodes previously rejected proposals. Leveraging MoFA, rejection memory, and a set of further enhancements, including 2-tap causal convolutions, prefix-only cross-attention, and non-uniform target layer selection, the final architecture, DNeo, achieves 1.3x faster generation speed than SoTA (DFlash-2) without impacting drafter footprint or generation quality. DNeo marks a new state of the art in high quality, fast LLM inference.
Community
Diffusion drafters need better feature encoding.
We discover that existing feature encoders destroy the feature space of the target model (residual mixing). Next, we show that feature encoders should possess working memory of previous mistakes (rejection memory). We also find that feature encoders should focus on only the most useful target model layers (the latter half).
The proposed architecture DNeo fixes these issues, using Mixture-of-Features Attention (MoFA) as a superior encoding mechanism, alongside other enhancements . DNeo exhibits up to 30% faster inference speeds than SoTA.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Contextual Correction of Parallel Drafts for Speculative Decoding (2026)
- D3D: Decoupled Diffusion Drafting for Asynchronous Speculative Decoding (2026)
- ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models (2026)
- Factorization-Guided Training and Inference for Diffusion-Drafted Speculative Decoding (2026)
- Draft in Parallel, Condition Through Depth: Adjacent Causal Injection for Speculative Decoding (2026)
- LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter (2026)
- DEdit: Iterative Draft Editing for Speculative Decoding (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.21925 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper

