P Wang

CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decoding

2024 · arXiv (Cornell University)

We present CREST (Compact Retrieval-Based Speculative Decoding), a redesign of REST that allows it to be effectively "compacted". REST is a drafting technique for speculative decoding based on retrieving exact n-gram matches of the most …