Transformer 推理为了不重算前面 token，会把所有 Key 和 Value 缓存下来——这就是 KV cache。短上下文时无所谓，但一旦到长上下文，事情失控: Llama2-13B 跑 128k context，KV cache 单独就吃 80GB——一张 A100 ...
Some results have been hidden because they may be inaccessible to you
Show inaccessible resultsSome results have been hidden because they may be inaccessible to you
Show inaccessible results