fix: copy decoder caches once per surviving beam, not per candidate
Each beam-search step deep-cloned the keys and values of every layer for every candidate it kept, so the cost grew quadratically with the sentence length and produced garbage faster than the unrolled dot() could win. Survivors are now chosen first; a beam whose parent has a single survivor takes its slices over (the parent is dead), and only a shared cache is cloned. The step's other buffers (query/key/value, hidden, logits) are reused across layers, beams and steps instead of being allocated afresh each time. TestRealModel confirms the output is unchanged.
This commit is contained in:
1 parent
e80ad634ef
commit
d6ee57130f
3 files changed
+74
-32
No files matched your search
@@ -21,6 +21,9 @@ version is in the `VERSION` file.
|
||||
power cut just after a save could leave an empty script). An existing file keeps its
|
||||
permissions, and saving through a symlink replaces the file it points to instead of
|
||||
replacing the link.
|
||||
- Translation is faster and uses much less memory: the beam search no longer copies
|
||||
every beam's decoder state for every candidate at every step, and reuses its
|
||||
work buffers instead of allocating new ones per layer and step.
|
||||
|
||||
## [1.4] - 2026-10-15
|
||||
|
||||
|
||||
Reference in new issue
Block a user