fix: copy decoder caches once per surviving beam, not per candidate

Each beam-search step deep-cloned the keys and values of every layer
for every candidate it kept, so the cost grew quadratically with the
sentence length and produced garbage faster than the unrolled dot()
could win. Survivors are now chosen first; a beam whose parent has a
single survivor takes its slices over (the parent is dead), and only a
shared cache is cloned. The step's other buffers (query/key/value,
hidden, logits) are reused across layers, beams and steps instead of
being allocated afresh each time. TestRealModel confirms the output is
unchanged.
This commit is contained in:
Kharec committed 2026-10-15 17:52:37 +02:00
1 parent e80ad634ef
commit d6ee57130f
3 files changed
+74 -32

No files matched your search

+3
View File
@@ -21,6 +21,9 @@ version is in the `VERSION` file.
power cut just after a save could leave an empty script). An existing file keeps its
permissions, and saving through a symlink replaces the file it points to instead of
replacing the link.
- Translation is faster and uses much less memory: the beam search no longer copies
every beam's decoder state for every candidate at every step, and reuses its
work buffers instead of allocating new ones per layer and step.
## [1.4] - 2026-10-15