Comment by huangsemao 7 hours ago What part of the optimization process gave you the biggest speed gain? 2 comments huangsemao Reply gitpusher42 5 hours ago Switching from mmap to parallel pread. From 0.5tok/sec to almost 4tok/sec. Running GPU work while reading missed experts also helped a lot, 4.4 -> 4.7
gitpusher42 5 hours ago Switching from mmap to parallel pread. From 0.5tok/sec to almost 4tok/sec. Running GPU work while reading missed experts also helped a lot, 4.4 -> 4.7
Switching from mmap to parallel pread. From 0.5tok/sec to almost 4tok/sec. Running GPU work while reading missed experts also helped a lot, 4.4 -> 4.7