LLM/airllm
1
0
Fork 0
mirror of https://github.com/0xSojalSec/airllm.git synced 2026-03-07 14:24:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

update wechat group bar code

Browse Source
This commit is contained in:
Yu Li
2023-09-22 22:30:07 -05:00
parent 3ab3de4e64
commit dcd2a861eb
2 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
anima_100k/README.md
View File

@@ -57,7 +57,7 @@ $`batch{\_}size \times num\_heads \times sequence\_len^2 \times float\_size = 32
$`batch\_size \times num\_heads \times sequence\_len \times block_c \times float\_size = 32\times100k \times 128\times2 = 782MB`$
[XEntropy](https://github.com/NVIDIA/apex/tree/master/apex/contrib/xentropy)可以把seq_len * 32000的ogit的内存分配变成inplace，从而节省一半的内存。
[XEntropy](https://github.com/NVIDIA/apex/tree/master/apex/contrib/xentropy)可以把seq_len * 32000的logit的内存分配变成inplace，从而节省一半的内存。
[Paged 8bit Adamw](https://github.com/TimDettmers/bitsandbytes), 可以通过用8 bit block-wise quantization把adam optimizer中的states, Momentum的内存占用从32 bit降到8 bit，降低4倍。

BIN
assets/wechat_group.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 52 KiB

After

Width:  |  Height:  |  Size: 246 KiB