Skip to content
Snippets Groups Projects
Commit 31552c61 authored by Riko Corwin Uphoff's avatar Riko Corwin Uphoff
Browse files

Updated batch size

parent 140033be
Branches
No related tags found
No related merge requests found
Pipeline #25298 passed
Stage: build
Hide whitespace changes
Inline Side-by-side
scripts/shell/pretrain_60m.sh
+ 2
− 2
View file @ 31552c61
...@@ -4,9 +4,9 @@ python3 main.py \ ...@@ -4,9 +4,9 @@ python3 main.py \
--mode pretraining \ --mode pretraining \
--optimizer galore \ --optimizer galore \
--model llama_60m \ --model llama_60m \
--batch_size 8 \ --batch_size 512 \
--num_epochs 1 \ --num_epochs 1 \
--num_training_tokens 10000 \ --num_training_tokens 1310000000 \
--max_length 256 \ --max_length 256 \
--shuffle false \ --shuffle false \
--dtype bf16 \ --dtype bf16 \
......
scripts/shell/pretrain_7b.sh
+ 1
− 1
View file @ 31552c61
...@@ -4,7 +4,7 @@ python3 main.py \ ...@@ -4,7 +4,7 @@ python3 main.py \
--mode pretraining \ --mode pretraining \
--optimizer galore8bit \ --optimizer galore8bit \
--model llama_7b \ --model llama_7b \
--batch_size 131000 \ --batch_size 512 \
--num_epochs 1 \ --num_epochs 1 \
--num_training_tokens 13100000 \ --num_training_tokens 13100000 \
--max_length 256 \ --max_length 256 \
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment