Skip to content
Snippets Groups Projects
Commit 178c7883 authored by Riko Corwin Uphoff's avatar Riko Corwin Uphoff
Browse files

Added FIXME

parent 5b2b84fa
No related branches found
No related tags found
No related merge requests found
Pipeline #25318 passed
......@@ -44,6 +44,7 @@ def load_data_finetune(args, tokenizer):
dataset = load_dataset(*arg_map[args.dataset])
def tokenize_function_finetune(batch):
# FIXME This fails for GLUE MNLI
return tokenizer(batch["sentence"], truncation=True, padding="max_length", max_length=args.max_length)
dataset = dataset.map(tokenize_function_finetune)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment