base_model: unsloth/mistral-7b-instruct-v0.3-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- mistral
- trl
license: apache-2.0
datasets:
- neph1/bellman-7b-finetune
- neph1/bellman-multiturn
language:
- sv
It's finetuned for prompt question answering, based on a dataset created from Swedish wikipedia, with a lot of Sweden-centric questions. New in this version is a multi-turn dataset of about 250 conversations, as well as a number of stories.
The name comes from the Swedish bard and poet Carl Mikael Bellman who lived in the 1700s. As with any bard, what this model says should be taken with a grain of salt. Even though it has the best of intentions.
Configuration:
Rank: 256
Alpha: 512
Learning rate (at start): 2e-5
Context length: 4096
Training length: ca 2 epochs
Important. Use correct prompt format for best results: [INST]Hur bakar jag en sockerkaka?[/INST]
TrainingArguments( per_device_train_batch_size = 6, gradient_accumulation_steps = 20, num_train_epochs=4, warmup_steps = 10, learning_rate = 2e-5, bf16 = true, logging_steps = 5, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, per_device_eval_batch_size = 6, eval_strategy="steps", eval_accumulation_steps = 20, eval_steps = 5, eval_delay = 0, save_strategy="steps", save_steps=5, report_to="none", output_dir="", )
Uploaded model
- Developed by: neph1
- License: apache-2.0
- Finetuned from model : unsloth/mistral-7b-instruct-v0.3-bnb-4bit
This mistral model was trained 2x faster with Unsloth and Huggingface's TRL library.
