How to up-merge?

by ucalyptus - opened May 9, 2024

Owner May 9, 2024

@mlabonne Hi, thanks for wonderful insights on llama-3b 120B and 225B.

How would the mergekit_config.yaml look like if the num_hidden_layers are doubled from 22 to 44 in premai-io/prem-1b-chat instead of halving as I did in this repository?

mlabonne

May 10, 2024

Layers that are duplicated are the ones that are shared in the layer_range, so you'd have the same config twice in your case (see https://huggingface.co/blog/mlabonne/merge-models#4-passthrough)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment