Finetuning llama 3.2 1B Quantized #231

HURIMOZ · 2025-04-16T11:54:17Z

HURIMOZ
Apr 16, 2025

Hi everyone, Iʻm getting issues upon inference on the llama 3.2 1B Quantized model I finetuned.
Hereʻs my training config:

# General settings
seed: 1234
share_vocab: true
save_data: "eole_processed_data/Llama-3.2-1B-Instruct_EN-FR-TY"

src_vocab: "models/Llama-3.2-1B-Instruct/tokenizer.json"

overwrite: true
report_every: 50

# Datasets
data:
  en_ty:
    path_src: "data/en-ty/en-ty-promptized.jsonl"
    weight: 1
    transforms: ['huggingface_tokenize']
  fr_ty:
    path_src: "data/fr-ty/fr-ty-promptized.jsonl"
    weight: 1
    transforms: ['huggingface_tokenize']
  en_haw:
    path_src: "data/en-haw/en-haw-everything-promptized-train.jsonl"
    weight: 3
    transforms: ['huggingface_tokenize']
  en_haw:
    path_src: "data/en-haw/sail-lab-promptized-train.jsonl"
    weight: 1
    transforms: ['huggingface_tokenize']
  en_pmt:
    path_src: "data/en-pmt/en-pmt-testing-always-train.jsonl"
    weight: 1
    transforms: ['huggingface_tokenize']
  en_mi:
    path_src: "data/en-mi/combined-en-mi-promptized-train.jsonl"
    weight: 1
    transforms: ['huggingface_tokenize']

  valid:
    path_src: "data/all-validation-data.jsonl"
    transforms: ['huggingface_tokenize']

skip_empty_level: silent

# Tokenizer and Preprocessing Configuration
transforms: []
transforms_configs:
  huggingface_tokenize:
    huggingface_model: "meta-llama/Llama-3.2-1B-Instruct"
    max_length: 512

# Training settings
training:
  # GPU dispatching
  world_size: 1
  gpu_ranks: [0]

  # Dropout settings
  dropout_steps: [0]
  dropout: [0.05]
  attention_dropout: [0.05]

  # Batching
  bucket_size: 512
  num_workers: 2
  batch_type: "tokens"
  batch_size: 96
  accum_count: [16]
  valid_batch_size: 96
  batch_size_multiple: 1

  # Optimization
  compute_dtype: "fp16"
  optim: "adamw"
  learning_rate: 0.0002
  warmup_steps: 200
  decay_method: "cosine"

  # Quantization (QLoRA)
  quant_layers: ['gate_up_proj', 'down_proj', 'up_proj', 'linear_values', 'linear_query', 'linear_keys', 'final_linear']
  quant_type: "bnb_NF4"

  # LoRA settings
  lora_layers: ['linear_values', 'linear_query', 'linear_keys', 'final_linear']
  lora_rank: 16
  lora_dropout: 0.05
  lora_alpha: 32
  lora_embedding: false

  # Training duration and checkpointing
  train_steps: 3000
  valid_steps: 100
  save_checkpoint_steps: 100
  keep_checkpoint: 15

  # Model paths
  train_from: "models/Llama-3.2-1B-Instruct"
  model_path: "finetune/Llama-3.2-1B-Instruct_EN-FR-TY_Laptop_v4"

# Tensorboard logging
tensorboard: true
tensorboard_log_dir: "logs/Llama-3.2-1B-Instruct_EN-FR-TY_Laptop_v4/tensorboard"
log_file: "logs/Llama-3.2-1B-Instruct_EN-FR-TY_Laptop_v4/training.log"

Here is the format of my training prompt (in jsonl format):

{"text": "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nIn Tahitian:\npatient demonstrates improved range of motion following six weeks of post-operative physical therapy, with flexion now at 120 degrees<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\ne ʻitehia ra te haʻamaitaʻiraʻa i te haʻutiʻutiraʻa o te taʻata maʻi a ono hepetoma i muri mai i te tāpūraʻa hia maoti te faʻahaʻutiʻutiraʻa tino, inaha e tīfene i te fāito 120 teteri<|eot_id|>"}

Here is my inference config file:

# Model info
model_path: "finetune/merged"

# HuggingFace tokenizer configuration
transforms: []
transforms_configs:
  huggingface_tokenize:
    huggingface_model: "meta-llama/Llama-3.2-1B-Instruct"
    max_length: 512

# Inference
seed: 1234
max_length: 512
batch_type: tokens
batch_size: 96
world_size: 1
gpu_ranks: [0]

# Quantization settings
#quant_layers: ['gate_up_proj', 'down_proj', 'up_proj', 'linear_values', 'linear_query', 'linear_keys', 'final_linear']
#quant_type: "bnb_NF4"
#compute_dtype: fp16

# Generation settings
temperature: 0.6
top_k: 50
top_p: 0.9
beam_size: 1
n_best: 1
report_time: true
src: None

and here is the format of my inference prompt (not in jsonl format):

<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nTranslate to Tahitian:\nPlaying gods with the cradle of life: French Polynesia’s president issues warning over deep-sea mining.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

And this is the output I get:

the ! s itu ations ! that ! are ! ! f acing ! ! the ! French ! ! people ! !ĊĊ {" text ": Ġ" <|begin_of_text|> import Ġre \n \n French Ġto ĠEnglish Ġtranslation :\ n Ãł Ġ 15 h 30

Can please tell me what Iʻm doing wrong?

vince62s · 2025-04-17T15:40:58Z

vince62s
Apr 17, 2025
Maintainer

your path_src dataset cannot be in jsonl format, it has to be plain text and you need to replace "\n" by the default.SEP token.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Finetuning llama 3.2 1B Quantized #231

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Finetuning llama 3.2 1B Quantized #231

Uh oh!

HURIMOZ Apr 16, 2025

Replies: 1 comment

Uh oh!

vince62s Apr 17, 2025 Maintainer

HURIMOZ
Apr 16, 2025

vince62s
Apr 17, 2025
Maintainer