Fine-Tuning GPT-2 on Your Own Data (A Practical)

1. Quick Context: Why Fine-Tune GPT-2 in 2025?

“Just because you have a hammer, doesn’t mean everything needs to be a nail.”
That’s how I see the GPT-4 hype sometimes.

Yes, GPT-4 is powerful. Yes, it’s everywhere. But here’s the deal: not every project needs a 100B+ parameter monster chewing through tokens on rented A100s. I still use GPT-2 — and not just as a toy model or for benchmarking. It earns its keep in production, especially for:

• Low-latency environments where every millisecond matters
• Projects with tight compute budgets
• On-premise deployments where sending data to an external API isn’t an option
• Fine-tuning niche tasks like chatbot personalization, internal ticket summarization, or domain-specific Q&A — where GPT-4 just doesn’t get the domain nuance, no matter how clever the prompt

I’ve personally fine-tuned GPT-2 for client projects where we needed full control over the model weights, repeatability, and… let’s be real — predictable costs.

So if you’re in that same boat — needing a model that’s lean, fast, and entirely yours — GPT-2 still makes a lot of sense. That’s exactly why I’m sharing this guide.

---

2. Environment Setup (No Shortcuts, Real Setup)

Let me save you a few hours of debugging: the environment is half the battle. Over the past couple of years, I’ve built this setup repeatedly for both local dev and cloud training (Colab Pro, Paperspace, and bare-metal boxes).

Here’s what works flawlessly for me — stable, reproducible, and friendly with both Hugging Face and PyTorch:

Python & Package Versions I Use

I stick to Python 3.10. Some Hugging Face features break on older versions, and I’ve run into edge cases with 3.11. For libraries:

• transformers==4.39.2
• datasets==2.18.0
• accelerate==0.27.2
• evaluate==0.4.1

Why these? Because I’ve had version mismatches wreck training scripts before — especially when models get checkpointed mid-run and suddenly start throwing tensor shape errors after an upgrade. Stick to fixed versions until your pipeline is stable.

CUDA & GPU Setup (What’s Worked for Me)

I’ve trained on both 8GB and 24GB VRAM cards. Trust me, anything under 12GB and you’ll be babysitting memory errors unless you tweak batch size, gradient accumulation, and use fp16.

• If you’re using NVIDIA 20xx/30xx, CUDA 11.8 and PyTorch 2.x is a good combo.
• Don’t forget to check nvidia-smi before you even think about training — I’ve wasted GPU time just by leaving a zombie kernel running.

Conda vs venv — My Take

I’ve used both, but for anything involving mixed CUDA/Torch/Hugging Face dependencies, I always go with conda. It’s more predictable, especially when you’re syncing across dev machines or team members.

My VSCode/Jupyter Setup

If you’re fine-tuning and monitoring metrics live, Jupyter in VSCode is great for quick edits and logs. But for longer runs and structured logging, I just trigger scripts through terminal + tmux, or use Weights & Biases for logging (more on that later).

Full Environment Setup

Here’s exactly what I use — no guesswork:

conda create -n gpt2-finetune python=3.10
conda activate gpt2-finetune

pip install transformers==4.39.2 \
            datasets==2.18.0 \
            accelerate==0.27.2 \
            evaluate==0.4.1 \
            torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

If you’re on Colab or a machine with CUDA mismatch, add this line before launching:

pip install xformers bitsandbytes

These help with memory efficiency, especially if you’re trying to run gpt2-xl on a tight VRAM budget.

---

3. Dataset Preparation (This Is Where Most People Screw Up)

“If you feed junk into the model, don’t be surprised when it learns garbage.”
— Something I remind myself every time I build a dataset.

You might be tempted to just grab some text files, tokenize them, and hit the train button. But let me tell you — the difference between a clean, efficient fine-tune and a frustrating week of loss debugging often comes down to the way your dataset is structured and tokenized.

3.1 Dataset Format — No Room for Assumptions

I like to keep things simple and transparent. I usually work with plain .txt files or a .csv if there’s some label or metadata I might need down the line.

But here’s what matters: the dataset should have a text column, plain and consistent.

Here’s what that looks like in a CSV:

id,text
1,"This is the first sample text."
2,"Here's another one with a different style and tone."
3,"GPT-2 fine-tuning is a balancing act of structure and nuance."

If you’re using JSONL, I keep it like this:

{"text": "This is the first sample text."}
{"text": "Here's another one with a different style and tone."}

Pro tip from experience: I always check for malformed lines or trailing commas — especially in JSONL — because the load_dataset function doesn’t always scream when it fails. Sometimes it just loads silently… wrong.

3.2 Preprocessing Pipeline — Tokenize Like You Mean It

This might surprise you: GPT-2 doesn’t have a padding token.
Yeah, I learned this the fun way when my model started throwing errors mid-epoch during multi-GPU training.

Here’s the tokenizer setup I always use now:

from datasets import load_dataset
from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token  # Hack: use EOS as padding

def tokenize_function(example):
    return tokenizer(
        example["text"],
        truncation=True,
        padding="max_length",
        max_length=512
    )

dataset = load_dataset("path/to/your_data.csv", split="train", delimiter=",")  # or load_dataset("json", data_files=...)
tokenized = dataset.map(tokenize_function, batched=True)

Let me break this down for you, based on what’s bitten me in the past:

🔥 Common Gotchas (I’ve hit every one of these)

• No padding token = runtime error: GPT-2 wasn’t trained with a pad_token_id, so you either set it manually (like above) or your trainer will crash.
• Memory bloat when batched=False: Always use batched=True when mapping tokenization. Otherwise, the Hugging Face datasets lib loads and processes line-by-line, which becomes a nightmare for anything over 10k samples.
• Long sequences truncate silently: You might lose valuable context if you don’t check your max length. I personally cap it at 512, unless I’m working with GPT-2 XL or have big VRAM.
• Unclean text = token soup: I always preprocess the raw text lightly — things like replacing weird Unicode characters, stripping HTML tags, and collapsing whitespace — before feeding it into the tokenizer. (If you’re interested, I can share my regex clean-up script.)

If you’re building your dataset from scratch, here’s a quick snippet I’ve used to convert a folder of .txt files into a usable CSV:

import os
import pandas as pd

texts = []
for fname in os.listdir("raw_texts"):
    with open(os.path.join("raw_texts", fname), "r", encoding="utf-8") as f:
        texts.append(f.read())

df = pd.DataFrame({"text": texts})
df.to_csv("prepared_dataset.csv", index=False)

So yeah, dataset prep isn’t glamorous — but it’s where the real work starts. I’ve seen perfectly good models go sideways because of padding errors, misaligned data columns, or tokenization issues that no one caught early.

---

4. Fine-Tuning GPT-2 Using Transformers (The Real Deal)

“A model is only as smart as the hands that tune it.”
— Probably someone who trained GPT-2 on a MacBook and lived to tell the tale.

If you’ve made it this far, you’re not looking for the usual BERT-style fine-tuning tutorial. You’re here for the nitty-gritty that actually matters when you’re working with autoregressive models like GPT-2, especially in 2025 where large-scale LLMs overshadow it—but you still want control, speed, and fewer surprises.

4.1 Model Selection — Choose Your Weapon Wisely

Here’s the deal: I’ve worked with gpt2, gpt2-medium, and gpt2-xl across various real-world tasks — some internal tooling, some client-facing generation systems. The choice always boils down to three factors: VRAM, latency, and context length tolerance.

My personal take on each:

• gpt2 (124M):
  ✅ Lightweight and fast.
  ❌ Starts forgetting structure with longer text (>300 tokens).
  🔧 Good for prototyping and light customization.
• gpt2-medium (345M):
  ⚖️ Sweet spot for balanced compute vs output quality.
  ✅ Still trainable on a single high-end GPU (like a 3090 or A6000).
• gpt2-xl (1.5B):
  💪 Output feels noticeably more coherent.
  ❌ Hugely memory-intensive — I’ve only run this with DeepSpeed + gradient checkpointing.
  ❗ Easily runs out of memory with batch size >1.

You might be wondering: what’s with the pad token thing again?
Here’s the catch — GPT-2 doesn’t have a pad_token_id. You need to handle this explicitly or your Trainer will throw.

from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained("gpt2")
model.resize_token_embeddings(len(tokenizer))  # Make room if you've modified tokenizer
model.config.pad_token_id = tokenizer.pad_token_id

I learned this the hard way. If you forget to set pad_token_id, evaluation can silently skew or crash altogether, especially with fp16 training.

4.2 Trainer API Approach — Fastest Way to Fine-Tune

When I need to get something up and running fast, Hugging Face’s Trainer is my go-to. I’ve used it in dozens of experiments, and it saves a ton of boilerplate. But here’s the thing — the defaults are not your friends.

Let me show you my usual configuration:

from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./results",
    evaluation_strategy="steps",
    eval_steps=500,  # Set based on dataset size
    save_strategy="steps",
    save_steps=500,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    num_train_epochs=3,
    logging_dir="./logs",
    logging_steps=100,
    fp16=True,  # Only if your GPU supports it (A100, 3090, etc.)
    save_total_limit=2,
    gradient_accumulation_steps=8,  # Critical for small-batch setups
    report_to="none",  # Turn off WandB unless you're actively logging
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["test"],
    tokenizer=tokenizer,
)

trainer.train()

A few things I’ve learned by doing this wrong before:

• Don’t skip gradient_accumulation_steps if your batch size is small — it makes a huge difference in training stability.
• I keep eval_steps and save_steps tight (e.g., 500) during early training so I can catch problems quickly without wasting compute.
• fp16=True is powerful, but it’ll backfire on older GPUs. If training crashes mid-loop, test with fp16=False first.

4.3 Accelerate + DeepSpeed (When You Go Big)

When I’m working with gpt2-xl or running fine-tunes on longer context windows (especially custom-trained GPT-2s with 1024+ token limits), Hugging Face’s Accelerate + DeepSpeed combo becomes essential.

This might surprise you: You don’t need to write a custom training loop anymore to use DeepSpeed — accelerate config handles it beautifully.

Here’s a basic accelerate_config.yaml I’ve used:

compute_environment: LOCAL_MACHINE
mixed_precision: fp16
deepspeed_config:
  zero_stage: 2
  offload_optimizer_device: cpu
  offload_param_device: cpu
  gradient_accumulation_steps: 8
  gradient_clipping: 1.0

Once that’s set up, just run:

accelerate launch train_gpt2.py

Where train_gpt2.py is essentially a Trainer-based script like the one above — but wrapped in Accelerate’s launch system.

In my experience:

• Offloading to CPU helps when VRAM is tight, but it slows down training — I only use it when desperate.
• Zero-2 is stable and gets you far, but for very large models, consider Zero-3 (with patience).

---

5. Custom Training Loop (If You Hate the Trainer API)

“Sometimes, you need to know exactly what the engine is doing under the hood — especially when the wheels come off.”

This might surprise you, but in many of my experiments, I’ve ditched the Trainer API completely. Not because it’s bad — it’s great for quick protos — but because it gets in the way when I want:

• Fine-grained control over logging, schedulers, or gradient clipping
• Clean multi-GPU logic (especially with DataParallel or DDP)
• A fully customizable training cycle, especially for curriculum learning or dynamic sampling

Let’s go step by step. Here’s how I typically set this up using PyTorch and Hugging Face’s core model/tokenizer objects.

5.1 The Classic Setup (Tokenizer + Model + Dataloader)

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer
from torch.utils.data import DataLoader
from datasets import load_dataset

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

dataset = load_dataset("your_dataset")
def tokenize(example):
    return tokenizer(example["text"], padding="max_length", truncation=True, max_length=512)

tokenized = dataset.map(tokenize, batched=True)
tokenized.set_format(type="torch", columns=["input_ids", "attention_mask"])

train_loader = DataLoader(tokenized["train"], batch_size=2, shuffle=True)

5.2 Token Shifting (Critical for GPT-2 Loss)

This is a gotcha I wish someone had warned me about when I first started working with autoregressive models:

# input_ids:    The input sequence
# labels:       Should be input_ids shifted by one (to predict the *next* token)

labels = input_ids.clone()
outputs = model(input_ids, attention_mask=mask, labels=labels)
loss = outputs.loss

You don’t need to manually shift them with Hugging Face — it does that internally — but you do need to pass the labels properly, or the loss function will go haywire.

5.3 Full Training Loop — Straight From My Scripts

from transformers import AdamW, get_scheduler
from torch.nn.utils import clip_grad_norm_

model = GPT2LMHeadModel.from_pretrained("gpt2")
model.resize_token_embeddings(len(tokenizer))
model.config.pad_token_id = tokenizer.pad_token_id
model.cuda()

optimizer = AdamW(model.parameters(), lr=5e-5)
lr_scheduler = get_scheduler("linear", optimizer=optimizer, num_warmup_steps=100, num_training_steps=1000)

model.train()
for epoch in range(3):
    for step, batch in enumerate(train_loader):
        input_ids = batch["input_ids"].cuda()
        attention_mask = batch["attention_mask"].cuda()
        labels = input_ids.clone()

        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()

        clip_grad_norm_(model.parameters(), max_norm=1.0)  # important
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()

        if step % 100 == 0:
            print(f"Epoch {epoch} | Step {step} | Loss: {loss.item():.4f}")

Why bother with all this?

Personally, I’ve needed custom loops when I wanted to:

• Log metrics to multiple backends (e.g., TensorBoard + console + JSON for dashboards)
• Implement dynamic sequence lengths based on dataset patterns
• Run adversarial training (yes, really — GPT-2 handles it better than you’d expect)

If that sounds like overkill, stick to Trainer. But if you’re chasing production-grade behavior or experimenting with custom loss functions, nothing beats your own loop.

---

6. Logging and Evaluation (The Real Metrics That Matter)

Let’s talk about metrics — because this is where many fine-tuning attempts look good on paper but fall flat when deployed.

6.1 Why I Don’t Trust Accuracy

Here’s the deal: token-level accuracy is almost meaningless in language modeling.

I’ve seen models that hit 98% token accuracy on eval — and still generate nonsense because they’re good at memorizing prefixes, not producing coherent sequences.

What I trust instead is loss and perplexity. Especially perplexity — not because it’s perfect, but because it’s consistent and explains how surprised the model is by the next token.

6.2 Eval Code I Actually Use

import math

eval_results = trainer.evaluate()
perplexity = math.exp(eval_results["eval_loss"])
print(f"Perplexity: {perplexity:.2f}")

When using a custom loop, just log loss after each eval batch, average it, and apply math.exp().

6.3 Saving + Reloading Models (Don’t Screw This Up)

When you save a fine-tuned GPT-2, make sure you’re saving both the model and tokenizer with your custom padding token and vocab size.

model.save_pretrained("fine_tuned_gpt2")
tokenizer.save_pretrained("fine_tuned_gpt2")

Then reload like this — especially if your tokenizer was modified:

model = GPT2LMHeadModel.from_pretrained("fine_tuned_gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("fine_tuned_gpt2")

I’ve made the mistake of saving a model and forgetting the tokenizer settings — it totally ruins sampling later.

---

7. Inference Pipeline (Prompting Your Fine-Tuned Model)

“A model that can’t talk is just weights. The magic is in the prompt.”

So, here’s the part I get asked about a lot — inference. Once you’ve trained your GPT-2, how do you actually use it in production-style pipelines? I’ve personally spent more time on this than I’d like to admit, trying to squeeze just the right outputs from models that technically worked, but didn’t say anything useful.

7.1 Basic Inference Setup

If you’re using Hugging Face, model.generate() is your go-to. But just calling generate() without tweaking the right knobs will either give you gibberish or robotic, ultra-safe responses.

Here’s what I always start with:

from transformers import GPT2LMHeadModel, GPT2Tokenizer

model = GPT2LMHeadModel.from_pretrained("fine_tuned_gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("fine_tuned_gpt2")
model.eval().cuda()

prompt = "The customer said:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    temperature=0.7,
    top_p=0.95,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

7.2 Choosing the Right Settings (My Defaults After Painful Trial-and-Error)

Let me break down what I actually use in production:

• max_new_tokens=50: I never use max_length anymore — it causes weird truncation when input_ids are long.
• temperature=0.7: Keeps it creative, but not wild.
• top_p=0.95: I use this over top_k because it’s more stable across different inputs.
• do_sample=True: Without this, you’re just getting greedy decoding — which, trust me, is boring and repetitive.

You might be wondering: How does this compare to base GPT-2?

Let me show you.

7.3 Fine-Tuned vs Original (Side-by-Side Sample)

base_model = GPT2LMHeadModel.from_pretrained("gpt2").eval().cuda()
fine_tuned_model = GPT2LMHeadModel.from_pretrained("fine_tuned_gpt2").eval().cuda()

prompt = "The customer said:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Base GPT-2
base_out = base_model.generate(**inputs, max_new_tokens=50, temperature=0.7, top_p=0.95, do_sample=True)
print("Base GPT-2:\n", tokenizer.decode(base_out[0], skip_special_tokens=True))

# Fine-tuned GPT-2
ft_out = fine_tuned_model.generate(**inputs, max_new_tokens=50, temperature=0.7, top_p=0.95, do_sample=True)
print("Fine-Tuned GPT-2:\n", tokenizer.decode(ft_out[0], skip_special_tokens=True))

In my experience, the fine-tuned model adopts the tone and vocabulary of your domain surprisingly well — especially if you’re training on customer support data, legal documents, or niche technical material. You’ll start seeing phrases you’ve never seen in the base model.

7.4 Other Tricks I Use During Inference

Here are a few hacks I keep in my inference pipeline toolbox:

• Post-processing: I often write a regex-based cleanup script to remove partial sentences or broken punctuation.
• Stop sequences: For structured tasks, I define custom stop tokens to end generation early.
• Input formatting: Adding newline tokens (\n\n) between prompt and completion can lead to more coherent generations in certain tasks.

---

8. Deployment Tips (Optional but Practical)

“Training gets the applause. Deployment pays the bills.”

If you’re shipping this thing — whether internally, for a client, or as part of a product — you’ll want three things:

• Efficient model formats (hello ONNX, TorchScript)
• A fast API (FastAPI > Flask, fight me)
• Awareness of GPU memory usage, because GPT-2 large can and will blow up your box if you’re not careful

Let’s break it down.

8.1 Exporting to TorchScript (Fast Path to Speedups)

Personally, I reach for TorchScript first — it’s native, fast, and integrates easily with PyTorch serving.

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

model = GPT2LMHeadModel.from_pretrained("fine_tuned_gpt2")
model.eval().cuda()

# Dummy input for tracing
inputs = torch.randint(0, 50257, (1, 32)).cuda()
traced = torch.jit.trace(model, (inputs,))

# Save it
traced.save("gpt2_traced.pt")

💡 Gotcha: You need to trace both input_ids and attention_mask if you’re using them — otherwise you’ll get runtime shape errors.

8.2 ONNX Export (Better for Interop + Edge Cases)

If you’re targeting inference outside of PyTorch — say in a C++ app or with something like Triton — ONNX is your friend.

from transformers import GPT2Tokenizer, GPT2LMHeadModel
from transformers.onnx import export
from pathlib import Path
import torch

model = GPT2LMHeadModel.from_pretrained("fine_tuned_gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("fine_tuned_gpt2")

export(
    preprocessor=tokenizer,
    model=model,
    output=Path("gpt2.onnx"),
    opset=13,
    tokenizer=tokenizer
)

🚨 Important: ONNX export sometimes fails for certain models or settings (e.g., dynamic padding). You might need to tweak export config or use a tool like onnxruntime-tools.

8.3 Hosting with FastAPI (Snappy and Clean)

Here’s a barebones inference server I’ve actually used — no unnecessary complexity.

from fastapi import FastAPI, Request
from pydantic import BaseModel
import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

app = FastAPI()

model = GPT2LMHeadModel.from_pretrained("fine_tuned_gpt2").cuda().eval()
tokenizer = GPT2Tokenizer.from_pretrained("fine_tuned_gpt2")

class PromptInput(BaseModel):
    prompt: str

@app.post("/generate")
def generate_text(data: PromptInput):
    inputs = tokenizer(data.prompt, return_tensors="pt").to("cuda")
    output = model.generate(**inputs, max_new_tokens=50, temperature=0.7, top_p=0.9, do_sample=True)
    result = tokenizer.decode(output[0], skip_special_tokens=True)
    return {"output": result}

To run it:

uvicorn myapp:app --host 0.0.0.0 --port 8000

💡 Why not Flask? Because FastAPI gives you auto docs, async support, and better speed — and I like writing code once and forgetting about it.

8.4 GPU Memory Benchmarks (Real-World Numbers)

Here’s what I’ve seen in real deployments (NVIDIA A100 40GB and RTX 3090):

Model	Peak VRAM (Batch=1)	Peak VRAM (Batch=4)
GPT-2 (124M)	~1.5 GB	~2.2 GB
GPT-2 Medium	~2.8 GB	~4.5 GB
GPT-2 Large	~5.1 GB	~8.7 GB
GPT-2 XL	~9.5 GB	>15 GB

If you’re on a smaller GPU (like a 6GB 1660 or a laptop card), you’re basically locked into GPT-2 small unless you run quantized or offloaded models.

What I Recommend (From Experience)

• For API deployments: Use FastAPI, run behind gunicorn or uvicorn with proper worker threads.
• For speed: Quantize with bitsandbytes or export to ONNX with onnxruntime-gpu.
• For multi-user traffic: Run inference in fp16 and batch requests with a queue like Redis + Celery.

---

9. Common Issues (And How I Fixed Them)

“Everyone talks about fine-tuning like it’s just plug and play. Reality? It’s mostly debugging.”

Here’s the deal: no matter how solid your dataset is or how clean your code looks, things will break. I’ve run into these issues personally, more than once. So instead of just listing the problems, let me walk you through how I dealt with them.

❌ CUDA Out of Memory (OOM)

You know the drill: everything runs fine for 2 steps… then boom — CUDA out of memory.

What I did:

• Reduced batch_size: Obvious, but necessary. Sometimes even batch=1 is what it takes.
• Used gradient checkpointing: This helped me a lot when working with GPT-2 Large on a 24GB card.

from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained("gpt2-large")
model.gradient_checkpointing_enable()

Saved me ~30-40% VRAM in training.

❌ Stuck at 0 Loss or Constant Loss

This one took me a full day to figure out the first time. My model just wasn’t learning anything — flat loss. No gradients. Just vibes.

What fixed it:

• I had forgotten to shift labels. If labels=input_ids, GPT2 ends up predicting the same token it’s looking at. That breaks the causal language modeling logic.

Here’s what I now always double-check:

labels = input_ids.clone()
outputs = model(input_ids=input_ids, labels=labels)
loss = outputs.loss

Or if you want to shift explicitly (sometimes necessary when customizing the loop):

labels = input_ids[:, 1:].contiguous()
inputs = input_ids[:, :-1].contiguous()
outputs = model(input_ids=inputs, labels=labels)

❌ Poor Outputs After Finetuning

This one hurts the most. You spend hours training, the loss goes down, but the generated text? Total garbage.

What helped me:

• Looked at the raw dataset — and yeah, it was noisy. Way too many repeated lines and out-of-context examples.
• Trained longer — going from 2 to 5 epochs made a visible difference.
• Lowered temperature + increased top_p during generation to get less chaotic completions.

model.generate(
    input_ids,
    max_new_tokens=50,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

Honestly, poor outputs are rarely about the model. It’s usually data quality, bad label alignment, or generation settings.

---

10. My Final Tips (Things You Only Learn by Doing)

“You don’t really know fine-tuning until you’ve wrecked your VRAM, waited 3 hours, and gotten gibberish in return.”

Let me share a few things I’ve learned the hard way — the kind of stuff that doesn’t usually show up in the docs.

🔹 Fine-Tuning Isn’t Always Worth It

This might surprise you, but sometimes… fine-tuning is the wrong choice.

If you’re just trying to make the model follow instructions better — go for prompt engineering or function-calling with GPT-4-style APIs.

I wasted two weekends training a GPT-2 model to answer FAQs when I could’ve used a good system prompt with few-shot examples. Now I ask myself: Do I need the model to know the data, or just reference it?

🔹 Low-Resource Fine-Tuning: My Go-To Now

When I’m tight on compute, I don’t do full fine-tuning anymore.

I use LoRA or PEFT adapters. Why? Because they:

• Need less than 1GB of extra VRAM
• Are modular — I can swap out task adapters like Lego bricks
• Train way faster — think 20x faster on a small GPU

from peft import get_peft_model, LoraConfig, TaskType

peft_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=4,
    lora_alpha=32,
    lora_dropout=0.1
)

peft_model = get_peft_model(model, peft_config)

Personally, I used this on a project fine-tuning GPT-2 for customer support transcripts — and it worked better than full fine-tuning. I even reused the same base model for 3 different departments using different LoRA adapters.

🔹 Always Save Checkpoints Smartly

I’ve lost progress more than once because I didn’t checkpoint often. Now I always save with save_total_limit, and I test reloading before training ends.

from transformers import TrainingArguments

training_args = TrainingArguments(
    save_total_limit=3,
    save_steps=500,
    output_dir="./checkpoints",
)

Learned this the hard way when a power cut wiped my 3-day run.