# How to Run Unsloth on AMD GPUs (ROCm) [2026 Guide]

> A late‑2026 compatibility matrix plus known‑good commands for getting Unsloth LoRA fine‑tuning to actually converge on Radeon and MI GPUs with ROCm.

- Canonical: https://www.kunalganglani.com/blog/unsloth-amd-gpu-rocm-fine-tuning-guide
- Author: Kunal Ganglani
- Published: 2026-09-27 · Updated: 2026-09-27
- Category: Technology · Tags: amd, rocm, fine-tuning, unsloth, pytorch

## TL;DR

You can fine-tune an open model on an AMD Radeon or Instinct GPU today, but only if your software versions actually match. This guide gives you a late‑2026 compatibility matrix for ROCm, PyTorch, and Unsloth, plus a small “sanity run” that proves your training is stable. The big idea: don’t start with fancy speedups. Get ROCm working, install the correct PyTorch ROCm build, run a tiny LoRA fine-tune in fp16, then add faster attention kernels only after you see clean loss curves. You’ll also learn the most common HIP and kernel errors and how to fix them.

You don’t need another “it installed!” ROCm post. You need Unsloth fine-tuning on an AMD GPU to **actually train**, with a loss curve that goes down for the right reasons.

By the end of this guide, you’ll have Unsloth fine-tuning running on an AMD GPU with ROCm, plus a quick sanity run that proves it’s converging (not just “it launched”). If your setup is close to supported, you can get from zero to a minimal LoRA run in ~45–90 minutes. If you’ve been stuck in “HIP error / no kernel image available” hell, plan for an afternoon.

This is an **Unsloth AMD GPU ROCm fine tuning guide** written for late‑2026 reality: ROCm support is way better than it was, but it’s still version-fragile, kernel-fragile, and “works” is not the same as “trains fast and converges”.

I’m going to be blunt. Most AMD fine-tuning failures aren’t because Unsloth is “broken”. They’re because your stack is a Jenga tower: kernel + driver + ROCm userland + PyTorch ROCm wheels + Triton/attention kernels + mixed precision. One mismatch and you get either a hard crash or, worse, silent slow fallbacks.

(Visual break: matrix screenshot)

## What is Unsloth?

Unsloth is a training and inference optimization toolkit that patches Hugging Face Transformers/TRL flows to make fine-tuning large language models faster and more memory-efficient, typically using LoRA and related parameter-efficient methods.

![a pair of black and silver graphics cards](https://cdn.sanity.io/images/vzekdneq/production/95206e1642fd6aecbafd1f976110eb243349bdf9-1200x675.webp)

Where Unsloth earns its keep is (a) smart model patching and (b) optional high-performance kernels. On NVIDIA/CUDA, those kernels are usually just… there. On AMD/ROCm, kernel availability depends on your exact GPU architecture and software versions. That’s the whole game.

If you’re still deciding whether you need fine-tuning at all, read my decision framework on [fine-tuning](/blog/fine-tuning-vs-rag-prompt-engineering) vs [RAG](/blog/fine-tuning-vs-rag-prompt-engineering) (retrieval-augmented generation) and prompt work.

## Unsloth AMD GPU ROCm fine tuning guide: the late‑2026 compatibility matrix

Here’s the matrix I wish existed the first time I tried to help someone debug Unsloth on a Radeon card. It’s not “official”. It’s the set of combinations that tend to behave in the real world, plus the landmines that waste your time.

![Two computer graphics cards on a yellow background](https://cdn.sanity.io/images/vzekdneq/production/5d15ccf93a993837c9e2ecd44834c1a4cbc0f5b2-1200x675.webp)

Two anchors before we start:

- **Treat the PyTorch ROCm wheel selector as the source of truth.** PyTorch’s install page currently shows Stable **PyTorch 2.7.0** and ROCm wheel availability for **ROCm 6.3** (that compute platform option exists right in the selector). If your installed ROCm runtime is 6.1 and you pip-installed a 6.3 wheel, you’re basically asking for missing `hip` libraries and runtime weirdness. See the official selector on [PyTorch](https://pytorch.org/get-started/locally/).
- **ROCm itself ships a published compatibility matrix and install guidance.** Start from AMD’s docs, not random Reddit incantations. [AMD ROCm documentation](https://rocm.docs.amd.com/) has an explicit “Compatibility matrix” section and per-component pages.
### Known-good stacks (pin these first)

“Known-good” means one thing: **you can run a small SFT, loss decreases smoothly, no NaNs, and throughput isn’t obviously crippled**. Anything else is a science project.

| GPU class | Example GPUs | OS baseline | ROCm baseline | PyTorch ROCm baseline | Attention kernel mode | Quantization path | Notes |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Consumer RDNA3 (24GB) | RX 7900 XTX (24GB) | Ubuntu 24.04 / kernel 6.x | ROCm 6.2–6.3 | torch 2.6–2.7 ROCm build | **SDPA** first, try FlashAttn later | **FP16 LoRA** | Most reliable Radeon path. SDPA is usually “good enough”. |
| Consumer RDNA3 (16GB) | RX 7800 XT (16GB) | Ubuntu 24.04 | ROCm 6.2–6.3 | torch 2.6–2.7 ROCm build | SDPA | FP16 LoRA + grad ckpt | VRAM is the constraint. Keep seq len sane (see VRAM section). |
| Consumer RDNA4 | RX 8800/8900 class | Ubuntu 24.04+ | ROCm 6.3+ | torch 2.7 ROCm build | SDPA | FP16 LoRA | Newer arch usually means better kernel coverage, but also sharper version coupling. |
| Datacenter CDNA (MI series) | MI250/MI300/MI350 | ROCm-certified distros | ROCm 6.2–6.3+ | torch matched to ROCm | FlashAttn **more likely** | FP16/BF16 LoRA | MI parts tend to have fewer “invalid device function” surprises. |

A few sharp edges I’ll save you from:

- If you’re on **Windows**: ROCm exists there, but Unsloth fine-tuning is still dramatically less predictable than on Linux. If your goal is “fine-tune this weekend”, use Linux.
- If you’re on **WSL2**: it’s great for a lot of dev. For ROCm fine-tuning, you’re stacking two “almost works” layers. If you want my honest recommendation: don’t.
Based on the benchmark data I maintain at [kunalganglani.com/llm-benchmarks](https://www.kunalganglani.com/llm-benchmarks), the pattern I keep seeing across non-NVIDIA platforms is: **getting the model to load is easy; getting stable throughput is the real work**. ROCm fine-tuning is the same story.

(Visual break: “known good” command snippet screenshot)

## Install ROCm and verify HIP works (rocminfo/hipcc)

This section is intentionally boring. Boring is the point. If you skip verification, you’ll spend the next 6 hours blaming Unsloth for a driver/runtime problem.

![A close up of a video card on a yellow background](https://cdn.sanity.io/images/vzekdneq/production/4b2ad617ac1e958085bcdea9d784cd3a8f36176d-1200x675.webp)

### 1) Pick a supported OS and kernel first

Start with a mainstream distro ROCm expects. In 2026, Ubuntu LTS is the pragmatic choice.

- Ubuntu **24.04 LTS** is a solid baseline.
- Linux kernel **6.x** is common.
The moment you go off-road (custom kernels, odd GCC/libstdc++ combos), you’re volunteering to debug C++ ABI errors instead of training models.

### 2) Install ROCm from AMD’s docs

Use AMD’s official install steps and their compatibility matrix: [AMD ROCm documentation](https://rocm.docs.amd.com/).

Practical advice I stick to:

- Keep your ROCm install **single-source**. Don’t mix distro packages + random tarballs + Docker bind mounts unless you know exactly why.
- If you’re using containers (often best for Radeon), make sure the host driver stack matches what the container expects. ROCm is not magic.
### 3) Verify `rocminfo` and `hipcc`

After install:

```bash
# should print GPU + agent info
/opt/rocm/bin/rocminfo | head -n 50

# should print ROCm/HIP compiler info
/opt/rocm/bin/hipcc --version
```

If `rocminfo` can’t see your GPU, stop. Don’t “try pip installs until it works”. Fix the driver/runtime first.

Common failure modes:

- `rocminfo: HSA runtime not found` → ROCm userland isn’t installed correctly or not on PATH.
- Permission issues accessing `/dev/kfd` → user not in the right group (often `video`/`render`).
## Install PyTorch for ROCm (matching wheel to ROCm)

This is the #1 root cause of “it imports, but training explodes”. The PyTorch ROCm wheel bakes assumptions about the ROCm runtime.

### The only rule that matters

**Your installed ROCm runtime version and your PyTorch ROCm build must match.**

PyTorch makes this easy if you actually use their selector. On the official page, the compute platform dropdown includes **ROCm 6.3** and generates the matching `--index-url` command. That selector is maintained by the people shipping the wheels. Use it: [PyTorch](https://pytorch.org/get-started/locally/).

Example (ROCm wheel install style):

```bash
python3 -m venv .venv
source .venv/bin/activate

# example only. generate your exact command from pytorch.org
pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3
```

### Verify PyTorch sees the AMD GPU

```bash
python - <<'PY'
import torch
print('torch', torch.__version__)
print('hip available', torch.cuda.is_available())
print('device', torch.cuda.get_device_name(0) if torch.cuda.is_available() else None)
PY
```

Yes, it still prints `cuda` in APIs. No, you’re not secretly using NVIDIA.

If `is_available()` is false:

- You installed a CPU wheel.
- Or you installed a ROCm wheel that doesn’t match your ROCm runtime.
If you see linker errors about missing `libamdhip64.so` or friends, you’re mismatched.

## Set up Transformers + TRL + Unsloth

Unsloth rides on top of Hugging Face Transformers and TRL.

- Unsloth repo: [Unsloth AI](https://github.com/unslothai/unsloth)
- TRL docs for trainer knobs (packing, clipping, etc.): TRL
### My opinionated dependency strategy for AMD

On CUDA I’m okay with “latest-ish” because the ecosystem is brutally well-tested.

On ROCm I pin. Always.

Pin your layers like you mean it:

- Pin `torch` to the ROCm wheel you installed.
- Pin `transformers`, `trl`, `accelerate`, `peft` to a tested set.
- Upgrade only one layer at a time.
Example:

```bash
pip install "transformers==4.56.*" "trl==0.12.*" "accelerate==1.0.*" "peft==0.13.*"

# install unsloth
pip install "unsloth[rocm]" || pip install unsloth
```

If the extra `[rocm]` doesn’t exist in the current Unsloth packaging when you read this, that’s fine. The point is: install Unsloth, then verify what optional kernels it pulled in.

### Attention kernels: assume SDPA first

One of the most common traps is people assuming FlashAttention is mandatory because every NVIDIA guide reads like it’s oxygen.

On AMD, **start with SDPA** (PyTorch scaled dot-product attention). Chase FlashAttention only after you have a converging baseline.

Why I’m so stubborn about this:

- FlashAttention availability on ROCm is architecture- and version-dependent.
- The failure mode is often not “missing package”. It’s “Triton compiled something, ran it, and now you get NaNs at step 200.” That’s the worst kind of failure.
## Run a minimal LoRA/SFT sanity check (prove it converges)

If you only do one thing from this post, do this. Don’t launch your full dataset run first. That’s how you burn an evening.

### The 5-step checklist (minimal moving parts)

1. Confirm `rocminfo` sees the GPU.
1. Confirm PyTorch ROCm sees the GPU (`torch.cuda.is_available()` is true).
1. Run a **single-batch forward+backward** in fp16.
1. Run **50–200 training steps** on a tiny dataset.
1. Confirm loss decreases smoothly and no NaNs appear.
That’s your “stack is sane” certificate.

### Minimal TRL SFTTrainer script (ROCm-friendly defaults)

This uses conservative knobs: fp16, gradient checkpointing, modest learning rate, clipping.

```python
# sft_rocm_sanity.py
import os
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")

import torch
from datasets import Dataset
from transformers import AutoTokenizer
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig

from unsloth import FastLanguageModel

model_name = "meta-llama/Meta-Llama-3-8B-Instruct"  # 8B class is a sane starting point

# Tiny dataset on purpose
data = Dataset.from_list([
  {"text": "Write a haiku about GPUs."},
  {"text": "Explain what ROCm is in one paragraph."},
  {"text": "Summarize the difference between LoRA and full fine-tuning."},
] * 200)

max_seq_length = 512

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=model_name,
    max_seq_length=max_seq_length,
    dtype=torch.float16,
    load_in_4bit=False,   # start without QLoRA on AMD
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=data,
    peft_config=lora_config,
    args=SFTConfig(
        output_dir="./out_rocm_sanity",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=2e-4,
        warmup_ratio=0.03,
        max_steps=200,
        logging_steps=10,
        fp16=True,
        bf16=False,
        gradient_checkpointing=True,
        max_grad_norm=1.0,
        packing=False,
        report_to=[],
    ),
)

trainer.train()
print("done")
```

Run it:

```bash
python sft_rocm_sanity.py
```

What “good” looks like:

- Loss starts high and trends down over the first **50–200 steps**.
- No `nan` loss. No sudden spikes to absurd values.
- GPU utilization is non-zero and stable.
If you get NaNs, don’t immediately blame ROCm. Do the boring trainer triage first:

- Lower LR from `2e-4` → `1e-4`.
- Ensure `max_grad_norm=1.0` is set.
- Turn off packing.
TRL documents these knobs and what they change: TRL.

## Troubleshoot common ROCm errors (HIP runtime, missing libs, kernel image issues)

Most guides turn this section into an error-message graveyard. That’s useless. You want a decision tree.

### 1) “HIP error / invalid device function / no kernel image available”

Translation: a compiled kernel doesn’t match your GPU architecture, or you installed a build that simply wasn’t compiled for your card.

Fix path:

- Confirm your GPU is actually supported by your ROCm version in AMD’s matrix: AMD ROCm documentation.
- Confirm your PyTorch wheel matches your ROCm runtime. Reinstall PyTorch using the official selector: PyTorch.
- If this happens when enabling FlashAttention/Triton kernels: force SDPA and disable fused attention paths.
Useful env knobs to try (these move around across versions, so treat this as a debugging tactic, not a forever setting):

```bash
export TORCH_LOGS="+dynamo,+inductor"
export TORCHDYNAMO_VERBOSE=1
```

If you see Triton compilation errors referencing HIP, that’s your sign to stop chasing fancy kernels and get a clean SDPA baseline.

### 2) Missing `hipblaslt` / `rccl` / shared library errors

This is almost always packaging mismatch:

- ROCm userland not fully installed.
- Container image expects a different ROCm stack than the host.
My stance: **containers are usually more reliable on Radeon**, because you can pin userland and Python deps together. But your host driver still matters.

If you want the broader ROCm vs CUDA story for [local AI](/blog/amd-rocm-vs-cuda-local-ai-open-source-guide), I wrote that comparison already. This post is narrower: making Unsloth fine-tuning stop failing.

### 3) bitsandbytes fails on ROCm (4-bit/8-bit)

This question shows up constantly: “Why does bitsandbytes 4-bit/8-bit quantization fail on ROCm?”

Because bitsandbytes has historically been CUDA-first. ROCm support has improved, but it’s still fragile, and it lags the newest training stacks.

Practical alternatives that don’t waste your weekend:

- **FP16 LoRA** (the baseline I recommend in this post).
- **8-bit optimizers** only if your ROCm build supports the exact kernel path you need.
- If your goal is cheap inference, skip training quantization entirely. Use GGUF for inference-only. I’ve written about quantization tradeoffs in [LLM quantization levels](/blog/llm-quantization-levels-q4-q8-fp16) and [GGUF vs GPTQ vs EXL2](/blog/llm-quantization-gguf-gptq-exl2).
If you’re on a 16GB Radeon card and trying to QLoRA a 13B model because TikTok told you it’s “easy”, stop. Use a smaller base model or add VRAM.

### 4) “It runs” but throughput is awful (kernel fallbacks)

This is the sneaky failure. No crash. No obvious error. Just performance that feels like you’re training through a straw.

Symptoms:

- GPU utilization is low.
- Tokens/sec is way lower than expected.
- You see unexpected CPU usage spikes.
How to diagnose:

- Turn on PyTorch compile logs (`TORCH_LOGS` above) to see if Inductor/Triton is bailing.
- Start from SDPA. If FlashAttention “works” but is slower, you’re probably hitting a fallback path.
Also: measure something real.

In my benchmark work for this site, throughput is the metric people hand-wave the most. That’s why I keep pushing tok/s and TTFT in posts like [local LLM benchmark methodology](/blog/local-llm-benchmark-methodology).

### 5) NaNs, divergence, and fake convergence

On AMD you’ll sometimes get “training” that doesn’t crash but is numerically unstable.

Things I check, in this order:

- Loss: do you ever see `nan`? If yes, don’t continue.
- Gradient clipping: set `max_grad_norm` to **1.0** (or even 0.5).
- Precision: prefer **fp16** first. bf16 can be great, but bf16 stability varies by GPU + stack.
- Sequence packing: turn it off until you’re stable.
Rule of thumb VRAM budgets (practical, not theoretical):

- **7B/8B** LoRA: 16GB can work with small batch, seq len ~512–1024, grad checkpointing.
- **13B** LoRA: 24GB is the sane starting point for non-suffering.
- Anything beyond: either multi-GPU (more pain on Radeon) or change strategy.
If you want the bigger hardware sizing story for [local LLM](/blog/local-llm-hardware-requirements-2026) work, I have a dedicated guide. This post stays focused on Unsloth + ROCm.

Here’s an official baseline walkthrough from AMD itself. It’s worth watching once to calibrate your environment expectations:

[Watch: How to Fine-Tune a Model on AMD GPUs using LoRA](https://www.youtube.com/watch?v=GNF32XuRAN4)

## Container vs bare-metal for Radeon: my default recommendation

If you’re on consumer Radeon, I generally recommend containers for fine-tuning.

Reasons:

- You can pin Python deps and ROCm userland together.
- You can share a known-good image with your team.
- You reduce “my system Python upgraded libstdc++ and now hip runtime crashes” incidents.
Bare-metal is fine too if you’re disciplined. Most people aren’t.

If you do bare-metal:

- Keep a dedicated venv per ROCm/PyTorch stack.
- Don’t upgrade blindly.
This mindset is similar to what I preach for reproducible dev envs in general, like in my [Python dev setup](/blog/python-ai-development-setup-2026) and [Linux migration plan](/blog/linux-migration-plan-developers) posts.

## My prediction for 2027: “ROCm works” won’t be enough

By 2027, “can I run it?” will be boring. It’ll be solved for AMD consumer GPUs.

The dividing line will be: **can you update your stack without accidentally losing fast kernels, falling back to slow paths, or reintroducing numerical weirdness?** NVIDIA has earned that kind of stability. AMD is getting closer, but Unsloth-style optimization stacks are exactly where the cracks still show.

If you’re building a serious fine-tuning workflow on Radeon today, don’t treat this as a one-off setup. Turn it into a pinned, scripted environment you can recreate on a fresh machine. That’s how you stop ROCm from becoming a recurring calendar event.

Photo by Brecht Corbeel on Unsplash.

## FAQ

### Can you run Unsloth on AMD GPUs with ROCm?

Yes, Unsloth can run on AMD GPUs via ROCm as long as your ROCm runtime, PyTorch ROCm build, and model stack are compatible. The most common reason it fails is a mismatched PyTorch ROCm wheel versus the installed ROCm version. Start with a minimal fp16 LoRA run using PyTorch SDPA attention, then optimize once you have a stable baseline.

### Does FlashAttention work on ROCm/AMD for Llama fine-tuning?

Sometimes, but it’s much more version- and GPU-dependent on ROCm than it is on CUDA. On consumer Radeon cards, you’re often better off starting with PyTorch’s SDPA attention and only enabling FlashAttention after a small run converges. If enabling FlashAttention triggers Triton compile errors or NaNs mid-run, treat that as a sign to fall back.

### Why does bitsandbytes 4-bit/8-bit quantization fail on ROCm, and what are alternatives?

bitsandbytes has historically been tightly coupled to CUDA kernels, so ROCm support is more fragile and can lag the newest training stacks. If 4-bit QLoRA fails, the most reliable alternative is fp16 LoRA with gradient checkpointing and smaller sequence lengths. For inference-only compression, consider GGUF-based quantization instead of training-time quantization.