By Shadrack John (Shadrackovsky), Antera Group.

Training a language model from scratch-tokenizer, architecture, pretraining corpus, all of it-is a serious undertaking, and it's rarely the right first move if your goal is a working Swahili assistant. In most cases, the faster and more resource-efficient path is to start from a strong open-weight base model and teach it your language and domain through parameter-efficient fine-tuning. This guide from ANTERA walks through exactly how to do that with LoRA and QLoRA on Mistral and Qwen models, with a specific eye toward the challenges that come up when your target language is low-resource.

Table of Contents

1. LoRA vs. QLoRA: What You're Actually Choosing Between

2. Choosing a Base Model: Mistral vs. Qwen for African Languages

3. Hardware and Environment Setup

4. Preparing African-Language Training Data

5. Configuring the Fine-Tune: Rank, Alpha, and Target Modules

6. The Training Script

7. Handling Code-Switching and Low-Resource Quirks

8. Evaluation: Don't Trust the Loss Curve

9. Merging, Exporting, and Deploying

10. Common Pitfalls

11. A Realistic Project Timeline

1. LoRA vs. QLoRA: What You're Actually Choosing Between

Both techniques fall under Parameter-Efficient Fine-Tuning (PEFT): instead of updating all of a model's billions of weights, you freeze the base model and train small "adapter" matrices injected into specific layers.

• LoRA (Low-Rank Adaptation) freezes the base model in its original precision (typically bf16 or fp16) and trains low-rank adapter matrices alongside it. It needs more VRAM than QLoRA but converges slightly more predictably.

• QLoRA takes this further by quantizing the frozen base model to 4-bit precision (using the NF4 format, which is tuned to how neural network weights are actually distributed) while still training the adapters in higher precision. This is what makes it possible to fine-tune a 7-8B model on a single consumer GPU with as little as 8-12GB of VRAM, or larger 13-30B models on a single 24GB card.

For a solo developer or small team without a fleet of A100s, QLoRA is almost always the right starting point. The quality gap between QLoRA and full fine-tuning is small enough on most tasks (commonly cited in the low single digits, as a percentage of benchmark score) that it's rarely worth the extra hardware cost, especially for a first pass on domain or language adaptation.

2. Choosing a Base Model: Mistral vs. Qwen for African Languages

Both are strong, permissively-licensed (Apache 2.0) choices, but they lean differently for this use case:

• Mistral models have historically been trained with strong multilingual coverage and have a reputation for saying "I don't know" rather than confidently hallucinating-useful if you're building anything RAG-adjacent (e.g. a Swahili question-answering system) on top of your fine-tune. Mistral's smaller dense models (in the 7-8B range) remain a practical, well-documented starting point for a first fine-tuning project, with newer, larger MoE releases (Mistral Small/Large) available if you need more headroom later.

• Qwen models (Alibaba's Qwen series) have consistently emphasized broad multilingual tokenizer coverage across a very large number of languages, and the smaller members of the family (in the 4-8B range) tend to be efficient to fine-tune while still being strong instruction-followers out of the box.

Practically: if you already have a working pipeline on Mistral-7B or Qwen2.5-7B, there's no urgent need to chase the newest release before your data pipeline and evaluation are solid. Model versions in this space move fast; treat the base model as a swappable component and invest your real engineering effort in the data and evaluation steps below, which transfer across versions.

3. Hardware and Environment Setup

The current standard stack for LoRA/QLoRA work is built around Python 3.11+, PyTorch, and the Hugging Face ecosystem (transformers, peft, trl, datasets), optionally accelerated with Unsloth, which rewrites key backpropagation kernels by hand to cut memory use and roughly double training speed versus a stock Hugging Face setup, while remaining compatible with the same PEFT/TRL workflow.

A minimal Unsloth-based setup:

bash

conda create -n antera-finetune python=3.11 -y
conda activate antera-finetune
PyTorch with CUDA support
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
Unsloth (pulls in bitsandbytes, transformers, peft, trl automatically)
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps xformers trl peft accelerate bitsandbytes
Evaluation tooling
pip install rouge-score nltk scikit-learn sacrebleu

Hardware guidance:

• 7-8B model, QLoRA, 4-bit: usable on a single 12GB consumer GPU (e.g., RTX 3060/4070) or a free-tier Colab T4.

• 7-8B model, LoRA (no quantization): more comfortable with 24GB (RTX 3090/4090).

• If you don't have local GPU access, on-demand cloud GPUs (RunPod, Lambda Labs, Vast.ai) let you rent an A100/H100 by the hour, which is often cheaper than buying hardware for a first project. Always run a short smoke test on a small data slice before committing to a longer, more expensive run.

4. Preparing African-Language Training Data

This is where most of the real work-and most of the leverage-lives, especially for low-resource languages where you can't just download a massive ready-made instruction dataset.

Data sourcing strategies, roughly in order of effort:

• Translate existing instruction datasets. Take an established English instruction-tuning dataset (Alpaca-style, Dolly, or similar) and machine-translate it into your target language, then have native speakers spot-check a sample for fluency and cultural fit. This is the fastest way to bootstrap volume, but translated data can carry over English sentence structure and idioms if you don't review it.

• Collect or license native-language text and convert it to instruction format. News archives, government publications, community forums, and public educational content can be reformatted into instruction/response pairs. Always check licensing before training on scraped or third-party content.

• Generate synthetic data with a strong multilingual model. Prompting a large model to generate instruction/response pairs in your target language, then filtering for quality, is a practical way to fill gaps-particularly for domain-specific tasks (e.g., Swahili customer support, agricultural advice) where no existing dataset comes close.

• Community and crowd-sourced collection. Organizations like Masakhane have shown that community-driven data collection-native speakers contributing and reviewing text directly-produces higher-quality, more representative data than scraping alone, even at a smaller scale.

Format. Whatever the source, converge on a consistent instruction format matching how you'll actually prompt the model in production:

json

{
"instruction": "Eleza kwa ufupi umuhimu wa kunywa maji ya kutosha kila siku.",
"input": "",
"output": "Kunywa maji ya kutosha kila siku husaidia mwili kufanya kazi vizuri..."
}

Data quality checklist before training:

• Deduplicate aggressively-repeated near-identical examples cause the model to overfit to specific phrasings.

• Balance register: include formal, conversational, and (if relevant to your use case) code-switched examples, not just news-style or religious text, which tends to dominate scraped Swahili corpora.

• Target at least a few thousand high-quality examples for noticeable behavior change; tens of thousands for more robust domain coverage. Quality consistently beats raw volume in low-resource fine-tuning-a smaller, carefully reviewed dataset will usually outperform a larger, noisier one.

5. Configuring the Fine-Tune: Rank, Alpha, and Target Modules

python

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/mistral-7b-instruct-v0.3-bnb-4bit",  # or a Qwen2.5/Qwen3 equivalent
max_seq_length = 2048,
load_in_4bit = True,   # this is what makes it QLoRA
)
model = FastLanguageModel.get_peft_model(
model,
r = 16,                     # rank: start here, go to 32-64 for harder domain shifts
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha = 16,            # commonly set equal to r as a starting point
lora_dropout = 0,
bias = "none",
use_gradient_checkpointing = "unsloth",
random_state = 3407,
)

Practical defaults for a first run:

• Rank (r): Start at 16. Language/dialect adaptation on top of an already-multilingual base model rarely needs more than 32; push higher only if you're teaching genuinely new domain knowledge, not just style and fluency.

• Alpha: Set equal to rank as a default; increase relative to rank if the adapter's influence feels too weak after evaluation.

• Target modules: Attention projections (q_proj, k_proj, v_proj, o_proj) plus the MLP projections (gate_proj, up_proj, down_proj) is a solid default that covers most of the model's capacity to shift behavior.

• Learning rate: LoRA typically needs a noticeably higher learning rate than full fine-tuning-around 2e-4 is a stable starting point-because far fewer parameters are being updated. Going much above 5e-4 risks instability.

6. The Training Script

Using TRL's SFTTrainer on top of the model configured above:

python

from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

dataset = load_dataset("json", data_files="swahili_instruct.jsonl", split="train")
def format_prompt(example):
return {
"text": f"### Maagizo:\n{example['instruction']}\n\n### Jibu:\n{example['output']}"
}
dataset = dataset.map(format_prompt)
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
dataset_text_field = "text",
max_seq_length = 2048,
args = TrainingArguments(
per_device_train_batch_size = 2,
gradient_accumulation_steps = 4,
warmup_steps = 20,
num_train_epochs = 3,
learning_rate = 2e-4,
fp16 = not torch.cuda.is_bf16_supported(),
bf16 = torch.cuda.is_bf16_supported(),
logging_steps = 10,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "cosine",
output_dir = "outputs",
save_strategy = "epoch",
),
)
trainer.train()

A few notes specific to low-resource fine-tuning: keep epochs modest (2-4) on smaller datasets to avoid overfitting to a narrow set of phrasings, and consider holding out a small validation split even from a small dataset-with limited data, it's tempting to train on everything, but you lose your only signal for catching overfitting early.

7. Handling Code-Switching and Low-Resource Quirks

If your target language involves heavy code-switching in real usage example Kiswaenglish for Kiswahili don't filter it out of your training data as "noise." A few practical steps:

• Include code-switched examples deliberately, ideally in the proportion they actually occur in your target use case-not zero, and not overrepresented either.

• Watch tokenizer fertility on your target language before committing to a base model. Run a quick script tokenizing a representative sample of your target-language text with each candidate model's tokenizer and compare average tokens-per-word; a model with dramatically higher fertility on your language will cost more to run and may generalize worse, independent of how much fine-tuning data you throw at it.

• If fertility is a serious problem, vocabulary augmentation (adding a few thousand target-language-specific tokens to the base tokenizer before fine-tuning) is a more advanced but effective option-though it requires resizing the model's embedding layer and is a bigger step than a standard LoRA workflow.

8. Evaluation: Don't Trust the Loss Curve

A dropping training loss tells you the model is memorizing your fine-tuning data-it does not tell you whether it's actually gotten more fluent, more accurate, or more useful in the target language. At minimum:

• Hold out a validation set the model never trains on, and check for the loss gap that signals overfitting.

• Have native speakers manually review a sample of generations on prompts outside the training distribution-this catches fluency and cultural-appropriateness issues that automated metrics miss entirely.

• Where available, run your fine-tuned model against any existing benchmark in your target language (translated MMLU subsets, Winogrande translations, or community-built benchmarks from initiatives like Masakhane) to get a comparable, if imperfect, quantitative signal.

• Re-check general capability, not just target-language performance: run a handful of English or general-reasoning prompts before and after fine-tuning to confirm you haven't badly degraded the base model's broader abilities-a known risk with aggressive fine-tuning on a narrow dataset.

9. Merging, Exporting, and Deploying

For QLoRA specifically, merging requires care: the base model must be de-quantized back to fp16/bf16 before the adapter weights are merged in. Calling a naive merge directly on the 4-bit model will silently degrade quality.

python

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained("outputs/checkpoint-final")
model = FastLanguageModel.for_inference(model)
Merge adapters into the base model at full precision, then save
model.save_pretrained_merged("swahili-mistral-merged", tokenizer, save_method = "merged_16bit")
Optional: export to GGUF for llama.cpp / Ollama deployment
model.save_pretrained_gguf("swahili-mistral-gguf", tokenizer, quantization_method = "q4_k_m")

From there, a GGUF export can be served locally with llama.cpp or Ollama for low-cost inference, which matters if you're deploying in a market where API costs per token are a real constraint.

10. Common Pitfalls

• Training on too little, too-uniform data and mistaking memorization for fluency-always evaluate on held-out, out-of-distribution prompts.

• Skipping the tokenizer fertility check and discovering after a full training run that your chosen base model is inefficient on your target language.

• Ignoring code-switching and ending up with a model that's fluent in "clean" formal Swahili but breaks down on the way people actually type.

• Merging QLoRA adapters directly on the 4-bit model instead of de-quantizing first, which silently degrades output quality.

• Chasing the newest base model release instead of shipping and iterating-open model releases in this space move on a roughly monthly cadence; a solid pipeline on a slightly older model beats an unfinished pipeline on the newest one.

11. A Realistic Project Timeline

For a solo developer working evenings and weekends:

• Week 1: Source and clean 2,000-5,000 instruction pairs; set up environment and run a smoke test on 50 examples.

• Week 2: Full training run, iterate on rank/learning rate based on validation loss and a handful of manual spot-checks.

• Week 3: Native-speaker evaluation pass, fix systematic errors by adding targeted examples to the dataset, retrain.

• Week 4: Merge, export, and deploy a small demo (a simple chat interface is enough to start collecting real usage feedback).

This mirrors, roughly, the shape of building something like a Swahili-focused assistant from an open base model rather than from scratch-most of the differentiation comes from data quality and evaluation discipline, not from exotic training tricks.