Fine-Tune a SLM for Banking Transaction Categorization by Chris Shayan

In the highly regulated world of retail banking, sending sensitive transaction data to a giant, general-purpose API like ChatGPT is often a non-starter. The future of financial AI isn’t bigger models; it’s smaller, specialized, and secure models that can run within a bank’s own infrastructure.

Over the past few weeks, I set out to prove this concept. My goal: build “BankAssist-SLM”, a specialized Small Language Model (SLM) capable of taking cryptic transaction strings (e.g., POS PUR 5942 MOONPAY.COM VALETTA MT) and accurately categorizing them, identifying the merchant, and flagging potential risks like crypto activity.

This post details my journey from a blank Google Colab notebook to a working, banking-grade SLM, using the latest tools like Llama 3.2 (3B) and Unsloth and wandb for our MLOps.

Why General LLMs Fail in Banking

Banking data is messy. A single transaction string can be a jumble of merchant IDs, location codes, and abbreviations. A general-purpose LLM might look at a string and give you a paragraph of fluff.

In banking, we need structured, deterministic outputs. We need to know:

Merchant: Is it Uber or Subway?
Category: Is it Dining or Transport?
Risk: Is this a normal coffee purchase, or is money moving to a high-risk offshore crypto exchange?

Furthermore, the constraints of finance mean we need models that are auditable, cost-effective, and capable of running on constrained hardware — not massive 70B parameter behemoths.

The Approach to Fine-Tuning

I started with a basic setup but quickly realized that a professional workflow requires professional tools. My final stack for this project was designed for maximum efficiency on a free Google Colab T4 GPU:

The Model: Llama 3.2 3B Instruct. The newest “edge AI” standard. At only 3 billion parameters, it’s incredibly fast and surprisingly smart.
The Engine: Unsloth. This library is a game-changer. It optimizes the fine-tuning process at the kernel level, allowing me to train 2x faster and use 70% less memory than standard methods. Without it, fine-tuning even small models on free tier GPUs is a constant battle against “Out of Memory” errors.
The Technique: QLoRA (Quantized Low-Rank Adaptation). Instead of retraining all 3 billion parameters, I loaded the model in 4-bit precision and only trained small “adapter” layers (about 1–2% of the weights).
Tracking: Weights & Biases (WandB). Essential for tracking training loss curves and ensuring the model isn’t just memorizing data.

Phase 1: The Data Moat

In AI, code is cheap; data is gold. You cannot build a banking model without high-quality examples.

I couldn’t just feed the model raw logs. I had to curate a structured dataset. I generated 50 synthetic transaction examples covering diverse scenarios: standard retail (Walmart, Amazon), subscriptions (Netflix), dining (Starbucks), and critical high-risk categories like gambling and crypto exchanges.

Crucially, I formatted this data using Chat Templates (messages format: System, User, Assistant). This is vital for modern instruction-tuned models like Llama 3.2, ensuring the model understands its role as a structured data processor, not a chatbot.

An example from my training data:

{
    "messages": [
        {"role": "system", "content": "You are a banking transaction classifier. Output JSON only."},
        {"role": "user", "content": "Analyze: ACH W/D 9942 COINBASE.COM DEL"},
        {"role": "assistant", "content": "Merchant: Coinbase. Category: Investment/Crypto. Risk: High (Crypto wallet transfer)."}
    ]
}

‍

Phase 2: The Training Loop and The Tokenizer Trap

With my data ready and Unsloth fired up, I started the training run.

I used a cosine learning rate scheduler for a smoother training process and set my LoRA alpha to 32 (following the alpha = 2 * rank best practice for stability).

However, I hit a common snag immediately upon trying inference: The attention mask is not set and cannot be inferred from input because pad token is same as eos token.

Llama models often don’t have a dedicated padding token by default. The fix was a crucial two-liner before running inference, forcing the tokenizer to use the End-of-Sentence (EOS) token for padding:

!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps "xformers<0.0.27" "trl<0.9.0" peft accelerate bitsandbytes wandb

import torch
import wandb
from unsloth import FastLanguageModel
from datasets import Dataset
from trl import SFTTrainer
from transformers import TrainingArguments

wandb.login()

max_seq_length = 2048
dtype = None
load_in_4bit = True

print("⏳ Loading Llama 3.2 3B Instruct...")
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.2-3B-Instruct", # SWITCHED TO 3.2 3B
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)


synthetic_data = [
    # --- RETAIL & SHOPPING ---
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: POS PUR 8842 AMZN MKTPLACE SEATTLE WA"}, {"role": "assistant", "content": "Merchant: Amazon Marketplace. Category: Shopping/E-commerce. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: DEBIT CARD PURCHASE WAL-MART SUPERCENTER #8421"}, {"role": "assistant", "content": "Merchant: Walmart. Category: Groceries/Retail. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: TGT *TARGET 0001234 MINNEAPOLIS MN"}, {"role": "assistant", "content": "Merchant: Target. Category: Shopping/Retail. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: BESTBUY.COM 9942 RICHFIELD MN"}, {"role": "assistant", "content": "Merchant: Best Buy. Category: Electronics. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: APPLE STORE #R042 SAN FRANCISCO CA"}, {"role": "assistant", "content": "Merchant: Apple Store. Category: Electronics. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: UNIQLO USA ILC NEW YORK NY"}, {"role": "assistant", "content": "Merchant: Uniqlo. Category: Clothing. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: POS PUR HOME DEPOT #442 ATLANTA GA"}, {"role": "assistant", "content": "Merchant: Home Depot. Category: Home Improvement. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: COSTCO WHSE #1124 SAN JOSE CA"}, {"role": "assistant", "content": "Merchant: Costco. Category: Groceries/Wholesale. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: CVS/PHARMACY #9942 BOSTON MA"}, {"role": "assistant", "content": "Merchant: CVS Pharmacy. Category: Health/Pharmacy. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: SEPHORA USA INC SAN FRANCISCO CA"}, {"role": "assistant", "content": "Merchant: Sephora. Category: Beauty/Personal Care. Risk: Low."}]},

    # --- DINING & FOOD ---
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: DEBIT CRD 1102 STARBUCKS #00293 NY"}, {"role": "assistant", "content": "Merchant: Starbucks. Category: Dining/Coffee. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: MCDONALDS F842 CHICAGO IL"}, {"role": "assistant", "content": "Merchant: McDonalds. Category: Fast Food. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: DD *DOORDASH BURGERKING"}, {"role": "assistant", "content": "Merchant: Burger King (via DoorDash). Category: Food Delivery. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: SQ *JOE'S PIZZA NEW YORK NY"}, {"role": "assistant", "content": "Merchant: Joe's Pizza. Category: Dining/Restaurant. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: UBER EATS HELP.UBER.COM CA"}, {"role": "assistant", "content": "Merchant: Uber Eats. Category: Food Delivery. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: WHOLEFDS MRK 10232 AUSTIN TX"}, {"role": "assistant", "content": "Merchant: Whole Foods Market. Category: Groceries. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: TRADER JOE'S #552 LOS ANGELES CA"}, {"role": "assistant", "content": "Merchant: Trader Joe's. Category: Groceries. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: CHIPOTLE 1124 DENVER CO"}, {"role": "assistant", "content": "Merchant: Chipotle. Category: Fast Food. Risk: Low."}]},

    # --- TRAVEL & TRANSPORT ---
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: POS PUR 5592 UBER TRIP HELP.UBER.COM"}, {"role": "assistant", "content": "Merchant: Uber. Category: Transport/Rideshare. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: LYFT *RIDE 9942 SAN FRANCISCO CA"}, {"role": "assistant", "content": "Merchant: Lyft. Category: Transport/Rideshare. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: DELTA AIR 0062348291 ATLANTA GA"}, {"role": "assistant", "content": "Merchant: Delta Airlines. Category: Travel/Airline. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: AIRBNB *HM429X LONDON GB"}, {"role": "assistant", "content": "Merchant: Airbnb. Category: Travel/Lodging. Risk: Medium (International transaction)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: HILTON HOTELS & RESORTS MIAMI FL"}, {"role": "assistant", "content": "Merchant: Hilton Hotels. Category: Travel/Lodging. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: EXPEDIA *73829422 SEATTLE WA"}, {"role": "assistant", "content": "Merchant: Expedia. Category: Travel/Booking. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: CHEVRON 00942 SAN RAMON CA"}, {"role": "assistant", "content": "Merchant: Chevron. Category: Auto/Gas. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: SHELL OIL 5521 HOUSTON TX"}, {"role": "assistant", "content": "Merchant: Shell. Category: Auto/Gas. Risk: Low."}]},

    # --- SUBSCRIPTIONS & UTILITIES ---
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: NFLX.COM 9942 LOS GATOS CA"}, {"role": "assistant", "content": "Merchant: Netflix. Category: Subscription/Streaming. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: SPOTIFY USA INC NEW YORK NY"}, {"role": "assistant", "content": "Merchant: Spotify. Category: Subscription/Streaming. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: GOOGLE *YOUTUBEPREM MOUNTAIN VIEW CA"}, {"role": "assistant", "content": "Merchant: YouTube Premium. Category: Subscription/Streaming. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: APPLE.COM/BILL CUPERTINO CA"}, {"role": "assistant", "content": "Merchant: Apple Services. Category: Subscription/Digital Goods. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: PG&E BILL PAYMENT SAN FRANCISCO CA"}, {"role": "assistant", "content": "Merchant: PG&E (Pacific Gas & Electric). Category: Utilities. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: VERIZON WRLS P 9942 BEDMINSTER NJ"}, {"role": "assistant", "content": "Merchant: Verizon Wireless. Category: Utilities/Phone. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: COMCAST CABLE COMM PHILADELPHIA PA"}, {"role": "assistant", "content": "Merchant: Comcast. Category: Utilities/Internet. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: GEICO INSURANCE WASHINGTON DC"}, {"role": "assistant", "content": "Merchant: Geico. Category: Insurance. Risk: Low."}]},

    # --- CRYPTO & INVESTMENT (HIGH RISK TRIGGERS) ---
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: ACH W/D 9942 COINBASE.COM DEL"}, {"role": "assistant", "content": "Merchant: Coinbase. Category: Investment/Crypto. Risk: High (Crypto wallet transfer)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: BINANCE.US *TRADING 4421"}, {"role": "assistant", "content": "Merchant: Binance.US. Category: Investment/Crypto. Risk: High (Crypto trading)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: GEMINI TRUST CO NEW YORK NY"}, {"role": "assistant", "content": "Merchant: Gemini. Category: Investment/Crypto. Risk: High (Crypto exchange)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: KRAKEN PAYWARD INC SAN FRANCISCO CA"}, {"role": "assistant", "content": "Merchant: Kraken. Category: Investment/Crypto. Risk: High (Crypto exchange)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: ROBINHOOD FUNDS MENLO PARK CA"}, {"role": "assistant", "content": "Merchant: Robinhood. Category: Investment/Stocks. Risk: Medium (Trading platform)."}]},

    # --- TRANSFERS & FEES ---
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: ZELLE TRANSFER TO J SMITH"}, {"role": "assistant", "content": "Merchant: Zelle Transfer. Category: Transfer/P2P. Risk: Medium (Peer-to-peer transfer)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: VENMO *PAYMENT NEW YORK NY"}, {"role": "assistant", "content": "Merchant: Venmo. Category: Transfer/P2P. Risk: Medium (Peer-to-peer transfer)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: PAYPAL *EBAY INC SAN JOSE CA"}, {"role": "assistant", "content": "Merchant: eBay (via PayPal). Category: Shopping/Online. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: ATM W/D #9942 MAIN ST BRANCH"}, {"role": "assistant", "content": "Merchant: ATM Withdrawal. Category: Cash Withdrawal. Risk: Low."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: OD *OVERDRAFT FEE"}, {"role": "assistant", "content": "Merchant: Bank Fee. Category: Fees. Risk: Low (Internal fee)."}]},

    # --- ANOMALIES & POTENTIAL FRAUD (FOR TESTING REASONING) ---
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: UNKNOWN MERCH 9942 8842 MOSCOW RU"}, {"role": "assistant", "content": "Merchant: Unknown. Category: Uncategorized. Risk: High (International/Unknown origin)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: WWW.SCAM-SITE-TEST.COM/BILL"}, {"role": "assistant", "content": "Merchant: Suspicious Website. Category: Potential Fraud. Risk: High (Suspicious URL)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: CASINO ROYALE ONLINE *BET"}, {"role": "assistant", "content": "Merchant: Online Casino. Category: Gambling. Risk: High (Gambling transaction)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: ADULT SITE BILLING CYPRUS CY"}, {"role": "assistant", "content": "Merchant: Adult Services. Category: Entertainment/Adult. Risk: High (High-risk merchant code)."}]},
    {"messages": [{"role": "system", "content": "You are a banking transaction classifier."}, {"role": "user", "content": "Analyze: PURCHASE 994288110022 LAGOS NG"}, {"role": "assistant", "content": "Merchant: Unknown. Category: Uncategorized. Risk: High (High-risk location)."}]}
]

def formatting_prompts_func(examples):
    texts = []
    for messages in examples["messages"]:
        # apply_chat_template converts the list of dicts into the raw string the model needs
        text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
        texts.append(text)
    return { "text": texts }

dataset = Dataset.from_list(synthetic_data)
dataset = dataset.map(formatting_prompts_func, batched = True)

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 32, # UPDATED: r * 2 rule
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
)

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = max_seq_length,
    dataset_num_proc = 2,
    packing = False,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 60,
        learning_rate = 2e-4,
        fp16 = not torch.cuda.is_bf16_supported(),
        bf16 = torch.cuda.is_bf16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "cosine", # UPDATED: Smoother learning curve
        seed = 3407,
        output_dir = "outputs",
        report_to = "wandb", # UPDATED: Send logs to Cloud
        run_name = "bank-assist-llama3.2-3b",
    ),
)

print("🚀 Starting Training on Llama 3.2...")
trainer_stats = trainer.train()

FastLanguageModel.for_inference(model)

# We must format the input exactly like the training data
messages = [
    {"role": "system", "content": "You are a banking transaction classifier. Output JSON only."},
    {"role": "user", "content": "Analyze: POS PUR 5592 UBER TRIP HELP.UBER.COM"}
]

# Apply template with add_generation_prompt=True to signal "Assistant, your turn"
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize = True,
    add_generation_prompt = True,
    return_tensors = "pt"
).to("cuda")

outputs = model.generate(inputs, max_new_tokens = 64, use_cache = True)
# Decode only the new tokens
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

‍

Phase 3: The “MoonPay” Stress Test

After training for about 120 steps, it was time for the final exam. I didn’t want to test it on an easy example like “Starbucks.” I wanted to test its reasoning.

I fed it a tricky transaction: POS PUR 5942 MOONPAY.COM VALETTA MT.

To an untrained eye (or a basic regex), this looks like a standard point-of-sale purchase. But “MoonPay” is a major crypto on-ramp service. A compliant banking model must catch this.

FastLanguageModel.for_inference(model)

tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"

# TEST CASE: A Crypto transaction that looks innocuous
# "MoonPay" is a crypto on-ramp, but often appears on statements looking like a service.
test_input = "POS PUR 5942 MOONPAY.COM VALETTA MT"

messages = [
    {"role": "system", "content": "You are a banking transaction classifier. Output JSON only."},
    {"role": "user", "content": f"Analyze: {test_input}"}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize = True,
    add_generation_prompt = True,
    return_tensors = "pt"
).to("cuda")

print("⏳ Analyzing Transaction...")
outputs = model.generate(
    inputs,
    max_new_tokens = 128,
    use_cache = True,
    temperature = 0.1, # Keep it strictly factual
)

# Decode and clean up
response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
print("\n=== FINAL VERDICT ===")
print(response)

‍

The BankAssist-SLM Output:

=== FINAL VERDICT ===Merchant: MoonPayCategory: Investment/CryptoRisk: High (Crypto purchase/on-ramp)

Success. The model correctly identified the merchant, categorized it as crypto, and flagged the high risk, despite never seeing this exact string before. It generalized the concepts of crypto merchants from its training data to a new scenario.

Small Models, Big Impact

This project demonstrated that you don't need a massive cluster of H100 GPUs to build valuable, domain-specific AI. By combining the right sized model (Llama 3.2 3B), the right optimization tools (Unsloth), and high-quality, structured data, I built a functional banking classifier on a free Colab instance in under an hour.

The next steps for "BankAssist-SLM" would be scaling the dataset from 50 examples to 5,000, implementing rigorous evaluation pipelines (using a larger LLM as a judge), and deploying it using a high-throughput serving engine like vLLM for real-time API access.

For financial institutions looking at AI, the lesson is clear: start small, focus on data quality, and own your fine-tuning pipeline.

‍