Pick an adapter size and watch a small model learn a new way of talking, then check what the shortcut cost.
A support inbox sorts messages into six buckets: billing, password reset, cancellation, order tracking, bug report, feature request. The starting model was trained on polite, well-punctuated writing. The messages it now has to handle are clipped, lowercase and full of typos, so it needs to adapt.
There are three ways to do that, and this page runs all three on the same test messages. Ask the model directly with no training at all. Retrain every weight it has. Or freeze the whole thing and train a tiny add-on beside it, which is what LoRA means. You choose how big that add-on is by typing a number, and the page shows how many settings each approach had to change and how well each one scored.
The last check is the one people skip. The add-on can be folded back into the original weights with plain arithmetic, so the finished model is one plain matrix again with nothing bolted on and nothing slower about it. The page proves the folded model gives exactly the same answers, to nine decimal places.
Everything trains here and now in a couple of seconds. It is small on purpose, and the output says exactly what was shrunk to make that possible.
FINE-TUNING PIPELINE prompt-only vs full fine-tune vs LoRA rank 16
==============================================================================
you chose adapter rank 16, alpha 16
alpha / r is held at 1, so a bigger rank buys
capacity without changing the size of each update step
RUN SIZE FOR THIS PAGE, and what the project uses when it is not on a clock:
feature width 128 dimensions (project: 256)
data 40 per intent generated (project: 95)
epochs 10 (project: 14)
rank sweep 5 configurations (project: 7)
the method is unchanged: one frozen base for both adapters, same
optimiser, same seed, same held-out test set.
------------------------------------------------------------------------------
1. DATA two banks, generated then filtered
------------------------------------------------------------------------------
six intents billing_question, password_reset, cancel_subscription, track_order, report_bug, request_feature
pretrain bank 120 generated -> 93 kept, formal phrasings (stands in for what the base already knew)
target bank 240 generated -> 152 kept, clipped and lowercase (the new domain)
filter 0 too short, 0 too long, 88 near duplicates (cosine >= 0.97 within a label)
label balance 1.61 to 1, within tolerance: False
the check reports, it does not delete: the fix for
imbalance is more of the small class, not less of the big
stratified split train 91 / validation 31 / test 30, seed fixed, built in 0.16 s
a few training rows
report_bug app crashes when i open csv export asap
request_feature hi suggestion: support the dashboard in the app chee
------------------------------------------------------------------------------
2. THE FROZEN BASE trained on the formal bank, then never touched again
------------------------------------------------------------------------------
features HashingEmbedder(dim=128), lexical, nothing downloaded
base head 774 weights, trained in 0.12 s, then frozen
base on the new domain accuracy 0.767, macro F1 0.760
that gap is what both methods have to close, and both
start from this identical head, so the comparison is
about the method and nothing else
------------------------------------------------------------------------------
3. PROMPT-ONLY no training at all
------------------------------------------------------------------------------
each intent is handed to the model as a numbered description and it picks
one. An unparseable reply counts as wrong rather than being retried or
quietly defaulted to the biggest class.
accuracy 0.700 macro F1 0.666 unparseable 0 of 30 0.01 s
------------------------------------------------------------------------------
4. WHAT EACH METHOD HAD TO CHANGE
------------------------------------------------------------------------------
method trainable frozen share of full train time
-----------------------------------------------------------------------------
prompt-only 0 - 0.0% 0.00 s
full fine-tune 774 0 100.0% 0.23 s
LoRA rank 16 2144 774 277.0% 0.63 s
the adapter is two small matrices: A is 16 by 128, B is 6 by 16.
B starts at exactly zero, so at step zero the adapted model is a bit-for-bit
copy of the base. Random values in both would perturb a working model before
training had even begun.
worth being blunt about the share column: this head has only 6 outputs, so
the adapter is only smaller than a full fine-tune while the rank stays under 6.
LoRA pays off when the matrix being adapted is large in both directions, as
it is inside a transformer. Same mechanism, smaller numbers.
------------------------------------------------------------------------------
5. TRAINING CURVE same optimiser, same seed, same data order
------------------------------------------------------------------------------
epoch full loss full val lora loss lora val
--------------------------------------------------
1 0.7717 0.839 0.8113 0.839
2 0.6219 0.839 0.7947 0.839
3 0.5077 0.935 0.7212 0.968
4 0.4309 0.968 0.5229 0.968
5 0.3699 0.968 0.3142 0.968
6 0.3230 0.968 0.1729 0.968
7 0.2837 0.968 0.0991 0.968
8 0.2545 0.968 0.0635 0.968
9 0.2285 0.968 0.0449 0.968
10 0.2084 0.968 0.0339 1.000
base weights untouched through LoRA training: True
no gradient ever reaches the base, which is why its logits are computed once
and reused every epoch. That is the cheap-training property, literally.
------------------------------------------------------------------------------
6. MERGE CHECK folding the adapter back into the base
------------------------------------------------------------------------------
W_merged = W + (alpha / r) * (B @ A). Exact arithmetic, not an approximation.
largest logit difference across the test set 8.882e-16
every prediction identical True
merged model parameters 774 (one matrix again)
so the adapter costs nothing at inference time once merged. Dropping the
alpha / r scaling is the classic way to get a merge that is nearly right and
quietly wrong, which is why it is asserted rather than assumed.
------------------------------------------------------------------------------
7. RESULTS one held-out test set, four systems
------------------------------------------------------------------------------
system trained accuracy macro F1
--------------------------------------------------------------
prompt-only, no training 0 0.700 0.666
frozen base, no adapter 0 0.767 0.760
full fine-tune 774 0.867 0.875
LoRA rank 16 2144 0.967 0.970
macro F1 is the number to argue with. Accuracy lets a model ignore a small
class and still look respectable; macro F1 charges it full price.
per class, LoRA rank 16
label support precision recall f1
-------------------------------------------------------
billing_question 6 1.000 0.833 0.909
password_reset 3 1.000 1.000 1.000
cancel_subscription 4 1.000 1.000 1.000
track_order 6 1.000 1.000 1.000
report_bug 6 1.000 1.000 1.000
request_feature 5 0.833 1.000 0.909
------------------------------------------------------------------------------
8. DOES A BIGGER ADAPTER HELP same base, same seed, alpha / r held at 1.0
------------------------------------------------------------------------------
rank trainable share of full accuracy macro F1 seconds
----------------------------------------------------------------
1 134 17.3% 0.867 0.872 0.06
2 268 34.6% 0.933 0.928 0.10
4 536 69.3% 0.967 0.970 0.18
8 1072 138.5% 0.967 0.970 0.34
16 2144 277.0% 0.967 0.970 0.64 <- you chose this
every row starts from a fresh copy of the same frozen base, so the table
measures the rank, not the order the rows ran in.
whole pipeline, data generation through the sweep, ran in 2.58 s here,
in plain Python with no tensor library and nothing written to disk.