Small Models, Big Savings
Not every task needs a giant AI model. Sorting support messages, spotting the language of a note, tagging sentiment or finding duplicate issues can be done by a small "encoder" model such as BERT, which is cheaper, faster, can run on your own machine and keeps data private. For a PM in 2026 this is a real product decision: it changes your running costs, your response times and your privacy story. And in an agentic setup, the smart pattern is to let a small model handle the easy 80 to 90 percent and call a large model such as Claude only when the task needs real reasoning. You do not need to code: you need to know what to ask for and how to judge the result.
The big idea in plain English
Two kinds of models. A large language model (LLM) such as Claude writes and reasons. An encoder model reads text and turns it into a label or a list of numbers (an "embedding") that captures meaning. Encoders are far smaller (tens to a few hundred million parameters, versus billions) so they are cheap to run, but they only do the narrow job they were trained for.
The family to know:
Model facts here come from each model's Hugging Face card at the time of writing (October 2026). Cards change, so always re-check before you decide.
What you'll be able to do
Choose the Right Size
Tell apart tasks that need an LLM from tasks a small model does well: classification, intent routing, sentiment, entity extraction (NER), language detection, semantic search and duplicate detection.
Do the Cost Math
Estimate cost per 1,000 requests for a small-model-first design and compare it with an all-LLM design, with clear assumptions.
Shortlist Models on Hugging Face
Filter by task, language and licence, read model cards, check evaluation results, and try models free in the browser before anyone writes code.
Write an Evaluation Requirement
Define a labelled test set, a target accuracy and an acceptable cost, so "good enough" is measurable and not a feeling.
Plan for Urdu and English
Ask the right questions about multilingual support, including mixed-language text, and test on real messages from your users.
Design the Routing Pattern
Specify "small model first, escalate to the LLM when unsure" with a confidence threshold, and decide when fine-tuning is worth it.
Where small models shine (and where they do not)
Great at
- Classifying tickets, emails or feedback into a fixed set of labels.
- Intent routing ("billing", "bug", "feature request").
- Sentiment and emotion tagging.
- Finding names, places, dates and products in text (NER).
- Detecting which language a message is in.
- Semantic search and "find similar" with embeddings.
- Dedupe: spotting near-duplicate issues or feedback.
Hand to an LLM instead
- Writing, summarising or rewriting text.
- Multi-step reasoning and planning.
- Open questions with no fixed answer set.
- Messy, ambiguous requests that need judgement.
- Anything where the label set keeps changing every week (a prompt is easier to change than a trained model).
The routing pattern and the cost math
Small model first, escalate when unsure. Every request goes to the small model, which returns a label and a confidence score. If the confidence is above a threshold (say 0.90), the answer is used. If not, the request is escalated to an LLM. You track how many requests escalate; that number is your savings dial.
Worked example (illustrative numbers only, not real prices)
Assume 100,000 support messages a month. Assumed LLM cost: $2.00 per 1,000 messages, so an all-LLM design costs $200 a month. Assumed small model, self-hosted or free-tier: $0.10 per 1,000 messages, so $10 a month to run it on all 100,000. If 85 percent are handled confidently by the small model and 15 percent escalate to the LLM: $10 + (15,000 x $2.00 / 1,000) = $10 + $30 = $40 a month, an 80 percent saving in this example.
These figures are made up to show the method. Your real numbers depend on the model, the message length, hosting and how many requests escalate. Ask your engineers for real token prices and measured escalation rates, and remember to include one-off costs such as labelling the test set and, if needed, fine-tuning.
Latency win
Small models answer in milliseconds on ordinary hardware, which makes live features such as search-as-you-type or instant routing feel snappy.
Privacy win
A small model can run inside your own environment, so sensitive text may not need to leave it. Always confirm with your security lead.
Multilingual: Urdu and English
Our team writes in both. The multilingual encoders above are trained on many languages at once, so one model can handle both. Coverage and quality are not equal across languages, though, so test with real Urdu, English and mixed-language messages. On their Hugging Face cards, XLM-RoBERTa base (94 language tags) and the multilingual MiniLM sentence model (50 languages) list Urdu, and mmBERT lists Urdu (code urd) among 1,800+ languages. For mBERT, read the language list on its card and check for yourself. Roman Urdu (Urdu written in English letters) is a special case: do not assume it works, test it.
How to pick a model on the Hugging Face Hub
The Hub hosts hundreds of thousands of models, so use a routine:
- Filter by task (for example Text Classification, Token Classification, Sentence Similarity). The Tasks page explains each in plain language.
- Filter by language (Urdu, English) and licence (Apache 2.0 and MIT are permissive; check anything else with your team before commercial use). See licences.
- Check downloads and likes as a rough signal of use. Popular is not proof of quality for your data.
- Read the model card: what data it was trained on, intended use, limits and bias. A card with no evaluation results deserves extra suspicion. See model cards.
- Look at the evaluation results. For embeddings, the MTEB leaderboard compares models across many tasks and languages; filter it to your language and size.
- Note the size (parameter count) and whether ONNX or quantised versions exist, since these run faster and cheaper. Ask your engineers to confirm.
- Try it free in the browser first: many model pages have a demo widget on the right, and Spaces are free hosted demos. Paste 20 real messages and see what happens before any engineering starts.
Example models to try in the widget: DistilBERT sentiment (English), multilingual Twitter sentiment, language detection. These are community models: check their cards and licences before relying on them.
The PM's job: requirement, evaluation, trade-off
1. Write the requirement
Name the task ("route each incoming message to one of 6 queues"), the languages, the label set, the speed needed, and the privacy rules. Say what happens when the model is unsure.
2. Build an evaluation set
Collect a few hundred real, anonymised examples with the correct answer written next to each, covering every language and the awkward cases. Keep it separate from anything used for training. Decide the pass mark up front, for example "at least 92 percent correct, and no queue below 85 percent".
3. Weigh accuracy against cost
A bigger model is usually a little more accurate and a lot more expensive. Ask: how much does one wrong answer cost us, and what does one right answer cost? Sometimes 90 percent at one tenth of the price is the better product.
4. Decide when to fine-tune
Try an off-the-shelf model first. If it misses the pass mark and you have a few hundred to a few thousand labelled examples, fine-tuning a small model on your own data often closes the gap. If the labels change often or you have almost no examples, stay with an LLM prompt. Fine-tuning is the engineers' work; your part is the labelled data and the pass mark.
Your certification path ($0)
Steps 1 and 4 give a free certificate you can show. The other steps are free but do not issue a certificate, and we say so for each one. Rough total for the recommended steps: about 15 hours, plus the practice.
1
Intro to Machine Learning (Kaggle Learn)
Provider: Kaggle · You earn: certificate of completion when you finish all lessons and exercises (Kaggle has announced downloadable course certificates; you need a free Kaggle account) · Cost: Free certificate · Time: about 3 hours (our estimate).
Models, training, testing and why we hold back data to evaluate. The exercises use Python in a browser notebook that Kaggle runs for you, so nothing is installed. Treat the code as read-and-run, not write-from-scratch.
OPEN COURSE
ABOUT CERTIFICATES
2
Machine Learning Crash Course (Google)
Provider: Google for Developers · You earn: knowledge only; the course page does not mention a certificate or badge · Cost: Free course · Time: pick the Classification, Embeddings and LLM modules, about 4 hours (our estimate; the page gives no hours).
The Classification module explains accuracy, precision and recall, which are the numbers you will use to judge any model. The Embeddings module explains what the "list of numbers" really is.
OPEN COURSE
3
Hugging Face LLM Course, chapters 1 and 2 (read along)
Provider: Hugging Face · You earn: no certificate; the course FAQ says "Currently we do not have any certification for this course" · Cost: Free course · Time: about 3 hours of reading (our estimate).
Written for people who know Python, so read chapter 1 (what Transformers are, and encoder vs decoder models) and skim chapter 2 for the ideas. You do not need to run the code.
CHAPTER 1
CHAPTER 2
4
Hugging Face AI Agents Course (fundamentals certificate)
Provider: Hugging Face · You earn: a free certificate of fundamentals after Unit 1; a certificate of completion needs Unit 1, a use-case assignment and the final challenge · Cost: Free certificate · Time: about 4 hours for Unit 1 (our estimate; the course gives about 3 to 4 hours a week per chapter).
This is your agentic step: how agents decide, use tools and call models. Needs a free Hugging Face account. The course expects basic Python, so aim for the fundamentals certificate and read along with the code.
OPEN COURSE
5
Introduction to natural language processing concepts (Microsoft Learn)
Provider: Microsoft Learn · You earn: a pass designation and achievement on your free Microsoft Learn profile when you pass the module assessment (not a formal certification) · Cost: Free badge · Time: about 1 hour (7 units; our estimate).
Beginner module on tokenization, statistical text analysis and semantic language models. A gentle recap of step 3.
OPEN MODULE
6
Optional: Large Language Models with Semantic Search (DeepLearning.AI)
Provider: DeepLearning.AI (with Jay Alammar and Luis Serrano) · You earn: course accomplishment, but the page says this needs a paid PRO membership, so it is not part of the $0 certificate path · Cost: Free to enroll and watch (the page says free for a limited time) · Time: 1 hour 32 minutes.
A clear, visual video course on embeddings and dense retrieval. Watch it for the ideas; skip the paid upgrade.
OPEN COURSE
Course catalogues and certificate rules change. Check each page before you start, and screenshot your completion. We checked these pages in October 2026.
Practice in GitLab
Do these on your real project board. No code and no new projects needed.
Exercise 1: A cost-reduction feature issue
Pick a real task that is probably done by an LLM today or done by hand (for example sorting incoming messages). Create an issue titled "Route [task] with a small model first". In the description write: the problem, the languages (Urdu, English), the label set, the current cost or time per 1,000 items, the target saving, and the privacy rules. Add an Evaluation set requirement section: how many examples, who labels them, where they are stored, and the pass mark (for example 92 percent correct).
Exercise 2: Model labels and a board
Create scoped labels model::candidate, model::testing, model::approved and model::rejected (Manage → Labels). Make an issue for each model you shortlist from the Hub (one issue per model), label it model::candidate, and set up a board with a list per label so you can drag a model through the stages.
Exercise 3: A model-selection comparison table in the wiki
In your project wiki create a page called Model Selection with a Markdown table. Columns: model name and link, task, languages, licence, size, evaluation score on our set, speed, estimated cost per 1,000 items, privacy notes, decision. Add 3 candidates, for example XLM-RoBERTa, the multilingual MiniLM and mmBERT. Link each row to its issue.
Exercise 4: A milestone with a decision gate
Create a milestone "Small-model pilot" with a due date. Put the feature issue and the model issues in it. Add a final issue "Go / no-go decision" that is closed only when the winning model has met the pass mark and the table is updated. Check the milestone burndown after a week.
The agentic angle: cheap first, Claude for the hard part
An agent is a loop that takes many small steps, and many of those steps are easy: "what language is this?", "which queue?", "have we seen this before?". If every step calls a large model, costs and delays add up fast. A well-designed agent uses a small model for the routine steps and calls Claude only for hard reasoning, planning and writing. It is the same routing pattern, applied at every step.
Practical exercise: draw the agent's routing map
Take a workflow you want an agent to run (for example "triage new feedback, link duplicates, draft a reply"). Ask Claude: "List each step, say whether a small classification or embedding model could do it or whether it needs an LLM, and suggest what confidence level should trigger escalation." Review its answer critically, correct it, and paste the result into the feature issue from Exercise 1 as a checklist. Mark each step small or LLM.
Free resource library