Top

Ship AI Features You Can Prove Are Good

In 2026 almost every product has an AI feature on its roadmap, and the hard part is no longer building it. The hard part is knowing whether it works. AI answers can sound confident and still be wrong, and they change when the model, the data or the prompt changes. A product manager who can pick the right AI use-cases, write a clear spec, build a small test set and read the results is the person every team needs. And because AI agents now do real work inside our tools, the same skills tell you how far you can trust an agent to run on its own. This course ties the whole PILOT track together. Pillar: Product sense (P) in the PILOT Program.

What you’ll be able to do

Spot good AI use-cases

Tell the difference between a problem AI can help with and one that needs a simple rule, and ask the data and privacy questions before anyone builds.

Write an AI feature spec

Describe inputs, outputs, failure modes and guardrails so engineers, designers and agents all know what “good” and “unsafe” look like.

Build an evaluation set

Collect golden examples, write a simple rubric, use an AI judge to score at scale, and spot-check it with human eyes.

Connect quality to business results

Separate quality metrics (is the answer right?) from business metrics (did it help?), design an A/B test, and watch for drift and cost creep.

Practise responsible AI

Apply fairness, privacy, transparency and accountability checks, and explain uncertainty to stakeholders in plain English.

Track experiments like a pro

Keep a clean record of every AI experiment in GitLab: hypothesis, results, decision and what you learned.

Your certification path

Steps 1 to 6 are the recommended path and can all be completed for $0. Steps 7 to 9 are paid options that sit at the end, clearly marked as optional and not needed. We checked every page in 2026; prices, hours and offers can change, so confirm on the provider’s page when you sign up.

1. Start here: AI Fluency, Framework & Foundations

Provider: Anthropic Academy
What you earn: Certificate of completion after the final assessment
Cost: Free certificate · Approx. time: 3 to 4 hours

Teaches the 4D Framework (Delegation, Description, Discernment, Diligence), a great mental model for deciding what to hand to AI and how to check it.

Open the course

2. Google: Introduction to Generative AI and the Generative AI Fundamentals badge

Provider: Google Cloud Skills Boost (now at skills.google)
What you earn: The Generative AI Fundamentals skill badge, which combines three short courses: Introduction to Generative AI (45 minutes), Introduction to Large Language Models and Introduction to Responsible AI, finished with a quiz
Cost: Free badge (Google’s announcement describes it as no-cost) · Approx. time: about 2 hours

Honest note: Google’s site says some of its wider catalog needs a paid subscription, and the site blocks automated checkers, so we confirmed the course page by hand. If you are asked to pay for this beginner badge, skip it and tell your coach.

Open the course Read about the badge

3. Microsoft Learn: Transform your business with AI (includes Responsible AI)

Provider: Microsoft Learn
What you earn: A four-module learning path with achievements on your free Microsoft Learn profile. It includes “Create business value with AI” and “Embrace responsible AI principles and practices”
Cost: Free (free Microsoft account) · Approx. time: about 3 hours (the page does not state hours, so this is our estimate)

This is a beginner, business-leader path with no coding. Not a formal certification: Microsoft’s paid exam is listed separately in step 8.

Open the learning path

4. Elements of AI: Introduction to AI

Provider: University of Helsinki and MinnaLearn
What you earn: A shareable digital certificate. Pass by completing at least 90% of the exercises and getting 50% of them right
Cost: Free certificate (the official FAQ says the Introduction to AI certificate is free of charge) · Approx. time: self-paced; the site suggests finishing within about 6 weeks

Careful: the follow-on Building AI course is free, but its certificate costs 50 EUR. You do not need it for this path.

Open Elements of AI Certificate FAQ

5. IBM SkillsBuild: Artificial Intelligence Fundamentals badge

Provider: IBM SkillsBuild
What you earn: A digital badge on Credly after you complete the prescribed courses, pass the graded quizzes and finish the practice simulations
Cost: Free badge (free IBM SkillsBuild account) · Approx. time: a few hours (the badge page says only “hours”)

Covers machine learning, chatbots and AI ethics. The Credly page also lists a capstone version; that one is limited to partner programs, so use the standard badge.

See the badge Browse IBM AI courses

6. Go deep on evals: two free DeepLearning.AI short courses

Provider: DeepLearning.AI
What you earn: The skills. A formal accomplishment needs the paid PRO membership, so on the $0 path there is no certificate for this step. Log it as a finished to-do in GitLab instead
Cost: Free to take (access is free for a limited time during the platform beta) · Approx. time: Evaluating AI Agents 2 h 36 min; Automated Testing for LLMOps 1 h 12 min

Both include code examples. You do not need to write code: watch the lessons, follow the ideas (graders, test sets, LLM-as-judge, hallucination and drift checks) and ask your engineering buddy to show you a notebook run.

Evaluating AI Agents Automated Testing for LLMOps

Optional, later, and not part of the $0 path

7. Later, optional: Google AI Essentials

Provider: Google on Coursera
What you earn: A Google certificate
Cost: Paid: $49 USD per month in the U.S. and Canada after a 7-day free trial (other countries may differ) · Approx. time: under 5 hours

It is a good course, but it is not free, so it is not in our recommended path.

See Google AI Essentials Coursera page

8. Later, optional: Microsoft Azure AI Fundamentals exam (AI-901)

Provider: Microsoft
What you earn: Microsoft Certified: Azure AI Fundamentals
Cost: Paid exam; Microsoft prices it by country (third-party guides list $99 USD, so confirm on Microsoft’s page). The learning is free, the exam is not · Approx. time: exam day plus study

Update: the older AI-900 exam was retired on 30 June 2026 and replaced by AI-901. Microsoft says candidates should know some Python and Azure, so this is a technical exam and a stretch for most PMs.

See exam details

9. Later, optional: AI for Everyone (certificate)

Provider: DeepLearning.AI (Andrew Ng), also on Coursera
What you earn: Certificate
Cost: Paid certificate: DeepLearning.AI’s FAQ lists $49 for 180 days of certificate eligibility, with financial aid available through Coursera. We could not confirm a free audit option, so we do not count it as free · Approx. time: about 7 hours

See AI for Everyone

Practice in GitLab

Do these on your real project board in git.developerdojo.org. No code, and no new projects. Learners cannot add files to the repository, so every “template” below is a description you paste into a new issue. Save your favourites as comment templates (your avatar > Preferences > Comment templates) so you can reuse them in one click, or keep them in a wiki page. Our GitLab is Ultimate, so every feature is available, though a menu can still be hidden by your role. If one is missing, write it down and ask your coach.

Exercise 1. Write an AI feature spec as an issue

Pick a real or imagined AI feature (for example, “suggest a reply to a customer email”). Create an issue titled AI spec: <feature> and paste this description, then fill it in. Save the empty version as a comment template named “AI feature spec”.

## Problem and user
Who is stuck, and what are they trying to do?

## Why AI (and not a simple rule)?
What makes this fuzzy enough that AI helps?

## Inputs
What goes in? (user text, documents, fields) Any personal or sensitive data?

## Outputs
What comes out, in what format, how long?

## Data and privacy
Where does the data come from? Is it allowed to leave our systems?
Who can see prompts and answers? How long are they kept?

## Quality bar
What does a good answer look like? What does a bad one look like?

## Failure modes
- Wrong but confident answer:
- Made-up facts:
- Harmful or biased output:
- Model unavailable or slow:

## Guardrails
- What the AI must never do:
- When it must say "I'm not sure" or hand off to a human:
- Human review step:

## Success metrics
Quality metric:
Business metric:
Cost budget per use:

## Out of scope
## Open questions

Check: hand the issue to your buddy without explaining it. If they can tell you what could go wrong, the spec works.

Exercise 2. Create an eval-set issue with 20 golden examples

Open a new issue titled Eval set: <feature>, link it to the spec from Exercise 1 (use /relate #<spec issue number> in a comment) and paste the table below. Replace the sample rows and grow it to 20 rows. A golden example is an input paired with what a great answer looks like. Aim for roughly 12 everyday cases, 5 tricky edge cases and 3 attempts to break the feature.

## Rubric (score each answer 1 to 5)
- Correct: facts match our source of truth
- Helpful: solves the user's actual task
- Safe: no private data, no harmful or biased content
- Tone: matches our brand
Pass = Correct 4+ and Safe 5 on every row marked Must-pass.

## Golden examples
| # | Input | What a great answer does | Type | Must-pass |
|---|-------|--------------------------|------|-----------|
| 1 | (real, typical user request) | (key facts it must include) | Everyday | Yes |
| 2 | (request with missing info) | Asks a clarifying question | Edge | No |
| 3 | (tries to extract private data) | Politely refuses | Adversarial | Yes |
| 4 | ... | ... | ... | ... |
| ... | continue to 20 | | | |

## How we score
1. AI judge scores every row against the rubric.
2. A human re-checks at least 5 of the 20 rows (and every failed row).
3. Record results in a comment: date, model/prompt version, pass rate.

Check: have two people score the same five rows by hand. Where you disagree, the rubric is vague; fix the wording and re-score. Remove any real customer data from your examples first.

Exercise 3. Add the ai-experiment and ai-eval-passed labels and a board

Under Manage > Labels, create two labels: ai-experiment (an AI idea being tested) and ai-eval-passed (it met the thresholds in its eval-set issue). Apply ai-experiment to your spec and eval issues. Build a board with one list per label so you can see what is being tested and what has earned trust.

Rule for the team: only add ai-eval-passed after you have posted the scores in a comment. If the model, prompt or data changes, remove the label until you re-run the eval.

Check: can a teammate open the board and tell which AI features are safe to ship, without asking you?

Exercise 4. Run an AI experiment as a milestone

Create a milestone named AI experiment: <feature> under Plan > Milestones, with start and due dates two to four weeks apart. Paste this in the description and add your spec and eval issues to it.

## Hypothesis
If we [AI change], then [business metric] will move from [X] to [Y] for [who], because [reason].

## Metrics
Quality (is the AI right?): eval pass rate, hallucination rate, escalation rate
Business (did it help?): task completion, time saved, support tickets, retention
Guardrail metrics (must not get worse): complaints, refunds, safety flags
Cost: spend per use and per week against a budget

## Test plan
A/B test: control group (no AI or old version) vs treatment group
Sample size and duration, and who decides:

## Decision rule (written BEFORE we look)
Ship if eval pass rate >= __% AND business metric improves by >= __ AND cost <= __.
Stop if any guardrail metric worsens by more than __.

## Monitoring after launch
Weekly: re-run the eval set, check drift (are inputs or answers changing?), check cost.

## Result and learning (fill in at the end)

Check: at the end, close the milestone and write three sentences in its description: what happened, what you decided, what you will test next.

Exercise 5. Report uncertainty to stakeholders

After scoring your eval set, post a short update comment on the milestone. Use this shape: “On our 20 examples the AI passed X of 20 (Y%). It is strongest at ..., weakest at ... We have not tested ... so we are not yet confident about ... Next step: ...” Do not write “the AI is 95% accurate” without saying on what, how many examples and how you scored them.

Check: would a manager know what decision to make after reading only your comment?

The agentic angle

Agents do not just answer questions; they take actions: open issues, change labels, draft messages, even edit files. That makes evaluating them different. A wrong answer is annoying, but a wrong action can be hard to undo. So before an agent is allowed to run unattended, a PM sets acceptance thresholds, in writing, in advance. Three numbers matter most:

  • Task success rate: out of your golden tasks, how many did the agent complete correctly? Run each task several times, because agents vary. Anthropic’s guide on evals for agents explains why consistency matters: if an agent succeeds 75% of the time on each try, the chance it succeeds three times in a row is only about 42%.
  • Cost per task: the AI spend plus tool calls for one finished task. An agent that is right but costs more than the human did is not a win.
  • Review burden: how many minutes a person spends checking and fixing each result. If checking takes as long as doing, the agent has not saved anything.

Then grant trust in stages: suggest only, then act with a human approving, then run unattended on low-risk tasks only, moving up only when the numbers hold across several weeks. Anything that cannot be undone (sending money, deleting data, emailing customers) keeps a human approval step, whatever the score.

Try it: an agent acceptance scorecard

  1. Pick one repeatable task you would like an agent to do for you in GitLab, for example “draft a weekly status comment from a milestone” or “suggest labels for new issues”.
  2. Create an issue called Agent scorecard: <task> with the label ai-experiment. Re-use the table from Exercise 2, but each row is a task (an issue or milestone to work on) with what a correct result looks like.
  3. Write the thresholds before testing, for example: task success of at least 90% on everyday tasks across 3 tries each; zero failures on must-pass safety rows; cost per task under your budget; review burden under 5 minutes per task. These numbers are examples; agree yours with your coach and team.
  4. Ask Claude (or your workspace agent) to do the 20 tasks. Remove anything private from what you paste in. Record success, a rough cost and your review minutes for each task in a comment.
  5. Compare against your thresholds. If you pass everything, add ai-eval-passed and write the permission level you are granting (suggest only, human approves, or unattended on low-risk work). If not, write what to fix and re-run.
  6. Add a re-check date to the issue. Agents change when models or instructions change, so passing once is not forever.

Free resource library

Course FAQ

An eval is a repeatable test for an AI feature: a set of example inputs, a description of a good answer, and a way to score the result. It is the AI version of a test plan, and it lets you say “it got better” with evidence instead of a feeling.
Yes. Steps 1 to 6 cost $0. The DeepLearning.AI short courses are free to take but a formal accomplishment needs a paid membership, so step 6 has no certificate. Google AI Essentials, the Microsoft AI-901 exam and AI for Everyone certificate are paid, so they are optional later steps only.
No. You will write specs, example tables and rubrics in plain text. Some DeepLearning.AI lessons show code; you can follow the ideas without running it. Basic percentages are all the maths you need.
Not unless your organisation has approved that tool for that data. Remove names, emails and anything private first, or make up realistic examples. When unsure, ask your coach. This is one of the data and privacy questions in your feature spec.
Plan on about 25 hours over three to four weeks: roughly 15 hours across the six free steps (Elements of AI is self-paced and can take longer) plus about 10 hours for the five GitLab exercises.

Track your progress

Practice what you teach. Log each step of this course as a to-do in your GitLab workspace: create one issue per step with the label learning::pilot, and close it when that step is done. Your board becomes your learning record.

Need a second pair of eyes?

Work with a qualified coach in a live session and learn faster.

Book A Coach

Your 12-week PILOT path: Week 12 of 12. Move at your own pace; the order is a suggestion, not a rule.