Someone applying for the Berlin evening told me on the form that they were already partway through a paid product management course. They wanted to watch the whole thing happen once, discovery through delivery, in a single session. They were paying for the knowing and applying to a free thing for the doing.
That gap is what this article is about.
There is more product AI training on sale right now than at any point I can remember. Bootcamps, certifications, memberships, five-day free workshop weeks that end with a discount code. Some of it is genuinely good. The problem is not quality. The problem is that you cannot tell which one you need until you have built one thing and found out where you actually get stuck.
So do that first. Here is exactly how, with a real example.
The two-hour build that tells you what to buy
Pick a problem you have, not a portfolio idea. The best ones are the small repetitive jobs you already resent. I will use a real one: every Monday morning a PM I know rebuilds the same status summary by hand from four different places.
Write the brief
Four parts, about thirty words, and no screens or features in it.
How might we help a product manager, on Monday morning with four browser tabs open, turn last week's raw updates into a summary she can send, in under five minutes?
Person, context, goal, finish line. Notice it does not say dashboard, or digest, or AI summarizer. Those are solutions and they are not yours to decide yet.
Name the one thing it has to prove
Not "it works". Something you can watch: she pastes last week's raw notes in, and what comes out is good enough to send without editing. That is your test, and you wrote it before you built anything, which is the entire trick.
Write the first prompt
Paste the brief in, then say what you want to see. Something like this, and yes, it is this plain.
Build a single page. A big text box where I paste messy weekly updates from four people. One button, "Make the summary". Below it, the output: three sections, what shipped, what slipped, what needs a decision.
Keep it plain. No dashboard, no charts. I want to test whether the output is good enough to send.
Build it
Ugly is fine. Half the flows missing is fine. You are not shipping this.
Run the test you named
Paste in a real week of real messages. Then read the output and ask the only question that matters: would I send this? Not "is it impressive".
Write down where it broke
One sentence per problem, in plain words. That last step is the whole reason you did this. Two hours in, you now have something almost nobody shopping for a course has: a specific, personal, honest list of what you cannot do yet.
Now let the gap pick your next step
Here is where I think most upskilling goes wrong. People buy a course to learn "AI", which is not a skill, and then they cannot tell whether it worked. Take the sentences you just wrote and run them through this.
“I never got started, I kept rewriting the brief.”
A framing gap. You are trying to be right instead of specific.
Nothing you can buy. Write three bad briefs in twenty minutes and pick one.
“It built the wrong thing and I could not say why.”
A brief gap. Your sentence named a solution, so you got someone else's.
The brief worksheet in the playbook, then try again on the same problem.
“I could not tell if the output was good enough to send.”
An evals gap. This one is real and genuinely course-shaped.
A proper evals course, or start by writing twenty test cases by hand.
“I could not get my own data into it.”
A tooling gap, and the smallest one on this list.
An afternoon with the API docs. Not a course.
“It looked finished and it was not.”
A scope gap. Demos flatter themselves.
Reps, and someone senior telling you where it will break.
“It worked but I could not explain it to my team.”
A framing gap again, at the other end.
Show it to a person, in a room, and watch what they ask.
Only two of those six rows are solved by buying something. That ratio has held every time I have watched someone go through it.
Three numbers you will meet while shopping, and none of them is a study
While you are deciding, you will be sold to. Three claims come up constantly and all three fall apart when you follow the citation.
“Only ten percent of training transfers to the job”
It comes from a 1982 article in the Training and Development Journal by David Georgenson, who cited nothing, because he was not reporting a finding. Robert Fitzpatrick traced it.
Georgenson had no need to, and did not, cite any evidence or authority for the 10% estimate; it is clear that he had used a rhetorical device to catch the reader's attention.
Forty years later it is still in decks, wearing a lab coat.
“Skills now have a five-year half-life”
Usually attributed to IBM or Deloitte. It is neither. It traces to a 2011 book, A New Culture of Learning, co-authored by the former Xerox chief scientist John Seely Brown. The shorter versions you see now, four years, two years in the AI era, come from a podcast remark by Kian Katanforoosh, who runs a company that sells skills assessment. It may still be true. It is not measured.
“Online courses have a three to seven percent completion rate”
This is the number the cohort-course industry uses to explain why its product is different. The peer-reviewed figure is a median of 12.6% across 221 courses, from Katy Jordan in 2015. And Harvard and MIT, looking at 4.5 million participants across 290 courses, found that among the 662,000 people who got at least halfway through, the median certification rate was 36%. The scary number is mostly a denominator problem: online courses count tyre-kickers as students.
Now look at the other side of that comparison. The claim that cohort-based courses hit seventy-five to ninety-six percent completion comes from three places. Andrew Chen at a16z, in the same post announcing a16z's investment in Maven. Wes Kao, who co-founded Maven, citing a course she co-founded. And Shaan Puri, an investor in Maven. I could not find one independent measurement anywhere.
Every one of those numbers may point in a true direction. Not one of them is evidence.
What the research does support
Learning by doing has real backing, and it is much better than the folklore.
In 2014 Scott Freeman and colleagues published a meta-analysis in PNAS covering 225 studies. Students in active learning conditions scored 0.47 standard deviations higher on exams and concept inventories. Failure rates were 21.8% under active learning against 33.8% under lecturing, which makes you about one and a half times more likely to fail if someone is just talking at you. A 2020 PNAS follow-up by Elli Theobald and colleagues found the same effect and added that it narrows achievement gaps by about a third on exam scores.
Be careful how far you carry that. It is undergraduate science and maths, and "active learning" there means group problem-solving and clicker questions, not shipping an app on a Tuesday night. It is a strong analogy. It is not a finding about you.
Now the part where I argue against myself
I run a three-hour evening. Here is the evidence that says three hours is not enough, and I would rather you read it from me.
In 2007 a team led by Kwang Suk Yoon reviewed more than 1,300 studies of teacher professional development for the US Department of Education. Nine survived the evidence screening. The three programmes with the least training, five to fourteen hours in total, showed no statistically significant effect at all. The ones above fourteen hours did. A single evening is three hours, which sits under the line where anybody has measured anything.
Then there is what happens afterwards. Alexander Nolte and colleagues tracked 592 projects from 73 hackathons with 1,912 participants. More than a third showed some continuation, but by day six only 17% were still active, and about 5% were still going after five months. That study measures projects rather than skills, and the same group points out that almost nobody has properly measured what participants retain. So it is not proof that the learning evaporates. It is a warning that the artifact usually does.
And spacing is real. Nicholas Cepeda and colleagues pulled together 839 assessments across 317 experiments and found that spreading practice out beats cramming, and that the ideal gap grows the longer you need to remember something.
So one build is a diagnosis, not an education. The practical version of all that research is boring and it works.
- Monday 1The two-hour build above. The small repetitive job you already resent.
- Monday 2Same shape, harder problem. Something with real data in it this time.
- Monday 3Now show it to someone. A colleague who did not build it. Stay quiet and watch.
- Monday 4Rebuild the one that went worst. Same brief, second attempt. This is where the learning actually lands.
That is twelve hours, spaced, on real work. It clears the threshold the research cares about, and it costs you four mornings.
What a hiring manager can actually see
One more reason to build before you buy, and this one is about the outside world rather than your own head.
In May this year Ithaka S+R ran a discrete choice experiment with 543 recruiters and hiring managers across 6,516 choice tasks. Moving a degree from in-person to online cost a candidate 2.5 percentage points of selection probability. One extra year of relevant work experience gained them 15.3 points. Six times bigger.
Candidates' work experiences and skills play a substantially larger role in hiring managers' decisions than degree program modality.
In June, Revelio Labs looked at people who took AI certifications between 2023 and 2025 and matched them against similar people who did not. The certificate holders were already ahead before they certified, and the certificate itself added about 2.8 percentage points to salary growth at their next move. Most of the apparent benefit is who takes courses, not what courses do.
Indeed's Hiring Lab measured AI mentions in US job postings at 4.2% in December 2025. That was the highest share ever recorded and 134% above early 2020, and it still means about one posting in twenty-four says the word. You have more time than the ads suggest.
You can buy the hours. You cannot buy the rep.
Where this leaves you
Four steps, in this order, and the order is the whole point.
Build one small thing this week
Two hours, a problem you actually have, and the test named before you start.
Write down what stopped you, in one sentence
Plain words. "I could not tell if the output was good enough to send" beats "I need to learn AI."
Run that sentence through the table above
Most of the time the answer is another rep, not a purchase. When it is a purchase, you now know exactly which one, which is a much easier decision.
Then repeat it, spaced
Four Mondays, four problems. That is what turns a diagnosis into a skill.
- 01You cannot pick the right course until you know your own gap, and the only way to find it is to build one small thing.
- 02Two hours: brief, name the test, prompt, build, run the test, write down where it broke.
- 03Most gaps are not course-shaped. Framing and scope problems need reps. Evals genuinely needs teaching.
- 04Three of the numbers this market sells on are not studies: the ten percent transfer figure, the five-year skill half-life, and the three-to-seven percent completion rate.
- 05One session is a diagnosis. Four spaced Mondays are an education. Hiring managers weight relevant experience about six times more than credential type.
The autumn build evenings run three hours, in English, and you leave with something working and one honest sentence about what stopped you. Pick a city.
The brief worksheet, the run sheet and the test-planning pages are free in the playbook.
Run one, then tell me how it went. You built it, you get the credit.
Frequently asked questions
What should I build first?
The smallest job you already resent doing by hand. A weekly summary you assemble from four places, a spreadsheet you re-sort every Friday, a message you rewrite for three audiences. Small and real beats clever and hypothetical, because you will know immediately whether the output is good enough.
How do I know if a course is worth taking?
Write your gap as one sentence first, then check whether the syllabus names that specific thing. If it does, and the teacher has shipped the kind of product you work on, it is a reasonable buy. If your sentence is “I have never shipped anything”, no course fixes that, and you should build before you buy.
Do AI certificates help you get hired?
A little, and less than the marketing says. Revelio Labs found in June 2026 that people who take AI certifications were already ahead before they certified, and that certifying added roughly 2.8 percentage points to salary growth at their next move. Ithaka S+R separately found hiring managers weight one extra year of relevant experience about six times more heavily than whether a credential was earned online.
Is one evening long enough to learn anything?
Not to learn a skill, no. The strongest study here, a US Department of Education review from 2007, found no measurable effect below fourteen hours of training. A three-hour evening is a rep and a reality check: you find out what building actually feels like and you leave with one honest sentence about what stopped you. Both are useful. Neither is a course.
I am not technical. Is this realistic?
Ten teams have built in my rooms so far and all ten shipped something working. Thirty-seven of the forty-three people involved do not write code for a living, and five of the ten teams had no engineer at all. Every one of those five shipped. The tools are not the hard part any more, which is exactly why the brief is.
What do I do after the first build?
Repeat it, spaced, on harder problems. Four Mondays in a row is twelve hours of real practice, which clears the threshold the research cares about and costs nothing. Keep the briefs. After a month you will have four working things and four honest notes on where you got stuck, which is a better portfolio than most certificates.
