A masterclass in crowdsourcing, w/Duolingo's Luis von Ahn

The very best founders I know are brilliant at building systems. They connect teams, they remove bottlenecks, and they eliminate single points of failure. And yet When it comes to their own wealth. Most are running a disconnected stack. A tax accountant here and a state attorney there, a wealth manager who doesn't talk to either one of them.

Creative planning was built to fix exactly that. One integrated team of tax professionals, state planners, investment specialists, all coordinated by a dedicated wealth manager who sees your full financial picture and keeps every piece working together. Proactive tax efficiency, state strategy, investments all under one roof. Creative planning where wealth works together. Learn more at creative planning dot com slash masters of scale.

Global hiring is such a headache when you're building fast, but companies like Eleven Labs are scaling effortlessly with deal. They quadrupled their workforce in twenty twenty five. Without hiring an HR team. Build your global team with deal. Visit deal.com slash MOS and start expanding your business today. That's D E L

dot com. Slash M O S. Hey, folks, Jeff Berman here. I am thrilled to share some of the new names who will be joining us at this year's Masters of Scale summit. This may be our biggest stage yet. Reed Hastings, Meredith Whitaker, Van Jones, Amjad Masad, and more.

Will be there with us October 20th through 22nd in San Francisco. If you're building something great, or you want to build something great, We want you there with us too. Join us at masters of scale dot com slash apply twenty six. That's mastersofscot com slash apply. Twenty six.

I was gonna make a gym. That was going to be free. Anybody could go there, imagine. It's a free gym. It's amazing. That's Luis Fon. Founder of Language Learning App, Duolingo.

Describing a game changing idea he had at the age of twelve to disrupt the fitness world. I thought this was a unique idea. that wasn't going to completely revolutionise the world. What was it? That made this idea so revolutionary.

It would harness the power of gym goers. Electricity We're gonna connect all the exercise equipment to the power grid. And we're gonna sell that electricity to the power company. And that's how we were gonna make money.

Young Luis. feverishly doodled weird and wonderful designs for its free gym. Exercise bikes hooked up to the grid. Treadmills wired to power converters. Bodybuilders could pump out the electrons as they pumped iron.

Yeah. En society could burn каларії. instead of fossil fuels. But

Luis's idea Hit a snag. It turns out this is actually not a very good idea. Humans are not very good at generating electricity this way. That's right.

Even the most dedicated gym rap. Can't. Generate enough electricity to make the scheme worthwhile. There was also Another flaw.

I have now realized a lot of gym make a lot of their money from people signing up and never showing up. And this is where young Luis's naivete shows through. Traditional gym are placing a bet that most of their customers will lose their drive. Whereas Luis wanted his gym's motivation to be aligned with his customers. Luis isn't the only person.

Who fell under the electrifying spell of the power pumping gym. Who knows how many thousands of people have had the same idea. Yeah, and uh actually I'm one more of those thousands to millions of people in my childhood. And just as you say, you learn that actually the math doesn't work. Right.

It's not just Unworkable math. That makes the gym idea an ineffective use of crowd sourcing. It's the lack of a shared mission. Bonding the crowd and the gym operator.

crowd is attracted by free gym time. but they're not interested in the fruit of their efforts, the free electricity. They have no shared pride in it. No commitment. And no reason.

To commit to keeping the lights on. and the energy flowing. That's why I believe. Crowdsourcing can scale your business in unexpected ways. As long as you perfectly align.

Your mission. With your crowds. Motivation. You gotta have incredible talent at every position. There are fires burning when you're going out. Can you believe that? Such an idiot. And then you go back to this is totally gonna be amazing. There are so many easy ways. I have no idea what to do. Sorry, we made a mistake. But you have to time it right. Oops. Working out of a free bedroom apartment. We haven't made just how you do it.

This is masters of scale. I'm Reed Hoffman, co-founder of LinkedIn. Partner Greylock. And your host.

And I believe crowd sourcing can scale your business in unexpected ways. As long as you perfectly align. Your mission. With your crowds. Motivation.

To picture what I mean, I want to remind you of a classic scene from the movie. Raiders of the Lost Ark. In this scene. Rakish archaeologist. Indiana Jones descends into an ancient room under the Egyptian desert.

Inside. There's a massive model of a long buried city. Indy has with him. An ancient artifact. called the staff of Raw.

It's basically a tall pole. with a bronze medallion at the top. He places it at a precise position on the map. At just the right time of day. Rays from the sun burst into the room.

Pass through the medallion on the staff. and send a ray of light. directly to the spot on the map. Where they'd find. The Ark of the Covenant.

Everything lined up. And this is the kind of alignment you need to get crowdsourcing right. Crowd sourcing is a way to tap skills. And scale. That you don't have in house.

When it works. It opens extraordinary opportunities. It lets you scale faster and farther than you ever could on your own. But If you get the alignment wrong.

You can end up with anything. From a weak fizzle. to a fiery disaster. I wanted to talk to Luis Fon on about this because his company, Duolingo. It's one of the most fascinating examples of crowd sourcing gone right.

Duolingo is the app that lets you learn language in bite sized lessons, motivated by achievements and prompts. It has over three hundred milyen users who complete over seven billion exercises a month. And much of the course content. is created by its passionate users. That's why alongside the likes of Spanish.

Chinese and Arabic. You can find more esoteric languages. Like Esperanto. Navajo And even

Cling on. Luisa's story is a master class in working with your audience. And he also knows. First hand how learning a language is Can change your life.

I was born in Guatemala and I I grew up there. And I came to college here in the US. But to come to college in the US you have to take an English proficiency test. The most common one is called a tofu. It's Stands for Test of English as a foreign language. I had to take a

This test. And I had the problem that The country of Guatemala ran out of seats for this test. And I Freaked out, I thought I couldn't apply to college anymore. Fortunately, I had enough means to be able to fly to El Salvador to take that test. Guatemala's kind of a dangerous.

Uh so I had to go there, I had to take the test. Flying into a war zone. Just to prove he can speak a language? Louise thought. There has to be a better way.

And at the time I thought I'm gonna do something to kill the tofu. On arriving in the US. Luis put aside his desire to kill the tofu. and focused on another ambition.

One he and I shared. I wanted to become a math professor. This is what I wanted to do. I read somewhere that you also wanted to become a professor. Philosophy. A little less disciplined than the math professor. So Luis left his home in Guatemala to study at Duke University.

Today. He's a consulting professor of computer science at Pittsburgh's Carnegie Mellon University. Or CMU. And it was at CMU, when Luis was completing his PhD, that he caught an impactful lecture by Yahoo's chief scientist. Udi Manbur.

When he came and gave a talk of ten problems that they didn't know how to solve at Yahoo. One of the problems really captivated my attention. So Yahoo was one of the first few places that would offer free email accounts and Spammers would write programs to obtain millions of email accounts.

Uh for free. 'Cause they wanted to send spam from Yahoo accounts. It was the year two thousand. Hanyahu was desperate. To stop spam bots from opening free accounts.

And then with my PhD advisor, a guy named Manuel Blum, who's amazing. We came up with a potential solution, which was these distorted characters that you have to type whenever you're Getting an account is just

We've all seen captions. When you're signing up for something online, you're presented with a set of random letters or numbers. Bent out of shape, as if they're melting. And you retype them. In a box.

With a bit of squinting and head tilting, it's not too hard for us humans. But at the time. It was impossible for machines. And this. Was the point.

Thoughts beware. Nun Sharpas. We built the first prototype and we actually gave it to Uri and he put it in Yahoo within a week. And I also now understand that for a very large company to do something in one week, it means it was a serious problem for them. So yeah, so from then on all kinds of websites started using it and it was pretty amazing. Here's something else that's pretty amazing.

Luis and Manuel gave away the capture technology. For free. That's right. Then let one of the world. Biggest internet companies have a tech they were desperate for.

With no payment. Why? It comes back. to that talk by Uti. Yahoo's chief scientist.

I now understand that this was a recruiting talk. Back then I did not realise that. I now understand the reason he came to see a meal was yeah, I'll just tell him some stuff that that's exciting and then um You know, maybe they'll come work for Yahoo. UD knew that the way to recruit engineers Wasn't to talk about how great it was to work for Yahoo.

Instead He presented these engineers with the one thing they couldn't resist. An unsolvable problem. Udi wasn't just recruiting in a very clever manner. He was also

Crowdsourcing for Yahoo. And his precise alignment between engineers motivation and Yahoo's mission is one you should keep in mind. Yahoo ran with Louisa's solution, and pretty soon captions were springing up Everywhere.

Wanna sign up for an online account? Capture. Join a mailing list? Capture. Post to a forum. Capture, capture, capture.

Charlie, Alpha, Papa, Dango, Charlie, Hotel, Alpha, Catch. If you want to tell somebody's human or not, you need a challenge response test and put a request to type words obscured and wavy The test fully automated

Alphabet ya, it's gonna get you your BOT you can take from me It's like a digital I can keep the co capture I'm at a robot. I'm a human being and know that you know what capture means It's completely automated public test to computers and the humans apart In terms of sheer numbers. It was a success.

I did a little back of the envelope calculation of the number of times somebody would Type of capture. online. And the number I came up with was about two hundred million times a day. first I was pretty proud of myself, I thought, look at the impact that this has had.

But then I started feeling bad because each time you type one of these you waste about ten seconds of your time. And two hundred million times ten seconds is half a million hours a day. So I started feeling bad about that. And not only that. People kinda hate.

Doing these things. Luis. started feeling responsible for millions of hours of lost productivity. But I think he's being far too hard on himself. The capture.

Did solve a problem. And a few seconds of annoyance is a price most of us are willing to pay for a life of a lot less spam. So the CAPTCHA was a successful, if simple, form of crowd sourcing. It's a decent alignment of crowd motivation. And company mission.

For a ten second task, it works. I think what really irked Luis was that there was a missed opportunity here. An inefficiency. And from speaking to Luis. I could tell he loaths.

Inefficiency. Follow me on a little tangent while I share Luis's response to one of our lightning round questions. The one thing you wish your phone Could do. You know, I started getting a lot of these smart home things.

And I have like a million Base stations plugged into other stuff and there's a million things and it doesn't work super well. I wish my phone could just make that work. I mean it should know where I am, turn on the lights right there. I shouldn't have to say, Okay, Google turn on the lights. I wish this was just farther along. I'm a big fan of this, but it's still pretty annoying. And people kind of make fun of me because you know, they're like you could just tap that switch. I'm like, no, no, no, no, no, listen to this. Okay, Google turn the lights red and then it works. But uh I wish my phone could just be

Better at that. It's the strong version. to inefficiency that got Luis thinking about how to iterate on the cap chat. So I thought can we make that time count for something useful?

During those ten seconds, the human brain that was solving this was doing something that computers could not do. So it was ten seconds of very valuable work in some sense. Luis started looking around for problems that could be solved. With ten seconds of human brain power. It was around this time that Google embarked on its quest to digitize every book in existence.

And They'd hit a snag. At the time, for older books, things that were written before nineteen eighty, the computer could not recognize about twenty five to thirty percent of the words. Luckily the computer knew it couldn't recognize the word. Luis thought to himself.

Well Humans can recognize those words. What if we could use those ten seconds of human brain power To help digitize. Old books.

We could take all those words and get people to read them for us while they're typing a capture on the internet. So that was the idea with recapture. And it turned out it worked. Luis called this new version. Recapture. And it was about this time that a lot of websites asked to use it.

Some of them. Or website you'll have heard of. The first one was not all that big, but actually for our size it was pretty big. A website was called Online Booty Call dot com. Well, Okay, you'll definitely have heard of the next one.

Pretty soon after that a website that was starting to get Big They contacted us. They were like, Hey, we're implementing a capture service. We looked around, this seems to be the best one. Can you handle high traffic? And of course we just said sure we can.

We didn't realise how high it was. That company? It was Facebook. Other big sites. like CNN and BuzzFeed signed up for Recapture's translation service.

And they were willing to pay. In two thousand and nine Google bougapture. And it's still harnessing its crowd sourcing power today. They're using it to improve things like the street view, the recognition of the addresses.

So basically anything that computers are not super good at, they kinda turn into a capture to get a lot of training data. I asked Luis about how the CAPTCHA experience Helped evolve his thinking about crowd sourcing. Did it occur to you this was like your gym idea? Where it was kind of not at first, but eventually I'm like, Oh, this is just like that gym. It's pretty similar. Yes. Yeah. And obviously as we get to Duolingo, it'll be a pattern, but what are the kind of the

Ways that you put this tool set together where you say, Oh, actually in fact There's ways that groups of humans can do this. in collaboration with computers that make all of this ecosystem better. What's the kind of questions you ask, or kind of the ideas that that occur to you in solving these problems that make you one of the very unique people of creating these ecosystems.

Yeah, I mean one of the hardest things is finding a problem that is kind of Goldilocks problem in that computers are not quite good at it, but humans are really good at it and humans can do it pretty quickly. At least in all the systems that I've done like this. I mean I I did a number of these. It was things that humans could do in five seconds, ten seconds. While Goldilocks wanted porridge was just the right temperature.

Luis wanted problems that were just the right complexity. They had to be too complex for computers, but easy enough for humans to do in a few seconds. And you can turn them into all kinds of different problems. I mean a capture is one where you can just uh the incentive for the human to do it is because they want to get an email account or buy tickets or whatever. The incentive is just this intrinsic thing. It's a blocker and they have to do it. Those captions

Only took ten seconds to do. But even a ten second blocker is frustrating. And frustrating your users is never good. There's another problem. With Recapture, users are actively helping achieve something.

Arguably something for the greater good. But they may not have been aware of it. And when they found out, well They weren't always thrilled. This is exactly the reaction one of our producers, Katie Clark Gray, had when she was working on this episode and realized for the first time.

That recaptures were more than just an annoyance. I had no idea until we were working on this episode that all of those thousands of recaptures I've done helped digitize books for Google. Now, I mean I love books, but I feel very conflicted that I wasn't even aware that I was helping and Like was it really a good thing? I mean, was it for Google's profit? Am I am I going to get free access to those books that I helped translate for the computer? Am I gonna get a credit? Well I even know which specific books that I help with. I I I mean, I have questions.

Though it's important to remember that one person's is another person's cool. Just listen to how our executive producer June Cohen. Reacted when she learned the hidden crowdsourcing.

Of the recapture. This is so odd because Katie and I agree on Most things. I carbon date to an earlier era in the web and in technology, I can actually remember the day I learned

the actual scope behind the CAPTCHA project that it wasn't just a clever way to make sure that I was a human, but it was this grand crowdsourcing scheme that was allowing us to digitize old books. That is Brilliant. Like the fact that Louis Van and his team had found a way to actually make effective use of this six seconds of my wasted time, I just thought. This is a masterwork in crowdsourcing. It had never occurred to me that you could put like

People's greatest annoyances to work for the world's greatest challenges. I love it. This makes me hate these six seconds much less. This is something to remember with crowdsourcing. Not everyone. We'll understand your mission.

And misunderstanding can cause a build up of bad sentiment that may poison your project. This is why you need to be clear with your contributors. Especially when their efforts take more than a mere ten seconds. And that brings us to Luis's next leap.

I wanted to do something related to education that could help a country like Guatemala. Which is why I thought of language learning. So I thought I would do something that would teach people English for free. And I didn't really know how to make money.

The first idea actually, it occurred to me it was something very similar to Recapture or very similar to the gym. Which was okay, we're not gonna charge people because I don't want to charge people because this is kind of supposed to help the world. So this was actually the first idea of Duolingo when we launched it. People would be learning a language, but then after a few lessons, we would tell them, Hey, you wanna practice what you just learned. Here's a news article that is related to what you just learned in the language that you're learning, can you translate it to your native language? In Duolingo, Luis had found a way to monetize the activity of his users in a way that furthered their mission, Rather than exploiting them. If you are crowdsourcing

For a for-profit business, you need to demonstrate that your mission is the same as that of your contributors. And that the money you make benefits your shared mission. But there was still a problem. For Duolingo.

There was again a flaw kinda similar to the gym flaw. Just can't make that much money off of translation, it turns out. I think it's kind of a shitty business model, but that was the original business model for Duolingo. Luisa's solution? Classic blitz scaling. Don't worry about the money right now.

Over time we Pretty much realized, look, the main thing we're gonna do is something to teach a language. We'll figure out how to make money later. And As time passed, we kept raising more and more venture capital and our investors would tell us the prevailing wisdom at the time was don't worry about making money, just grow.

Fortunately. For Luis. And his investors. Duolingo users. We're motivated by the shared mission.

Making it easy and free for anyone. To start learning a language. They were exactly the passionate core you need if you are letting money take a back seat as you speed down the blitz scaling highway. This passionate core was also critical in the next vital development in Duolingo's business model. When you've built substantial wealth through your business, it's often tied up in a single equity position.

The upside is real, but so is the risk, and knowing when to act isn't always obvious. Creative planning works with business owners to build a strategy around concentrated equity. When to diversify, how to manage tax risk, and how to protect what you've spent years. Building. Creative planning where wealth works together.

Learn more at creative planning dot com slash masters of scale. Hey listeners, Bob here. If you listen to Rapid Response on Masters of Scale, you may be missing half the show. Because every Friday we release a second rapid response exclusively in the Rapid Response feed. The guests and topics are just as compelling and timely from Ford CEO to NASA's administrator to the lessons from The Devil Wears Prada. It takes about 10 seconds to find, just search rapid response wherever you listen to podcasts and hit follow to make sure you never miss an episode. I hope to see you there.

Humans will never be more intelligent than AI. There's gonna be two types of companies. Those are great at AI and those that went out of business because they weren't. How do we build a future? That is human centered. I'm Rana El Calyubi. And on my podcast, Pioneers of AI, we answer that question and so many more. As an AI scientist, entrepreneur, and investor, I know what it takes to build AI that works for everyone.

Every week, I sit down with the pioneers shaping our future. And we take you behind the scenes of the AI that's transforming our lives. Find pioneers of AI wherever you tune in. Before the break, we heard how Luis Van found a Duolingo. To help people learn languages. For free.

Duolingo had tried to make money by capturing the efforts of its language learners to offer a translation service. However, this turned out. To be a non starter. The translation business was hard. But I would also say it was another

Bigger problem. although now the motivations of the company were in the same ballpark as those of its users. They were not totally aligned. You can be a passionate language learner without caring a jot. about providing translation services.

So Luis put aside this crowd sourcing idea for Duolingo. However, he was about to hit upon a more unexpected use of crowdsourcing. One that would see Duolingo's motivations. Perfectly aligned with those of its users. And that would push Duolingo.

to become the world's most popular language learning app. When we launched Duolingo, you could only learn Spanish. And German. And I made the Spanish course. I'm a native Spanish speaker. And then my co-founder is from Switzerland, Swiss German, and he made the German course. And then we hired somebody to do a Portuguese course and we hired somebody to do a French course.

This handcrafted approach. was an undeniable. An unexpected success. Apple named us the iPhone app of the year. We didn't expect that.

And it just kept growing and growing and people started asking for a ton of languages. And Literally every day we would get requests. Multiple requests for a can you add This language or that language. So we

quickly realized there was no way we were gonna be able to add all the languages that people wanted. And also it didn't even make much business sense. I mean, adding a course to learn Esperanto. Yeah, just not gonna get that many users. The people. who reached out to Luis were deeply motivated to share the wonder of their native languages with other people.

Luis thought back to the theme running through his previous ideas. It just occurred to me at some point. In the past I've worked on crowd sourcing. Maybe we can crowdsource this. Maybe we can get People to actually help us add the course. themselves. And so the first thing I did is for one of these requests randomly.

I just responded, I said, look. This is a very small language. Would you be able to help? They responded pretty much immediately saying, I'd love to And I have friends and I'm willing to help.

Note here how Luis uncovered a hugely valuable resource. In fact, It was the resource that would shoot duolingo to massive scale. And he found it by reaching out to his passionate crowd of users. We realized actually we could get a good number of people helping us add languages to Duolingo.

In part because it was a free way to learn languages. I mean people really saw it. We were very mission driven and people really saw that. It's the holy grail. Or should that be The Rosetta Stone of Crowdsourcing. rallying people who share your mission to gather beneath your banner.

To keep them there, you have to make sure you're directing their efforts towards something that engages them. Not something that has no relation to them. like generating electricity they'll never use or helping to digitize books. They'll never read. What we did is we opened up our tools for creating languages for Duolingo, for adding languages to Dualingo to the world. In the first week about fifty thousand people applied to our a language.

Keep this in mind when you reach out to users and offer to be partners with them. In a shared mission. You raise the stakes. If you let them down, they'll lose faith. In your commitment.

And your product. And that powerful crowd. You've worked so hard to assemble. Will disperse. As will.

Their willingness to work with you. This is why Luis put so much time and effort Into ensuring that every user generated course Duolingo puts out meets his rigorous standards. To do this. He had to find a way to measure their effectiveness.

But first. We would measure just user engagement. So you know average time that they spent on the course, user retention. So did they come back the next day, et cetera. I did not realize how hard it would be to measure How well people learn. I thought, how hard can it be? You do a pre and a post test. You just ask them beforehand, you do a little a few questions, then you let them learn, and then you ask a few questions afterwards.

It is impressively hard to do that. It's taken us years and A significant number of people with PhDs to figure this out. This is a problem that all mission driven founders Must face. How much people like your product.

Is not the same as how effective it is. in achieving your mission. for some fraction of the users, we would give them a little test on the app, and then we would have them use the app for a while, and then we would give them another test and we would see how much they learned. And then we thought, okay, well we can A B test against this. So If we teach a little differently, we can see

Did the scores go up more or less? But They soon found a problem with this method. At first The numbers told a story of success.

But when they looked a little closer. The story was tottion. One of the problems we had Probably the easiest way to get your scores to move up. is to make the people who are not very good drop away.

But that's not teaching better. That's just like if you just make the course really frustrating. Actually the less able people drop away. And your scores go up. And then we realized what was going on was

You know, why we were just making the course impossibly difficult. So the ones that stuck around kinda already knew the language. So it turns out to be pretty hard, but by now we have a much more sophisticated ways of measuring how well people learn. We have a number of measurements and we can tell uh, you know, the quality of the different courses. Using these measurements. Duolingo can test a course on a selection of users and tweak it to perfection before releasing it.

Maintaining quality is vital to keeping users and contributors engaged. But so is getting out of the way. If your efforts to keep quality up end up blocking more than they help. Then your crowd.

We'll find somewhere else. To realize their passion. I think the reason we have so many volunteers working on some of these kind of smaller courses on Duolingo is Because of the mission. I mean, we've shown them over the years that we are not going to suddenly put a paywall in the front of Duolingo and just nobody will be able to use their stuff.

So one of the biggest things is just really people do it because they believe in the mission. The mission is so important that Duolingo has taken clear Legally binding steps. to assure contributors that they're fully committed. At first.

We did not have a legal contract with our contributors. It is unclear who owned this data. I mean it didn't really occur to us. After a while there was a couple of volunteers that started thinking, Well, wait, who owns this? And we had to spend effort and we had to get our lawyers to really craft a good contract where now the contract is so the volunteers actually own all their work.

But we get license forever to use it. Despite his long history of crowdsourcing, This was something new for Luis. And I believe this legal agreement between Duolingo and his users is a great innovation, launching the company. to the next level.

Of scale. This is the first time I really actually had to deal with that. I think part of the reason is because unlike my other projects most of the effort that people put in is these five second things that people don't care all that much. In this case, you know, the volunteers spend sometimes years helping us add something and so this is the first time we had to really deal with that. This clear contract. that contributors retain ownership of all content they submit.

makes them feel secure. That Duolingo is aligned. With their mission. And if you want to harness the power of your crowd, the most important thing you'll need to do is to make sure your mission and that of your contributors is totally in sync. But there are also other things you can do to keep your crowd motivated.

I want to spend the rest of this episode focusing on one way Luis does this. It's something. That duolingo is famous for. Gamification For example.

Adding elements from video games. Such as leveling up. Leaderboards. and scoring points. One of the things that I have found with Becavio Economics is

In my sense the people who know most about how to influence behavior I would say are not these people who are PhDs in behavioral economics, no offense. I would say it's people who make games. Yep. Those people are amazing. So we take from Duolingo, I mean we spend a lot of time since it's funny, but you can see a lot of the different items in Duolingo.

You can trace a lot of the things we do based on what game our product managers are currently playing. So for a while they were playing Clash Royale. which has these chests. Well, Dueling was full of chests these days. So it's this funny thing you can trace a lot of our elements just to which game people were playing. I do similar thing to uh even product managers at LinkedIn and other things like you must be playing some of these simple games to get a sense of them. Right. Because you will learn things from them. Even if you go, Well, the game's kinda boring. It's not that interesting. Like, No. Right. Pay attention to how actually even a very simple game

Actually retain some people. Because of the mechanics that they use. It's amazing. Now I'll tell you, don't get your product organization playing class royale. Got it. We have a little leaderboard. Very powerful. We have these little game dynamics.

It works a lot. Any lessons from those game dynamics? Yeah, I think Leaderboards and progress bars. People just want to fill up progress bars.

Early on we realized the hardest thing about learning a language is staying motivated. And so we rely heavily on gamification for keeping people motivated. So we do a lot of things For the first four years of Du Lingo, the only thing we did was improve user retention. And it's basically buying gamification feature. So we by now we run about two thousand A B tests a year. And most of them are on just little game mechanics.

And it it really has helped us. Another person who has thought deeply about gamification. is Elizabeth Sampat. Game designer and author of the book. Empathy engines, design games.

That are personal. Political And profound. Gamification works best when it's really tied deeply into the user's needs or desires. And when it really fails is when the gameified solutions

are actually about the needs and desires of the person making the product. as opposed to the person that's actually going to be using it. Elizabeth says Duolingo is a good example of Of gamification done right. The thing about Duolingo is

The vast majority of people don't have the goal To learn a language. They have the goal to know a language. And the part that's really painful, the part where all the friction is, is having to get to the point where you know the language. So by making that as uh distracting and sort of lighthearted.

As you can It really helps smooth out the friction towards people's goal. And that's really what game design is. It's about figuring out what the actual goal of something is and how to get there. And so much of the time, and this is true for gamification as as much as it is for games, when you have a system and

That isn't giving you the output that you want. The the real question is have I named the correct goal? What is the actual goal that we're trying to reach here? Because oftentimes you've actually created a really solid system. You've just created a system for the wrong goal.

But even thoughtful gamification. can spin out of control. Constant course correction is needed in order to keep gaming elements. Aligned with your mission. It's something.

Both Luis and I have grappled with. One of the things that we found at LinkedIn was also where game dynamics can lead you astray. So for example When we started, we wanted to create a game to get people to be more connected. So we put your number of connections in a prominent way. And so what happened is everyone started playing it just like progress bars. But then some people

Went crazy. Like Twenty three thousand, twenty five thousand. Like nobody knows twenty three thousand people. I mean literally nobody. But it's like I'm trying to win that game. And we had a protest when we later retrofitted this and said, Okay, at five hundred, it goes to five hundred plus and we're done. Uh-huh. Like we end the game there is a way that we kind of

Patch the game dynamic. Have you guys had any kind of similar experiences where the game dynamic led you astray and you needed to alter it in some way in order to have the better outcome for the ecosystem and the community and the people? Just last year, we took a concept from gaming where the idea is you actually make a leaderboard with strangers, but with strangers that are equally committed to you. And the way to implement that is this concept called leagues. You are with fifty Randos in the Bronze League. Last one week the top ten people go to the Silver League. And then if you're at the top ten there, you got to the next league, et cetera. And so

This was very powerful at Duolingo. In fact, just adding this leaderboard increased the average time spent by users on Duolingo by twenty percent. Which is a huge number given that we have so many users. But it created some people who were Basically Only motivated by this leaderboard. And they started doing things that were not good for their learning. I mean, for example, some of them discovered that you can just redo the first lesson.

a million times and it happens to be an extremely easy first lesson. And so they just redo that like crazy. We've started patching this and now, you know, if you redo Any lesson more than twice we stop giving you points. It's stuff like that. And again, people have complained about this type of stuff. We've had to patch it. And in general We wanna do what's good for the users in terms of actually learning and we don't wanna get these people always learning the word from man.

A trillion times. Yeah, the game dynamic is to help them. Yeah. Stay engaged, stay motivated, not to create a different outcome. Yeah. But these are all nudges. The overall drive for your crowd. always comes back to the wellspring of your shared mission. It was this shared mission that let Duolingo hold off from needing to generate revenue.

And focus on. Attracting mission focused creators. Because we had this mission that we didn't want to really charge for content. Our hands were a little tight on what we could make money with. I mean the standard way to make money in education is just to charge for People learning.

So we thought well the first thing we're gonna do is just gonna add up programmatic ad. at the end of a lesson. We have enough users that that was significant. I mean it's tens of millions of dollars from those little programmatic ads. Then we added a subscription that the only thing it did was turn off the ads at first. Now it does a few more things, but at first just turned off the ads. Quickly the subscription made more money than the ads. Generating revenue.

didn't disrupt the mission to bring free education. To as many people as possible. The main concern for Luis is to keep Duolingo focused on the mission. I worry about this. I wholeheartedly believe that Our business model.

It's actually Better than the Just standard charging for content. This may not be intuitive, and I understand many investors will just never agree to this, but in the education category, we are the top grossing app. I mean, we make more money than any other education app and we are like the one that doesn't charge for the actual content.

Luis has also been able to revisit his early ambition to kill the tofu. That anachronistic English test. He and millions of others have to take If they want to study in the US. We as a company have our own version of the test. So what happened is

About four years ago, we started getting emails from people saying, Hey, thank you for teaching me English. You know, I couldn't learn it before you guys, it's free, thank you. But now I have a problem. I need to certify that I know English. And we got enough of these emails that we started looking into this. Now Duolingo offers an English test. For forty-nine dollars and is accepted by Yale.

Columbia. John Hopkins and around five hundred other institutions. And the best part is you don't have to cross borders or brave war zones to take it. As it's all done.

Online. I kinda had forgotten my whole TOEFL experience, you know, when I had to take that test in Guatemala, but I quickly remembered and when we looked at this business, it's a pretty crazy business. It's about somewhere between five and ten billion dollars a year are spent by people Proving that they know English. This is another unexpected place where the crowd took Luis and Duolingo too. And this is what you want.

When you take your product To the crowd. And the power of it. They are invested. and will do their best to help you improve your product.

Feedback will not be coming from just the disgruntled few. Or if you're in dire straits. The disgruntled many. Your users will be invested in your sharion. And they will help you course correct

Well before you hit the jagged rocks. I'm Reed Hoffman. Thank you for listening. Masters of Scale is a Wait What original. The show is recorded remotely with sanitized audio gear.

Our executive producers are June Cohen and Darren Treff. Our supervising producer is Jay Punjabi. Our producers are Chris McLeod, Adam Skuse, Jenny Cataldo, Jordan McCloud. Catherine Clark Gray. Hali Bondi, Christina Gonzalez, and Ben Manila. Our editor at large is Bob Safi.

Original music and sound design by Ryan Holliday. And Daniel Nissenbaum. Gaming Music by Allison Leighton Brown. Audio editing. by Keith J Nelson.

Mixing and mastering. by Brian Pugh. Special thanks. Appreciate Elisa Shriber. David Sanford, Saida Sepieva.

Adam Heiner Emily McManus. Kelsey Capitano. Tim Cronin Sarah Sandman.

Charlie Menesses, and Colin Howard. Visit MastertheScale dot com to find the transcript. For this episode. And be sure.

To subscribe. or email newsletter.

What you see above is a preview of the first minutes. One unlock costs 3 credits and covers this episode forever: full segment and word-level timestamps on this page, plus .txt, .srt, .vtt and word-level JSON downloads, as many times as you like.