We’re always striving to improve our product and ensure a world-class learning experience, and one of the primary ways we do so is by being Duolingo learners ourselves and using our product as much as possible. In this blog post, we’ll dive deeper into how we do this through a process called dogfooding.
What is dogfooding?
No, we’re not literally talking about the kind that you feed your dog, although the term “dogfooding” comes from a Kal Kan executive who actually ate the brand’s dog food in a meeting, proving he was a consumer of the product, not just selling it!
At Duolingo, we are all learners, too, and spend hours in the app to test features, find bugs, and ensure the user experience is the best it can be.
How internal dogfooding works at Duolingo
Duolingo employees are encouraged to download the latest internal version of the app, whether on Android, iOS, or Web clients—or all three!—and use it regularly. This internal app runs the most recent app build, and through it we can test new features and app improvements before they reach users.
At Duolingo, everyone is encouraged to dogfood, not just those building user-facing features. Right now, more than 70% of the company are dogfooders, including our CEO, who dogfoods our app every day across multiple courses on multiple device types 😮
In addition to creating a culture of dogfooding at all levels, Duolingo also hosts a twice-yearly Language Challenge which provides a financial incentive for employees to dogfood language courses consistently during the 6-month period. Our coworkers offer lots of thoughtful feedback while completing their lessons!
Dogfooding data
As soon as a new app build is available, Duolingo employees are encouraged to download it and begin testing—this starts the collection of telemetry from the new build: performance indicators like App Not Responding (ANR) events, app crashes and Out of Memory (OOM) events, and frame rates. We also leverage Google Firebase Crashlytics for segmenting app crash data across clients and app versions.
In fact, we get so much data from dogfooding builds alone that we built tooling to glean early insights from telemetry and scale our engineering operations. For app performance indicators, we created a Release Dashboard that displays telemetry for new builds as soon as they enter dogfooding—we’re usually able to get a reliable signal for each within a few hours.
This allows engineering teams to monitor performance of upcoming releases and, if metrics are elevated, to investigate and further understand where regressions may have been introduced.
We also built an internal tool called Jeeves, which collects user feedback from dogfooding bug reports and external feedback—this includes inputs like our beta program, Reddit, Twitter, Google Play and the App Store, and Zendesk. Jeeves uses AI to aggregate this data into “spikes,” or trending words and phrases.
Our QA and engineering teams can filter for spikes generated from dogfooding reports, which allows teams to identify, triage, and fix issues that might have impacted the user experience before they are ever seen by learners.
Squash bugs early and often
Internal teams get immense value from bugs found during dogfooding. To make it easier to file and investigate bugs, we created an internal tool called Shake-to-Report, which allows employees to quickly report bugs so engineering teams can start addressing them before they reach users.
Shake-to-Report works pretty much exactly how it sounds: if an employee encounters a bug while using the app, they can shake the device (or on Web clients, click a button), which pulls up an internal bug report form.
To help diagnose bugs, reports automatically contain a screenshot and a host of metadata, including the experiments in which the user is currently treated, the user’s device and app version, and the user’s specific course and lesson data. We include a unique Fullstory recording in every bug report that shows what a user was doing in the app before they experienced a bug. Shake-to-Report bugs also contain a log file, which is particularly useful as engineering teams often add log statements at different points to help debug issues.
For example, in advance of our launch of the Math and Music courses in the iOS app, Duolingo employees dogfooded an internal build of these new courses for months! A few weeks before the official launch, the team behind the Music course noticed persistent dogfooding bug reports about crashes in the Music course, coinciding with elevated crash rates in Crashlytics.
To diagnose the crashes, the team added additional statements logging users’ Music course information, song data, and course interactions. As dogfooding bug reports continued to come in, the team analyzed the more detailed logs and identified the root case: an edge case resulting from pausing songs in a specific pattern. With only a few weeks to go before the launch of the Math and Music courses on iOS, they implemented a fix, thanks to logging additional data in dogfooding bug logs!
The Release Dashboard, Jeeves, and Shake-to-Report are maintained by Duolingo’s Test and Release Infrastructure team, who work closely with client engineers and QA to ensure a seamless app release process. On Monday mornings, before rolling out the newest public app version, our internal QA team checks the bugs that have been reported from dogfooding over the weekend. We don’t begin app rollout unless there are no bugs that are blocking, or that significantly impact, the user experience. (Non-blocking bugs are triaged by the QA team.)
By creating a culture of dogfooding company-wide, we are able to maintain a constant stream of data that helps us maintain app quality and continually improve our app experience!
Want to join a culture of dogfooding and maintain app quality for users worldwide? We're hiring!