How Duolingo uses Twilio Verify WhatsApp with SMS Fallback

How Duolingo uses Twilio Verify WhatsApp with SMS Fallback
November 14, 2023
Written by
Twilion

At SIGNAL 2022, Duolingo shared how they deployed Twilio's Verify application to authenticate users and connect them with their friends. Furthering our relationship, Twilio invited Duolingo and Nico Sacher, Principal Product Manager, to run an experiment in select countries using WhatsApp as a primary verification channel with SMS as a fallback for One-time Passcodes (OTPs). This blog post will describe the results of that experiment.

Duolingo saw the following results from sending OTPs with WhatsApp and SMS as a fallback channel:

  • Improved OTP verification conversion/success rate

  • Lower OTP costs

  • No code implementation

Our research shows that the experimental WhatsApp with SMS fallback treatment performed better than the SMS-only control treatment, demonstrating a 12% improvement in verification success rate while reducing cost by 87%*. Twilio and Duolingo saw that it took fewer OTP messages to verify a phone number going through WhatsApp with SMS fallback treatment.

* Based on internal testing data. Experiment design detailed below.

Using WhatsApp as an OTP channel

What's the best channel to send One-time Passcodes to your end users? Email? SMS? An Over-the-top (OTT) messaging application like WhatsApp? The answer can sometimes be more complex, with varying customer preferences and regional nuances.

What if your OTPs for multi-factor authentication were delivered to users based on the most cost-effective channel or the channel most likely to convert? While SMS verification is commonly used for OTPs, SMS costs vary by country, and not all users have reliable cellular coverage.

WhatsApp is one of the most popular messaging apps, with over 2 billion users across 180 countries. WhatsApp usage is prevalent in some countries and can be much less expensive for businesses than sending an SMS (up to 90%). On the other hand, consumers are less accustomed to receiving OTPs from WhatsApp, and some don't have the app.

“For those in the US, it's often hard to imagine an SMS OTP wouldn't arrive immediately, but it's common in many other geographies. I've experienced the frustration of trying multiple times to get an OTP in Latin America, so it's not shocking that WhatsApp is a much better experience for many users.”

-Nico Sacheri, Duolingo Principal Product Manager

At Twilio, we wanted to explore using WhatsApp as an OTP delivery channel with SMS as a fallback to improve conversion rates and decrease customer costs.

Experiment design

During the experiment, we enabled a new Twilio Verify feature called “WhatsApp with SMS Fallback” for OTP message delivery in select countries. When Duolingo made the Verify API call to send an OTP via SMS, Verify attempted to send an OTP via WhatsApp instead.

Verify identified if the WhatsApp message attempt resulted in delivery to the end user's handset or a failure. If the WhatsApp OTP failed because the recipient didn’t have a WhatsApp account, Verify automatically sent an SMS in under 3 seconds (P99). The end-user received the same code from WhatsApp or SMS to input into Duolingo’s existing UI.

If Duolingo made another Verify API call to send an OTP via SMS to the same mobile number (end-user) in the same verification session (ten-minute window), Verify sent the OTP via SMS and did not attempt a send via WhatsApp.

“Initially, we were concerned the WhatsApp UI was not as integrated with the device OS to pull OTPs from a message. However, we quickly learned that the increased delivery rate significantly outweighed any drop due to additional friction.”

-Nico Sacheri, Duolingo Principal Product Manager
WhatsApp with SMS Fallback
WhatsApp with SMS Fallback

Experiment results

The A/B experiment ran for two weeks, where 50% of Duolingo’s verification volume used the feature, and 50% was unaffected. In that time, Duolingo saw improved OTP conversion rates and lower costs per OTP.

  • Improved verification conversion rates:
  • Out of 12 countries tested, nine countries showed higher success rates and lower costs when WhatsApp was used, compared to SMS-only
“We're excited to see how we can continue to use failover OTP channels to improve user experience and our costs.”

-Nico Sacheri, Duolingo Principal Product Manager

Conclusions

  • We saw improved OTP conversion in some markets where WhatsApp is a preferred communication channel
  • We saw decreased OTP costs by sending via WhatsApp instead of SMS in some markets
  • We recommend implementing an omnichannel fallback system

Along with Duolingo, Twilio Verify has conducted similar experiments across other customers, countries, end-user platforms, and messaging channels, powering an algorithm to predict which channel is optimal for OTP delivery. Cross-channel fallback is also useful when one channel has an outage.

Twilio Verify offers omnichannel user verification across a growing selection of channels. Stay tuned for more from the Twilio Verify team on intelligently sending OTPs to your end-users on their preferred channels.

Learn more about Twilio’s Verify WhatsApp and contact Sales to enable the feature.