Background

As a part of our annual plan we are focused on increasing the retention of a new generation of readers on our website, allowing a new generation to build a lasting connection with Wikipedia, by exploring opportunities for readers to more easily discover and learn from content they are interested in.
We are beginning the fiscal year by experimenting with a series of experiments of browsing experiences to determine which we would like to scale for production use, and on which platform (web, apps, or both). We will then focus on scaling these experiments and testing their efficacy in increasing retention in production environments. Our goal by the end of the year is to launch at least two experiences on representative wikis and to accurately measure a 5% increase in reader retention for readers engaged in these experiences.

OKR Hypothesis

This work is apart of the 2024-2025 Annual Plan Wiki Experiences 3.1 work.

Official Hypothesis:
If we introduce a personalized rabbit hole feature in the Android app and recommend articles based on the types of topics and sections a user is interested in, we will learn if the feature is sticky enough to result in multi-day usage by 10% of users exposed to the experiment over a 20-day period, and a higher pageview rate than users do not engage with the feature.

Secondary Hypothesis:
If we compare rabbit holes that are accessible based on initiation in the search field vs. through readinglists we will see higher engagement rates and retention with content presented in the search as opposed to reading lists.

How will we know we were successful

Validation
10% of unique users that engage with experiment do so more than once in a 30 day period
5% higher internal referral clicks of recommended articles from people that engage with the experiment compared to those that did not engage with feature
5% higher clicks on suggestion based on search queries vs. click throughs to articles from suggested reading lists
5% higher clicks to view suggested reading list vs users that hit enter on search queries
65% or higher of feedback scores are positive

Guardrails
60% of feedback from in-app users are negative
10 community members from target group express negative sentiments

Curiosities
Do we see a difference in the retention rate for logged in users vs logged out users
How does the metrics from Rabbit hole compare to Recommended Content
Do we see higher pageviews for users that engage with the feature vs those that do not

User Stories

Reading List
As someone who read the Katherine Goble article and the NASA article, I want to receive a list of recommended of articles to read related to Women in Science and NASA, so that I can come back and read them later.

Search
As someone who is reading the Philanthropy section of the Beyonce article, I want to see a recommendation prepopulated in search that reads “Beygood Partnerships” and when I press search it provides highlights of the organizations BeyGood has partnered with and I can decide to read more about those partners or go to another article, so that I can decide which article I want to go deeper into.

In-App Feedback
As a tester of the rabbit hole experiments, I would like to provide feedback about recommendations without leaving the app, so that the feedback gets to the dev team and recommendations can improve over time.

Must Haves

• Run as ABC test
  • A group is the control
  • B group sees a recommended search query from article view in the search
  • C group receives dialog encouraging them to see and save their recommended reading list
• Interface should be clear that recommendations are based on user interest
• Constrained to Sub Saharan Africa and South Asia
• User able to provide feedback about the quality of recommendations in-app
• Recommendations should pull from Categories, Topics or MoreLike. When doing instrumentation, there must be a way for us to know which selection a user made was from which API.
• After 20 days the experiment should be removed from the app

Nice to Haves

• Display summaries of articles in search and reading list interface

Target Quant Regions and Languages

South Asia & Sub Saharan Africa

User Testing Languages

• English
• Hindi
• French
• Arabic

User Testing Considerations

• Impact for screenreaders
• Impact for RtL readers
• Preferences based on Age

Designs

Reading list

Rabbit holes

Qual Research Questions by Designs

Reading List

• Should reading lists be dynamic?
• If so what cadence should they update
• Do people want reminders that their reading lists have been updated?
• Do people want us to guess their reading list based on history? How does that weigh against them setting categories/topics they are interested in and getting list based on those topics?
• If they want to select what they are interested in, how should they adjust their interests?
• Should there be articles recommended to be added to their existing lists based on its current list?

Rabbit Holes

• Should the query recommendation take users straight to an article or launch search?
• How do people feel about seeing a summary with the query?
• Should the cursor be at the front or the back of the query?
• Should we combine elements of rabbit holes and recommended content?

Resources

Work in Progress Product Deck

---

Release Checklist

• Publish project page and get page translated to target languages before release @ARamadan-WMF
• Instrumentation @SNowick_WMF
• QA @ABorbaWMF
• Design Review @SChekfa-WMF
• Review of Copy @Dbrant
• Send message in Release Announcements channel that project has been released to Beta then update thread once released to Production @Dbrant

Post-Release Checklist

• Conduct analysis 20 days after release @SNowick_WMF
• Conduct usability testing based on information in Rabbit holes deck @SChekfa-WMF
• Use 20 day analysis to submit Asana report @Dbrant
• Create code cleanup task @Dbrant
• Update MediaWiki page with results of experiment after review with Haley @ARamadan-WMF
• Mark project complete on Android annual planning page @ARamadan-WMF