Build the A/B test infrastructure for the table of contents.
Acceptance criteria
- Generalize all existing A/B test JavaScript code which is currently tied to the sticky header A/B test
- For initial page views, bucketing should be determined by page ID
- For follow up page views, all links should be intercepted to add the tableofcontents query string using the current value
- When the tableofcontents query string is present that should control the bucket used.
Developer notes
https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/759367
QA steps
The AB test has been enabled on Beta with the following bucket distribution:
unsampled (not part of experiment): 0% control (should see TOC in content): 50% treatment (should see TOC in sidebar): 50%
Testing the bucketing
- Based on the distribution above, a general sweep across the article pages (e.g. https://en.wikipedia.beta.wmflabs.org/wiki/Dog, https://en.wikipedia.beta.wmflabs.org/wiki/Cat, etc) on beta should be performed asserting that you always either see the old TOC in the content or the new TOC in the sidebar but never both. Roughly half the articles should show the new TOC and the other half should show the old TOC. Because the bucketing is done with the article id instead of the user id, the assigned bucket for a particular article should remain constant regardless of whether the user is logged in or not.
If having trouble, check that the skin-vector-toc-experiment class is on the body tag. This determines whether the AB test is enabled for that page. It's possible anonymous pages might be cached and not have the experiment enabled yet.
Testing link hijacking
Per the AC, part of this ticket involved attempting to hijack every link on the page in an effort to have the user remain in a particular bucket across pages when they click a link. This is expected to not be robust (for example, if the user clicks a link from Google there is a chance they won't remain in the same bucket), however it should mostly work when clicking links that lead to articles with a TOC. Therefore, the following steps can be performed, for example:
Link hijacking the Dog article
- Go to https://en.wikipedia.beta.wmflabs.org/wiki/Dog . Assert that the old TOC shows in the content.
- Click on the "domestic" link in the article
- Assert that a tableofcontents=0 query param is appended to the end of the url and the old TOC is shown when the page is loaded.
Link hijacking the Cat article
- Go to https://en.wikipedia.beta.wmflabs.org/wiki/Cat . Assert that the new TOC shows in the sidebar.
- Click on the "domesticated" link in the article
- Assert that a tableofcontents=1 query param is appended to the end of the url and the new TOC is shown when the page is loaded.
Testing query param override
When the AB test is enabled, using the tableofcontents query param should override the particular bucket that would have otherwise been chosen.
Query param override of Dog article
- Go to https://en.wikipedia.beta.wmflabs.org/wiki/Dog?tableofcontents=1
- Assert that the new TOC is shown in the sidebar
- Go to https://en.wikipedia.beta.wmflabs.org/wiki/Dog?tableofcontents=0
- Assert that the old TOC is shown in the content
Query param override of Cat article
- Go to https://en.wikipedia.beta.wmflabs.org/wiki/Cat?tableofcontents=1
- Assert that the new TOC is shown in the sidebar
- Go to https://en.wikipedia.beta.wmflabs.org/wiki/Cat?tableofcontents=0
- Assert that the old TOC is shown in the content
QA Results - Beta
| AC | Status | Details |
|---|---|---|
| 1 | ✅ | T302046#7981781 |
| 2 | ✅ | T302046#7981781 |
| 3 | ✅ | T302046#7981781 |
QA Results - Prod
| AC | Status | Details |
|---|---|---|
| 1 | ✅ | T302046#7981802 |
| 2 | ✅ | T302046#7981802 |
| 3 | ✅ | T302046#7981802 |