QA Search preview instrumentation
Closed, ResolvedPublic
Actions

Description

Structured data team deployed the instrumentation to track search previews (T307572).
The instrumentation spec has been defined in this document.

Instrumentation note

The related events will be stored in mediawiki_searchpreview schema.

QA summary

Schema QA in betalabs (T321069#8462307 by @Etonkovidova)

What has been checked	Status	Note
wiki	✅ pass	recorded
action	✅ pass	see the comment for the examples of recorded events
platform	✅ pass	2 types of source are captured: mobile and desktop
result_display_position	✅ pass	fixed - go to other search page (next 20 search results for example), get `result_display_position: -1`
session_id	✅	fixed - null on mobile (it's always not null on desktop)
meta	✅ pass	recorded
dt	✅ pass	recorded

Schema QA on production

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T307038 [Placeholder EPIC] SDAW Search Improvements
Resolved	None	T306341 [EPIC] Quickview/Search Preview panel on the special:search page
Resolved	cchen	T307571 Analytics for SDAW Search Improvements
Resolved	cchen	T311166 Special:Search Quickview analytics
Resolved	cchen	T307572 [M] Instrumentation for SDAW search improvements
Resolved	cchen	T324707 QA Search preview instrumentation

Event Timeline

cchen triaged this task as Medium priority.Dec 7 2022, 8:44 PM

cchen created this task.

cchen moved this task from Triage to Upcoming Quarter on the Product-Analytics board.

CBogen added a project: SDAW-Search-Improvements (Milestone 2: QuickView MVP).Dec 8 2022, 3:20 PM

CBogen moved this task from Triage to Analytics on the Structured-Data-Backlog board.Dec 12 2022, 3:57 PM

cchen mentioned this in T321069: [L] SearchPreview instrumentation - Create a Schema.Dec 14 2022, 6:54 PM

cchen updated the task description. (Show Details)Dec 16 2022, 9:41 AM

cchen added a subscriber: Etonkovidova.

CBogen edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.Dec 16 2022, 2:29 PM

CBogen moved this task from Incoming to Needs QA on the Structured-Data-Backlog (Current Work) board.

Examples of click-article-section and click-interwiki-commons events:

$schema: /analytics/mediawiki/searchpreview/1.0.0
action: click-article-section
result_display_position: 0
wiki_id: enwiki
platform: desktop
session_id: e10ed74a0f372fad0deelbvfs3l3
meta:
  stream: mediawiki.searchpreview
  domain: en.wikipedia.beta.wmflabs.org
  id: 2f1ea285-d1d6-4d56-9a44-1a22b144b510
  dt: '2022-12-19T23:38:43.931Z'

$schema: /analytics/mediawiki/searchpreview/1.0.0
action: click-interwiki-commons
result_display_position: 0
wiki_id: enwiki
platform: desktop
session_id: e10ed74a0f372fad0deelbvfs3l3
meta:
  stream: mediawiki.searchpreview
  domain: en.wikipedia.beta.wmflabs.org
  id: ac0c8561-95f6-4d14-a12c-5abcf0de4eb4
  dt: '2022-12-19T23:39:09.457Z'

Example of Mobile session_id is recorded:

$schema: "/analytics/mediawiki/searchpreview/1.0.0"
action: "open-searchpreview"
dt: "2022-12-20T00:21:44.082Z"
meta: {stream: 'mediawiki.searchpreview', domain: 'en.wikipedia.beta.wmflabs.org'}
platform: "mobile"
result_display_position: 2
session_id: null
wiki_id: "enwiki"


$schema: "/analytics/mediawiki/searchpreview/1.0.0"
action: "click-image"
dt: "2022-12-20T00:19:37.176Z"
meta: {stream: 'mediawiki.searchpreview', domain: 'en.wikipedia.beta.wmflabs.org'}
platform: "mobile"
result_display_position: 1
session_id: null
wiki_id: "enwiki"

Etonkovidova moved this task from Needs QA to Verify on Production on the Structured-Data-Backlog (Current Work) board.Dec 20 2022, 12:26 AM

cchen moved this task from Upcoming Quarter to Kanban on the Product-Analytics board.Jan 9 2023, 6:44 AM

cchen edited projects, added Product-Analytics (Kanban); removed Product-Analytics.

cchen moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

Re-tested in betalabs the presence of all actions based on mediwiki.searchpreview schema in betalabs :

action	desktop	mobile
open-searchpreview	✅
close-searchpreview	✅
click-image	✅
click-snippet	✅
click-article-section	✅
click-interwiki-commons	✅
click-interwiki-links	❌

click-interwiki-links on Desktop was not recorded
did not see any events recorded from mobile

The data QA of the instrumentation in production done on 2023-01-31. See QA notebook for details.

What has been checked	Status	Note
date	✅ pass	events available on since 2023-01-05. Count of daily events and sessions significantly increase after new-session events added on 2023-01-26.
wiki	✅ pass	3 wikis recorded
action	✅ pass	7 types of events recorded: new-session (starting from 2023-01-26), open-searchpreview, close-searchpreview, click-image, click-snippet, click-article-section, click-interwiki-commons. `click-interwiki-links` on Desktop was not recorded (T321078)
is_anon	✅ pass	True or False or Null ( less than 0.1% after `isAnon` is added to searchsatisfaction schema on 2023-01-27)
platform	✅ pass	only on desktop in January
result_display_position	❓	-1 for new-session events; >= 0 for all the other events. Some open-searchpreview and close-searchpreview events everyday has `result_display_position = -1`

@SimoneThisDot there are some open-searchpreview and close-searchpreview events everyday have result_display_position = -1. Are these events for refreshing pages?

cchen mentioned this in T326663: [L] Update instrumentation & schema to track things for which we're currently relying on searchSatisfaction.js.Jan 31 2023, 9:43 PM

Update data QA on 2023-03-15, see QA notebook for details.

What has been checked	Status	Note
action	✅ pass	8 types of events recorded: new-session (starting from 2023-01-26), open-searchpreview, close-searchpreview, click-image, click-snippet, click-article-section, click-interwiki-commons, click-interwiki-links (starting in March)
platform	✅ pass	Desktop and mobile events recorded.

QA Search preview instrumentationClosed, ResolvedPublicActions