Add automated accessibility tests in CI to generate accessibility benchmarks for Vector
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bwang
	Feb 7 2022, 8:57 PM

Description

Background

Adding automated accessibility testing to our CI tests will allow us to quantify a11y issues over time. When used with manual testing, automated testing can help catch new issues, and be used to evaluate the impact of a11y improvements to Skins.

Proposal

Create automated accessibility tests using pa11y and axe-core. The tests will focus on common Skin UI rather than article content or user generated content, which would generate a lot of noise. The tests will also not cause CI to fail when violations are found, but will instead record data from the tests to be used in Grafana. Automated a11y tests will be run daily along with the other daily selenium tests.

Ideally as much of the accessibility testing boilerplate and config can be in core, so that setting up tests in different Skins is as easy and consistent as possible.

Proof of concept

Patch in Vector: https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/771476
Grafana: https://grafana-rw.wikimedia.org/d/9ejT-XA7k/readers-web-a11y?orgId=1&from=now-7d&to=now

Currently the POC only sends a very simple count of a11y violations & warnings to Graphite via the statsv endpoint. In the future we should be able to log more details to be viewed in Grafana. We also should migrate to Prometheus as Graphite is deprecated

Details

Subject	Repo	Branch	Lines +/-
Add selenium daily job to Vector	integration/config	master	+7 -0
a11y: create report in LOG_DIR or 'a11y/'	mediawiki/skins/Vector	master	+6 -5
a11y: create report in log folder	mediawiki/skins/Vector	master	+1 -1
Add automated a11y tests with pa11y	mediawiki/skins/Vector	master	+1 K -10
POC Add automated a11y tests to selenium with axe-core	mediawiki/skins/MinervaNeue	master	+181 -190

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T170096 Evaluate automated accessibility integration test solutions and implement them
		Resolved		bwang	T301184 Add automated accessibility tests in CI to generate accessibility benchmarks for Vector

Event Timeline

bwang created this task.Feb 7 2022, 8:57 PM

Restricted Application added subscribers: Masumrezarock100, Aklapper. · View Herald TranscriptFeb 7 2022, 8:57 PM

Change 740684 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[mediawiki/skins/MinervaNeue@master] POC Add automated a11y tests to selenium with axe-core

https://gerrit.wikimedia.org/r/740684

gerritbot added a project: Patch-For-Review.Feb 7 2022, 8:58 PM

Volker_E awarded a token.Feb 7 2022, 10:54 PM

Jdlrobson added a project: Web-Team-Backlog (Needs Prioritization (Tech)).Feb 7 2022, 11:12 PM

Jdlrobson moved this task from Backlog to Vector code cleanup on the Web-Team-Backlog (Needs Prioritization (Tech)) board.Feb 8 2022, 5:18 PM

Quiddity awarded a token.Feb 10 2022, 9:50 PM

zeljkofilipin added a project: User-zeljkofilipin.Feb 11 2022, 11:44 AM

zeljkofilipin moved this task from Backlog 🪒 to Deep work 🌊 on the User-zeljkofilipin board.

zeljkofilipin subscribed.

Patch in MinervaNeue: https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/758961

Is this the wrong link? Should it link to this? https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/740684/

zeljkofilipin moved this task from Deep work 🌊 to Waiting 🏹 on the User-zeljkofilipin board.Mar 2 2022, 3:35 PM

Change 740684 abandoned by Bernard Wang:

[mediawiki/skins/MinervaNeue@master] POC Add automated a11y tests to selenium with axe-core

Reason:

Abandoning in favor of pa11y approach
https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/771476

https://gerrit.wikimedia.org/r/740684

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2022, 5:11 PM

Change 771476 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[mediawiki/skins/Vector@master] POC Add automated a11y tests with pa11y

https://gerrit.wikimedia.org/r/771476

gerritbot added a project: Patch-For-Review.Mar 30 2022, 2:23 PM

zeljkofilipin moved this task from Waiting 🏹 to Deep work 🌊 on the User-zeljkofilipin board.Mar 30 2022, 3:07 PM

bwang updated the task description. (Show Details)Apr 4 2022, 4:54 PM

bwang updated the task description. (Show Details)Apr 4 2022, 6:23 PM

Change 771476 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Add automated a11y tests with pa11y

https://gerrit.wikimedia.org/r/771476

Maintenance_bot removed a project: Patch-For-Review.Apr 6 2022, 8:29 PM

This is currently being worked on so moving into board for tracking.

Jdlrobson triaged this task as Medium priority.Apr 6 2022, 9:44 PM

Change 777884 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[integration/config@master] Add selenium daily job to Vector

https://gerrit.wikimedia.org/r/777884

gerritbot added a project: Patch-For-Review.Apr 6 2022, 9:51 PM

Jdlrobson removed bwang as the assignee of this task.Apr 18 2022, 10:15 PM

Jdlrobson moved this task from Needs Analysis to Upcoming on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

We need help from Zeljko to merge https://gerrit.wikimedia.org/r/777884

Volker_E mentioned this in T291945: [Spike] Automated accessibility testing for Codex.Apr 25 2022, 3:00 AM

Jdlrobson moved this task from Needs triage to Technical on the Vector (legacy skin) board.Apr 25 2022, 9:38 PM

zeljkofilipin removed zeljkofilipin as the assignee of this task.Apr 26 2022, 1:38 PM

zeljkofilipin moved this task from Deep work 🌊 to Q2 👔 on the User-zeljkofilipin board.

The selenium-daily-beta-Vector job now sets WMF_JENKINS_BEACON_URL=https://meta.wikimedia.org/beacon/statsv/?

Change 777884 merged by jenkins-bot:

[integration/config@master] Add selenium daily job to Vector

https://gerrit.wikimedia.org/r/777884

Jdlrobson assigned this task to bwang.Apr 28 2022, 11:26 PM

Jdlrobson moved this task from Blocked on Others to Doing on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

zeljkofilipin moved this task from Q2 👔 to Watching 📺 on the User-zeljkofilipin board.Apr 29 2022, 12:19 PM

bwang moved this task from Doing to Ready for Signoff on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.May 2 2022, 3:24 PM

The tests are running well inside the new selenium-daily job for Vector, and results are showing in grafana. However, the reports are still not showing up in Jenkins, I'm still figuring out why the log artifacts arent showing.

bwang moved this task from Ready for Signoff to Needs More Work on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.May 2 2022, 3:54 PM

zeljkofilipin moved this task from Watching 📺 to Deep work 🌊 on the User-zeljkofilipin board.May 4 2022, 1:27 PM

From the job terminal output (selenium-daily-beta-Vector/31):

Archiving artifacts
No artifacts found that match the file pattern "log/". Configuration error?

From the job template:

    publishers:
...
     - junit:
         results: 'log/junit*.xml,log/WDIO.xunit*.xml'

Looks like the job is only archiving some xml files.

Looks like archive-log-allow-empty publisher is actually archiving the log folder.

job-templates.yaml

publishers:
 - archive-log-allow-empty

macro.yaml

- publisher:
    name: archive-log-allow-empty
    publishers:
     - archive:
        artifacts: 'log/'
        allow-empty: true

Change 789186 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/skins/Vector@master] a11y: create report in log folder

https://gerrit.wikimedia.org/r/789186

Change 789186 merged by jenkins-bot:

[mediawiki/skins/Vector@master] a11y: create report in log folder

https://gerrit.wikimedia.org/r/789186

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.12; 2022-05-16).May 4 2022, 4:01 PM

Change 789643 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/skins/Vector@master] a11y: create report in LOG_DIR or 'tests/a11y/log'

https://gerrit.wikimedia.org/r/789643

Change 789643 merged by jenkins-bot:

[mediawiki/skins/Vector@master] a11y: create report in LOG_DIR or 'a11y/'

https://gerrit.wikimedia.org/r/789643

Jdlrobson moved this task from Needs More Work to Doing on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.May 5 2022, 5:05 PM

zeljkofilipin moved this task from Deep work 🌊 to Watching 📺 on the User-zeljkofilipin board.May 6 2022, 8:47 AM

hashar unsubscribed.May 6 2022, 4:24 PM

bwang moved this task from Doing to Ready for Signoff on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.May 6 2022, 6:57 PM

zeljkofilipin mentioned this in T308446: Set up automated functional testing skeleton for Codex.May 17 2022, 2:48 PM

LGoto reassigned this task from bwang to • nray.May 23 2022, 5:20 PM

Great job @bwang !

zeljkofilipin mentioned this in T232638: Add accessibility check for Homepage .Jun 8 2022, 9:28 AM

bwang mentioned this in T319080: Add automated accessibility testing to Minerva.Sep 30 2022, 8:25 PM

KStoller-WMF mentioned this in T301790: Add an accessibility testing tool.Nov 8 2022, 4:25 PM

bwang renamed this task from Add automated accessibility tests in CI to generate accessibility benchmarks for Skins to Add automated accessibility tests in CI to generate accessibility benchmarks for Vector.Mar 9 2023, 7:47 PM

bwang added a parent task: T170096: Evaluate automated accessibility integration test solutions and implement them.

bwang updated the task description. (Show Details)

Maintenance_bot removed a project: Patch-For-Review.Mar 9 2023, 8:11 PM

Add automated accessibility tests in CI to generate accessibility benchmarks for VectorClosed, ResolvedPublicActions