[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Page MenuHomePhabricator
Create Task
Maniphest T220479

Improve rollback confirmation selenium tests
Closed, InvalidPublic1 Estimated Story Points
Actions

Description

Right now, the tests related to the rollback confirmation prompt (specifically tests/selenium/specs/rollback.js) use browser.pause() to solve issues with delays in JavaScript loading and elements appearing after animations have been executed.

In order to solve the issues with the delayed loading of JavaScript modules, we should implement something similar to what has been done in the Cite extension to ensure that mediawiki.api is always loaded before we try to make API calls:

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cite/+/502212/

Additionally, instead of using pause to wait for the buttons to become clickable, we should use waitForVisible() to ensure the elements are there without relying on arbitrary time constraints.

Details

SubjectRepoBranchLines +/-
selenium: Skip another flaky Rollback testmediawiki/coremaster+1 -1
selenium: Asynchronous methods should be returned to browser.call()mediawiki/coremaster+1 -1
selenium: Remove LoginPage.waitForScriptsToBeReady() methodmediawiki/coremaster+3 -8
Temporarily skip rollback testmediawiki/corewmf/1.34.0-wmf.3+1 -1
Temporarily skip rollback testmediawiki/corewmf/1.34.0-wmf.4+1 -1
Temporarily skip rollback testmediawiki/coremaster+1 -1
Improve rollback tests setup by extracting repeating logic to HistoryPage objectmediawiki/coremaster+15 -23
Replace pauses in rollback tests w/ dynamic waitsmediawiki/coremaster+29 -8
Customize query in gerrit

Related Objects

Event Timeline

• Tim_WMDE created this task.Apr 9 2019, 9:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 9 2019, 9:15 AM
• Tim_WMDE added a project: WMDE-Fundraising-Tech.Apr 9 2019, 9:36 AM
gerritbot added a comment.Apr 15 2019, 10:24 AM

Change 503951 had a related patch set uploaded (by Tim Eulitz; owner: Tim Eulitz):
[mediawiki/core@master] Replace pauses in rollback tests w/ dynamic waits

https://gerrit.wikimedia.org/r/503951

gerritbot added a project: Patch-For-Review.Apr 15 2019, 10:24 AM
• Tim_WMDE set the point value for this task to 2.Apr 15 2019, 10:25 AM
• Tim_WMDE added a project: WMDE-FUN-Sprint-2019-04-01.
• Tim_WMDE moved this task from Todo to Review on the WMDE-FUN-Sprint-2019-04-01 board.
kai.nissen removed a project: WMDE-Fundraising-Tech.Apr 15 2019, 11:12 AM
kai.nissen added a project: WMDE-FUN-Sprint-2019-04-15.Apr 15 2019, 1:35 PM
kai.nissen changed the point value for this task from 2 to 0.
kai.nissen moved this task from Todo to Review on the WMDE-FUN-Sprint-2019-04-15 board.
• Tim_WMDE changed the point value for this task from 0 to 1.Apr 15 2019, 3:25 PM
• Tonina_Zhelyazkova_WMDE claimed this task.Apr 26 2019, 8:26 AM
• Tonina_Zhelyazkova_WMDE moved this task from Review to Doing on the WMDE-FUN-Sprint-2019-04-15 board.
gerritbot added a comment.Apr 26 2019, 10:06 AM

Change 506623 had a related patch set uploaded (by Tonina Zhelyazkova; owner: Tonina Zhelyazkova):
[mediawiki/core@master] Improve rollback tests setup by extracting repeating logic to HistoryPage object

https://gerrit.wikimedia.org/r/506623

• Tonina_Zhelyazkova_WMDE removed • Tonina_Zhelyazkova_WMDE as the assignee of this task.Apr 26 2019, 12:36 PM
• Tonina_Zhelyazkova_WMDE moved this task from Doing to Review on the WMDE-FUN-Sprint-2019-04-15 board.
• Tonina_Zhelyazkova_WMDE subscribed.
kai.nissen added a project: WMDE-FUN-Sprint-2019-04-29.Apr 29 2019, 10:04 AM
kai.nissen moved this task from Todo to Review on the WMDE-FUN-Sprint-2019-04-29 board.
• Tim_WMDE moved this task from Review to Done on the WMDE-FUN-Sprint-2019-04-29 board.Apr 29 2019, 1:36 PM
• Tonina_Zhelyazkova_WMDE moved this task from Done to Doing on the WMDE-FUN-Sprint-2019-04-29 board.Apr 29 2019, 2:42 PM
• Tim_WMDE moved this task from Doing to Review on the WMDE-FUN-Sprint-2019-04-29 board.Apr 30 2019, 9:05 AM
gerritbot added a comment.Apr 30 2019, 2:03 PM

Change 503951 merged by jenkins-bot:
[mediawiki/core@master] Replace pauses in rollback tests w/ dynamic waits

https://gerrit.wikimedia.org/r/503951

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.3; 2019-04-30).Apr 30 2019, 3:00 PM
• Tonina_Zhelyazkova_WMDE moved this task from Review to Done on the WMDE-FUN-Sprint-2019-04-29 board.Apr 30 2019, 3:19 PM
gerritbot added a comment.May 4 2019, 5:07 PM

Change 508090 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Remove LoginPage.waitForScriptsToBeReady() method

https://gerrit.wikimedia.org/r/508090

Krinkle subscribed.May 4 2019, 5:08 PM

Please convert to mw.loader.using() so that it doesn't become someone else's problem when the tests fail again if mediawiki.api isn't used by the log in page for some reason. This is a test for the rollback feature, not for the login page. And neither of these enqueues mediawiki.api. It is needed for an inline snippet, so should be ensured by that inline snippet.

WMDE-Fisch subscribed.May 6 2019, 9:01 AM
In T220479#5157518, @Krinkle wrote:

Please convert to mw.loader.using() so that it doesn't become someone else's problem when the tests fail again if mediawiki.api isn't used by the log in page for some reason. This is a test for the rollback feature, not for the login page. And neither of these enqueues mediawiki.api. It is needed for an inline snippet, so should be ensured by that inline snippet.

Hey, you're right this is something we did not consider yet ( and I guess it's also where we were talking about different things but thought we meant the same ^^' ). So I had a quick talk with the team. In the end we need an approach that does a bit both.

  • We have to use mw.loader.using() to make sure the modules are not only there but also will be loaded in the first place.
  • We also have to use some wait step to make sure that the MW JS core lib is loaded so that even can use mw.loader

We will change the waitForScripts method so that it does both. And this method can then live as a general "wait for and load a module" method in the Utils.

Thanks again for all the time spend in reviewing this. I hope we found a good ending now.

gerritbot added a comment.May 6 2019, 9:09 AM

Change 506623 merged by jenkins-bot:
[mediawiki/core@master] Improve rollback tests setup by extracting repeating logic to HistoryPage object

https://gerrit.wikimedia.org/r/506623

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.4; 2019-05-07); removed MW-1.34-notes (1.34.0-wmf.3; 2019-04-30).May 6 2019, 10:00 AM
gerritbot added a comment.May 7 2019, 8:50 PM

Change 508714 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/core@master] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508714

gerritbot added a comment.May 7 2019, 9:50 PM

Change 508714 merged by jenkins-bot:
[mediawiki/core@master] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508714

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.5; 2019-05-14); removed MW-1.34-notes (1.34.0-wmf.4; 2019-05-07).May 7 2019, 10:00 PM
gerritbot added a comment.May 8 2019, 3:39 PM

Change 508849 had a related patch set uploaded (by Krinkle; owner: Jdlrobson):
[mediawiki/core@wmf/1.34.0-wmf.4] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508849

gerritbot added a comment.May 8 2019, 3:39 PM

Change 508850 had a related patch set uploaded (by Krinkle; owner: Jdlrobson):
[mediawiki/core@wmf/1.34.0-wmf.3] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508850

Krinkle added a project: Wikimedia-production-error (ARCHIVED -- Shared Build Failure).May 8 2019, 3:40 PM
Krinkle moved this task from Untriaged to Resolved on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.
gerritbot added a comment.May 8 2019, 3:58 PM

Change 508849 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.4] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508849

gerritbot added a comment.May 8 2019, 3:59 PM

Change 508850 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.3] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508850

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.4; 2019-05-07); removed MW-1.34-notes (1.34.0-wmf.5; 2019-05-14).May 8 2019, 4:00 PM
gerritbot added a comment.May 8 2019, 9:11 PM

Change 508938 had a related patch set uploaded (by Krinkle; owner: Jdlrobson):
[mediawiki/core@master] selenium: Asynchronous methods should be returned to browser.call()

https://gerrit.wikimedia.org/r/508938

gerritbot added a comment.May 9 2019, 12:33 PM

Change 508090 abandoned by Krinkle:
selenium: Remove LoginPage.waitForScriptsToBeReady() method

Reason:
Superseded by rMW3990e0bbc709: Improve rollback tests setup by extracting repeating logic to HistoryPage object, which already fixed the issue of the extra method that was added in the public wdio-mediawiki package.

https://gerrit.wikimedia.org/r/508090

gerritbot added a comment.May 9 2019, 12:42 PM

Change 508938 merged by jenkins-bot:
[mediawiki/core@master] selenium: Asynchronous methods should be returned to browser.call()

https://gerrit.wikimedia.org/r/508938

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.5; 2019-05-14); removed MW-1.34-notes (1.34.0-wmf.4; 2019-05-07).May 9 2019, 1:00 PM
xSavitar subscribed.May 11 2019, 10:43 AM

Got another instance:

11:09:48 1 failing
11:09:48 
11:09:48 1) Rollback with confirmation should offer a way to cancel rollbacks:
11:09:48 element (".mw-rollback-link .jquery-confirmable-button-no") still not visible after 5000ms
11:09:48 running chrome
11:09:48 Error: element (".mw-rollback-link .jquery-confirmable-button-no") still not visible after 5000ms
11:09:48     at Context.<anonymous> (tests/selenium/specs/rollback.js:47:37)
11:09:48     at Promise.F (node_modules/core-js/library/modules/_export.js:36:28)
11:09:48     at elements(".mw-rollback-link .jquery-confirmable-button-no") - isVisible.js:54:17
11:09:48     at isVisible(".mw-rollback-link .jquery-confirmable-button-no") - waitForVisible.js:73:22

Patch: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/505282

Reedy removed a project: Patch-For-Review.May 17 2019, 7:44 PM
awight mentioned this in T225163: Spurious timeout in MediaWiki's rollback browser test.Jun 6 2019, 10:54 AM
Krinkle added a comment.Jun 13 2019, 5:20 PM

And again at https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/516308/

[chrome #0-2]   Rollback with confirmation
[chrome #0-2]       ✓ should offer rollback options for admin users
[chrome #0-2]       - should offer a way to cancel rollbacks
[chrome #0-2]       - should perform rollbacks after confirming intention
[chrome #0-2]       1) should verify rollbacks via GET requests are confirmed on a follow-up page
[chrome #0-2]
[chrome #0-2]   Rollback without confirmation
[chrome #0-2]       ✓ should perform rollback via POST request without asking the user to confirm
[chrome #0-2]       ✓ should perform rollback via GET request without asking the user to confirm
[chrome #0-2]
[chrome #0-2]
[chrome #0-2] 3 passing (22s)
[chrome #0-2] 2 pending
[chrome #0-2] 1 failing
[chrome #0-2]
[chrome #0-2] 1) Rollback with confirmation should verify rollbacks via GET requests are confirmed on a follow-up page:
[chrome #0-2] Expected rollback page to appear.
[chrome #0-2] Error: Expected rollback page to appear.
[chrome #0-2]     at getText("#firstHeading") - at elements("#firstHeading") - getText.js:18:17
[chrome #0-2]
gerritbot added a comment.Jun 13 2019, 5:21 PM

Change 516810 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Skip another flaky Rollback test

https://gerrit.wikimedia.org/r/516810

gerritbot added a project: Patch-For-Review.Jun 13 2019, 5:21 PM
gerritbot added a comment.Jun 13 2019, 5:44 PM

Change 516810 merged by jenkins-bot:
[mediawiki/core@master] selenium: Skip another flaky Rollback test

https://gerrit.wikimedia.org/r/516810

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.10; 2019-06-18); removed MW-1.34-notes (1.34.0-wmf.5; 2019-05-14).Jun 13 2019, 6:00 PM
Krinkle added a comment.EditedJun 17 2019, 12:53 AM

I guess it's still not well-enough disabled. Still failing builds at gerrit 517348

 1) Rollback with confirmation should offer rollback options for admin users:
An element could not be located on the page using the given search parameters (".mw-rollback-link .jquery-confirmable-text").
running chrome
Error: An element could not be located on the page using the given search parameters (".mw-rollback-link .jquery-confirmable-text").
    at getText() - index.js:312:3

.. and again at gerrit 512067

1) Rollback without confirmation should perform rollback via GET request without asking the user to confirm:
Expected rollback page to appear.
running chrome
Error: Expected rollback page to appear.
    at elementIdText("0.007272742124479814-1") - getText.js:35:50
Krinkle moved this task from Resolved to Untriaged on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.Jun 17 2019, 12:53 AM
Krinkle moved this task from Untriaged to Resolved on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.Jun 20 2019, 4:30 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 20 2019, 5:10 PM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM
Krinkle added a comment.Oct 8 2019, 8:45 PM

See also T226144: Fix flaky Selenium test: Rollback without confirmation should perform rollback via GET request without asking the user to confirm

Jdforrester-WMF changed the subtype of this task from "Production Error" to "Task".Oct 8 2019, 8:46 PM
Krinkle edited projects, added ci-test-error (WMF-deployed Build Failure); removed Wikimedia-production-error (ARCHIVED -- Shared Build Failure).Apr 18 2020, 10:53 PM
Krinkle removed projects: ci-test-error (WMF-deployed Build Failure), MW-1.34-notes (1.34.0-wmf.10; 2019-06-18).Apr 19 2020, 5:37 PM
thiemowmde added projects: WMDE-TechWish-Maintenance, Browser-Tests.Jan 10 2022, 5:25 PM
WMDE-Fisch added a comment.Jan 26 2022, 2:09 PM

Currently browser tests for rollback confirmation got disabled and removed due to flackyness. It might make sense to re-add at least one browser tests for that functionality in the future. This currently de-prioritized by WMDE.

See T300141: Add basic browser test for rollback confirmation

WMDE-Fisch closed this task as Invalid.Jan 26 2022, 2:12 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits