[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Page MenuHomePhabricator

1.39.0-wmf.12 deployment blockers
Closed, ResolvedPublic5 Estimated Story PointsRelease

Details

Backup Train Conductor
hashar
Release Version
1.39.0-wmf.12
Release Date
May 16 2022, 12:00 AM

2022 week 20 1.39-wmf.12 Changes wmf/1.39.0-wmf.12

This MediaWiki Train Deployment is scheduled for the week of Monday, May 16th:

Monday May 16thTuesday, May 17thWednesday, May 18thThursday, May 19thFriday
Backports only.Branch wmf.12 and deploy to Group 0 Wikis.Deploy wmf.12 to Group 1 Wikis.Deploy wmf.12 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.12 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • If you have a risky change in this week's train add a comment to this task using the Risky patch template
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.39.0-wmf.11
Next: 1.39.0-wmf.13

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptApr 1 2022, 12:52 AM
Risky Patch! ๐Ÿš‚๐Ÿ”ฅ
thcipriani triaged this task as Medium priority.
thcipriani updated Other Assignee, added: hashar.
thcipriani set the point value for this task to 5.
Risky Patch! ๐Ÿš‚๐Ÿ”ฅ
  • Change: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/767769
  • Summary: Changing connection management to the databases
  • Test plan: Errors related to database should not go up.
  • Places to monitor:
  • Revert plan: Probably rolling back train would be ok. Quick fix in cases would be to replace getConnection with getConnectionInternal and the patch better not to be reverted.
  • Affected wikis: all
  • IRC contact: Amir1, Krinkle
  • UBN Task Projects/tags: DBA Performance-Team
  • Would you like to backport this change rather than ride the train?: No because I want it to be tested in group0 first.

Change 792557 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792557

Change 792557 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792557

Mentioned in SAL (#wikimedia-operations) [2022-05-17T07:22:07Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.12 refs T305218

Mentioned in SAL (#wikimedia-operations) [2022-05-17T07:39:23Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.12 refs T305218

Mentioned in SAL (#wikimedia-operations) [2022-05-17T07:53:58Z] <jnuche@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.12 refs T305218 (duration: 14m 35s)

Change 792563 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792563

Change 792563 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792563

Mentioned in SAL (#wikimedia-operations) [2022-05-17T08:06:02Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.12 refs T305218

Change 792976 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792976

Change 792976 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792976

Change 792978 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792978

Change 792978 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792978

Mentioned in SAL (#wikimedia-operations) [2022-05-18T08:26:06Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.12 refs T305218

Mentioned in SAL (#wikimedia-operations) [2022-05-18T08:27:01Z] <jnuche@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.12 refs T305218 (duration: 00m 53s)

There is a huge spike of "MediaWiki\\Parser\\ParserObserver::notifyParse: Possibly redundant parse!","type":"mediawiki","channel":"DuplicateParse" errors in production. These are not visible on the new-errors dashboard due to the large message payload being truncated, and thus not indexed by OpenSearch, but they are instead visible on the mediawiki-errors dashboard.

The start of this correlates exactly with the 09:00 UTC rollout of group1 wikis to 1.39.0-wmf.12.

I believe this is train-blocking for reasons of exessive logging and making telemetry unstable/unusable. Secondarily, the warning itselfg may also be significant and train-blocking, depending on what investigation finds. Tertiarily, the warning should be fixed post-hoc to not exceed the allowed payload size, but that is not urgent.

Screenshot (739ร—1 px, 111 KB)
Screenshot (663ร—1 px, 97 KB)

This one was probably caused by yours truly: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/791702

I think I know what part caused it, we probably should replace it with hash

Change 793120 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] ParserObserver: Avoid pushing the whole content to the debug log

https://gerrit.wikimedia.org/r/793120

Change 792665 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.39.0-wmf.12] parser: Avoid pushing the whole content to ParserObserver debug log

https://gerrit.wikimedia.org/r/792665

Change 792665 merged by jenkins-bot:

[mediawiki/core@wmf/1.39.0-wmf.12] parser: Avoid pushing the whole content to ParserObserver debug log

https://gerrit.wikimedia.org/r/792665

Mentioned in SAL (#wikimedia-operations) [2022-05-18T22:30:03Z] <ladsgroup@deploy1002> Synchronized php-1.39.0-wmf.12/includes/parser/ParserObserver.php: Backport: [[gerrit:792665|parser: Avoid pushing the whole content to ParserObserver debug log (T305218)]] (duration: 00m 52s)

Change 793120 merged by jenkins-bot:

[mediawiki/core@master] parser: Avoid pushing the whole content to ParserObserver debug log

https://gerrit.wikimedia.org/r/793120

The json truncated thing is fixed now:

image.png (380ร—979 px, 42 KB)

Change 793476 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/793476

Change 793476 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/793476

Mentioned in SAL (#wikimedia-operations) [2022-05-19T13:52:49Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.12 refs T305218