[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Page MenuHomePhabricator
Create Task
Maniphest T247786

MWExceptionHandler reqId sometimes differs from php-wmerrors reqId
Closed, ResolvedPublic
Actions

Description

https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-deploy-2020.03.16/mediawiki?id=AXDky3uCKWrIH1QRCes2

timestamp: 2020-03-16T19:21:42.517620+00:00
host: mw1378
type: mediawiki
channel: exception
caught_by: /etc/php/php7-fatal-error.php via php-wmerrors
reqId:  Xm-RwwpAMNwAAlOE5gwAAABA # <-- different
message:

[Xm-RwwpAMNwAAlOE5gwAAABA] PHP Fatal error: Allowed memory size of 692060160 bytes exhausted (tried to allocate 1179648 bytes) in /srv/mediawiki/php-1.35.0-wmf.23/includes/cache/MessageCache.php on line 283

And the exact same error instance reported by the MW side:
https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-deploy-2020.03.16/mediawiki/?id=AXDky3vkh3Uj6x1zZRwi

timestamp:  2020-03-16T19:21:42+00:00
host: mw1378
type: mediawiki
channel: exception
caught_by:  mwe_handler
reqId: Xm-RwwpAMFwAALzTp1IAAACY # <-- different
unique_id: Xm-RwwpAMNwAAlOE5gwAAABA # <!-- (interesting)
message:

[Xm-RwwpAMFwAALzTp1IAAACY] /w/api.php?…   PHP Fatal Error: Allowed memory size of 692060160 bytes exhausted (tried to allocate 1179648 bytes) in /srv/mediawiki/php-1.35.0-wmf.23/includes/cache/MessageCache.php on line 283

This should be the same. Without that, the standard practice of searching by reqId is missing results.

Details

SubjectRepoBranchLines +/-
WebRequest: Avoid unused wfRandomString() callmediawiki/coremaster+8 -11
mediawiki: Fix reqId in php7-fatal-error.php to consider X-Request-Idoperations/puppetproduction+6 -3
Customize query in gerrit

Related Objects

Event Timeline

Krinkle created this task.Mar 16 2020, 7:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 16 2020, 7:31 PM
Krinkle added a project: Platform Engineering.EditedMar 16 2020, 7:50 PM
Krinkle added a subscriber: • WDoranWMF.

@WDoranWMF This may be a good issue for Clinic Duty to look into how core's MWExceptionHandler works. I covered most if it already, but this would apply it in practice. I haven't investigated yet, but am happy to help whomever does.

• Gilles moved this task from Inbox, needs triage to Radar on the Performance-Team board.Mar 16 2020, 9:26 PM
• Gilles edited projects, added Performance-Team (Radar); removed Performance-Team.
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.Mar 16 2020, 9:26 PM
Anomie added a project: SRE.Mar 17 2020, 1:44 PM
Anomie moved this task from Inbox to Triage Meeting Inbox on the Platform Engineering board.
Anomie subscribed.

Within MediaWiki, the request ID is determined inside WebRequest::getRequestId() and considers three sources (in order):

  1. If $wgAllowExternalReqID is true (as it is on Wikimedia sites) and the X-Request-Id HTTP header is present, its value is used.
  2. If $_SERVER['UNIQUE_ID'] is set (e.g. from Apache mod_unique_id), its value is used.
  3. A random value is generated on the first call.

The description above mentions /etc/php/php7-fatal-error.php; checking that on mwmaint1002, I see it only considers $_SERVER['UNIQUE_ID'] when generating "reqId" and prefixing the message. It would have to be updated to also consider X-Request-Id first before falling back to UNIQUE_ID. I note that file is in operations/puppet, where CPT in general doesn't have merge rights (so tagging SRE).

As for "unique_id" in the log data, that comes from the vendor class Monolog\Processor\WebProcessor and also reflects only $_SERVER['UNIQUE_ID'].

• AMooney assigned this task to • holger.knust.Mar 17 2020, 7:49 PM
• AMooney triaged this task as Medium priority.
• AMooney edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.
• AMooney moved this task from Inbox to Doing(WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.
gerritbot added a comment.Mar 24 2020, 1:08 AM

Change 582947 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] WebRequest: Avoid unused wfRandomString() call

https://gerrit.wikimedia.org/r/582947

gerritbot added a project: Patch-For-Review.Mar 24 2020, 1:08 AM
gerritbot added a comment.Mar 24 2020, 1:27 AM

Change 582948 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/puppet@production] mediawiki: Fix reqId in php7-fatal-error.php to consider X-Request-Id

https://gerrit.wikimedia.org/r/582948

• holger.knust moved this task from Doing(WIP:5) to Waiting for Review on the Platform Team Workboards (Clinic Duty Team) board.Mar 24 2020, 2:00 PM
gerritbot added a comment.Mar 24 2020, 2:21 PM

Change 582948 merged by Alexandros Kosiaris:
[operations/puppet@production] mediawiki: Fix reqId in php7-fatal-error.php to consider X-Request-Id

https://gerrit.wikimedia.org/r/582948

Anomie closed this task as Resolved.Mar 24 2020, 2:24 PM
Anomie reassigned this task from • holger.knust to Krinkle.
Anomie added a subscriber: • holger.knust.
• WDoranWMF moved this task from Waiting for Review to Done on the Platform Team Workboards (Clinic Duty Team) board.Mar 24 2020, 2:44 PM
gerritbot added a comment.Mar 24 2020, 9:41 PM

Change 582947 merged by jenkins-bot:
[mediawiki/core@master] WebRequest: Avoid unused wfRandomString() call

https://gerrit.wikimedia.org/r/582947

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2020, 10:11 PM
Krinkle mentioned this in T253674: WikimediaDebug "Find in XHGui" can't find profiles.May 26 2020, 6:57 PM
Krinkle mentioned this in T253675: Remove mod_unique_id from app servers.May 26 2020, 7:04 PM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits