After 13:30 UTC the 19th March we have generated more than 2 million warnings: https://logstash.wikimedia.org/goto/7da1cb514026e05e2cc2abc8c630d4c6
@timestamp 2020-03-20T06:02:39 t @version 1 t _id AXD2iWDQKWrIH1QRLiGl t _index logstash-mediawiki-2020.03.20 # _score - t _type mediawiki t channel DBQuery t exception.class RuntimeException t exception.code 0 t exception.file /srv/mediawiki/php-1.35.0-wmf.24/includes/libs/rdbms/database/Database.php:2108 t exception.message t exception.trace #0 /srv/mediawiki/php-1.35.0-wmf.24/includes/libs/rdbms/database/Database.php(3144): Wikimedia\Rdbms\Database->normalizeUpsertKeys(array) #1 /srv/mediawiki/php-1.35.0-wmf.24/includes/ActorMigration.php(324): Wikimedia\Rdbms\Database->upsert(string, array, array, array, string) #2 /srv/mediawiki/php-1.35.0-wmf.24/includes/Revision/RevisionStore.php(857): ActorMigration->{closure}(integer, array) #3 /srv/mediawiki/php-1.35.0-wmf.24/includes/Revision/RevisionStore.php(575): MediaWiki\Revision\RevisionStore->insertRevisionRowOn(Wikimedia\Rdbms\DatabaseMysqli, MediaWiki\Revision\MutableRevisionRecord, Title, integer) #4 /srv/mediawiki/php-1.35.0-wmf.24/includes/Revision/RevisionStore.php(516): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer) #5 /srv/mediawiki/php-1.35.0-wmf.24/includes/libs/rdbms/database/Database.php(4179): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string) #6 /srv/mediawiki/php-1.35.0-wmf.24/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure) #7 /srv/mediawiki/php-1.35.0-wmf.24/includes/libs/rdbms/database/DBConnRef.php(641): Wikimedia\Rdbms\DBConnRef->__call(string, array) #8 /srv/mediawiki/php-1.35.0-wmf.24/includes/Revision/RevisionStore.php(518): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure) #9 /srv/mediawiki/php-1.35.0-wmf.24/includes/Storage/PageUpdater.php(996): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef) #10 /srv/mediawiki/php-1.35.0-wmf.24/includes/Storage/PageUpdater.php(766): MediaWiki\Storage\PageUpdater->doModify(CommentStoreComment, User, integer) #11 /srv/mediawiki/php-1.35.0-wmf.24/includes/page/WikiPage.php(1907): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer) #12 /srv/mediawiki/php-1.35.0-wmf.24/includes/EditPage.php(2390): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer) #13 /srv/mediawiki/php-1.35.0-wmf.24/includes/EditPage.php(1659): EditPage->internalAttemptSave(array, boolean) #14 /srv/mediawiki/php-1.35.0-wmf.24/includes/api/ApiEditPage.php(422): EditPage->attemptSave(array) #15 /srv/mediawiki/php-1.35.0-wmf.24/includes/api/ApiMain.php(1591): ApiEditPage->execute() #16 /srv/mediawiki/php-1.35.0-wmf.24/includes/api/ApiMain.php(523): ApiMain->executeAction() #17 /srv/mediawiki/php-1.35.0-wmf.24/includes/api/ApiMain.php(494): ApiMain->executeActionWithErrorHandling() #18 /srv/mediawiki/php-1.35.0-wmf.24/api.php(84): ApiMain->execute() #19 /srv/mediawiki/w/api.php(3): require(string) #20 {main} t facility user t host mw1394 t http_method POST t ip 10.64.16.152 t level WARNING t logsource mw1394 t message Wikimedia\Rdbms\Database::normalizeUpsertKeys called with deprecated parameter style: the unique key array should be a string or array of string arrays t mwversion 1.35.0-wmf.24 t normalized_message Wikimedia\Rdbms\Database::normalizeUpsertKeys called with deprecated parameter style: the unique key array should be a string or array of string arrays t phpversion 7.2.26-1+0~20191218.33+debian9~1.gbpb5a340+wmf1 t program mediawiki t referrer https://commons.wikimedia.org/wiki/Category:TEL_letter_combinations t reqId XnRcfwpAEJgAAlbtMEQAAAAA t server commons.wikimedia.org t severity warning t shard s4 t tags input-kafka-rsyslog-udp-localhost, rsyslog-udp-localhost, kafka, es, es, throttle_warning t timestamp 2020-03-20T06:02:39+00:00 t type mediawiki t unique_id XnRcfwpAEJgAAlbtMEQAAAAA t url /w/api.php t wiki commonswiki
Looks like this started right at:
13:32 hashar@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.24 (duration: 01m 07s) 13:31 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.24