Error: Typed property GrowthExperiments\NewcomerTasks\AddLink\LinkRecommendationUpdater::$linkRecommendationTaskType must not be accessed before initialization
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	Aklapper
	Oct 12 2023, 8:40 AM

Description

Error

service.version: 1.41.0-wmf.30
trace.id: f9af90088827d1f065d5bb5d
Find trace.id in Logstash

labels.normalized_message

[f9af90088827d1f065d5bb5d] [no req]   Error: Typed property GrowthExperiments\NewcomerTasks\AddLink\LinkRecommendationUpdater::$linkRecommendationTaskType must not be accessed before initialization

error.stack_trace

from /srv/mediawiki/php-1.41.0-wmf.30/extensions/GrowthExperiments/includes/NewcomerTasks/AddLink/LinkRecommendationUpdater.php(266)
#0 /srv/mediawiki/php-1.41.0-wmf.30/extensions/GrowthExperiments/includes/NewcomerTasks/AddLink/LinkRecommendationUpdater.php(158): GrowthExperiments\NewcomerTasks\AddLink\LinkRecommendationUpdater->getLinkRecommendationTaskType()
#1 /srv/mediawiki/php-1.41.0-wmf.30/extensions/GrowthExperiments/includes/NewcomerTasks/AddLink/LinkRecommendationUpdater.php(87): GrowthExperiments\NewcomerTasks\AddLink\LinkRecommendationUpdater->evaluateTitle(MediaWiki\Title\Title, MediaWiki\Revision\RevisionStoreRecord, boolean)
#2 /srv/mediawiki/php-1.41.0-wmf.30/extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php(285): GrowthExperiments\NewcomerTasks\AddLink\LinkRecommendationUpdater->processCandidate(MediaWiki\Title\Title, boolean)
#3 /srv/mediawiki/php-1.41.0-wmf.30/extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php(160): GrowthExperiments\Maintenance\RefreshLinkRecommendations->processCandidate(MediaWiki\Title\Title, boolean)
#4 /srv/mediawiki/php-1.41.0-wmf.30/maintenance/includes/MaintenanceRunner.php(703): GrowthExperiments\Maintenance\RefreshLinkRecommendations->execute()
#5 /srv/mediawiki/php-1.41.0-wmf.30/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#6 /srv/mediawiki/multiversion/MWScript.php(159): require_once(string)
#7 {main}

Impact

Notes

Details

	Subject	Repo	Branch	Lines +/-
	LinkRecommendationUpdater: Update $linkRecommendationTaskType declaration	mediawiki/extensions/GrowthExperiments	master	+1 -1
	LinkRecommendationUpdater: Update $linkRecommendationTaskType declaration	mediawiki/extensions/GrowthExperiments	wmf/1.41.0-wmf.30	+1 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	hashar	T347081 1.41.0-wmf.30 deployment blockers
		Resolved	PRODUCTION ERROR	phuedx	T348719 Error: Typed property GrowthExperiments\NewcomerTasks\AddLink\LinkRecommendationUpdater::$linkRecommendationTaskType must not be accessed before initialization

Event Timeline

Aklapper created this task.Oct 12 2023, 8:40 AM

Restricted Application added a project: Growth-Team. · View Herald TranscriptOct 12 2023, 8:40 AM

Aklapper triaged this task as Unbreak Now! priority.Oct 12 2023, 8:41 AM

hashar subscribed.Oct 12 2023, 8:58 AM

All the errors came from mwmaint2002 from the GrowthExperiments script maintenance/refreshLinkRecommendations.php.

There were some errors at 8:26 and 8:27 followed by a constant stream of errors from 8:29 to 8:34 which caused me to roll back the train. Then I am not sure whether the new code was related since apparently the exceptions stopped just when I started the rollback but it was only effective at 8:53.

GrowthExperiments_maintenance_script.png (383×796 px, 20 KB)

The periodic job from Puppet:

profile::mediawiki::periodic_job { "growthexperiments-refreshLinkRecommendations-${name}":
    command  => "/usr/local/bin/foreachwikiindblist 'growthexperiments & ${name}' extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php --verbose",
    interval => '*-*-* *:27:00',
}

There were no exceptions previously. So I guess it is indeed related to wmf.30. It does not look like anything meaningful got introduced in the extension.

We have promoted the wikis to wmf.30 at 8:28 which matches the start of the exception. I have triggered the rollback at 8:34 which for some reason matches the end of the exception stream but maybe it is unrelated. The sync of code happend between 8:49-8:50. So I don't know :(

Maybe it is because some wmf.29 cached values got retrieved by the wmf.30 code and are not compatible leading to the error?

TheresNoTime subscribed.Oct 12 2023, 9:18 AM

Aklapper added subscribers: kostajh, Urbanecm_WMF.Oct 12 2023, 9:19 AM

Change 965476 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/GrowthExperiments@master] LinkRecommendationUpdater: Update $linkRecommendationTaskType declaration

https://gerrit.wikimedia.org/r/965476

gerritbot added a project: Patch-For-Review.Oct 12 2023, 10:04 AM

Novem_Linguae subscribed.Oct 12 2023, 10:48 AM

Change 965476 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] LinkRecommendationUpdater: Update $linkRecommendationTaskType declaration

https://gerrit.wikimedia.org/r/965476

Maintenance_bot removed a project: Patch-For-Review.Oct 12 2023, 11:30 AM

FWIW, this patch is good in the sense that I agree that linkRecommendationTaskType should be nullable, but in local dev, it looks like the task type still isn't getting set correctly. I now get the exception thrown in the getLinkRecommendationTaskType helper:

'link-recommendation' is not a link recommendation task type

This may be local config; I'm double checking now.

Okay, yes, I verified that I was missing some configuration for link recommendations specifically, in my GrowthExperiments config. Locally,
docker compose exec mediawiki php extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php --verbose
is now processing topics and exiting with status 0.

Change 965217 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/GrowthExperiments@wmf/1.41.0-wmf.30] LinkRecommendationUpdater: Update $linkRecommendationTaskType declaration

https://gerrit.wikimedia.org/r/965217

gerritbot added a project: Patch-For-Review.Oct 12 2023, 12:10 PM

@kostajh tracked down the cause – the erroneous typehint was added in rEGREda4b9991cdaa: Migrate several IDatabase::select calls.

Thanks for testing the change, @jsn.sherman!

Nice thank you! I will approve and deploy the backport then I guess promote the group2 wikis again.

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.1; 2023-10-17).Oct 12 2023, 1:00 PM

Change 965217 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.41.0-wmf.30] LinkRecommendationUpdater: Update $linkRecommendationTaskType declaration

https://gerrit.wikimedia.org/r/965217

Maintenance_bot removed a project: Patch-For-Review.Oct 12 2023, 1:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-10-12T13:11:55Z] <hashar@deploy2002> Started scap: Backport for [[gerrit:965217|LinkRecommendationUpdater: Update $linkRecommendationTaskType declaration (T348719)]]

Mentioned in SAL (#wikimedia-operations) [2023-10-12T13:13:15Z] <hashar@deploy2002> phuedx and hashar: Backport for [[gerrit:965217|LinkRecommendationUpdater: Update $linkRecommendationTaskType declaration (T348719)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-10-12T13:18:47Z] <hashar@deploy2002> Finished scap: Backport for [[gerrit:965217|LinkRecommendationUpdater: Update $linkRecommendationTaskType declaration (T348719)]] (duration: 06m 51s)

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.30; 2023-10-10).Oct 12 2023, 2:00 PM

I have promoted the wikis. The cron job will starts at 14:27 UTC (23 minutes from now).

Mentioned in SAL (#wikimedia-operations) [2023-10-12T14:11:44Z] <urbanecm> mwmaint2002: stop previous instance of refreshLinkRecommendations maintenance job (T348719)

The scheduled job should have triggered at 14:27 and I don't see the error in the logs which thus looks like the issue got fixed.

Maybe the maintenance job had nothing to do for this iteration. If the issue has not been fixed, the next time it does process things that will log an error which will point us back to this task for more investigation.

Lets assume that got fixed for real (and that got tested via MediaWiki docker).

phuedx awarded a token.Oct 12 2023, 3:46 PM

	F38214959: GrowthExperiments_maintenance_script.png
	Oct 12 2023, 9:04 AM

Error: Typed property GrowthExperiments\NewcomerTasks\AddLink\LinkRecommendationUpdater::$linkRecommendationTaskType must not be accessed before initializationClosed, ResolvedPublicPRODUCTION ERRORActions