papers.ssrn.com is another one: (user report here). This appears to be an actual block, AFAIK. https://www.mediawiki.org/wiki/Topic:Ybg5cq7nal5z0dom

In T362379#10351488, @Robertsky wrote:

In T362379#10343746, @Mvolz wrote:

In T362379#10343726, @Mvolz wrote:

In T362379#10343714, @Robertsky wrote:

FYI, it seems that straitstimes.com may be blocked now as well.

Yup. Very clear block starting Nov 5th 2024 visible: https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-3M%2Cto%3Anow))

We're not sending them more than 30 requests/day so :/ The day they blocked us, only 3 requests!

Tested locally and it doesn't work either - might have been an infrastructure change rather than a deliberate block.

Was the user agent string changed around then as well? If so, maybe we can drop an email to them to see if they can unblock the user agent string?

Once we do the switchover,

In T362379#10343726, @Mvolz wrote:

In T362379#10343714, @Robertsky wrote:

FYI, it seems that straitstimes.com may be blocked now as well.

Yup. Very clear block starting Nov 5th 2024 visible: https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-3M%2Cto%3Anow))

We're not sending them more than 30 requests/day so :/ The day they blocked us, only 3 requests!

In T362379#10343714, @Robertsky wrote:

FYI, it seems that straitstimes.com may be blocked now as well.

Citoid now natively supports the restbase-like pattern, so this should now be unblocked. (Deployed 20-11-2024)

FYI I took this opportunity to switch from master to main for the default branch name, so that might require some local updating!

In T369547#10332988, @ppelberg wrote:

Next steps
In light of the challenges @Mvolz and @dchan identified with testing this patch on beta, we're going to deploy this change re-rest it once it's live in production.

In T95388#10333039, @ppelberg wrote:

Next step
Editing Team to make a clear choice about whether to move forward with this approach or to let it go for now.

Perhaps premature but deployed today without apparent mishap. :) If we'd like update https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/973362 but not doing any path remapping, should now be possible.

I don't know how to do this myself. I had a look at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated

In T369547#10323793, @dchan wrote:

Thanks @EAkinloose, that's really helpful. It looks like the Citoid service might be returning 404 on beta cluster. I'm seeing a 404 error from the REST API:

https://en.wikipedia.beta.wmflabs.org/api/rest_v1/data/citation/mediawiki/https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2F9%2F98%2FColoring_page_for_Wikipedia_Day_2019_in_NYC.pdf?action=query&format=json

That means we never see a 415 error, and therefore never see the more specific error text. @Mvolz is this an expected issue on beta cluster?

It's working now (though outdated.)

Example get response:

WSJ.com is one of the publishers we can't cite because of CloudFront. I'm not sure if there are others.

Resolving as deployed (Went out to group 1 and 2 Tuesday.)

Any news?

In T378461#10309264, @fnegri wrote:

• for refill I have just opened a phab ticket: T378686: refill: review citoid usage
• for citations we have the on-wiki bug report opened by @Mvolz

I think both have been resolved/addressed. @Mvolz do you think this task can be resolved?

Which graph would these data go into? The main graph, or the scholarly metadata one?

@ppelberg we only keep the last 3 months of data. For the 6 months, should I download now and wait until we collect the next 3 months worth of data and compile it, or just do 3 months.

Now we have better tooling we can examine this more in depth; NPR requests are actually timing out and responding with 504. This would be slightly weird for a deliberate block as you're more likely to get a 403 and for the response to be relatively fast where indeed these are actually timing out.

I see in the other thread you don't have Turnilo access, sorry. The user-agent is 'reFill/2 (http://en.wikipedia.org/wiki/User:Zhaofeng_Li/reFill)'

In T378686#10282814, @Curb_Safe_Charmer wrote:

There is certainly nothing in the service log that indicates that reFill was called 30,000 times in any 12 hours.

In T369547#10079818, @ppelberg wrote:

In T369547#10063836, @Mvolz wrote:

I find this message kind of confusing because the statements aren't explicitly connected.

"We couldn't make a citation for you because the resource is in a format we don't support" sounds clearer to me.

Great spot, @Mvolz . What do you think about the proposed version below?

Separately: what page do you think should be linked from the format we don't support portion of the message?

We couldn't create a citation for you.<br>
The resource is in a [[TBD | format we don't support]]. Create a citation manually below.

Should we trial this for a month or something? I don't like it for the long term but maybe worth seeing if there's an impact?