If I give a bit more context about this request, I would invite you to read https://phabricator.wikimedia.org/T349972. In this ticket, the WMF has implemented in the mobile-html output, the original thumbnailing width for use/Kiwix, so we are in position to render the real thumbnail size as given by the wikitext. See https://github.com/openzim/mwoffliner/issues/1925 for the upstream (MWoffliner) ticket.

@abi_ Thank you! Seems to work fine!

Error is in the config file: git@git@github.com:kiwix/apple.git, this is not a valid address.

@abi_ I'm a bit puzzled by the situation. Translatewiki bot has access to the whole Kiwix organisation on Github and have good permissions for the kiwix/apple

In T351949#9370182, @abi_ wrote:

@Kelson Our current implementation of Apple file format does not support multiline strings such as these

"loc-external-alert" = "An external link is tapped. \
However, your current setting does not allow it to be loaded.";
"loc-article-cell-template" = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, \
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.";
"loc-ZimFileDetail-Alert-Delete" = "The zim file and all bookmarked articles \
linked to this zim file will be deleted.";

Would it be easy to avoid using of these literal new lines?

In T351949#9367633, @abi_ wrote:

In T351949#9359466, @Kelson wrote:

3. Once the project is added on translatewiki, please add message documentation for the project.

I have created the ticket at https://github.com/kiwix/apple/issues/561. Do we have an example of macOS/iOS software with that?

See: https://github.com/wikimedia/wikipedia-ios/tree/main/Wikipedia/Localizations/qqq.lproj

Currently blocked on this due to: T352300: Newlines in "Apple" format not supported

@abi_ We are working on it. See https://github.com/kiwix/apple/issues/571. I will come back to you within a few days when this will be done.

In T351949#9359151, @abi_ wrote:

1. Do you want translatewiki to commit to feature branch and then create a PR to main branch?

@matmarex Thank you very much for this fix!

Yes, this is on the radar, but we face pretty much difficulties around mobile-html new end-point for the moment. So not a TOP prio our side. I honestly don't think we will be able to tackle this before 2024.

No sure what is the exact status here, but on the size of Kiwix we have implemented the support of the new mobile-html REST end-point. Unfortunately we are not really able to create ZIM files because of https://phabricator.wikimedia.org/T349972. We still rely therefore on the mobile-section end-point.

Actually this ticket should be a bug, image resolutions are far too big for a mobile end-point and seems to be a waste of bandwidth. I have open a ticket at https://phabricator.wikimedia.org/T349972

@MatthewVernon That would be welcome on MWoffliner side! For the rest I can not say, but respecting the RFC sounds here quite important.

@ssastry AFAIK what is described seemscorrect to me. @vadim-kovalenko and @cscott would certainly be able to tell more as they both know MWoffliner and the Wikimedia backend infra.

@MSantos Thank you very much

@akosiaris @MSantos May I underline Vadim's request: carifying if we (at Kiwix) can still benefit from the mobile-sections API until we finish with the MWoffliner code update is an important and relatively time-dependant question.

@daniel Thank you! Things are clearer now for me. At the core of my problem is that we have always been using $wgContentNamespaces to know which namespaces should be scraped... and here it does not work that well (anymore).

@Arlolra Thank you for your technical explanation but I have a hard time to use it to answer my questions which are:

• Is the "Story" namespace really only a JSON, so the JsonContentHandler is appropriate?
• If this is JSON, is that really appropriate to have this namespace in https://www.mediawiki.org/wiki/Manual:$wgContentNamespaces?
• Is that normal ("it's a feature, not a bug") that https://id.wikipedia.org/api/rest_v1/page/html/Story%3AWisata_Wajib_di_Toraja returns an error and not the corresponding HTML (like for end-users)?
• If this is normal/wanted that no HTML is returned, then is that appropriate to have a link pointing to https://phabricator.wikimedia.org/T324711 (seems a bit unclear to me).

@cscott @ssastry Really sure this is fixed? Assuming this has been deployed, this is still failing https://id.wikipedia.org/api/rest_v1/page/html/Story%3AWisata_Wajib_di_Toraja. Actually this bug seems to be the reason why we can not generated ZIM offline snapshot of id.wikipedia.org since two quarters. More details about the bug at MWoffliner level at https://github.com/openzim/mwoffliner/issues/1853.

@Ladsgroup Thank you!

@Dzahn I reopen the task because it seems people can still post and subscribe to the mailing list. This should not be the case (and as moderator I would like to not get notices for this).

@abi_ Thank you for confirming, we will work on this in the next days. Thank you in advance for your patience.

@abi_ Kind of suspect that the strings should be stored in multiple files (like one per language)

@RBrounley_WMF Thank you for opening this ticket, I will follow up.

@abi_ THX!!!

@abi_ I confirm that the github "translatewiki" has write access to the repository and I have just created the qqq.jsonfile, see https://github.com/openzim/mwoffliner/pull/1771.

We are cursed...

@Rgaudin Do you have any hint about this last comment from @Hokwelum?

Just FYI, no need anymore on MWoffliner side as we use an online object cache meanwhile.

This ticket is really old. We have made many improvement in our toolchain software and infrastructure (many backoff strategies) in the meantime. I can not assess if the problem is still there or if it has been fixed; but whatever the current status, we don't suffer of it anymore. I would be fine to close it.

@abi_ Thank you for the PR, currently under review!

At MWoffliner we have started to study the problem. Because of this ticket but as well because our unstable experience with the usage of this end-point. See https://github.com/openzim/mwoffliner/issues/1730. Hopefuly Wikimedia does provide other better end-points allowing us to retrieve the same information.

@abi_ Thank you! I will create a qqq.json and remove the metadata. When can we expect a first PR from Translatewiki to check the process works fine?

I guess this ticket can be closed?

In T324866#8459496, @Ladsgroup wrote:

I can think of several (I don't know the details of your system and might have missed something):

• Use Wikimedia Enterprise (@RBrounley_WMF in this ticket is PM of WME), I think your case can be easily justified for a free tier: https://dumps.wikimedia.org/other/enterprise_html/

@Ladsgroup The MWoffliner scraper has already been quite optimised over years. I have no obvious improvement in mind for the moment but we will consider any concrete new proposal.

Would that https://github.com/openzim/mwoffliner/issues/1664 fix the issue, so far we are really not familiar with this new API end-point?

On my end, this looks better now. Thx!

The reported bug seems indeed to have "vanished". Thank you for the good work.

If you can delete the whole thread https://lists.wikimedia.org/hyperkitty/list/wikifr-l@lists.wikimedia.org/thread/J2FU23D4C5ERWIK2LWBYUBTYK3O6KP6Y/, then problem would be solved. It's a pity we can not hide it. I don't like that we manipulate the archive.

@Ladsgroup Thank you but I don't see these two options below the "Add to favorites"... although I'm owner on the list.

See also https://github.com/openzim/mwoffliner/issues/1482

@Dzahn Great, thank you very much!

I have achieved to create account an retrieve admin access to the list, but really no clue how to remove the few messages (from the public archive). Not even sure this is possible at all. As a consequence, I have put the whole ml archive in private. Unfortunately, this seems to fail, archive is still accessible at https://lists.wikimedia.org/hyperkitty/list/wikifr-l@lists.wikimedia.org/latest

In T260223#8311261, @Hokwelum wrote:

In T260223#8302299, @Kelson wrote:

During September we have detected that one of the mirror was full and therefore rsync client was keeping download and redownloading things. As a consequence a significant part of the bandwidth and the rsync daemon slots were always busy. We still lack of monitoring solution for this on our side honestly. We should figure a tool and a process to better monitor the server/service. If you have propositions, they are welcome.

Hello Kelson, We would like to know if the space problem has been resolved as we received the same error message today.

@nskaggs Not sure this is a question to me, but in the case it needed, could you please change the ZIM mirroring rsync command to:

• request on master.download.kiwix.org in place of download.kiwix.org
• sync the following repositories in addition to wikipedia: wikibooks, wikinews, wikiquote, wikisource, wikiversity, wiktionary, wikivoyage

It seem that this bug partly explains why ZIM files of Wiktionary in English are crappy, see https://github.com/openzim/mwoffliner/pull/1665

It seems the bug reported in T274359 is the same as this one, so reputting here the bug description:

@matmarex Strange to close this ticket considering the bug described in the ticket is not fixed!

During September we have detected that one of the mirror was full and therefore rsync client was keeping download and redownloading things. As a consequence a significant part of the bandwidth and the rsync daemon slots were always busy. We still lack of monitoring solution for this on our side honestly. We should figure a tool and a process to better monitor the server/service. If you have propositions, they are welcome.

@komla What would be the way to proceed:

• Can you delete everything except the DB (no sure what is the same BTW)?
• Should we somehow migrate the DB first?