In T304548#7956082, @MGerlach wrote:

In T304548#7953084, @Trizek-WMF wrote:

I'm keeping the two wikis in my freezer.

@MGerlach, when could we start checking on the models?

Anytime. The models should be available for inspection locally on one of the stat-machines. @kevinbazira ran the training pipeline for these models and could probably share where the files for the models are located.
Maybe it would be worth to open a separate task for this work to be discussed in more detail? I think this could involve substantial amount of effort to find the error and try to fix it since we dont yet know what exactly we are looking for.

Thank you for opening the new task.
How should we proceed now (consider that I have not idea of what's going on)?

Hi @MGerlach thanks for the potential starting points on how to manually inspect the model.
Which person (or team) should take on this task?

Hi @KStoller-WMF thanks for this!

I have a question here.

This type of manual inspection can be time-consuming, and the ownership of model improvement tasks such as this one is under discussion. For this specific case, I wanted to ask: is this high priority on your end? Would you prefer to have these model issues fixed, or is it a possibility to leave the jawiki model (and others with same issues) undeployed for now, and focus our efforts on deploying to other wikis where the current pipeline works?

Thanks!

@Miriam I'll check with others on the team to be sure they agree, but if this will be a large effort, then it likely makes sense to move forward with the wikis where the current pipeline works first. Thanks for the reply!

Thanks @KStoller-WMF !

Comment on jawiki: I believe the problem comes from the tokenization to generate n-grams which we use as anchor-candidates. Our current approach (here) uses NLTK's standard tokenizer. Among other things, it relies on whitespaces to identify boundaries of words. However, as I understand this does not work in Japanese as it is written (mostly) without whitespaces (see here or here). Thus, we need to improve the tokenizer for Japanese (and potentially other languages with similar properties too). A starting point might be this blogpost which gives an overview on different techniques for tokenization.

Let's focus on all the wikis at once. It is much more reasonable than doing it as they come.

@Miriam We don't think we have the resources to do this. Lets chat