Out of the ~1600 (cluster, wiki) pairs run so far we have 48 that failed the count check after reindexing.
Example failure from mediawikiwiki.codfw.reindex.log:
general index...
...
Setting index identifier...mediawikiwiki_general_1583750799
...
Task: efGikGagRvaI7V5vLMKdBQ:33189921 Search Retries: 0 Bulk Retries: 0 Indexed: 708700 / 739104
Task: efGikGagRvaI7V5vLMKdBQ:33189921 Search Retries: 0 Bulk Retries: 0 Indexed: 718900 / 739104
Task: efGikGagRvaI7V5vLMKdBQ:33189921 Search Retries: 0 Bulk Retries: 0 Indexed: 727900 / 739104
Task: efGikGagRvaI7V5vLMKdBQ:33189921 Search Retries: 0 Bulk Retries: 0 Indexed: 734000 / 739104
Task: efGikGagRvaI7V5vLMKdBQ:33189921 Search Retries: 0 Bulk Retries: 0 Indexed: 739104 / 739104
Verifying counts...Not close enough! old=739107 new=495600 difference=0.32946109291348
Failed to load index - counts not close enough. old=739107 new=495600 difference=0.32946109291348. Check for warnings above.Looking at the index afterwards though, the counts seem reasonable:
ebernhardson@mwmaint1002:/home/mstyles/cirrus_log$ curl -s https://search.svc.codfw.wmnet:9243/_cat/indices | grep mediawikiwiki_general
green open mediawikiwiki_general_1564414369 fkIH5anxRAWcrTX3y9IEbQ 1 2 739612 79734 6.4gb 2gb
green open mediawikiwiki_general_1583750799 pq77hdudTlygUBHAPawZQg 1 2 739104 0 4.3gb 1.4gb
ebernhardson@mwmaint1002:/home/mstyles/cirrus_log$ curl https://search.svc.codfw.wmnet:9243/mediawikiwiki_general_1564414369/_count?pretty
{
"count" : 739612,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
}
}
ebernhardson@mwmaint1002:/home/mstyles/cirrus_log$ curl https://search.svc.codfw.wmnet:9243/mediawikiwiki_general_1583750799/_count?pretty
{
"count" : 739104,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
}
}