I'm getting a little confused, so will leave some breadcrumbs about where various error stats come from in the current code:

• MediaWiki.FileImporter.specialPage.execute.fail.* - Permission and user block errors when first opening the special page are tallied as , which should be used with care because it almost overlaps with the ...fail.plan.* below.
• MediaWiki.FileImporter.specialPage.execute.fail.plan.total and detailed MediaWiki.FileImporter.specialPage.execute.fail.plan.byType.* tally errors when building the ImportPlan.
• Mediawiki.FileImporter.import.result.exception is produced by Importer::import called when action=submit.

The grafana dashboard shows all these errors now, and sums correctly over the past 24hr. Next steps are to distinguish between recoverable and unrecoverable errors during the planning stage. Then we can provide more granularity where desired--probab best if @Lea_WMDE looks over the dashboard to help scope where we need this granularity.

Notes from discussion:

• Definitely go ahead with creating a MediaWiki page to document what the buckets are and what specific errors are in each one.
• Link to our documentation from the "i" information popup on the failure graph, and summarize to help users with interpretation.
• Minimum granularity for this task is to split out unrecoverable from recoverable errors, and to expose exact counts for AbuseFilter matches.

Notes on next implementation steps:

• Our exceptions should all respond to getCode, with a string constant naming the error type.
• Take recoverable errors out of the planning phase error count, report it separately.
• FileImporter.import.result.exception' is missing two errors, "bad edit token" and "bad import hash". Move import submit stats logging up to the exception handler to catch these.
• Drop the "plan" vs. "submit" distinction, this doesn't seem to matter.
• Report time taken to fail?

Report time taken to fail?

I don't think we need that yet. Right now we are interested in step 1: Knowing how often which errors appear.

Change 517081 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/FileImporter@master] Give all import exceptions a code

https://gerrit.wikimedia.org/r/517081

Change 517061 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/FileImporter@master] Standardize error buckets

https://gerrit.wikimedia.org/r/517061

Epic catalog of error codes is evolving here: https://www.mediawiki.org/wiki/Extension:FileImporter/Errors

Change 517081 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Give all import exceptions a code

https://gerrit.wikimedia.org/r/517081

Confirmed that AbuseFilter errors are reported with error codes like abusefilter-disallowed, so they'll be easy to visually group in Grafana.

Reminder that grouping automatically is not trivial until we have tagged metrics.

Split out obsessive documentation into T225951, a non-blocking subtask.

This task will need a short iteration after merge and beta deployment, to update and test Grafana panels for the new metrics.

Change 517061 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Standardize error buckets

https://gerrit.wikimedia.org/r/517061

Now we're testing with beta cluster Grafana, then updating the production. Oh and ¡surprise!, these changes were merged just in time to get dropped into the MediaWiki train tonight (testwiki) and tomorrow. We'll need to monitor that on Wednesday.

Trouble testing on beta: I'm unable to see any FileImporter metrics whatsoever on https://grafana-labs-admin.wikimedia.org/dashboard/db/awight-beta-fileimporter?orgId=1

I ran into this when building the production graphs, T226100. I'm not ready to trust the data due to this glitch--either something I'm doing wrong, or Graphite bugging out.

Worked around the glitch, I think the graphs are correct now. After demo and approval, we can remove the old graphs.

Change 517877 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/FileImporter@master] Document new error metrics

https://gerrit.wikimedia.org/r/517877

Change 517877 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Document new error metrics

https://gerrit.wikimedia.org/r/517877

The dashboard is looking great now!
I just have one thing:

• The graph contains both recoverable and unrecoverable errors. Let's focus on unrecoverable errors for now (but if it is a 2 minute thing to do, I would be interested in a seperate graph for recoverable errors, too)

In T225602#5282034, @Lea_WMDE wrote:

The dashboard is looking great now!
I just have one thing:

• The graph contains both recoverable and unrecoverable errors. Let's focus on unrecoverable errors for now (but if it is a 2 minute thing to do, I would be interested in a seperate graph for recoverable errors, too)

Sure, that's easy. I have a crude workaround the lack of tagged metrics (keyword bingo!), so we can filter on byRecoverable as part of the metric path.

Ready for re-review.