Thanks @Clement_Goubert! Yeah, --sort-by=.metadata.creationTimestamp is my go-to for ordering.

In T380925#10360022, @Tgr wrote:

You can definitely break maintenance scripts by changing the right env variable, but as long as the user needs to be explicit about which variables they want to set / copy, that seems like a "don't do that then" problem. From a security perspective, I don't think it makes any difference as you can already execute arbitrary code via shell.php (and mwscript also lets you run ad hoc PHP files).

In T341553#10358961, @Urbanecm_WMF wrote:

Do we plan to fix the race conditions related to --attach?

Drive-by: The reason is that sudo doesn't preserve the original environment by default -- so in

sudo FOO=bar command

the command sees $FOO, but in

FOO=bar sudo command

it doesn't.

In T341553#10358334, @Urbanecm_WMF wrote

1. Hitting control-c no longer terminates the job (so blindly following the script's guidance will execute the job)

[...]
Since mwscript-k8s now prints guidance on ctrl+c, the first issue is probably a low-priority thing (we likely do not want to hardcode stuff specific to mwscript-k8s to userOptions.php itself, and hitting ctrl+c will print the right guidance; the only thing that could be done is explicitly asking the user about their intention, as I suggested at T341553#10263470, but I understand the concerns previously raised by @RLazarus).

In T341553#10357946, @Tgr wrote:

Is there a way to pass an env variable? The need came up in T380575: Make SUL3 authentication domain mode available from CLI; old mwscript doesn't do it smoothly either, but it can be made to work, which I suspect won't be the case with mwscript-k8s.

As currently implemented, if the shell dies (or the network is disconnected, or anything else interrupts the stream) then stdin is closed in the container.

This is now supported, and documented at https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Input_from_a_file.

(For the avoidance of doubt: We'll need some form of solution to this problem before turning off the mwmaint hosts, but I'm not working on it as a mwscript-k8s feature.)

In T341553#10268045, @Michael wrote:

Recording here that I'm noticing myself still running one-off scripts on the maint-hosts because, as I understand it, for the new way of running them, I would need deployer-rights, and I do not have (and do not really want) those.

So going forward, I have to either ask for more priviledges which allow me to do many more things than I actually want to do, or I have to ask someone else to run the maintenance scripts for me in the future. Both are not great, but maybe these tradeoffs are worth it.

In T341553#10263470, @Urbanecm_WMF wrote:

at the very least, --help should have that link.

In T341553#10262044, @Clement_Goubert wrote:

Suggestion: Catch ctrl-c and print out something like

⚠️ It looks like you may have wanted to stop your script's execution, use:
kube-env mw-script-deploy codfw; kubectl delete job mw-script.codfw.foo

then proceed to exit mwscript-k8s ?

In T341553#10260574, @bd808 wrote:

[...] needing to remember to setup the correct credentials with kube_env mw-script-deploy codfw before being able to use kubectl ... commands is a cognitive burden that could be replaced by mwscript-k8s making that environmental change as part of its normal operation.

Sorry this happened. Unfortunately it's kind of working as intended -- not because it's supposed to be hard to kill a job when you want to kill it, but because the job is supposed to keep working after the mwscript-k8s launcher terminates. (Thus preventing the "oops, I forgot to start it in a tmux and now I'm stuck" scenario.)

The shape of this sounds right to me. Similarly we can have the mw-script helmfiles gate on the same conftool value for defense-in-depth, but also read it in the wrapper script and exit early with a friendlier error message.

Note that while the announcement is only for "manual maintenance scripts", it's probably safe to assume invoking mwscript from other scripts that invoke mwscript falls under a similar umbrella.

This is ready to use, and documented (including the JSON output format) at https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Shelling_out_to_mwscript-k8s. @EBernhardson please give this a try and let me know how it works for you! Happy to iterate as needed.

Thanks -- that was in reference to:

Instead of

In T341553#10217525, @Urbanecm_WMF wrote:

[...]
The mediafiles can be very large – I've certainly uploaded files that had dozens of GBs in total. As long as mwmaint had enough space and sufficient sleep was allowed for videoscalers to catch up, things worked well.

This is now supported!

Ha, attachAccount.php specifically blocks this from working:

A mwscript-k8s flag to log to SAL is on my to-do list -- I hadn't gotten around to filing a task, thanks.

And if for whatever reason, we end up with a different namespace from the currently implemented as systemd timers recurring scripts

This is MWScript.php behavior, and it's actually unchanged:

If this changed in mwscript-k8s, it's unintended (but definitely not impossible) -- I'll dig into it today and get back to you.

In T341553#10202653, @EBernhardson wrote:

And a bug report. When running mwscript-k8s from an arbitrary directory it leaves behind a .kube directory. I often move to the current deploy dir, such as /srv/mediawiki/php-1.43.0-wmf.25, to get tab completion of the maintenance scripts. Running mwscript-k8s from that dir left a .kube directory behind. That directory caused scap to fail with a permission denied error since the dir is not world readable. My guess would be that should be created in $HOME/.kube. Plausible $HOME isn't being passed down to the k8s commands?

In T341553#10200997, @EBernhardson wrote:

Could --follow also exit with the same return code as the script that was executed? The most notable example of this on our side is the CirrusSearch UpdateOneSearchIndexConfig.php maintenance script which signals to cirrus-reindex-orchestrator via the exit code one of three possible results of the update operation.

In T341553#10192994, @taavi wrote:

Is it possible to include a text file from disk in the container where a script runs in? Some scripts (like extensions/CentralAuth/maintenance/attachAccount.php) use those for lists of things to process.

Thanks! In general you shouldn't need to do this, even if the job was a mistake. Kubernetes cleans up the job automatically a week after it terminates -- whether it completed or failed -- and there's nothing wrong with leaving it until then.

The image version is now copied from the mw-web deployment.

So, mwscript-k8s is still in medium-early development, some snags still expected and it shouldn't be anyone's primary workflow yet. There'll be wider announcements when it's ready for adoption (Soon TM but Not Yet TM); in the meantime, thanks for giving it a try and feedback is still welcome, just don't panic when it isn't ready to use full-time.

Thanks both! Sorry for missing the earlier task.

See also T359130 for the cookbook work. We aren't as far as I expected we'd be, so we can revisit which of those steps for cronjobs-on-k8s need to be accomplished before the switchover, but I agree it's a good idea to add this to mw-cli-wrapper.py in the meantime and start using it.

This looks related to https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1060867 -- at a guess, should

Yeah, the usual standard, for any time you change something on a debug host, is a note in #wikimedia-operations to say something like "I'm grabbing mwdebug1002 to test an Apache config change" -- that ensures no one else is testing a conflicting change at the same time, such as (for example) a deployment during a deployment window. :)

From -serviceops IRC logs, I think this was during the time @Ottomata was working on an unrelated Apache config change, and testing it on a mwdebug host. That would explain why the tests failed on one host only.

I haven't dug for logs, but workers were saturated and latency was up: https://grafana.wikimedia.org/goto/h67ix_uSg

Someone in serviceops probably knows the answer to this but I don't, at least not confidently. Here's a rough stab:

Script output is visible through kubectl logs, and mwscript-k8s can be invoked with -f to immediately start tailing the script output (under the hood, it just invokes that kubectl command). If you don't launch with -f it prints out the kubectl command so you can copy and paste it.

This is fixed, thanks again for testing!

Thanks for the report! It's actually not because of the successful exit; the script handles that.

Disregard the above scap, I got too carried away with "never run helmfile across all mw deployments, use scap instead" but obviously that rule doesn't apply here. :)

Thanks. At present the controller monitors all namespaces, but ignores pods other than in mw-script. So if I were estimating memory usage I'd base it on the total number of pod events in the cluster, not just in the namespace.

That sounds reasonable! Note for the future that helm diff has a --suppress-output-line-regex which does exactly what you'd like it to do, but it's not available in the version we're currently running.

If we're really worried about that race condition, is it plausible to do this?

Clinic duty SRE here -- I/F, can you start investigating this at the MTA end? Triaging this to High in case it's widespread, but feel free to decrease if it turns out it's not.

@AndyRussG Welcome back!