Use env var to control if notebooks are executed and add docs, slim, slimfast, and slimgallery variants to Makefile #646
Conversation
|
I tested this locally and it worked! We could set the env var in a new |
|
Looking at prep_docs, I bet it's the mermaid diagrams that are slow! |
|
Ok so it works! Also, Now if you run this on CI (aka fresh clone) it might not complete--I haven't tested. but locally it seems good! |
|
Absolutely awesome @psobolewskiPhD! I'm a bit confused about the mermaid diagrams — if we skip prep_docs altogether and they're not cached, would it crash the build or would it just be rendered as text? |
|
Overall this is super super exciting — after we merge this we can add a separate circleci config for slimfast (😂) which would allow really fast feedback from CI builds — gamechanger! |
Dunno. it's on the todo to test with a fresh clone where prep_docs hasn't been run. |
|
Question: |
For sure! and I assume also skips gallery? |
|
Latest update:
|
|
Oh, I tested locally with a fresh clone. edit: making stub files is probably the easiest. Have the files generated by prep_docs there, but just placeholders. |
|
@psobolewskiPhD I will take a look at this tomorrow. We should be able to get the API build running smoother. Let me take a looks at the scripts and think about optimization. |
|
I'm almost ready to merge a commit that adds |
|
Boom. Works nicely. |
bd58d53 to
ab69fe9
Compare
|
If this goes through, I have a prototype of:
|
|
This is marvelous! 🤩 Truly incredible. And it looks way less janky than I expected. Code blocks are still there, just nothing obviously ran (maybe this is obvious but I sort of expected a blank spot). Also, anywhere there is a stub it nicely informs "This is a stub. The real file is autogenerated in a full build." Which I really like! Might be worth adding Now all that's left is me not having to rethink Git Bash every time because I'm on Windows. I'm sure this will help me finally learn the like 2 extra steps I have to do that I always forget... |
I think there are 2 main workflows for a fresh clone/new contributor:
Transitioning to a longer term contributor? may as well run @TimMonko did you want me to revert the |
|
Would be cool to come up with a way to do a quick gallery build, of only the examples you edit -- so like the |
Quiet is ok as long as it tells users a useful error (like a myst formatting issue or whatever). I honestly I have no idea what this sphinx doc --quiet is supposed to mean. Your overall synopsis of the usage is 💯 Pretty much can be dropping in to the contributing documentation page as is |
|
Also to be most forward. I trust your judgement, wisdom, and experience much more than mine, so do whatever you think is best |
So on unix systems at least (macos/linux for sure) the normal console display for interactive shells prints both standard error and standard output. So --quiet will silence all the normal progress etc output, when everything runs normally, but it will print errors and warnings when things are whack, because the typical display will also show standard error. The distinction is useful when piping commands or running things headless. Then you may want to monitor errors in particular, hence a different output channel (standard error) for them. |
|
I think I can get a gallery autobuild. You will need to run a gallery build once (but don't need prep-docs or notebook execution) and then it should update edited examples. I'll test it and then push if it works. |
|
Ok! I think I got examples to work.
and to make it even easier to contribute a single example:
neither of these require the full examples to be built in advance (I wanted to build stubs and md5s for the gallery, but it was too annoying) |
|
Using (oh btw i reverted the --quiet mode so everything looks the same when running) |
| # no generation from prep_docs scripts, no gallery | ||
| # will not remove existing gallery files | ||
| docs: clean prep-stubs | ||
| NAPARI_APPLICATION_IPY_INTERACTIVE=0 sphinx-build -M html docs/ docs/_build -D plot_gallery=0 -D sphinx_gallery_conf.examples_dirs=$(GALLERY_PATH) $(SPHINXOPTS) |
There was a problem hiding this comment.
(1) is sphinx_gallery_conf.examples_dirs needed with plot_gallery=0?
(2) I don't understand now how to build both docs and gallery? ie what is the incantation for a "full build" now?
(3) it might be worth mentioning that in the comment. "For gallery, look at XYZ. For a single gallery example, look at ABC. For a full build, look at ZYX."
There was a problem hiding this comment.
@psobolewskiPhD I'll merge this in the morning my time because my questions above are minor, but if you have a chance to look at them I appreciate it. 😊 🙏
There was a problem hiding this comment.
-
my understanding from the docs is that examples_dir is a needed config, while plot_gallery is an execution option:
https://sphinx-gallery.github.io/stable/configuration.html
The settings are the same as existing make commands that didn't build gallery, so I didn't want to mess with it. -
this hasn't changed. it's still make html.
-
Do you mean add comments to the existing make options? I can do that. I will definitely update the contribution guide before this is merged, just wanted some feedback on the options.
There was a problem hiding this comment.
@jni I update the contribution guide for the new make variants. I tried to make it clear which you'd want to run depending on what you were trying to contribute.
|
@psobolewskiPhD this is amazing work. 😍 I had a minor question when looking over the full makefile, but improvements are optional at this point and can come in followup PRs. |
it's confusing and redundant with all the other live variants.
|
I update the contribution guide for the new make variants. I tried to make it clear which you'd want to run depending on what you were trying to contribute. |
…targets (#649) # References and relevant issues Followup to #646 # Description In a clean repo, `prep_stubs` was generating warnings due to missing targets. The stub files were written, but lacked targets. I fix this in this PR and add `clean-prep` and `clean-full` to be able to ensure all autogenerated content can be cleaned if needed.
…#669) # References and relevant issues Depends on #671 # Description In #646 new make targets were added for faster builds that skip certain parts of the process. In this PR I make them available on CI using a comment: `napari-bot make <target>` (add the `@` in front of napari-bot) Here's the breakdown: 1. circleci/config.yml : Here I add a parameter `make_target` to make the make target configurable on-run. Default is `slimfast` (no gallery, no outside stuff, no notebook execution) 2. build_docs.yml : I added this back, as a reusable workflow, so that there is a build-only workflow that can be run on PR push and triggered with different make_target. (The build-deploy job needs to use the full build for obvious reasons). I made it so that this build-only job also be manually triggered by dispatch. Whether triggered or by manual dispatch, this workflow accepts a make target, one of: html html-noplot docs slimfast slimgallery. The default is `slimfast`, so on (every) push/open of PR it will use `slimfast`, but can be triggered with the other options. Note: at present the triggered state is not saved, so push always used the default. 3. build_trigger.yml : This is the workflow that detects PR comment, checks make target vs a list, and triggers the CircleCI and build_docs builds with the make target from the comment. It also has a step to report back that the triggered build_docs workflow finished and provide a link to the run. 4. build_and_deploy.yml : I modified this to not run on PR push anymore, because it's a full build -- we do need full build for deployment and it's triggered by napari/napari as well. I also modified the PR template to give info on the CI builds, as well as the contributor guide where we talk about docs building on CI. Note: I don't think the new features will trigger in this PR, because it's from a branch. You should be able to see this in action and test it here in my fork, where this PR branch is the default branch: psobolewskiPhD#74 You can see builds triggered by comments from Juan and Carol: https://github.com/psobolewskiPhD/napari-docs/actions/workflows/build_trigger.yml Followups: - have the trigger add a reaction to the comment, so the user knows it triggered - have the trigger add a label to the PR and then use the label as default for subsequent runs (e.g. push ones) --------- Co-authored-by: Tim Monko <timmonko@gmail.com> Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
References and relevant issues
closes: #625
Description
In #625 (comment) Juan noted that there is a flag in Myst NB to control whether notebooks are executed.
This is a proof of concept using that.
Basically, the flag is set via ENV var. If it's not set, the flag is set to
auto, which is the default. See:https://myst-nb.readthedocs.io/en/latest/configuration.html#execution
The effect is that now you can do something like:
NB_EXECUTION_MODE="off" make html-noplotand notebooks will not be executed.Additional slowness is from
prep_docswhich builds files for events, preferences, and the UI arch diagrams. All of the source material is from outside napari/docs.As a result, I've added
--stubsflag to all of the scripts, which will generate a stub file if the .md file is missing. This ensures that there are no warnings when trying to build the docs.In order to give more control over building docs, I've added to the Makefile:
prep-stubs-- this runsprep_docs.py --stubs, to generate stubs instead of the real files. Will not overwrite, so you can manually runprep-docsif you want.docs-- this will do just napari/docs, but run notebooks (pops up napari GUIs). Meaning: no gallery, prep_docs --stubs, nothing from napari/napari basicallydocs-live-- same as above, but autobuildslim-- same as docs, but without notebook executionslimfastwhich isslimbut with parallelization (-j auto). For me this one runs in ~35 s.slimfast-live-- live version ofslimfast. Withslimfastbeing fast, it runs an initial build. Great way to get started with editing non-codecell content.html-liveto be calledhtml-noplot-live, reflecting its actual role for consistency and to useprep-stubs, because those files get ignored anyways. It will still execute notebooks during autobuild.We have so many examples, that it's slow to build them all.
As a result, I modified
cleanto not remove them -- you can useclean-galleryto do that.Then I added:
slimgallery-- will clean and then build the docs with the examples gallery, but not run all the notebook cells in the rest of the docs. So only the examples will pop the viewer.slimgallery-livewill do the above, but as an autobuild. Will build all examples that arn't built--the whole gallery. Then, only the example you edit will be re-built when you edit it. Other pages will also rebuild, but notebook cells will not run.slimgallery-%--%is a capture, to a variable. This is a target forbuild-single-exampleHere this will clean and build just a single example, so you would do:slimgallery-vortexfor thevortex.pyexample.slimgallery-live-%-- this is the same as above, but will autobuild the single example. Editing other examples will be ignored. Editing other pages will work and trigger rebuild.Finally, I updated the documentation and README for the new make options.