8000 DOC: enhance documentation for first-time users by stsievert · Pull Request #191 · htcondor/htmap · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

DOC: enhance documentation for first-time users #191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 57 commits into from
May 6, 2020
Merged

DOC: enhance documentation for first-time users #191

merged 57 commits into from
May 6, 2020

Conversation

stsievert
Copy link
Contributor
@stsievert stsievert commented Mar 20, 2020 •
edited
Loading

What does this PR implement?
It adds a link back to the HTCondor homepage on the front page of the HTMap documentation.

I was looking for a piece of documentation specific to HTCondor ("what does a HOLD tag on a job mean?"). I went looking for link back to the HTCondor documentation, and the front page was the first place I looked.

It also implements the following:

  • Adds page on the CLI
  • Adds page on how to effectively use HTCondor (requesting CPUs/disk/GPUs/etc, some useful shell commands, etc)
  • Reorganizes documentation (rename "recipes" to "advanced tutorials")
  • Adds example about processing results from completed job to Map.component_by_status
  • Adds note about how string parameters are processed (including note about strings with quotes)

All of these come from my first (real) usage of HTMap and HTCondor.

edit This PR will...

@stsievert stsievert changed the title DOC: Add link to HTCondor pool DOC: Add link to HTCondor pool documentation Mar 20, 2020
JoshKarpel
JoshKarpel sugge 8000 sted changes Mar 20, 2020
View reviewed changes
Copy link
Contributor
@JoshKarpel JoshKarpel left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made a suggestion on the FAQ text to make it a little meatier.

Otherwise, LGTM! Let me know if there's any other places you want to touch up or if I should go ahead with a merge.

@JoshKarpel JoshKarpel added the docs Documentation-related label Mar 20, 2020
@codecov
Copy link
codecov bot commented Mar 20, 2020 •
edited
Loading

Codecov Report

Merging #191 into v0.6.0 will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           v0.6.0   #191   +/-   ##
=====================================
  Coverage      78%    78%           
=====================================
  Files          19     19           
  Lines        1967   1967           
  Branches      315    315           
=====================================
  Hits         1548   1548           
  Misses        356    356           
  Partials       63     63

@stsievert
Copy link
Contributor Author
stsievert commented Mar 20, 2020 •
edited
Loading

Let's hold off a little bit on merging; I'd like to at least get a basic test up and running. This PR can hold the documentation fixes I make in the meantime.

I have perhaps a naive usage question. I have this script on submit2.chtc.wisc.edu:

def job(x):
    return x

if __name__ == "__main__":
    mapped = map(job, range(4))
    results = list(mapped)
    print(results)

    import htmap as ht

    future = ht.map(job, range(4))
    print(future)
    results = list(future)
    print(results)

I submit this script with a miniconda install on submit2.chtc.wisc.edu (in which I think I've only installed htmap==0.5.1 besides the default packages). It throws an error because ExitCode =!= 0. Here's stdout with the traceback:

(base) [stsievert@submit2 htmap-trial]$ python submit.py
[0, 1, 2, 3]
Map(tag = super-proper-rig)
Traceback (most recent call last):
  File "submit.py", line 11, in <module>
    results = list(future)
  File "/home/stsievert/miniconda3/lib/python3.7/site-packages/htmap/maps.py", line 422, in __iter__
    yield from self.iter()
  File "/home/stsievert/miniconda3/lib/python3.7/site-packages/htmap/maps.py", line 439, in iter
    yield self._load_output(component, timeout = timeout)
  File "/home/stsievert/miniconda3/lib/python3.7/site-packages/htmap/maps.py", line 343, in _load_output
    self._wait_for_component(component, timeout)
  File "/home/stsievert/miniconda3/lib/python3.7/site-packages/htmap/maps.py", line 309, in _wait_for_component
    raise exceptions.MapComponentHeld(f'Component {component} of map {self.tag} is held: {self.holds[component]}')
htmap.exceptions.MapComponentHeld: Component 0 of map super-proper-rig is held: [3] The job attribute OnExitHold expression 'ExitCode =!= 0' evaluated to TRUE
(base) [stsievert@submit2 htmap-trial]$

Why isn't this working?

@stsievert @JoshKarpel
Update docs/source/faq.rst
dbc4c5e 
Co-Authored-By: Josh Karpel <josh.karpel@gmail.com>
@JoshKarpel
Copy link
Contributor

I snooped around in your .htmap directory on submit2 (hope you don't mind), since that error is a little odd.

It turns out you're falling through a crack in the mechanism that detects execution errors. The problem is that your Docker image doesn't have cloudpickle installed. If you run future.stderr[0] you can see the actual error; if you run future.stdout[0] you'll see the the installed packages.

Basically, the problem is that HTMap is failing execute-side before it can manage to produce an execution error for you. I'm not quite sure how to tackle this, but I've created an issue to track it #194.

For now, you probably need to make a new image (based on the one you're using) with cloudpickle (or preferably htmap itself) installed.

@stsievert
Copy link
Contributor Author

I snooped around in your .htmap directory on submit2 (hope you don't mind), since that error is a little odd.

Thank you – that resolved my issue. I've added some debugging tips in error-handling.ipynb for when a job is held. Now, I have the following to complete:

  1. Launching with Docker
  2. Machine specification to run jobs
  3. Running my job
  4. (possibly) checkpointing.

I'll likely have more documentation changes in this process. I've modified PR title to reflect that.

@stsievert stsievert changed the title DOC: Add link to HTCondor pool documentation WIP: DOC: Add link to HTCondor pool documentation Mar 21, 2020
@JoshKarpel JoshKarpel changed the base branch from master to v0.6.0 March 23, 2020 19:56
@stsievert
Copy link
Contributor Author

I've consolidates the tutorials and recipes into one page with a divider. I can't test this change right now (htcondor doesn't install on macOS).

I did this because I remembered seeing a Docker tutorial, and spent a couple minutes looking for it. I first looked in "Tutorials", then I looked in "Dependency Management", then I bounced around a couple places before looking in the recipes. I moved it to the first place I looked.

@JoshKarpel
Copy link
Contributor

Seems quite reasonable!

If you need an environment to build the docs in, the testing Docker container should just need the docs requirements installed in it to be able to do a docs build. Start the container by running ./dr bash from the top of the repository, pip install -r docs/requirements.txt, and then you should be able to cd docs and make html.

I should probably encapsulate that in another container for ease of use... (I run into the same problem on Windows, since we don't have pip-installable HTCondor Python bindings on Windows).

@stsievert stsievert mentioned this pull request Mar 24, 2020
Closed
@JoshKarpel
Copy link
Contributor

Quick thought - be careful on HTCondor documentation, because a lot of things ends up being site-specific. The WantFlocking option, for example, is CHTC-only. We need to avoid CHTC-isms in the docs.

That does make me think that we should eventually have a "HTMap in CHTC" page on https://github.com/CHTC/chtc-website-source where we can put that kind of information.

JoshKarpel
JoshKarpel suggested changes May 5, 2020
View reviewed changes
Copy link
Contributor
@JoshKarpel JoshKarpel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love it! Comments have a bunch of small nitpicks/personal-preference edits, but I love the reorganization, and the wealth of little tips and tricks coming from your experience getting started is awesome.

stsievert and others added 23 commits May 5, 2020 20:04
@stsievert @JoshKarpel
Update docs/source/tips-and-tricks.rst
1ff95ae 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert @JoshKarpel
Update htmap/options.py
2d61be9 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert @JoshKarpel
Update docs/source/htcondor.rst
f63a4ef 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert @JoshKarpel
Update docs/source/htcondor.rst
eff2060 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert @JoshKarpel
Update docs/source/htcondor.rst
26f3b2a 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert @JoshKarpel
Update docs/source/htcondor.rst
8b3348c 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert
Add noteson job state
e3f7e1d
@stsievert
Merge branch 'doc-link' of https://github.com/stsievert/htmap into do…
e48a380 
…c-link
@stsievert @JoshKarpel
Update docs/source/htcondor.rst
e0bd0de 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert @JoshKarpel
Update docs/source/htcondor.rst
0e984e9 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert @JoshKarpel
Update docs/source/tips-and-tricks.rst
a9300bd 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert
Edits
d77538f
@stsievert
Merge branch 'doc-link' of https://github.com/stsievert/htmap into do…
66baf1d 
…c-link
@stsievert
Add note about sep
8f5fb97
@stsievert @JoshKarpel
Update docs/source/tips-and-tricks.rst
7bc7c3f 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert @JoshKarpel
Update docs/source/tips-and-tricks.rst
6a888d1 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert
Review
8df7824
@stsievert @JoshKarpel
Update docs/source/tips-and-tricks.rst
602745a 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert
Merge branch 'doc-link' of https://github.com/stsievert/htmap into do…
07dc595 
…c-link
@stsievert @JoshKarpel
Update docs/source/tips-and-tricks.rst
524ccc9 
Co-authored-by: Josh Karpel <josh.karpel@gmail.com>
@stsievert
newline
5388d87
@stsievert
Merge branch 'doc-link' of https://github.com/stsievert/htmap into do…
67b6677 
…c-link
@stsievert
Variable name change
ac6c149
@JoshKarpel
Copy link
Contributor

Looks like everything is resolved! I'll let it simmer overnight for any final changes and merge it tomorrow.

(Don't worry about the CI builds... there's a flaky test that I've been trying to hammer out.)

@stsievert
Copy link
Contributor Author

I'll try to give it another review tonight too.

@JoshKarpel JoshKarpel merged commit 3c0bedd into htcondor:v0.6.0 May 6, 2020
@JoshKarpel
Copy link
Contributor

Thank you so much @stsievert !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JoshKarpel JoshKarpel JoshKarpel approved these changes

Assignees
No one assigned
Labels
docs Documentation-related
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@stsievert @JoshKarpel
0