libjars #1342
Merged
libjars #1342
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
May 27, 2016 16:24
also fixed a typo having to do with the combiner for hadoop_tmp_dir, should note this in CHANGES.txt
(looks like the master node setup script currently fails) We probably want to monitor the master node setup script like we do with other steps, and just be careful about how this affects self._log_interpretations
still to do: - StepFailedException should say "Master node setup step" not "Step 0 of ..." - num_steps is incorrect for error reporting - step log parsing should fall back to parsing stderr if no job ID in syslog
also suppressed warnings about missing job ID when parsing errors from script-runner.jar
added 12 commits
June 24, 2016 17:28
pretty much just need to test _build_steps() and _wait_for_step_to_complete()
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes the
libjarsoption work for both Hadoop and EMR (sorry, Dataproc!). Fixes #198.To make this work on EMR, I had to introduce the concept of "master node setup," which is basically an EMR step that runs a simple script on the master node using
script-runner.jar. In theory, we should be able to run anything here (like bootstrap and setup; see #1336), but for now it just copies jars from S3 into a local directory.This also makes sure that generic args created by mrjob (e.g.
-D mapreduce.job.reduces=0) always come beforehadoop_extra_args, which could contain a combination of generic and JAR-specific args.(fixes #1331, #1332).