This repository was archived by the owner on Jan 8, 2019. It is now read-only.
This repository was archived by the owner on Jan 8, 2019. It is now read-only.
Open
Description
There are currently lots of shell outs being done for HDFSDU prep operations, plus a default 30s timeout for waiting for the service to come up. We should use the existing dir_creation bits plus make some other minor changes to mitigate these delays somewhat.
So far, @ronny-macmaster and I have come up with the following suggestions, which should involve minimal effort but reduce the overall runtime significantly:
- Create and manage the HDFSDU user in
dir_creationwithNO_QUOTAor some sane number that won't interfere with ops - Figure out a way to speed up the HDFS fetch operation. Maybe another jruby script that only imports the needed HDFS classes and does a small number of tasks.
- Also add a method for doing the stat
- Change the timeout of the service wait block to something much lower, maybe 3-5 seconds by default? No more than 10 I'd hope.
The steps above should not be overly complicated, but it'd be nice to get feedback here from @cbaenziger et al.