cmd/reload: Calling this when long-running job is running causes commands to time out thereafter #283
Comments
How strange! |
|
OOPS! This is not true, after all. I improperly tested that case before ...
I apologize for the false alarm. Jobber only gets blocked if I do not force the job into the background. |
|
I have an idea (more like an educated guess) as to what might be causing this problem with certain commands blocking the job runner. I am not a very experienced Go programmer, so I might be on the wrong track, but just in case I hit upon something ... Look at lines 77 through 87 in the file common/exec.go. And then check out the discussion here: It seems like something like Output() or CombinedOutput() needs to be called or Goroutines need to be utilized, instead of using the two ReadAll() calls, because otherwise, the stdout or stderr pipes could block ... and this seems to be what is happening in the case that I am reporting here. Is this a possibility? |
|
I think the following solution might work. Just replace lines 76-84 of common/exec.go with this code ... I tested this locally with the latest code base, and it seems to work. However, I don't have enough experience with Go to feel confident about submitting this as a PR. |
|
This is awesome. I will check this out. I think you’re onto something.
… On May 13, 2020, at 1:57 PM, Lloyd Zusman ***@***.***> wrote:
I think the following solution might work. Just replace lines 76-84 of common/exec.go with this code ...
var stdoutBytes []byte
var stderrBytes []byte
go func() {
stdoutBytes, _ = ioutil.ReadAll(stdout)
}()
go func() {
stderrBytes, _ = ioutil.ReadAll(stderr)
}()
I tested this locally with the latest code base, and it seems to work. However, I don't have enough experience with Go to feel confident about submitting this as a PR.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
... but now I'm not sure whether this really fixed it. :( Now I have more info about when the problem seems to occur. If a long-running jobber-initiated job is in progress, I can do a I'm not sure if this happens all the time under those circumstances, because I don't have time for in-depth testing at the moment. But this might provide more useful info for diagnosing the problem. |
|
After a quick perusal of Again, I'm not sure about this, and I don't have time right now for more in-depth investigation, but perhaps the job sockets need to be handled differently (somehow ... ???) in order to allow jobs to run and reloads to take place at the same time. |
If, while a job is running, the user does 'jobber reload', this command will fail with timeout, and then subsequently all commands will also fail with timeout. This commit fixes this problem by having Jobber cancel all running jobs when the user does 'jobber reload'.
|
I think I have a fix for this. Could you please install this package -- https://github.com/dshearer/jobber/suites/690922681/artifacts/6521232 -- and let me know if it works? |
|
To @dshearer ... This indeed seems to have fixed the problem. Thank you! I scheduled my long-running job in Then, I waited for the job to fire off, which it indeed did at the correct time. So, as far as I'm concerned, it looks like the problem has indeed been fixed. |
If, while a job is running, the user does 'jobber reload', this command will fail with timeout, and then subsequently all commands will also fail with timeout. This commit fixes this problem by having Jobber cancel all running jobs when the user does 'jobber reload'.
reload when long-running job is running causes commands to time out
reload when long-running job is running causes commands to time outreload when long-running job is running causes commands to time out thereafter
reload when long-running job is running causes commands to time out thereafterIf, while a job is running, the user does 'jobber reload', this command will fail with timeout, and then subsequently all commands will also fail with timeout. This commit fixes this problem by having Jobber cancel all running jobs when the user does 'jobber reload'.
If, while a job is running, the user does 'jobber reload', this command will fail with timeout, and then subsequently all commands will also fail with timeout. This commit fixes this problem by having Jobber cancel all running jobs when the user does 'jobber reload'.
Uh oh!
There was an error while loading. Please reload this page.
Most jobs that I submit through Jobber on my Ubuntu 18.4 machine run with no problem, but certain jobs cause problems ...
When one of those jobs starts up, I can no longer run any jobber commands. Every jobber command that I try hangs, and then after a while, it prints "Call to Jobber timed out." This includes invocations of
jobber list,jobber log,jobber pause, and all other jobber commands.Most jobs don't cause this to occur, but one job which reliably causes this problem is when I run a program called "luckyBackup", which is a front end to an
rsync-based backup. I run that job once a day in order to back up my machine to a filesystem on a mounted USB drive. That job caused no ill effects when it ran under cron, but for some reason, it consistently causes Jobber commands to hang and time out when it is running.Also, this is not simply an effect of a job running for a long time. I can run
/bin/sleep 300as a Jobber job, and this does not occur.Also, I can specify my job as follows, and it still causes the problem:
cmd: ( /path/to/my/job args ... & ) </dev/null 1>/dev/null 2>&1In this case, the command returns immediately, and Jobber thinks it ran. Nonetheless, the fact that it is running in the background still leads to jobber commands timing out.
As soon as I kill the process, jobber starts working again.
This seems to be due to some characteristic of the actions that the job itself is performing. Since the problematic job involves running a program that does an
rsync-based backup of my entire machine, I'm guessing that something thatrsyncdoes at a low level might be interfering with Jobber's job management code.The text was updated successfully, but these errors were encountered: