tests: tentative fixes for sporadic host and kvm failures #3434

lucab · 2016-12-01T15:04:16Z

This PR tentatively fixes the following flakes:

multiple jenkins-master failures, possibly due to debug output leading to a children blocked on write()
kvm stop failures, possibly due to a too short timeout (or kvm: too low thread limit #3382, to be determined)

s-urbaniak

LGTM, just two nits.

s-urbaniak · 2016-12-01T15:11:39Z

tests/rkt_tests.go

 func waitOrFail(t *testing.T, child *gexpect.ExpectSubprocess, expectedStatus int) {
+	bufOut := []string{}
+	ttyIn, ttyOut := child.AsyncInteractChannels()
+	close(ttyIn)


👍 Can we leave a TODO to change gexpect.AsyncInteractChannels to accept channels rather than returning them?

Do you mean opening a ticket to gexpect or a just a reminder for us here?

I suggest just a TODO here, I'll set up a PR in gexpect and some documentation around the usage patterns for these channels.

s-urbaniak · 2016-12-01T15:15:40Z

tests/rkt_tests.go

+	bufOut := []string{}
+	ttyIn, ttyOut := child.AsyncInteractChannels()
+	close(ttyIn)
+	for line := range ttyOut {


(I guess) this relies on the fact that the ttyOut chan will be closed when a real err happens or io.EOF in https://github.com/ThomasRooney/gexpect/blob/8e96656e8fb794d716ee75b5a8cecbc15cde4393/gexpect.go#L167. I think we should document this here and in gexpect.

It does, I'll note this down.

This commit fixes waitOrFail to drain output from the child, in order to avoid a deadlock situation where the children is blocked on a write toward the parent, while the parent is blocked waiting for the children.

This raises the pod-stop timeout from 5 to 15 seconds. Sporadic test failures on KVM flavor may be due to a too strict limit, so we just 3x here to make sure failures are not due to bad timing.

lucab · 2016-12-02T09:45:48Z

Rebased, PTAL.

s-urbaniak

LGTM

lucab added area/testing needs/review labels Dec 1, 2016

lucab added this to the v1.21.0 milestone Dec 1, 2016

s-urbaniak suggested changes Dec 1, 2016

View reviewed changes

lucab changed the title ~~tests: teantative fixes for sporadic host and kvm flavors~~ tests: tentative fixes for sporadic host and kvm flavors Dec 1, 2016

lucab changed the title ~~tests: tentative fixes for sporadic host and kvm flavors~~ tests: tentative fixes for sporadic host and kvm failures Dec 1, 2016

lucab added 2 commits December 1, 2016 15:59

tests: drain output in waitOrFail

a04bfbf

This commit fixes waitOrFail to drain output from the child, in order to avoid a deadlock situation where the children is blocked on a write toward the parent, while the parent is blocked waiting for the children.

tests/stop: raise the pod-stop timeout

46e9c15

This raises the pod-stop timeout from 5 to 15 seconds. Sporadic test failures on KVM flavor may be due to a too strict limit, so we just 3x here to make sure failures are not due to bad timing.

lucab force-pushed the to-upstream/tests-fixes branch from fd8ee7a to 46e9c15 Compare December 1, 2016 15:59

s-urbaniak approved these changes Dec 2, 2016

View reviewed changes

s-urbaniak merged commit a4706c1 into rkt:master Dec 2, 2016

lucab mentioned this pull request Dec 2, 2016

tests: fix some potential gexpect hangs #3443

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: tentative fixes for sporadic host and kvm failures #3434

tests: tentative fixes for sporadic host and kvm failures #3434

tests: tentative fixes for sporadic host and kvm failures #3434

tests: tentative fixes for sporadic host and kvm failures #3434

Conversation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment