8000 fix(core): prevent peas from hanging on close by jacobowitz · Pull Request #2824 · jina-ai/serve · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

fix(core): prevent peas from hanging on close #2824

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 1, 2021
Merged

Conversation

jacobowitz
Copy link
Contributor

We noticed that this test was failing a lot (but randomly) in ci

I was running this test locally a lot and saw it less often, but it happened as well occasionally. The reason for the failure is that the flow closing hangs, which does so because the Pea closing hangs.
After discussion with @JoanFM we decided to remove the alive check the process as it seems to be unnecessary and was just added recently and may be the reason for the issue.
Another possible workaround was a time.sleep() after terminating a Pea, but we would rather try to avoid this.

I was running this changed version locally hundreds of time and did not encounter the issue anymore.

@jina-bot jina-bot added size/XS area/core This issue/PR affects the core codebase area/network This issue/PR affects network functionality component/peapod labels Jul 1, 2021
@codecov
Copy link
codecov bot commented Jul 1, 2021

Codecov Report

Merging #2824 (ac2d45f) into master (65e7ec1) will increase coverage by 7.78%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2824      +/-   ##
==========================================
+ Coverage   80.96%   88.74%   +7.78%     
==========================================
  Files         138      138              
  Lines        9352     9351       -1     
==========================================
+ Hits         7572     8299     +727     
+ Misses       1780     1052     -728     
Flag Coverage Δ
daemon 46.05% <100.00%> (-1.12%) ⬇️
jina 88.17% <100.00%> (+7.80%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
jina/peapods/peas/__init__.py 93.58% <100.00%> (-0.05%) ⬇️
jina/flow/base.py 88.28% <0.00%> (-0.79%) ⬇️
jina/peapods/runtimes/zmq/zed.py 92.44% <0.00%> (+0.88%) ⬆️
jina/types/routing/table.py 93.39% <0.00%> (+0.94%) ⬆️
jina/logging/logger.py 94.17% <0.00%> (+0.97%) ⬆️
jina/jaml/helper.py 85.00% <0.00%> (+2.50%) ⬆️
jina/peapods/runtimes/gateway/http/app.py 93.05% <0.00%> (+2.77%) ⬆️
jina/types/ndarray/sparse/__init__.py 100.00% <0.00%> (+3.57%) ⬆️
jina/jaml/__init__.py 95.09% <0.00%> (+3.92%) ⬆️
jina/peapods/runtimes/jinad/client.py 90.53% <0.00%> (+4.14%) ⬆️
... and 33 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65e7ec1...ac2d45f. Read the comment docs.

@jacobowitz
Copy link
Contributor Author

hangs again :( https://github.com/jina-ai/jina/pull/2824/checks?check_run_id=2960786051

Will add the sleep, but try again reproducing locally as well

@jacobowitz
Copy link
Contributor Author

hangs again :( https://github.com/jina-ai/jina/pull/2824/checks?check_run_id=2960786051

Will add the sleep, but try again reproducing locally as well

actually we may use an old docker image here due to 7978a39

I will rebase the other pr and check what happens

@jacobowitz
Copy link
Contributor Author

hangs again :( https://github.com/jina-ai/jina/pull/2824/checks?check_run_id=2960786051
Will add the sleep, but try again reproducing locally as well

actually we may use an old docker image here due to 7978a39

I will rebase the other pr and check what happens

passing: https://github.com/jina-ai/jina/pull/2799/checks?check_run_id=2960905711

@jacobowitz jacobowitz marked this pull request as ready for review July 1, 2021 09:14
@jacobowitz jacobowitz requested a review from a team as a code owner July 1, 2021 09:14
@jacobowitz
Copy link
Contributor Author 8000

this should be merged, despite the failing tests:

  • test_common fails because we test against old jinad version due to ci bug (fixed in fix(daemon): fix flexible root workspace mount in docker #2799). I reran ci with the fixed version in the other PR and it fixed the bug we are fixing here.
  • the rolling_update integration test failure is some other flaky test failing often in ci, not related to this change (we should fix that one as well though)

@jacobowitz jacobowitz requested review from hanxiao and nan-wang and removed request for maximilianwerk and Kelton8Z July 1, 2021 09:43
@hanxiao hanxiao merged commit e7b7c83 into master Jul 1, 2021
@hanxiao hanxiao deleted the fix-hanging-pea-exit branch July 1, 2021 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/core This issue/PR affects the core codebase area/network This issue/PR affects network functionality component/peapod size/XS
Projects
None yet
3D98
Development

Successfully merging this pull request may close these issues.

4 participants
0