8000 fix(importer): add custom sys meta path finder by numb3r3 · Pull Request #2953 · jina-ai/serve · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

fix(importer): add custom sys meta path finder #2953

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 27, 2021
Merged

fix(importer): add custom sys meta path finder #2953

merged 2 commits into from
Jul 27, 2021

Conversation

numb3r3
Copy link
Member
@numb3r3 numb3r3 commented Jul 15, 2021 •
edited
Loading

This PR aims at fixing the (internal) extra dependency issues in our hub .

For instance, base on the executor from https://github.com/jina-ai/executor-audio-VGGishEncoder/tree/fix-module-not-found , i found some weird observations.

from jina import Executor, Flow
from jina.executors import BaseExecutor
import threading

# # => case1: works
# from vggish_audio_encoder import VggishAudioEncoder
# encoder = VggishAudioEncoder()

# # => case2: works
# def run():
#     encoder = BaseExecutor.load_config('config.yml')
# threading.Thread(target=run).start()

# # => case3: works
encoder = BaseExecutor.load_config('config.yml')

# # => case4: works
# f = Flow().add(uses='config.yml')
# with f:
#     f.block()

All of the use cases above work perfectly. However, if we want to use jina executor/pod/pea cli to load executor, we will get errors:

jina executor --uses config.yml   # => Failed, throw exception: `vggish` module cannot be found

@jina-bot jina-bot added size/S area/core This issue/PR affects the core codebase area/helper This issue/PR affects the helper functionality component/jaml labels Jul 15, 2021
@codecov
Copy link
codecov bot commented Jul 15, 2021 •
edited
Loading

Codecov Report

Merging #2953 (0c13339) into master (06d2f2e) will decrease coverage by 0.07%.
The diff coverage is 81.25%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2953      +/-   ##
==========================================
- Coverage   90.03%   89.96%   -0.08%     
==========================================
  Files         138      138              
  Lines        9513     9545      +32     
==========================================
+ Hits         8565     8587      +22     
- Misses        948      958      +10
Flag Coverage Δ
daemon 43.71% <46.87%> (+0.01%) ⬆️
jina 89.95% <81.25%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
jina/importer.py 84.90% <76.00%> (-2.75%) ⬇️
jina/jaml/__init__.py 95.26% <100.00%> (+0.16%) ⬆️
jina/clients/base/grpc.py 63.82% <0.00%> (-6.39%) ⬇️
jina/peapods/runtimes/gateway/prefetch.py 93.15% <0.00%> (-1.37%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06d2f2e...0c13339. Read the comment docs.

@github-actions
Copy link
github-actions bot commented Jul 15, 2021 •
edited
Loading

Latency summary

Current PR yields:

  • 😶 index QPS at 2014, delta to last 1 avg.: +2%
  • 😶 query QPS at 25, delta to last 1 avg.: -1%
  • 😶 dam extend QPS at 77203, delta to last 1 avg.: +1%
  • 😶 avg flow time within 1.1653 seconds, delta to last 1 avg.: +0%
  • 😶 import jina within 0.3011 seconds, delta to last 1 avg.: +2%

Breakdown

Version Index QPS Query QPS DAM Extend QPS Avg Flow Time (s) Import Time (s)
current 2014 25 77203 1.1653 0.3011
2.0.7 1972 25 76383 1.1622 0.2949

Backed by latency-tracking. Further commits will update this comment.

JoanFM
JoanFM reviewed Jul 15, 2021
View reviewed changes
jina/importer.py Outdated
_loader = ModuleLoader()


class PathFinder:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be better to rename this to JinaExecPathFinder, according to https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer_meta_path_finders.html there is a PathFinder class involved and could become confusing?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for your suggestions. I have renamed it as ExtensionPathFinder

@numb3r3 numb3r3 force-pushed the fix-internal-imports branch from 4d02e07 to 0c13339 Compare July 16, 2021 04:10
@numb3r3 numb3r3 requested a review from JoanFM July 16, 2021 10:46
@numb3r3 numb3r3 changed the title fix(importer): draft a path finder fix(importer): a custom sys meta path finder Jul 21, 2021
@numb3r3 numb3r3 changed the title fix(importer): a custom sys meta path finder fix(importer): add custom sys meta path finder Jul 21, 2021
@numb3r3 numb3r3 requested a review from hanxiao July 22, 2021 08:36
winstonww
winstonww approved these changes Jul 26, 2021
View reviewed changes
Copy link
Contributor
@winstonww winstonww left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just a quick question for my understanding.

jina/importer.py
load_path = module.__spec__.origin
init_file = '__init__.py'
if load_path.endswith(init_file):
module.__path__ = [load_path[: -len(init_file) - 1]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my understanding, what's the use case for setting the module.__path__ variable?

@hanxiao hanxiao marked this pull request as ready for review July 27, 2021 07:10
@hanxiao hanxiao requested a review from a team as a code owner July 27, 2021 07:10
@hanxiao hanxiao merged commit 75c28fc into master Jul 27, 2021
@hanxiao hanxiao deleted the fix-internal-imports branch July 27, 2021 07:14
@tadejsv tadejsv mentioned this pull request Aug 10, 2021
Closed
@tadejsv tadejsv mentioned this pull request Aug 18, 2021
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@winstonww winstonww winstonww approved these changes

@nan-wang nan-wang Awaiting requested review from nan-wang

@cristianmtr cristianmtr Awaiting requested review from cristianmtr

@JoanFM JoanFM Awaiting requested review from JoanFM

@hanxiao hanxiao Awaiting requested review from hanxiao

Assignees
No one assigned
Labels
area/core This issue/PR affects the core codebase area/helper This issue/PR affects the helper functionality component/jaml size/S
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@numb3r3 @winstonww @JoanFM @hanxiao @jina-bot
0