Introduce Docker to ease local development #659
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Prepare all the python versions tox wants to use using pyenv. These were sourced by the latest Travis builds * Match Travis mongo version * Provide an ENTRYPOINT that will ensure MongoDB is running pylint and py34 commands still failing.
py34 test execution would fail due to a UnicodeDecodeError exception being raised by the coverage package from a previous python versions run. Per the nose docs, the `--cover-erase` option will "erase previously collected coverage statistics before run" https://nose.readthedocs.io/en/latest/plugins/cover.html#cmdoption-cover-erase No Travis impact is expected, as no coverage statistics would exist in that run context. Full traceback: ```python py34-pymongo-pyexecjs run-test: commands[0] | nosetests -x -s --with-coverage --cover-package=mongomock --show-skipped Traceback (most recent call last): File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/coverage/data.py", line 293, in read_file self.read_fileobj(f) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/coverage/data.py", line 271, in read_fileobj data = self._read_raw_data(file_obj) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/coverage/data.py", line 311, in _read_raw_data go_away = file_obj.read(len(cls._GO_AWAY)) File "/root/.pyenv/versions/3.4.10/lib/python3.4/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 98: ordinal not in range(128) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mongomock/.tox/py34-pymongo-pyexecjs/bin/nosetests", line 10, in <module> sys.exit(run_exit()) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/nose/core.py", line 121, in __init__ **extra_args) File "/root/.pyenv/versions/3.4.10/lib/python3.4/unittest/main.py", line 92, in __init__ self.parseArgs(argv) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/nose/core.py", line 145, in parseArgs self.config.configure(argv, doc=self.usage()) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/nose/config.py", line 346, in configure self.plugins.configure(options, self) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/nose/plugins/manager.py", line 284, in configure cfg(options, config) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/nose/plugins/manager.py", line 99, in __call__ return self.call(*arg, **kw) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/nose/plugins/manager.py", line 167, in simple result = meth(*arg, **kw) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/nose/plugins/cover.py", line 155, in configure self.coverInstance.load() File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/coverage/control.py", line 677, in load self.data_files.read(self.data) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/coverage/data.py", line 653, in read data.read_file(self.filename) File "/mongomock/.tox/py34-pymongo-pyexecjs/lib/python3.4/site-packages/coverage/data.py", line 297, in read_file filename, exc.__class__.__name__, exc, coverage.misc.CoverageException: Couldn't read data from '/mongomock/.coverage': UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 98: ordinal not in range(128) ```
The container would use python3.4 by default which would generate the following error: ```python pylint run-test: commands[0] | pylint --rcfile=tox.ini --load-plugins pylint_quotes mongomock tests ************* Module mongomock.gridfs mongomock/gridfs.py:13:0: E0012: Bad option value 'import-outside-toplevel' (bad-option-value) mongomock/gridfs.py:20:0: E0012: Bad option value 'import-outside-toplevel' (bad-option-value) ----------------------------------- Your code has been rated at 9.99/10 ``` Using python3.6 instead, returns the code rating back to "10.00/10"!
Codecov Report
@@ Coverage Diff @@
## develop #659 +/- ##
===========================================
+ Coverage 95.28% 95.30% +0.02%
===========================================
Files 18 18
Lines 3395 3410 +15
===========================================
+ Hits 3235 3250 +15
Misses 160 160
Continue to review full report at Co 8000 decov.
|
pcorpet
requested changes
Aug 18, 2020
There was a problem hiding this comment.
Thanks a lot for this! I should admit I have a local Dockerfile doing half of what you're proposing and was thinking about committing it for a while.
I have a few comments inline, and have also some general comments:
- please add something to the README to document how to use Docker
- I think we could add a CMD in the Dockerfile to launch tox by default
- the documentation or the default CMD should allow easily to launch both tox for all envs as well as for only one env especially the default one (python 3.8.5 with pymongo and pyexecjs)
- add your name to the list of contributors if you're OK with that
- I'm OK for you to add the docker-compose.yml file now already (it makes mounting the local volume way easier)
Again, thanks a lot, this is super useful.
|
Given what I found here, I'm going to start on a side branch off this of the docker-compose version...this way I get it out of my head and something tangible for you to review/play with. I see a couple of options:
I'll wait to add documentation about this stuff once we land somewhere a littler firmer, but I believe I've addressed the rest of your comments. |
Let's do that. We're trying to ease the local development and remove the need of installing a trillion stuff on the machine. Asking for docker-compose in addition to docker is really small and I'm totally for that option. |
* Move .travis.yml comment about keeping versions in sync for docker * Update README with some notes about docker-compose usage
Alright - here we are in docker-compose land!
juannyG
commented
Aug 18, 2020
| @@ -30,7 +30,7 @@ class BulkOperationsTest(TestCase): | |||
| def setUp(self): | |||
| super(BulkOperationsTest, self).setUp() | |||
| if self.test_with_pymongo: | |||
| self.client = pymongo.MongoClient() | |||
| self.client = pymongo.MongoClient(host=os.environ.get('TEST_MONGO_HOST', 'localhost')) | |||
There was a problem hiding this comment.
All of these host definitions are the cost of defining mongo as another service in docker-compose: it no longer lives on localhost in this setup.
Benefit: locally, we can define the version we want to use without having to worry about installation gotchas like path or tarball file structure changes.
pcorpet
approved these changes
Aug 19, 2020
|
My pleasure! |
* Add comments for keeping OS+distro in sync between Dockerfile + travis * Update README to use latest and greatest in docker-compose example * No need to expose mongo ports to host in docker-compose * Remove default Dockerfile command
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While starting local development on a feature, I found myself wrestling with certain dependencies that weren't explicitly clear to me (Mongo needs to be running for tests; what version? for example) - so I thought I might do some digging in the recent Travis builds to try and create a Docker image to help make my experience more portable - and also give me a better idea as to how Travis will behave when it sees my changes 😄
The Dockerfile itself uses pyenv to prime all the python versions
toxneeds and also prepares the desired MongoDB installation (in this case 4.1.1).This is a completely optional development route and should have no impact on any active development environments (Travis or otherwise).
The only "gotchas" I encountered have to do with the second and third commits:
--cover-eraseto test commands which removes coverage statistics from previous runs. This was due to thepy34run experiencing a UnicodeDecodeError from a previous run's.coverageartifact. This shouldn't effect Travis, as all the version executions are isolated from one another - so this should only affect local environments.pylintcommand to python3.6 to ensure the score remained a 10.00/10. python3.6 matches the python version Travis uses for thepylintcommand. For some reason, the container would use python3.4 by default, which, because of its deprecation notice, caused the score to go down to a 9.99/10To use after cloning, one can simply enter the directory containing the
Dockerfile, build, and run!FYI: If this is deemed acceptable, I'll probably do a follow up PR with another secondary optional route to use
docker-compose- this would require basically everything of what is here anyway, but moved around, for cleaner organization. This would allow for a docker only setup, or a docker-compose setup, providing even more flexibility.