8000 Dockerfiles Python 3 cleanup by nicknezis · Pull Request #3601 · apache/incubator-heron · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
This repository was archived by the owner on Mar 3, 2023. It is now read-only.

Dockerfiles Python 3 cleanup #3601

Merged
merged 11 commits into from
Aug 21, 2020
Merged

Dockerfiles Python 3 cleanup #3601

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6c42a34
Initial commit
nicknezis Aug 6, 2020
bc558e2
Updates to Dockerfiles
nicknezis Aug 7, 2020
3cc27c6
Updating minimum version of supported Python
nicknezis Aug 7, 2020
da5af49
More updates to use venv
nicknezis Aug 9, 2020
d73fbc7
More Docker image updates
nicknezis Aug 9, 2020
c8ce25f
Update to use newer venv instead of virutalenv
nicknezis Aug 9, 2020
1f16afe
Updates to travis.yml
nicknezis Aug 9, 2020
d87c59d
Removing commented out line
nicknezis Aug 10, 2020
35f49c7
Update bazel_configure.py
nicknezis Aug 10, 2020
165ca46
Adding venv test
nicknezis Aug 16, 2020
0aa5b10
Removing obsolete Dockerfiles
nicknezis Aug 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,9 @@ addons:
packages:
- libtool-bin
- libcppunit-dev
- python3
- pkg-config
- python3-dev
- python3-wheel
- python3-venv
- wget
- zip
- zlib1g-dev
Expand Down
19 changes: 0 additions & 19 deletions WORKSPACE
8000
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,10 +181,6 @@ REQUESTS_SRC = "https://pypi.python.org/packages/d9/03/155b3e67fe35fe5b6f4227a8d

SETUPTOOLS_WHEEL = "https://pypi.python.org/packages/a0/df/635cdb901ee4a8a42ec68e480c49f85f4c59e8816effbf57d9e6ee8b3588/setuptools-46.1.3-py3-none-any.whl"

VIRTUALENV_SRC = "https://pypi.python.org/packages/d4/0c/9840c08189e030873387a73b90ada981885010dd9aea134d6de30cd24cb8/virtualenv-15.1.0.tar.gz"

VIRTUALENV_PREFIX = "virtualenv-15.1.0"

WHEEL_SRC = "https://pypi.python.org/packages/c9/1d/bd19e691fd4cfe908c76c429fe6e4436c9e83583c4414b54f6c85471954a/wheel-0.29.0.tar.gz"

http_file(
Expand Down Expand Up @@ -229,21 +225,6 @@ http_file(
urls = [SETUPTOOLS_WHEEL],
)

http_archive(
name = "virtualenv",
build_file_content = "\n".join([
"load(\"@rules_python//python:defs.bzl\", \"py_binary\")",
"py_binary(",
" name = 'virtualenv',",
" srcs = ['virtualenv.py'],",
" data = glob(['**/*']),",
" visibility = ['//visibility:public'],",
")",
]),
sha256 = "02f8102c2436bb03b3ee6dede1919d1dac8a427541652e5ec95171ec8adbc93a",
strip_prefix = VIRTUALENV_PREFIX,
urls = [VIRTUALENV_SRC],
)
# end pex repos

# protobuf dependencies for C++ and Java
Expand Down
9 changes: 8 additions & 1 deletion bazel_configure.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
import datetime
import platform
import subprocess
import tempfile

sys.path.append('third_party/python/semver')
import semver
Expand Down Expand Up @@ -293,6 +294,11 @@ def discover_jdk():
print('Using %s:\t%s' % ('JDK'.ljust(20), jdk_path))
return jdk_path

def test_venv():
with tempfile.TemporaryDirectory() as tmpdirname:
if subprocess.run(["python3", "-m", "venv", tmpdirname]).returncode != 0:
fail("Python3 venv module is not installed.")

######################################################################
# Discover the linker directory
######################################################################
Expand Down Expand Up @@ -413,7 +419,8 @@ def main():
env_map['AUTOMAKE'] = discover_tool('automake', 'Automake', 'AUTOMAKE', '1.9.6')
env_map['AUTOCONF'] = discover_tool('autoconf', 'Autoconf', 'AUTOCONF', '2.6.3')
env_map['MAKE'] = discover_tool('make', 'Make', 'MAKE', '3.81')
env_map['PYTHON3'] = discover_tool('python3', 'Python3', 'PYTHON3', '3.4')
env_map['PYTHON3'] = discover_tool('python3', 'Python3', 'PYTHON3', '3.6')
test_venv()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should probably check that python -m venv returns non-zero (the version will be same as the python interpreter)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a suggestion of how best to do this? Seems the discover_tool method wouldn't work for what you describe, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might take a custom test, which I think there's already some of. I'm not sure if there is a nice portable test for venv other than something like:

import subprocess, tempfile
def test_venv():
    with tempfile.TemporaryDirectory() as tmpdirname:
        return subprocess.run(["python3", "-mvenv", tmdpirname]).returncode == 0

The reason I think this is because I suspect there will be difference with how distribution maintainers can choose how this is included, and trying to cater for nuances doesn't sound good.

trivia: Debian make venv and pip separate packages, whereas the cpython codebase can let you install venv along with python, and then pip will be created using vendoered version that ensurepip unfurls (debian patches this out).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the test function. Let me know if it looks good.

if platform == 'Darwin':
env_map['LIBTOOL'] = discover_tool('glibtool', 'Libtool', 'LIBTOOL', '2.4.2')
Expand Down
1 change: 0 additions & 1 deletion docker/compile/Dockerfile.centos7
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ RUN yum -y install \
libtool \
make \
patch \
python \
python3-devel \
zip \
unzip \
Expand Down
57 changes: 0 additions & 57 deletions docker/compile/Dockerfile.ubuntu14.04
View file
Open in desktop

This file was deleted.

60 changes: 0 additions & 60 deletions docker/compile/Dockerfile.ubuntu16.04
View file
Open in desktop

This file was deleted.

5 changes: 3 additions & 2 deletions docker/compile/Dockerfile.ubuntu18.04
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,19 @@ RUN apt-get update && apt-get -y install \
libunwind8 \
libcppunit-dev \
patch \
python \
python3-dev \
python3-venv \
pkg-config \
wget \
zip \
virtualenv \
unzip \
git \
curl \
tree \
openjdk-11-jdk-headless

RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10

ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64

ENV bazelVersion 3.4.1
Expand Down
4 changes: 2 additions & 2 deletions docker/compile/Dockerfile.ubuntu20.04
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,12 @@ RUN apt-get update && apt-get -y install \
libunwind8 \
libcppunit-dev \
patch \
python-is-python3 \
python3-dev \
python \
python3-venv \
pkg-config \
wget \
zip \
virtualenv \
unzip \
git \
curl \
Expand Down
4 changes: 3 additions & 1 deletion docker/test/Dockerfile.ubuntu18.04
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,17 @@ RUN apt-get update && apt-get -y install \
libcppunit-dev \
patch \
python3-dev \
python3-venv \
wget \
zip \
virtualenv \
unzip \
git \
curl \
tree \
openjdk-11-jdk-headless

RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is linking python to python3 needed - was it that transient dependency which needed it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got this error:

ERROR: An error occurred during the fetch of repository 'pip_deps':
   pip_import failed:  (src/main/tools/process-wrapper-legacy.cc:75: "execvp(python, ...)": No such file or directory
)
ERROR: no such package '@pip_deps//': pip_import failed:  (src/main/tools/process-wrapper-legacy.cc:75: "execvp(python, ...)": No such file or directory
)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes it sound like bazel (that .cc is a part of it) is trying to run python on the system, rather than python3 for some reason, which would be a preexisting issue. Maybe it's because test --host_force_python=PY3 isn't in tools/bazel.rc?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just tested with the following, but it still fails:

test --genrule_strategy=standalone
test --host_force_python=PY3
test --ignore_unsupported_sandboxing
test --spawn_strategy=standalone

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be related to this line. Maybe we don't need it? Testing it now with it removed.

incubator-heron/WORKSPACE

Line 398 in ee73293

pip_deps()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it's needed for some of the tar and docker rules.


ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64

ENV bazelVersion 3.4.1
Expand Down
6 changes: 3 additions & 3 deletions scripts/packages/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -604,8 +604,9 @@ genrule(
"export RELEASE_FILE_DIR=$$(pwd)",
"export TMP_DIR=$$(mktemp -d -t heronpy.XXXXX)",
"echo $$TMP_DIR",
"$(location @virtualenv//:virtualenv) --no-download --quiet --clear $$TMP_DIR/venv",
"PS1= source $$TMP_DIR/venv/bin/activate",
# Create the virtual environment
'python3 -m venv $$TMP_DIR/venv --clear',
"VIRTUAL_ENV_DISABLE_PROMPT=1 source $$TMP_DIR/venv/bin/activate",
"export HERONPY_DIR=$$TMP_DIR/heronpy",
"export HERONPYAPI_UNZIP=$$TMP_DIR/heronapiunzip",
"export HERONPYSTREAMLET_UNZIP=$$TMP_DIR/heronstreamletunzip",
Expand Down Expand Up @@ -661,7 +662,6 @@ genrule(
"//heronpy/connectors:heron-pythonconnectors-package",
"//heronpy/proto:proto-py-package",
"//heronpy/streamlet:heron-python-streamlet-api-package",
"@virtualenv",
],
)

Expand Down
8 changes: 3 additions & 5 deletions tools/rules/pex/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,9 @@ LINUX_EXECUTE = [
POST_EXECUTE = [
'ln -sf "$$OUTDIR" "$$TMPF"',
'VENV="$${TMPF}/venv"',
'$(location @virtualenv//:virtualenv) --no-download --quiet --clear "$$VENV"',
'# this is just to keep the activate script happy',
'PS1=',
'source "$$VENV/bin/activate"',
# Create the virtual environment
'python3 -m venv $$VENV --clear',
'VIRTUAL_ENV_DISABLE_PROMPT=1 source "$$VENV/bin/activate"',

'pip install pex \
--quiet --no-cache-dir --no-index --build $(@D)/pexbuild \
Expand Down Expand Up @@ -70,6 +69,5 @@ genrule(
executable = True,
message = "Bootstrapping pex",
output_to_bindir = True,
tools = ["@virtualenv"],
visibility = ["//visibility:public"],
)
2 changes: 1 addition & 1 deletion website2/website/scripts/python-doc-gen.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ VENV=./venv/
echo $VENV
PIP_LOCATION=${HERON_ROOT_DIR}/bazel-bin/scripts/packages

virtualenv "$VENV"
python3 -m venv "$VENV"
source "$VENV/bin/activate"
pip3 install pdoc==0.3.2
pip3 install --ignore-installed six
Expand Down
0