8000 fix: Support platform requirements in generated requirements files by danfairs · Pull Request #2302 · replicate/cog · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

fix: Support platform requirements in generated requirements files #2302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

danfairs
Copy link
Contributor
@danfairs danfairs commented May 7, 2025

This PR allows environment markers from user requirements files to be propagated into the generated output requirements file.

It also lays the foundation for future support of --hash. These are currently removed, since their presence triggers the --require 10000 -hashes mode, and that would break existing builds.

This looks a fairly chunky PR, because we refactor the requirements processor to use a new PythonRequirement struct to capture everything we need to know about a requirement, rather than manipulating strings directly. This has allowed the generation of the requirements file itself to be separated from the version handling logic. I have also added a lot more tests.

I have taken care to maintain the ordering of requirements in the output, so that they continue to reflect the user's own requirements file (not to mention the test cases!). This bookkeeping adds a small amount of extra complexity.

While this change passes all tests, and my own image now builds, it would benefit from testing on others' images.

For example, I have a requirements.txt file - an extract of which looks like this:

python-dateutil==2.9.0.post0 \
    --hash=sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3 \
    --hash=sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427
pywin32==310 ; sys_platform == 'windows'
setuptools==79.0.1 ; python_full_version >= '3.12' \
    --hash=sha256:128ce7b8f33c3079fd1b067ecbb4051a66e8526e7b65f6cec075dfc650ddfa88 \
    --hash=sha256:e147c0549f27767ba362f9da434eab9c5dc0045d5304feb602a0af001089fc51

Here's the build output with --debug set. Note that both the platform requirements are preserved for the pywin32 package (and others), and the hashes are omitted:

Setting CUDA to version 12.4 from Torch version
Setting CuDNN to version 12.4
Building Docker image from environment in cog.yaml...
Generated requirements.txt:
--extra-index-url https://download.pytorch.org/whl/cu124
contourpy==1.3.2
cycler==0.12.1
filelock==3.18.0
fonttools==4.57.0
fsspec==2025.3.2
jinja2==3.1.6
kiwisolver==1.4.8
markupsafe==3.0.2
matplotlib==3.10.1
mpmath==1.3.0
networkx==3.4.2
numpy==2.2.5
nvidia-cublas-cu12==12.4.5.8 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-cuda-cupti-cu12==12.4.127 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-cuda-nvrtc-cu12==12.4.127 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-cuda-runtime-cu12==12.4.127 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-cudnn-cu12==9.1.0.70 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-cufft-cu12==11.2.1.3 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-curand-cu12==10.3.5.147 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-cusolver-cu12==11.6.1.9 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-cusparse-cu12==12.3.1.170 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-nccl-cu12==2.21.5 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-nvjitlink-cu12==12.4.127 ; platform_machine == 'x86_64' and sys_platform == 'linux'
nvidia-nvtx-cu12==12.4.127 ; platform_machine == 'x86_64' and sys_platform == 'linux'
opencv-python==4.11.0.86
packaging==25.0
pillow==11.2.1
pyparsing==3.2.3
python-dateutil==2.9.0.post0
pywin32==310 ; sys_platform == 'windows'
setuptools==79.0.1 ; python_full_version >= '3.12'
six==1.17.0
sympy==1.13.1
torch==2.5.1
triton==3.1.0 ; python_full_version < '3.13' and platform_machine == 'x86_64' and sys_platform == 'linux'
typing-extensions==4.13.2
$ docker buildx build --provenance false --platform linux/amd64 --load --cache-to type=inline --file - --tag cog-condense-sam2-base --progress auto .
[+] Building 26.6s (16/16) FINISHED                                                                                                                                                                                     docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => => transferring dockerfile: 1.07kB                                                                                                                                                                                                  0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1.4
... etc ...

This fixes #2078.

If there are any standards/expectations around individual commits, let me know. I'm used to squash-and-merge, and this PR might be a bit messy if you just use merge commits.

danfairs added 12 commits May 7, 2025 09:08
Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
Unfortunately the presence of a single hash triggers the strict —require-hashes
mode, which may break a lot of builds if users are unprepared for this.

Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
@danfairs danfairs marked this pull request as ready for review May 7, 2025 15:19
@danfairs danfairs marked this pull request as draft May 7, 2025 15:34
danfairs added 2 commits May 7, 2025 16:36
Signed-off-by: Dan Fairs <dan@condensereality.com>
Signed-off-by: Dan Fairs <dan@condensereality.com>
@danfairs danfairs marked this pull request as ready for review May 7, 2025 15:46
Signed-off-by: Dan Fairs <dan@condensereality.com>
@danfairs
Copy link
Contributor Author
danfairs commented May 9, 2025

@allcontributors please add @danfairs for code

Copy link
Contributor

@danfairs

I've put up a pull request to add @danfairs! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

python_requirements environment constraints
1 participant
0