8000 ImportError when running with coverage measurement activated · Issue #6584 · pydantic/pydantic · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

ImportError when running with coverage measurement activated #6584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
1 task done
Mr-Pepe opened this issue Jul 11, 2023 · 4 comments · Fixed by pydantic/pydantic-core#1003
Closed
1 task done

ImportError when running with coverage measurement activated #6584

Mr-Pepe opened this issue Jul 11, 2023 · 4 comments · Fixed by pydantic/pydantic-core#1003
Assignees
@davidhewitt
Labels
bug V2 Bug related to Pydantic V2

Comments

@Mr-Pepe
Copy link
Contributor
Mr-Pepe commented Jul 11, 2023 •
edited by pydantic-hooky bot
Loading

Initial Checks

  • I confirm that I'm using Pydantic V2

Description

I upgraded a library (A) to Pydantic v2. Now I want to upgrade a second library (B) that uses A to Pydantic v2. However, pytest runs fail with:

.tox/py39-test/lib/python3.9/site-packages/library_a/schemas.py:10: in <module>
    import pydantic
.tox/py39-test/lib/python3.9/site-packages/pydantic/__init__.py:3: in <module>
    import pydantic_core
.tox/py39-test/lib/python3.9/site-packages/pydantic_core/__init__.py:6: in <module>
    from ._pydantic_core import (
E   ImportError: PyO3 modules may only be initialized once per interpreter process

The tests run fine if I run with the --no-cov flag.

I would like to produce a minimal example but I don't know where to start and the error message does not tell me anything. Library B imports some Pydantic models from library A in its conftest.py and the run crashes on importing B's conftest.py. Unit tests for library A work without problems and with activated coverage though.

How should I go about analyzing the issue? What raises the import error?

Example Code

No response

Python, Pydantic & OS Version

pydantic version: 2.0.2
pydantic-core version: 2.1.2 release build profile
python version: 3.9.13 (main, Jul 18 2022, 09:11:04)  [GCC 12.1.0]
platform: Linux-6.4.2-arch1-1-x86_64-with-glibc2.37
optional deps. installed: ['typing-extensions']

Selected Assignee: @dmontagu
@Mr-Pepe Mr-Pepe added bug V2 Bug related to Pydantic V2 unconfirmed labels Jul 11, 2023
@Mr-Pepe
Copy link
Contributor Author
Mr-Pepe commented Jul 12, 2023

I think I got a minimal example running. Would be nice if someone could confirm that they see the same behavior.

# src/my_package/__init__.py

import pydantic
# setup.cfg

[metadata]
name = my-package

[options]
packages = find:
package_dir =
    =src
install_requires =
    pydantic>=2.0
    pytest
    pytest-cov
    coverage[toml]

[options.packages.find]
where = src

# pyproject.toml

[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"

Create a virtual environment and run pip install -e . and pytest src --doctest-modules --cov=my_package.a.
Running with --cov=my_package works, but any sub-package (my_package.xxx, regardless of whether it exists or not) leads to the Pydantic error.

@dmontagu dmontagu assigned davidhewitt and unassigned dmontagu Jul 25, 2023
@dmontagu
Copy link
Contributor

@davidhewitt as the resident pyo3 expert do you have any idea what's going on?

@davidhewitt
Copy link
Contributor

The cause of the interaction with coverage seems to be caused by nedbat/coveragepy@2603597#diff-7ab8fe55706697d3772483ba055af46f5aacd86b110458a7cc824ae7e2324447R274

This is even documented as expected behaviour of coverage: https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst#version-602--2021-10-11

In general the rule is that PyO3 modules aren't safe be imported more than one, because they can contain C statics and this is particularly bad in interaction with subinterpreters. (Data can cross sub-interpreters or otherwise cause wonky state.) Maybe we can loosen this check (or give modules an opt-in way to disable it). Further thoughts welcome.

@davidhewitt davidhewitt mentioned this issue Sep 1, 2023
Closed
@tonal
Copy link
tonal commented Sep 4, 2023 
ImportError while loading conftest '/home/tonal/lang/projects/promsoft/epool/props_service/tests/conftest.py'.
conftest.py:13: in <module>
    from data_props import TreeConf, TreeManager
../data_props/__init__.py:3: in <module>
    from .data_props import *
../data_props/data_props.py:19: in <module>
    from pydantic import Field, validate_arguments
/home/tonal/.pyenv/versions/epool-props_service/lib/python3.11/site-packages/pydantic/__init__.py:3: in <module>
    import pydantic_core
/home/tonal/.pyenv/versions/epool-props_service/lib/python3.11/site-packages/pydantic_core/__init__.py:6: in <module>
    from ._pydantic_core import (
E   ImportError: PyO3 modules may only be initialized once per interpreter process

requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile
#
aiodns==3.0.0
    # via aiohttp
aiohttp[speedups]==3.8.5
    # via -r requirements.in
aiosignal==1.3.1
    # via aiohttp
annotated-types==0.5.0
    # via pydantic
anyio==4.0.0
    # via httpcore
async-timeout==4.0.3
    # via
    #   -r requirements.in
    #   aiohttp
attrs==23.1.0
    # via aiohttp
brotli==1.0.9
    # via aiohttp
certifi==2023.7.22
    # via
    #   httpcore
    #   httpx
    #   sentry-sdk
cffi==1.15.1
    # via pycares
charset-normalizer==3.2.0
    # via aiohttp
click==8.1.7
    # via -r requirements.in
cython==3.0.2
    # via -r requirements.in
dnspython==2.4.2
    # via pymongo
fastnumbers==5.0.1
    # via natsort
filelock==3.12.3
    # via -r requirements.in
fluent-logger==0.10.0
    # via -r requirements.in
frozenlist==1.4.0
    # via
    #   aiohttp
    #   aiosignal
h11==0.14.0
    # via httpcore
httpcore==0.17.3
    # via httpx
httpx==0.24.1
    # via -r requirements.in
idna==3.4
    # via
    #   anyio
    #   httpx
    #   yarl
more-itertools==10.1.0
    # via -r requirements.in
motor==3.3.1
    # via -r requirements.in
msgpack==1.0.5
    # via fluent-logger
multidict==6.0.4
    # via
    #   aiohttp
    #   yarl
mysqlclient==2.2.0
    # via -r requirements.in
natsort[fast,icu]==8.4.0
    # via -r requirements.in
numpy==1.25.2
    # via -r requirements.in
pycares==4.3.0
    # via aiodns
pycparser==2.21
    # via cffi
pydantic==2.3.0
    # via
    #   -r requirements.in
    #   pydantic-settings
pydantic-core==2.6.3
    # via pydantic
pydantic-settings==2.0.3
    # via -r requirements.in
pyicu==2.11
    # via natsort
pymongo[snappy,srv,zstd]==4.5.0
    # via
    #   -r requirements.in
    #   motor
python-dotenv==1.0.0
    # via pydantic-settings
python-snappy==0.6.1
    # via pymongo
pyyaml==6.0.1
    # via -r requirements.in
sentry-sdk==1.30.0
    # via -r requirements.in
sniffio==1.3.0
    # via
    #   anyio
    #   httpcore
    #   httpx
sqlparse==0.4.4
    # via -r requirements.in
typing-extensions==4.7.1
    # via
    #   pydantic
    #   pydantic-core
urllib3==2.0.4
    # via sentry-sdk
uvloop==0.17.0
    # via -r requirements.in
yarl==1.9.2
    # via aiohttp
zstandard==0.21.0
    # via pymongo

@davidhewitt davidhewitt mentioned this issue Sep 11, 2023
Merged
@davidhewitt davidhewitt mentioned this issue Oct 3, 2023
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davidhewitt davidhewitt

Labels
bug V2 Bug related to Pydantic V2
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update PyO3 to 0.20 pydantic/pydantic-core
4 participants
@tonal @davidhewitt @Mr-Pepe @dmontagu
0