8000 [BR]: fail2ban relies on deprecated, soon-to-be-removed setuptools install functionality · Issue #3999 · fail2ban/fail2ban · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[BR]: fail2ban relies on deprecated, soon-to-be-removed setuptools install functionality #3999

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
2 tasks done
thesamesam opened this issue May 1, 2025 · 10 comments
Open
2 tasks done

[BR]: fail2ban relies on deprecated, soon-to-be-removed setuptools install functionality #3999

thesamesam opened this issue May 1, 2025 · 10 comments
Labels
3rd party issue

Comments

@thesamesam
Copy link
thesamesam commented May 1, 2025 •
edited
Loading

Environment:

  • Fail2Ban version : 1.1.0, master
  • OS, including release name/version : Gentoo Linux
  • Fail2Ban installed via OS/distribution mechanisms
  • You have not applied any additional foreign patches to the codebase

The issue:

fail2ban currently relies on setuptools setup.py install, but this has been marked deprecated for quite some time. In setuptools-80.1.0, they've finally set a removal date for 2025-10-31.

Steps to reproduce

/tmp/fail2ban $ python setup.py install --prefix="/usr" --root="/tmp/foo"
/usr/lib/python3.13/site-packages/setuptools/_distutils/dist.py:289: UserWarning: Unknown distribution option: 'test_suite'
  warnings.warn(msg)
running install
/tmp/fail2ban/setup.py:111: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  install.initialize_options(self)
[...]

After 2025-10-31, that deprecation warning will become fatal, and setup.py install won't work at all.

In summary, the Python ecosystem decided that the setup.py install way of doing things was too haphazard. There are two models now:

  1. Installing software via PEP517, which has restrictions on the installation of data files and other auxiliary files, and is intended for software that is meant to be imported as a library, or a Python tool with little-to-no configuration or extra files;
  2. Installing software via a real build system like Meson, that happens to just be installing software written in Python.

A lot of software written in Python was relying on setup.py install because it seemed like the right thing to do, but there's no reason that software just written in Python that isn't intended to be imported or must live in site-packages has to be installed by the Python packaging setup (especially if it's not intended to be installed via pip or on pypi at all).

I personally recommend Meson for this case.

Expected behavior

No deprecation warnings.

Observed behavior

Deprecation warnings indicating imminent breakage.

Any additional information

See also #3361, pypa/setuptools#2088 (comment), and pypa/packaging-problems#576 (comment)
@thesamesam thesamesam added the bug label May 1, 2025
@sebres
Copy link
Contributor
sebres commented May 2, 2025 •
edited
Loading

You seem to misunderstand the deprecation: it is not the setuptools what is deprecated, but setup.py install method.
See Python Packaging :: Should setup.py be deleted?.
With other words, one have to use python 8000 -m pip install . instead of python setup.py install.

Particularly if I try this on my box with newest setuptools, I don't see any warning and it simulates the installation without any issue:

sudo python3 -m pip install --dry-run --break-system-packages --report /tmp/inst-report.txt .

Also note that the packaging is rather a matter of distribution maintainers and their common mechanisms or tools used by distribution.
For instance debian maintainers (and we when building our releases) use pybuild/dh to build fail2ban. Other distros may use other stuff to build it.

We have surely to adjust the README and the wiki for "new" syntax of manual install, but the assumption "fail2ban relies on deprecated, soon-to-be-removed setuptools install functionality" is not correct.

@sebres sebres added 3rd party issue and removed bug labels May 2, 2025
@eli-schwartz
Copy link
Contributor

Hello.

Please show me the contents of your systemd service file that is installed by pip install.

@thesamesam
Copy link
Author
thesamesam commented May 2, 2025 •
edited
Loading

I haven't confused them. Please see the links I gave and my description above. The wheel format can't actually handle auxiliary data files like configuration in /etc correctly at all. Using pip install will use setuptools-via-PEP517.

Also note that the packaging is rather a matter of distribution maintainers and their common mechanisms or tools used by distribution.

I am such a distribution maintainer, and I'm reaching out to explain the problem. Right now, fail2ban's build system requires manual hacks (which every distribution will have to do) to handle the files fail2ban needs outside of site-packages.

@sebres
Copy link
Contributor
sebres commented May 2, 2025

Please show me the contents of your systemd service file that is installed by pip install.

We don't install it (and never did it) by manual install - as the README and wiki say, the build folder would contain fail2ban.service which could be then copied (if needed) to /etc/systemd/system/.
Or do you mean the new build facilities would not generate fail2ban.service from fail2ban.service.in?
Although the task is and was always the task of maintained packages (not of common manual install).

Right now, fail2ban's build system requires manual hacks (which every distribution will have to do) to handle the files fail2ban needs outside of site-packages

Doesn't gentoo have build tools like pybuild/dh? Or is that mentioned Meson such a tool?

And are this "manual hacks" not just few rules for packaging tool to package paths like this?

./config ==> /etc/fail2ban
./build/scripts ==> /usr/bin/

Anyway, why don't you provide your build scripts (Meson or whatever), so other maintainers would not require manual hacks?
Welcome with a PR...

@eli-schwartz
Copy link
Contributor

As was mentioned by the setuptools maintainer on the linked ticket:

I'd really like to know why a drop-in replacement for setup.py install using PEP 517 isn't viable today.

The issue is miscellaneous packages that were relying on setup.py install kind of "just because this thing is written in Python" and install auxiliary data files, as mentioned in #2088 (comment). fail2ban is an example, cloud-init is another.

I ran build -w on fail2ban and it built a wheel.

Is that wheel not viable for installation? I can see it has usr/share/doc and etc/fail2ban, which would require special handling by a distributor's installer, but it sure seems close to being drop-in replacement.

My reply there was very specific about the issues which exist in practice, for fail2ban itself:

fail2ban creates build/fail2ban.service, which already needs to be manually installed as it isn't included in a wheel nor even copied over as a data_files. Really it should have, historically, been in data_files so that it would be installed to /usr/lib/systemd/system/. But if it did, that wouldn't help install from a wheel, because... here are the contents of the file when using python -m build:

[Unit]
Description=Fail2Ban Service
Documentation=man:fail2ban(1)
After=network.target iptables.service firewalld.service ip6tables.service ipset.service nftables.service
PartOf=iptables.service firewalld.service ip6tables.service ipset.service nftables.service

[Service]
Type=simple
Environment="PYTHONNOUSERSITE=1"
ExecStartPre=/bin/mkdir -p /run/fail2ban
ExecStart=build/bdist.linux-x86_64/wheel/fail2ban-1.1.1.dev1.data/scripts/fail2ban-server -xf start
# if should be logged in systemd journal, use following line or set logtarget to sysout in fail2ban.local
# ExecStart=build/bdist.linux-x86_64/wheel/fail2ban-1.1.1.dev1.data/scripts/fail2ban-server -xf --logtarget=sysout start
ExecStop=build/bdist.linux-x86_64/wheel/fail2ban-1.1.1.dev1.data/scripts/fail2ban-client stop
ExecReload=build/bdist.linux-x86_64/wheel/fail2ban-1.1.1.dev1.data/scripts/fail2ban-client reload
PIDFile=/run/fail2ban/fail2ban.pid
Restart=on-failure
RestartPreventExitStatus=0 255

[Install]
WantedBy=multi-user.target

Those paths are totally wrong, and are normally deduced by python setup.py install --prefix=/usr. That needs replacing here.

You can see this buried in the logs that you pasted:

running install_scripts
creating build/bdist.linux-aarch64/wheel/fail2ban-1.1.1.dev1.data/scripts
copying build/scripts-3.13/fail2ban-server -> build/bdist.linux-aarch64/wheel/fail2ban-1.1.1.dev1.data/scripts
copying build/scripts-3.13/fail2ban-regex -> build/bdist.linux-aarch64/wheel/fail2ban-1.1.1.dev1.data/scripts
copying build/scripts-3.13/fail2ban-client -> build/bdist.linux-aarch64/wheel/fail2ban-1.1.1.dev1.data/scripts
copying build/scripts-3.13/fail2ban-testcases -> build/bdist.linux-aarch64/wheel/fail2ban-1.1.1.dev1.data/scripts
WARNING: Cannot find root-base option, check the bin-path to fail2ban-scripts in "fail2ban.service" and "fail2ban-openrc.init".
Creating build/fail2ban.service (from fail2ban.service.in): @BINDIR@ -> build/bdist.linux-aarch64/wheel/fail2ban-1.1.1.dev1.data/scripts
Creating build/fail2ban-openrc.init (from fail2ban-openrc.init.in): @BINDIR@ -> build/bdist.linux-aarch64/wheel/fail2ban-1.1.1.dev1.data/scripts
creating fail2ban-python binding -> build/bdist.linux-aarch64/wheel/fail2ban-1.1.1.dev1.data/scripts

If I run in legacy mode and pretend it's a Makefile:

$ python setup.py install --prefix=/usr --root=$PWD/pkg

running install_scripts
creating /tmp/fail2ban/pkg/usr/bin
copying build/scripts-3.12/fail2ban-testcases -> /tmp/fail2ban/pkg/usr/bin
copying build/scripts-3.12/fail2ban-regex -> /tmp/fail2ban/pkg/usr/bin
copying build/scripts-3.12/fail2ban-server -> /tmp/fail2ban/pkg/usr/bin
copying build/scripts-3.12/fail2ban-client -> /tmp/fail2ban/pkg/usr/bin
Creating build/fail2ban.service (from fail2ban.service.in): @BINDIR@ -> /usr/bin
Creating build/fail2ban-openrc.init (from fail2ban-openrc.init.in): @BINDIR@ -> /usr/bin
creating fail2ban-python binding -> /tmp/fail2ban/pkg/usr/bin

Note the paths are now /usr/bin

@eli-schwartz
Copy link
Contributor

So, using pip install produces wheels, which tells fail2ban to produce the wrong file contents for the systemd service and openrc init file.

They can't be used. Gentoo manually copies them over, but we also use setup.py install to make sure the file contents are correct.

@eli-schwartz
Copy link
Contributor

Note this problem cannot be solved by pip because pip now requires that you do NOT install software directly, but first build a generic "wheel" package with unfixed relative paths.

As discussed at pypa/packaging-problems#576 (comment) -- the python packaging community has officially deprecated and removed support for installing system integration. If your software doesn't install into a python virtualenv as standalone, it isn't a good fit for pip. If you have /etc files, you aren't supported. If you need to install GUI /usr/share/applications/*.desktop, you're not supported. If you need to install systemd services, those won't work in ~/.venvs/app_foo/lib/systemd/system/ so you are, once again, not supported.

Anyway, why don't you provide your build scripts (Meson or whatever), so other maintainers would not require manual hacks?
Welcome with a PR...

I may look into this soon, if nobody else beats me to it. :)

sebres added a commit that referenced this issue May 3, 2025
@sebres
setup.py: enhanced with 2 new commands, f2b-build and f2b-install…
2b27d52 
… (not yet ready) to fulfill the build and installation processes

if deprecated `install` method gets removed and pip (setuptools-via-PEP517) wouldn't install anything excepting library;
closes gh-3999
@sebres
Copy link
Contributor
sebres commented May 3, 2025 •
edited
Loading

How about this one?..
In branch fix-pythonic-build-install I extended setup.py with 2 new commands - build-ex and install-ex (not yet ready, only builds at the moment), that internally uses setuptools (build and egg_info) to build the library and hereafter create the fail2ban.service and fail2ban-openrc.init files, and copy the auxiliary data (/etc, /var, etc) to --build-base=... folder (default build in root of fail2ban).

For instance, to make build folder (very similar to deprecated install):

python3 setup.py build-ex --build-base=build --without-tests

(it is also covered in GHA now - see actions/runs/14814502138 for the output)

Use build-ex --help for more detailed usage with all options:

$ python3 setup.py build-ex --help
usage: setup.py comand [options]
Commands:
  build-ex        build the package underneath ./build (or --build-base)
  install-ex      will install the package
Options:
  --quiet (-q)    run quietly (turns verbosity off)
  --dry-run (-n)  don't actually do anything
  --help (-h)     show detailed help message
Options of 'install-ex' and 'build-ex' commands:
  --prefix=            installation prefix
  --build-base= (-b)   base directory for build (default ./build)
  --root=              install everything relative to this
                       alternate root directory
  --lib=               directory for library
  --bin=               build directory for binary
  --without-tests      don't enclose fail2ban test-suite

(--prefix and --root are not implemented at the moment, but new options --lib and --bin which act relative --build-base)

This doesn't rely on deprecated mechanisms anymore.

If it is OK, I'd fulfill the install-ex method for manual install and release it.

@sebres
Copy link
Contributor
sebres commented May 3, 2025

small amend: commands f2b-build and f2b-install renamed in build-ex and install-ex (comment above adjusted)

@sebres
Copy link
Contributor
sebres commented May 6, 2025

Option --prefix and command install-ex (with --root) are also implemented now:

python3 setup.py install-ex --root=./pkg --build-base=./build --prefix=/usr --without-tests

This would firstly build to folder "./build" (using build-ex command) and then install everything from ./build to given --root ("./pkg").
The files fail2ban.service and fail2ban-openrc.init remain as by original install only in folder "./build", but in opposite to deprecated install, they would be generated now on the phase of build-ex...
A 5CE6 nd because it also handles --prefix now, one could also use build-ex instead of install-ex now to prepare the package.

Anyway it looks completed to me now, reviews and tests are welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3rd party issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sebres @eli-schwartz @thesamesam
0