8000 Add basic mercurial support by phlogistonjohn · Pull Request #197 · fsfe/reuse-tool · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add basic mercurial support #197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Apr 21, 2020

Conversation

phlogistonjohn
Copy link
Contributor

Fixes #99 as far as running the tool on a couple of repos of my own went.

I won't claim that this is comprehensive support for Mercurial as compared to the current git support but it got the tool working for me and generating sensible reports (vs checking my build artifacts). I did try to add some tests as far as code paths that seemed relevant to running hg versus git goes.

I could probably add a few more tests if needed but I am not sure what you would think are tests that really need a specific check when run using mercurial versus git.

@phlogistonjohn
Copy link
Contributor Author

The 'pypy3-test' failure appears to be due to the (rather old) version of mercurial that lacks the '--terse' flag. I'm curious what the project's stance is toward either requiring a newer version of mercurial, or trying to adapt the code to lack of "collapsing" directories. Specifically, I used the terse option to meet the behavior of the git function that: "If a whole directory is ignored, don't return all files inside of it." As far as I can tell old versions of mercurial just don't do that at all.

So if you're interested in these patches I'd like to know what direction you think it worth pursuing.

I have not checked the reason for the failure of test, 3.5, macos-latest yet.

@carmenbianca
Copy link
Member

Hey @phlogistonjohn thank you so much for this PR!

I added 9929af6 in which I extracted a lot of code into its own module. I should have done this much, much earlier, and the fact that the code was all over the place probably made your life quite a bit harder. And although there is still some code in places that I would like to be elsewhere, this change should make it a lot easier for additional VCSs to be implemented.

The change still needs cleaning up, but your code looks excellent—thank you!

I'll look into the PyPy thingamajig as well. That image has persistently given me trouble.

@carmenbianca
Copy link
Member

Specifically, I used the terse option to meet the behavior of the git function that: "If a whole directory is ignored, don't return all files inside of it." As far as I can tell old versions of mercurial just don't do that at all.

This behaviour is a performance thing. Calling Git once to get a list of all files is much quicker than calling Git every time you need to figure out whether something is ignored or not. I suspect the same would also be true for Mercurial.

I'm perfectly happy with requiring newer versions of Mercurial.

@carmenbianca carmenbianca force-pushed the jjm-mercurial-basic-support branch 3 times, most recently from ae69fcc to baf471c Compare April 6, 2020 17:16
@carmenbianca
Copy link
Member

The 'pypy3-test' failure appears to be due to the (rather old) version of mercurial that lacks the '--terse' flag.

Interestingly, the opposite is true. The version of Mercurial is too new. On Fedora, I have Mercurial version 4.9, and everything works fine. But when I do pip install mercurial>=5.3, the same test starts failing as in the PyPy image.

The test is:

    @hg
    @posix
    def test_all_files_hg_ignored_contains_newline(hg_repository):
        """File names that contain newlines are also ignored."""
        (hg_repository / "hello\nworld.pyc").touch()
        project = Project(hg_repository)
       assert Path("hello\nworld.pyc").absolute() not in project.all_files()

I manually redid the test, and this is the output:

$ hg status --ignored --terse=i --no-status                                   
build/
src/custom.pyc
src/exception.pyc
src/source_code.pyc

hello\nworld.pyc is missing from this list. I do not know why. But when I remove --terse=i, it shows up again in the list.

Not using the --terse=i flag would be fine, but incur a performance penalty for big directories that are entirely ignored.

phlogistonjohn and others added 9 commits April 21, 2020 12:45
Typically the underscore indicates that a function is "private"
and not meant to be used outside of the module. This function is
meant to be used outside of _util.py thus the rename.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Signed-off-by: John Mulligan <jmulligan@redhat.com>
This is a minimal but sufficient change to make the too usable in
the small mercurial repo I am testing against.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Copy and modify some of the git specific tests to ensure that mercurial
(hg) repos behave similarly when it comes to ignoring files.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
Using the strategy pattern for this makes reading and using the code a lot
easier.
Updated documentation to reflect the changes to VCS. Also added John Mulligan as
author and copyright holder.
This flag is causing some grief in later versions of Mercurial.

Signed-off-by: Carmen Bianca Bakker <carmenbianca.bakker@liferay.com>
@carmenbianca carmenbianca force-pushed the jjm-mercurial-basic-support branch from d61f740 to 44fea39 Compare April 21, 2020 11:47
Signed-off-by: Carmen Bianca Bakker <carmenbianca.bakker@liferay.com>
@carmenbianca carmenbianca merged commit a0eaee4 into fsfe:master Apr 21, 2020
@marcoXbresciani
Copy link

In order to be ready to check this solution, I'm trying to restore the environment I've used while I found the #97 and working on the #99.
Not sure what is happening: everything seems fine, but actually no output.

F:\Programmi\aamfp>pip3 install fsfe-reuse
Requirement already satisfied: fsfe-reuse in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (1.0.0)
Requirement already satisfied: reuse in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from fsfe-reuse) (0.8.0)
Requirement already satisfied: requests in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (2.23.0)
Requirement already satisfied: Jinja2 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (2.11.1)
Requirement already satisfied: binaryornot in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (0.4.4)
Requirement already satisfied: license-expression in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (1.2)
Requirement already satisfied: boolean.py in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (3.7)
Requirement already satisfied: python-debian in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (0.1.36)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse->fsfe-reuse) (2019.11.28)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse->fsfe-reuse) (1.25.8)
Requirement already satisfied: chardet<4,>=3.0.2 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse->fsfe-reuse) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse->fsfe-reuse) (2.9)
Requirement already satisfied: MarkupSafe>=0.23 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from Jinja2->reuse->fsfe-reuse) (1.1.1)
Requirement already satisfied: six in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from python-debian->reuse->fsfe-reuse) (1.14.0)
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

F:\Programmi\aamfp>python -m pip install --upgrade pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.0.2

F:\Programmi\aamfp>pip3 install --user fsfe-reuse
Requirement already satisfied: fsfe-reuse in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (1.0.0)
Requirement already satisfied: reuse in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from fsfe-reuse) (0.8.0)
Requirement already satisfied: boolean.py in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (3.7)
Requirement already satisfied: Jinja2 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (2.11.1)
Requirement already satisfied: requests in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (2.23.0)
Requirement already satisfied: license-expression in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (1.2)
Requirement already satisfied: python-debian in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (0.1.36)
Requirement already satisfied: binaryornot in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse->fsfe-reuse) (0.4.4)
Requirement already satisfied: MarkupSafe>=0.23 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from Jinja2->reuse->fsfe-reuse) (1.1.1)
Requirement already satisfied: chardet<4,>=3.0.2 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse->fsfe-reuse) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse->fsfe-reuse) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse->fsfe-reuse) (2019.11.28)
Requirement already satisfied: idna<3,>=2.5 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse->fsfe-reuse) (2.9)
Requirement already satisfied: six in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from python-debian->reuse->fsfe-reuse) (1.14.0)

F:\Programmi\aamfp>python3 -m reuse --help

F:\Programmi\aamfp>python3 -m reuse init

F:\Programmi\aamfp>python3 -m reuse download

F:\Programmi\aamfp>python3 -m reuse download --all

F:\Programmi\aamfp>python3 -m reuse lint

I know I'm probably missing something, but I'm not a Python guy, so I don't know where to check.

Hints?

@carmenbianca
Copy link
Member

@marcoXbresciani Not really sure. Following remarks:

  1. You should be installing reuse instead of fsfe-reuse.
  2. The contents of this PR are not in a stable release yet. I want to make a stable release later today.
  3. I have absolutely no idea why there's no output on the Windows console. Seems like a Windows problem to me.

@marcoXbresciani
Copy link

Hello @carmenbianca,

  1. tried also with reuse. Same result.
  2. I know these modification are not available, yet. I was just trying to prepare the environment to check them. 😄
  3. 😭 python 3.8 and 3.8.2 and reuse 0.8.0 and 0.8.1. Same: no ouputs. 😞 I'll try to understand, somehow.

Thank you.

@carmenbianca
Copy link
Member

Maybe you can try Powershell instead of cmd.exe as a sanity check? I don't know. I haven't used Windows in forever :x

@marcoXbresciani
Copy link

Nothing. Not even as admin. 😭 I'll try the classic Windows solution: uninstall and reinstall... might be. Who knows. It worked fine, when I played with #97 and #99 ...

@marcoXbresciani
Copy link

Still no luck. Can't understand why it doesn't work anymore. :(

F:\Programmi\aamfp>pip3 install reuse
Collecting reuse
  Downloading reuse-0.10.0-py3-none-any.whl (131 kB)
     |████████████████████████████████| 131 kB 1.7 MB/s
Requirement already satisfied: python-debian in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (0.1.36)
Requirement already satisfied: requests in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (2.23.0)
Requirement already satisfied: binaryornot in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (0.4.4)
Requirement already satisfied: boolean.py in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (3.7)
Requirement already satisfied: license-expression in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (1.2)
Requirement already satisfied: Jinja2 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (2.11.1)
Requirement already satisfied: chardet in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from python-debian->reuse) (3.0.4)
Requirement already satisfied: six in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from python-debian->reuse) (1.14.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse) (2019.11.28)
Requirement already satisfied: idna<3,>=2.5 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse) (2.9)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse) (1.25.8)
Requirement already satisfied: MarkupSafe>=0.23 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from Jinja2->reuse) (1.1.1)
Could not build wheels for python-debian, since package 'wheel' is not installed.
Could not build wheels for requests, since package 'wheel' is not installed.
Could not build wheels for binaryornot, since package 'wheel' is not installed.
Could not build wheels for boolean.py, since package 'wheel' is not installed.
Could not build wheels for license-expression, since package 'wheel' is not installed.
Could not build wheels for Jinja2, since package 'wheel' is not installed.
Could not build wheels for chardet, since package 'wheel' is not installed.
Could not build wheels for six, since package 'wheel' is not installed.
Could not build wheels for certifi, since package 'wheel' is not installed.
Could not build wheels for idna, since package 'wheel' is not installed.
Could not build wheels for urllib3, since package 'wheel' is not installed.
Could not build wheels for MarkupSafe, since package 'wheel' is not installed.
Installing collected packages: reuse
Successfully installed reuse-0.10.0

F:\Programmi\aamfp>python3 -m reuse

F:\Programmi\aamfp>

I've even installed that wheel package, but no luck.

F:\Programmi\aamfp>pip3 install reuse
Collecting reuse
  Using cached reuse-0.10.0-py3-none-any.whl (131 kB)
Requirement already satisfied: boolean.py in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (3.7)
Requirement already satisfied: license-expression in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (1.2)
Requirement already satisfied: binaryornot in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (0.4.4)
Requirement already satisfied: requests in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (2.23.0)
Requirement already satisfied: python-debian in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (0.1.36)
Requirement already satisfied: Jinja2 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from reuse) (2.11.1)
Requirement already satisfied: chardet>=3.0.2 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from binaryornot->reuse) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse) (2.9)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse) (2019.11.28)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from requests->reuse) (1.25.8)
Requirement already satisfied: six in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from python-debian->reuse) (1.14.0)
Requirement already satisfied: MarkupSafe>=0.23 in c:\users\marco.bresciani\appdata\roaming\python\python38\site-packages (from Jinja2->reuse) (1.1.1)
Installing collected packages: reuse
Successfully installed reuse-0.10.0

F:\Programmi\aamfp>python3 -m reuse

F:\Programmi\aamfp>

If you have any idea, please help.

@marcoXbresciani
Copy link
marcoXbresciani commented May 4, 2020

Seems working: instead of using python3 -m reuse as suggested in #97, Windows people can even use (with latest Reuse and latest Python3), simply reuse from the PowerShell or command prompt command line.
Should work.

@marcoXbresciani
Copy link

Fixes #99 as far as running the tool on a couple of repos of my own went.

I won't claim that this is comprehensive support for Mercurial as compared to the current git support but it got the tool working for me and generating sensible reports (vs checking my build artifacts). I did try to add some tests as far as code paths that seemed relevant to running hg versus git goes.

I could probably add a few more tests if needed but I am not sure what you would think are tests that really need a specific check when run using mercurial versus git.

It seems your solution does not consider the .hgtags file and sees it as "unlicensed". Am I missing some configuration pieces?

F:\Programmi\aamfp>reuse lint
# MISSING COPYRIGHT AND LICENSING INFORMATION

The following files have no copyright and licensing information:
* .hgtags


# SUMMARY

* Bad licenses:
* Deprecated licenses:
* Licenses without file extension:
* Missing licenses:
* Unused licenses:
* Used licenses: FSFAP, GFDL-1.3-or-later, GPL-3.0-or-later
* Read errors: 0
* Files with copyright information: 7 / 8
* Files with license information: 7 / 8

Unfortunately, your project is not compliant with version 3.0 of the REUSE Specification :-(

F:\Programmi\aamfp>

@carmenbianca
Copy link
Member

@marcoXbresciani Glad you got it working.

It seems your solution does not consider the .hgtags file and sees it as "unlicensed". Am I missing some configuration pieces?

The file is missing a copyright header.

If .hgtags is the sort of file that doesn't have a copyright header, you can put the header in .hgtags.license. To be frank though, I'm not really sure what .hgtags does. Should this file be added as an exception?

@phlogistonjohn
Copy link
Contributor Author

IMO .hgtags should probably be handled like .hgingore and .gitignore. It's going to be very common in any mercurial project that does releases with tags. It slipped my mind when I added the initial code though.

The file contains meta-data meant for the VCS rather than the code but is tracked with the rest of history as a file.

@carmenbianca
Copy link
Member

.hgignore and .gitignore aren't currently ignored themselves. Looking at some examples of .hgtags, though, it makes sense to ignore it.

@phlogistonjohn
Copy link
Contributor Author

Ah, you're right of course. I was misremembering and thinking of the .git and .hg dirs themselves not the -ignore files.
Those are meant to be edited by the user unlike .hgtags (except for those .001% cases :-)).

@marcoXbresciani
Copy link
marcoXbresciani commented May 4, 2020

In the meanwhile I got the REUSE compliance for my little project by just adding a .hgtags.license file. :) https://savannah.nongnu.org/forum/forum.php?forum_id=9730 and here it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for mercurial version control system
3 participants
0