GCovr generates no coverage data for CMake out of source builds #230

landofcake · 2018-03-02T01:17:27Z

This is a follow on issue ticket from #64, as I'm currently experiencing exactly the same issue, and I can't for the life of me make gcovr correct pick up any coverage data from any gcov data generated from a standard CMake structured build (created in CLion). I am using Windows 10, with my code being built using the MinGW/MSYS g++ package. I am invoking gcovr from MinGW bash.

In my configuration, I've set the correct gcov compiler flags and managed to get output gcda and gcno files from a compile and test run.

The project is structured like this:

Project Root
________Subproject1
________________src
________________________(cpp files end up here)
________________include
________________________(h files end up here)
________Subproject2
________Subproject3

CMake automatically does it's business to generate makefiles for the build, and on build, it creates an output setup like this (where cmake-build-test is the build configuration I've set up which builds using the correct compiler flags for coverage).

Project Root
________cmake-build-test
________________Subproject1
________________________CMakeFiles
________________________________${Subproject1name}.dir
________________________________________src
________________________________________________(gcno and gcda files end up here)
________________Subproject2
________________Subproject3

I've tried multiple different combinations of the --root option, and I've tried collating the gcda and gcno files in a separate directory to no avail.

I've attached the log from a run of gcovr using the following:

gcovr -v -r .

I ran this from inside the project root (for reference, in the logs, it's C:\dev\hs\MSN, and the subproject directories are C:\dev\hs\MSN\MSN, C:\dev\hs\MSN\MSNTest, C:\dev\hs\MSN\MSNLib).

It may be that I'm simply not understanding the specifics of how the root flag is supposed to be used, but if this is genuinely a problem with what should be a supported setup, please let me know if there's any more useful information I can provide.

landofcake · 2018-03-02T01:20:27Z

gcovr.txt

latk · 2018-03-02T13:56:05Z

Thank you for this detailed and useful report, I'll try to look into it more deeply over the weekend.

Are you using gcovr 3.4? If not, does anything change when you update?

This might probably be a Windows issue, not an out-of-source issue. The log shows some filenames as c:\dev\hs\MSN but others as C:\dev\hs\MSN, note that case. As exclusion of files is checked by regexes, the case must match. It is not obvious to me why the case is mixed here, that may be indicative of a gcovr bug.

landofcake · 2018-03-02T23:21:58Z

Thanks for the reply @latk , I can confirm that I'm using 3.4.

For a test, I threw up a tiny little test project that simply contains one source file and outputs the object files to the same directory, and gcovr correctly generated everything.

landofcake · 2018-03-03T00:09:25Z

I think you're possibly right about the windows drive names being inconsistent @latk , I hacked the code a bit to normalise all of the drive names to have the same case and I get coverage data correctly.

Initially I tried setting them to always be upper case (e.g. c:\bla\bla to C:\bla\bla), but that didn't work, as it seems that the root parameter is coming in to the compiled root_filter regex as lower case. So I then tried setting all of the paths (currdir, data_fname, source_fname, root_dir and fname) to have lower case drive names and everything seems fine.

I'm not a great programmer so it's not a brilliant fix, so I won't try and set up a push request, don't worry!

Here's the new output log including some debugging prints I put in to see what was going on:

gcovr.txt

(Turns out I wasn't getting very good coverage)

latk · 2018-03-03T17:09:37Z

It doesn't matter if it's a “brilliant” fix. If it fixes a bug, I'd like to have it – even if only to see where you had to insert checks.

I'm still a bit confused why these lower-case drive letters even enter the system. The root filter is obtained from doing an abspath() on the --root parameter (which is the current directory . in most cases). Our testing never picked up this issue before.

Which Python version are you using?

inorton · 2018-03-03T17:24:20Z

I have a similar issue but actually get less output. I'm using a rather large cmake oos build,

main/
   various-sources/...
   cmake-build-gcov/...various-folders/various*.gc*

Except while gcovr managed to read 140 gcno/gcda files it seems to think none of them match my source folder. I've also tried various combinations of -r and -f values with no success.

I've also had gcovr open in the pycharm debugger and to me it looks like that gcovr doesn't manage to find the source file and then runs gcov, gcov then simply prints it's usage message and no output

latk · 2018-03-03T17:34:33Z

@inorton It sometimes helps to run gcovr from your build directory, and point the -r parameter to the source directory. Does that help?

If not, please open a separate issue in which you also include your OS, Python version, gcovr version, and the gcovr output when running with the --verbose flag.

If you can create a minimal CMake project that demonstrates these problems, I could turn that into a test case and think about a fix. This would be most helpful. There are many complaints in connection with CMake, but so far no one has provided a clear test case that I can work with.

landofcake · 2018-03-05T19:41:10Z

Hi @latk ,

I'm sorry to bother you again, but I'm getting things set up so I can put the fixes in and create a pull request, but I'm struggling a little bit to get all of the tests passing on my windows development machine.

It seems that a number of the tests fall over because the run function can't pick up make from the environment. I can't see anything about this listed in the contribution guidelines, and I've had a look at the gcovr CI and it looks like everything is OK there, is there anything I need to do to make sure that the tests can correctly pick up what's in my %PATH% ?

The specific error is:

E AssertionError: assert False
E + where False = run(['make', 'clean'])

I'm running the tests from MinGW bash using 'python -m pytest -v' inside the test directory.

Thanks!

latk · 2018-03-05T20:25:39Z

@landofcake Unfortunately I'm not very familiar with Windows myself. There were similar problems when we added Appveyor builds, see #189. I think we had to set a MAKE = mingw32-make environment variable to get some tests to work, but most worked before that. Have you double-checked that your PATH contains the mingw directory with the make.exe?

If you have difficulty testing locally, you can sign up for Travis CI and Appveyor for your fork, then look at their build results when you push. What I do:

hack some code
push it to some branch on my personal fork
get a build failure email
attempt a fix
git commit --amend changed-file.py to overwrite the last commit
git push --force-with-lease to overwrite my remote branch and trigger the next build

This isn't very efficient, but it works. You can prevent smaller mistakes by running flake8 and gcovr --help before committing, if you (like me) sometimes forget to close a parenthesis.

Please let me know if there's anything else I can help you with.

landofcake · 2018-03-05T21:10:44Z

Thanks for the information @latk , I've been experimenting and I'm now close to getting all of the tests working correctly in my Windows MinGW environment. The main issues are:

Make and CMake need to be in the system path, even if you run the tests from a MinGW/MSYS bash which has them in the path, they don't propagate through the subprocess.call call in the test runner.
By default, the call to CMake in the cmake_oos test uses the standard generator for the system, which on my Windows install caused a visual studio project to be generated and not Makefiles. I've solved this by using a Preload.cmake and forcing the generator to output Makefiles using set (CMAKE_GENERATOR "Unix Makefiles" CACHE INTERNAL "" FORCE). I'll look into a more sustainable solution which actually works out whether it needs to do this at build time based on the system information.
It seems that parsing the output of 'uname' to work out the BASE_OS in the shared_lib Makefile possibly needs a few more checks. In my build, I'm using both MinGW and MSYS but uname outputs MINGW32_NT-6.2, which doesn't match any of the existing options (MSYS_NT or CYGWIN_NT). Setting BUILD_OS to MSYS_NT if uname detects MINGW32 works fine.

There are a few more failures left caused by some regressions in the coverage output, so I'll investigate these next.

latk · 2018-03-05T21:49:34Z

@landofcake Thank you, this info is useful and I'll update the contribution guide at some point. Don't bother putting too much effort in the failing tests, as you can just exclude them for now: python -m pytest -k 'not test_name' where test_name is some failing test. The -k option can combine substrings of tests name with Python-like logic operators (and or not).

latk · 2018-03-06T14:28:31Z

I just found that this issue had previously been reported as #86. The solution there was to add a normcase() call. I think this is needed in only two places, as of the current code on the master branch:

the build_filter() function in gcovr.utils, before the filter regex is compiled
the apply_filter_include_exclude() function in gcovr.gcov, where case normalization should be applied to the given filename and perhaps the abs_filename?

landofcake · 2018-03-07T00:13:39Z

Hey @latk ,

I had a bit of a play with this, and it seems that if you wholesale normcase() everything as it comes in it seems to play some strange games with some of the regex matching which I haven't perfectly worked out yet.

The approach I have which works currently is to specifically sanitise the windows drive letter and keep the rest of the paths as the original case. Specifically, the following call:

re.sub(r"^[a-zA-C](?=[~:])", lambda m: m.group(0).lower(), path)

(Actually the ~ isn't strictly necessary here, I was testing to see whether we also need to sanitise windows drive letters at the start of data_fname as well, but it doesn't appear to be necessary).

Using this approach and sanitising all of the paths when we're inside process_gcov_data appears to allow everything to work correctly on my example. Specifically, I've sanitised:

source_fname
currdir
root_dir
fname

One interesting thing I discovered is that in some of the tests we can get into process_gov_data with source_fname being None (e.g. html-nested2-use-existing). Is this intentional?

landofcake · 2018-03-07T00:22:23Z

Current status is that this is nearly passing all of the tests, I'm having some slightly strange windows related test issues ... specifically:

Some tests might need a broader check of the output from uname to ensure that correct parameters are used (e.g. ifeq ($(BASE_OS),MSYS_NT) doesn't catch MinGW/Cygwin).
There seems to be some strange reproducibility problems with the 'linked' test. I'm currently in the state where the only failing test is txt-linked, which fails with a full test run using python -m pytest -v but passes when run specifically using python -m pytest -s -k 'txt-linked' -vv.

I guess this can probably be dropped into a separate issue report which I'll do when I've got time.

inorton · 2019-07-05T15:24:35Z

Hello from the future! I have managed to get gcov and gcovr to play well with cmake by setting the correct compiler flags.

  set(GCOV_LINK_FLAGS "-lgcov --coverage")
  set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${GCOV_LINK_FLAGS}")
  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O0 -g --coverage -fprofile-arcs -ftest-coverage")
  # work around gcc 4.9.2 bug
  # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64277
  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-array-bounds")

Set these in my own toolchain or early in your cmakelists.

This gives me gcno files in my cmake build folder (next to .o files) and when I run my unit tests the gcda files appear there too.

You then simply need to run gcovr inside your sources root (not your build folder)

gcovr -r . cmake-build-dir --print-summary

blueyed · 2019-07-06T09:12:04Z

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O0 -g --coverage -fprofile-arcs -ftest-coverage")

I think --coverage is enough (-fprofile-arcs -ftest-coverage is not needed additionally).

set(GCOV_LINK_FLAGS "-lgcov --coverage")

Not sure about the explicit -lgcov, but it might also not be needed.

inorton · 2019-07-06T09:56:18Z

@blueyed in later gcc --coverage is more or less all that is needed, I'm using 4.9 though

ingomueller-net · 2020-11-02T11:46:31Z

I am trying the approach described by @inorton using gcovr 4.1 on my project but still get an empty report. Any suggestions how I can debug this?

Update: Using the --verbose flag, I saw that llvm-cov wasn't called correctly. As suggested here, I created a small script cov.sh with the following content and used the gcovr flag --gcov-executable $PWD/cov.sh to use it:

#!/usr/bin/bash
exec llvm-cov-9.0 gcov "$@"

With that, the approach works fine for me.

Spacetown · 2022-01-01T20:46:17Z

I close the issue since our tests are running with cmake and the actions are also running under Windows.

landofcake changed the title ~~GCov generates no coverage data for CMake out of source builds~~ GCovr generates no coverage data for CMake out of source builds Mar 2, 2018

latk added Type: Bug Platform: Windows Filters related to filters, include/exclude, path handling labels Mar 5, 2018

latk mentioned this issue Mar 6, 2018

gcovr in automated builds windows/mingw no output due to drive letter issue #86

Closed

landofcake mentioned this issue Mar 7, 2018

Tests should check for more BASE_OS types in Windows #240

Closed

latk mentioned this issue May 24, 2019

Windows absolute paths are garbled when passed to shutil #313

Closed

Spacetown closed this as completed Jan 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

GCovr generates no coverage data for CMake out of source builds #230

GCovr generates no coverage data for CMake out of source builds #230

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GCovr generates no coverage data for CMake out of source builds #230

GCovr generates no coverage data for CMake out of source builds #230

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!