-
Notifications
You must be signed in to change notification settings - Fork 429
Implement test coverage via kcov in our CI #15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
You could just cache the compiled kcov folder. This is our Circle-CI implementation (should be similar on travis): dependencies:
cache_directories:
- kcov-34
override:
- |
if [[ ! -d "kcov-34" ]]; then
wget 'https://github.com/SimonKagstrom/kcov/archive/v34.zip' -O /tmp/kcov.zip
unzip /tmp/kcov.zip
cd kcov-34
cmake .
make -j$(nproc)
else
cd kcov-34
fi
sudo make install |
Not sure if this is the right place to ask, but does anyone have an example (command and index.html output) of kcov working with bats??? Coverage Report Any and ALL help is appreciated. Thanks, |
@alexRambo I'm having a hard time parsing what you're doing, but here's an example: https://travis-ci.org/mbland/go-script-bash/jobs/390296132#L641 which corresponds to: https://coveralls.io/builds/17410861 Also note that it's using kcov v35: https://travis-ci.org/mbland/go-script-bash/jobs/390296132#L521 @particleflux I definitely need to steal that for both bats-core and go-script-bash! 🙂 |
Hi @mbland, thanks for the quick reply and useful links. Trying to understand some of the flags your CI is using... they seem to coming from "kcov --uncommon-options"... not sure if there are some 'special flags' that will allow me to get kcov code coverage for .bats files... Basically, I have a .bats file that is calling my .sh file: Looking through your comments. |
Ah. The latest kcov has issues. I put a bunch of notes in mbland/go-script-bash#244 that I'll follow up on eventually. I'd recommend trying to use v35 explicitly and see if that helps. |
Apologies, running kcov v33 as per this github docker image: Update: (bad news) |
Update: Thanks, |
The problem almost certainly is with $ kcov --bash-dont-parse-binary-dir \
--include-pattern="$include_pattern " \
--exclude-pattern="$exclude_pattern" \
path/to/coverage/results/dir \
bats path/to/tests For example, the command from https://travis-ci.org/mbland/go-script-bash/jobs/390296132, which posted coverage results to: https://coveralls.io/builds/17410861
|
@mbland, thanks for the quick feedback. Folder structure: Contents of test file "/usr/src/xyz/Bats_Tests/tests/hello1.bats": @test "hello without arguments" { @test "hello with world argument" { Contents of test file "/usr/src/xyz/Bats_Tests/hello.sh": if [[ "$1" == "world" ]]; then Again, running the hello1.bats file succeeds in running the unit test (UT). It shows 1 success and 1 failure for the "/usr/src/xyz/Bats_Tests/tests/hello1.bats"; this is expected as we'd like to see 50% code coverage for 'sanity check' later on. One naive questions:
Note: kcov --bash-dont-parse-binary-dir --include-pattern=/user/src/xyz/Bats_Tests/,/user/src/xyz/Bats_Tests/tests,/usr/src/xyz/,/user/src/xyz/,/user/src/,/user/src/ /usr/src/xyz/coverage/ /usr/src/xyz/Bats_Tests/tests/hello1.bats |
I think I might see what's happening here...it's an expectations thing.
However, I've never been concerned with collecting coverage for This would explain why you're only seeing coverage for the files under test, and why I thought you might be invoking So it's true that if you want coverage for the test files themselves, Bats may not be the best choice. However, I'd suggest considering whether collecting coverage for |
@mbland, I'm confused by your previous post...
For clarity, I want to get coverage on the CODE UNDER TEST file "hello.sh"; which is being called from UNIT TEST file hello1.bats. Hope this clears things up... and you understand my frustration with getting such a simple file (hello.sh) to show coverage :) |
@alexRambo Sorry about that...but now I've finally reproduced what's happening locally and have a concrete recommendation. First, though, I never answered your question about Second, I updated your test files to the following, where
#!/bin/bash
if [[ "$1" == "world" ]]; then
echo "Hello World"
else
echo "Hello unknown"
fi
#!/usr/bin/env bats
# Works when "$PWD" is equivalent to "$BATS_TEST_DIRNAME/.."
HELLO_CMD='./hello.sh'
# Works from any "$PWD"
#HELLO_CMD="$BATS_TEST_DIRNAME/../hello.sh"
# An alternative to the above that chomps off the last path component
#HELLO_CMD="${BATS_TEST_DIRNAME%/*}/hello.sh"
@test "hello without arguments" {
run "$HELLO_CMD"
[ "$output" == "Hello unknown" ]
}
@test "hello with world argument" {
run "$HELLO_CMD" world
[ "$output" == "Hello World" ]
} I cloned bats-core v1.1.0 under # When `HELLO_CMD='./hello.sh'`, we must execute them in `~/src/tmp`.
$ cd ~/src/tmp
# Remove the coverage directory between runs, or else the data accumulates.
$ rm -rf ~/src/tmp/coverage
# Removed --bash-dont-parse-binary-dir as it's a v35 option.
# Note --include-path here, instead of --include-pattern.
$ ~/src/SimonKagstrom/kcov/src/kcov --include-path=~/src/tmp ~/src/tmp/coverage \
~/src/bats-core/bats-core/bin/bats ~/src/tmp/tests/
1..2
ok 1 hello without arguments
ok 2 hello with world argument
$ google-chrome "file://$HOME/src/tmp/coverage/index.html" Happily, Best I can tell, Let me know whether or not this helps. |
Update 07/12/2018: No coverage still :/ Here's the '0%' coverage output from command "vi ~/src/tmp/coverage/index.json": SUGGESTION: Watch out:
Dockerfile: Hope someone can help still as kcov and Bats seems to "work" for everyone else :/ Thanks, OLD: On the positive side, I learned more about the need of "$BATS_TEST_DIRNAME". Thanks @mland. |
Since it was mentioned here - my example for kcov and bats on Circle CI has been updated to support kcov 36 and Circle CI 2: https://github.com/particleflux/kcov-bats-circleci-codeclimate/ There is some trickery in the script to correctly report the coverage with the Codeclimate reporter - may be this is relevant for Coveralls too: One thing to note though: I am for whatever reason not able to get it to run inside docker - it needs to use the |
@particleflux @mbland Tested environments:
Here's the example Ubuntu container:
Calling kcov like this {
"files": [
{"file": "/tmp/bats.6999.src", "percent_covered": "27.59", "covered_lines": "8", "total_lines": "29"},
{"file": "/usr/bin/bats", "percent_covered": "86.21", "covered_lines": "25", "total_lines": "29"},
{"file": "/usr/lib/bats-core/bats-preprocess", "percent_covered": "70.59", "covered_lines": "24", "total_lines": "34"},
{"file": "/usr/lib/bats-core/bats-exec-test", "percent_covered": "65.47", "covered_lines": "146", "total_lines": "223"},
{"file": "/usr/lib/bats-core/bats", "percent_covered": "43.42", "covered_lines": "33", "total_lines": "76"}
],
"percent_covered": "60.36",
"covered_lines": 236,
"total_lines": 391,
"percent_low": 25,
"percent_high": 75,
"command": "bats",
"date": "2019-06-25 09:09:52"
} So kcov never seems to find the "script under test", and it doesn't seem to matter which script I call, neither my own or @particleflux's, the output stays similar. |
It could be a problem with kcov. I wrote a comment here. |
Not sure if this is in any way relevant but is Part of my motivation for asking is my ongoing saga in search of libraries to use with bats-core. After looking at asserts and mocks I am now looking in code coverage... |
Hi everyone. I am currently trying to run kcov v38 with BATS 1.3.0 with a following command kcov --bash-dont-parse-binary-dir --include-pattern=.sh/ --exclude-pattern=lib/bats coverage \ What I get is: kcov: error: Can't write helper If I decide to specify full paths then the kcov launches, however it measures the coverage of the bats scripts ( as described in this thread by @fuero. Maybe I am being naive, but has anybody came up with a solution for this problem? I have been trying only on MacOS Big Sur 11.2.3 on zsh 5.8 (x86_64-apple-darwin20.0) and GNU bash, version 5.0.18(1)-release. If you would guys require more details, I am happy to provide answers to any questions, but unfortunately I am bit novice to this topic. |
@alexRambo Did you resolved this issue? As I am getting the same error!! |
I have this working (sort of) Here is the command I run. Note I am sending output to an apache2 web directory to view output report
kcov does its job, and creates a report. I have not run into the issues others have been having.
But, doing so, kcov only sees EDIT Disregard the problem I stated, I misread what kcov was doing for output. bats works with kcov properly. I just need to run my script without sudo. so the first command is what worked. |
The following approach worked for me:
I am able to check the content in folder |
@brunolnetto thank you for sharing your solution! However, I had troubles running your command with
Without I share with you the Dockerfile I used to run kcov. FROM kcov/kcov
RUN apt update && \
apt install -y git && \
git clone https://github.com/bats-core/bats-core.git && \
cd bats-core && \
./install.sh /usr/local
COPY dgoss /src/dgoss
COPY test/dgoss.bats /src/tests/dgoss.bats
WORKDIR /src
CMD kcov \
--clean \
--report-only \
--bash-dont-parse-binary-dir \
--exclude-pattern=tests \
--include-path=. ./coverage/ bats ./tests |
Insightful! :-) |
I've used https://github.com/SimonKagstrom/kcov to great effect to collect coverage on mbland/go-script-bash and publish it to Coveralls.
The only wrinkle is that I clone and build the kcov repo fresh every time, since the Ubuntu package is a bit old. For this project and that one, I should look into finding a way to install an updated binary.
The text was updated successfully, but these errors were encountered: