Testing for kernel performance regressions

Posted Aug 4, 2012 8:33 UTC (Sat) by fhuberts (subscriber, #64683)
In reply to: Testing for kernel performance regressions by gregkh
Parent article: Testing for kernel performance regressions

How about crowd-sourcing that driver testing then?
I'd be willing to donate CPU cycles on most of my machines to test the kernel, drivers, etc. if the results of that would be aggregated somewhere...

Testing for kernel performance regressions

Posted Aug 4, 2012 10:11 UTC (Sat) by Cato (guest, #7643) [Link] (2 responses)

CPAN Testers already does this for testing CPAN modules and Perl on a vast range of OSes (and hardware) - http://wiki.cpantesters.org/

Organising this for Linux would be harder given a bootable system is required, but it could be done.

https://lwn.net/Articles/446382/

Testing for kernel performance regressions

Posted Aug 4, 2012 11:49 UTC (Sat) by jnareb (subscriber, #46500) [Link] (1 responses)

> CPAN Testers already does this for testing CPAN modules and Perl on a vast range of OSes (and hardware)

This works so well because by default CPAN client does tests when installing modules, and send those results to CPANtesters. So it is very easy to become CPANtesters contributors.

Perhaps a request at install / upgrade time to perform regression benchmarks of one's system before first run would be a good idea for Linux testers project?

Testing for kernel performance regressions

Posted Aug 4, 2012 15:09 UTC (Sat) by Cato (guest, #7643) [Link]

That's a good idea - there are already some projects that do functional testing of hardware on new systems, what's missing is the feedback into central test results repository. Though it would need to be a distro-independent test approach so it can be used with the latest kernels as well as distros.

Testing for kernel performance regressions

Posted Aug 4, 2012 10:27 UTC (Sat) by siim@p6drad-teel.net (subscriber, #72030) [Link]

i'd also be willing to test on the few machines i have if there was a straightforward way to run the testsuite and to report the results so that they would be visible to interested parties.

Testing for kernel performance regressions

Posted Aug 4, 2012 14:42 UTC (Sat) by gregkh (subscriber, #8) [Link] (5 responses)

> How about crowd-sourcing that driver testing then?

That's exactly what we do, and what we expect when we do the -rc releases.

You are running them to test that nothing breaks on your machine, right?

If not, please do so.

Testing for kernel performance regressions

Posted Aug 4, 2012 21:34 UTC (Sat) by smoogen (subscriber, #97) [Link]

Well usually the machines that are going to be affected are those in production so running random rc kernels aren't being done on that. My guess is something similar to "Tragedy of the Commons"... people don't test until they have to because they have more important things to do (usually what their boss tells them is important). They then expect that the vendors and coders will do it for them (eg that somewhere there is a huge building filled with racks of machines that vendor X uses every day for every change..) and all for free too.

Testing for kernel performance regressions

Posted Aug 6, 2012 10:24 UTC (Mon) by geertj (subscriber, #4116) [Link] (3 responses)

> > How about crowd-sourcing that driver testing then?
>
> That's exactly what we do, and what we expect when we do the -rc releases.
> You are running them to test that nothing breaks on your machine, right?

Wrong. I am not testing -rc releases, because i have other stuff to do. And i'm not complaining that my hardware doesn't work either, which makes my behavior wholly consistent. Just pre-empting that comment.. :)

There is a lot more that could be done to make this "crowdsourced testing" more effective. Currently it is quite difficult to test out -rc releases. You have to know how to compile a kernel, and how to install and run it in your distribution. Certainly not rocket science, but not easy for the average distro user, which is who you'd need to go after for large-scale outsourced testing.

Just an idea.. What if bootable live test images could be created for -rc releases? Ideally they would need no local storage, but optionally they could use a dedicated partition. The live image could do a whole bunch of tests and send back the results, together with information the hardware the tests ran it. Those could be analyzed for problems. Also the tests could include performance tests.

Every time a new -rc would be released, you'd rebuild the live image, and ask people via G+, Facebook, Twitter, the mailing list, etc, to burn it to a CD and boot their system with it. The CD runs for a few hours overnight, sends back the results, and then says "Thank you, i'm done". I bet you that this could increase your testing base by 10x. Of course you want to be pretty sure that it is safe and put a lot of safeguards in place to make sure it is (which of course doesn't mean you don't need a pretty scary disclaimer before running the CD).

Testing for kernel performance regressions

Posted Aug 6, 2012 11:14 UTC (Mon) by niner (subscriber, #26151) [Link]

The idea sounds great. But this way cannot do much more than smoke tests. You can test if the kernel boots on the target's hardware. What you cannot really test is if btrfs will work on the user's drbd device which runs on top of LVM which runs on top of some special hardware RAID controller or such interesting setups. It's at least very difficult to test if suspend/resume works or if the user will experience some performance regression in his favourite game. You cannot test if the external display works automagically when the laptop is put into it's docking station.

In short: it would be a step forward but there's still plenty of stuff which users would have to test manually. But of course: the perfect is the enemy of the good.

Testing for kernel performance regressions

Posted Aug 8, 2012 16:33 UTC (Wed) by broonie (subscriber, #7078) [Link]

Lots of the distros do actually have packaged versions of latest -rc or similar kernels available for installation.

Testing for kernel performance regressions

Posted Feb 7, 2013 10:52 UTC (Thu) by rbrito (guest, #66188) [Link]

> There is a lot more that could be done to make this "crowdsourced testing" more effective. Currently it is quite difficult to test out -rc releases. You have to know how to compile a kernel, and how to install and run it in your distribution. Certainly not rocket science, but not easy for the average distro user, which is who you'd need to go after for large-scale outsourced testing.

Having to compile the kernels is a burden indeed, especially for those with weaker machines.

At least for Debian-based disributions it seems that Canonical provides daily compiled kernels, which is cool to have in mind (I only remembered that when I read your comment):

http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/

Testing for kernel performance regressions

Posted Aug 4, 2012 23:11 UTC (Sat) by krakensden (subscriber, #72039) [Link] (1 responses)

Phoronix (insert 5 minute hate) actually already built this. It's just that nobody who does any work on anything it tests ever looks at the results, or talks to them to try and make it more useful for developers.

It's a little sad.

Testing for kernel performance regressions

Posted Aug 5, 2012 4:43 UTC (Sun) by dirtyepic (guest, #30178) [Link]

Every time I've tried to point out flaws in their compiler benchmarks I've been ignored. I know the signal to noise ratio on their forums is very high but they've continued to make the same mistakes for years now.