Testing for kernel performance regressions

Posted Aug 4, 2012 14:42 UTC (Sat) by gregkh (subscriber, #8)
In reply to: Testing for kernel performance regressions by fhuberts
Parent article: Testing for kernel performance regressions

> How about crowd-sourcing that driver testing then?

That's exactly what we do, and what we expect when we do the -rc releases.

You are running them to test that nothing breaks on your machine, right?

If not, please do so.

Testing for kernel performance regressions

Posted Aug 4, 2012 21:34 UTC (Sat) by smoogen (subscriber, #97) [Link]

Well usually the machines that are going to be affected are those in production so running random rc kernels aren't being done on that. My guess is something similar to "Tragedy of the Commons"... people don't test until they have to because they have more important things to do (usually what their boss tells them is important). They then expect that the vendors and coders will do it for them (eg that somewhere there is a huge building filled with racks of machines that vendor X uses every day for every change..) and all for free too.

Testing for kernel performance regressions

Posted Aug 6, 2012 10:24 UTC (Mon) by geertj (subscriber, #4116) [Link] (3 responses)

> > How about crowd-sourcing that driver testing then?
>
> That's exactly what we do, and what we expect when we do the -rc releases.
> You are running them to test that nothing breaks on your machine, right?

Wrong. I am not testing -rc releases, because i have other stuff to do. And i'm not complaining that my hardware doesn't work either, which makes my behavior wholly consistent. Just pre-empting that comment.. :)

There is a lot more that could be done to make this "crowdsourced testing" more effective. Currently it is quite difficult to test out -rc releases. You have to know how to compile a kernel, and how to install and run it in your distribution. Certainly not rocket science, but not easy for the average distro user, which is who you'd need to go after for large-scale outsourced testing.

Just an idea.. What if bootable live test images could be created for -rc releases? Ideally they would need no local storage, but optionally they could use a dedicated partition. The live image could do a whole bunch of tests and send back the results, together with information the hardware the tests ran it. Those could be analyzed for problems. Also the tests could include performance tests.

Every time a new -rc would be released, you'd rebuild the live image, and ask people via G+, Facebook, Twitter, the mailing list, etc, to burn it to a CD and boot their system with it. The CD runs for a few hours overnight, sends back the results, and then says "Thank you, i'm done". I bet you that this could increase your testing base by 10x. Of course you want to be pretty sure that it is safe and put a lot of safeguards in place to make sure it is (which of course doesn't mean you don't need a pretty scary disclaimer before running the CD).

Testing for kernel performance regressions

Posted Aug 6, 2012 11:14 UTC (Mon) by niner (subscriber, #26151) [Link]

The idea sounds great. But this way cannot do much more than smoke tests. You can test if the kernel boots on the target's hardware. What you cannot really test is if btrfs will work on the user's drbd device which runs on top of LVM which runs on top of some special hardware RAID controller or such interesting setups. It's at least very difficult to test if suspend/resume works or if the user will experience some performance regression in his favourite game. You cannot test if the external display works automagically when the laptop is put into it's docking station.

In short: it would be a step forward but there's still plenty of stuff which users would have to test manually. But of course: the perfect is the enemy of the good.

Testing for kernel performance regressions

Posted Aug 8, 2012 16:33 UTC (Wed) by broonie (subscriber, #7078) [Link]

Lots of the distros do actually have packaged versions of latest -rc or similar kernels available for installation.

Testing for kernel performance regressions

Posted Feb 7, 2013 10:52 UTC (Thu) by rbrito (guest, #66188) [Link]

> There is a lot more that could be done to make this "crowdsourced testing" more effective. Currently it is quite difficult to test out -rc releases. You have to know how to compile a kernel, and how to install and run it in your distribution. Certainly not rocket science, but not easy for the average distro user, which is who you'd need to go after for large-scale outsourced testing.

Having to compile the kernels is a burden indeed, especially for those with weaker machines.

At least for Debian-based disributions it seems that Canonical provides daily compiled kernels, which is cool to have in mind (I only remembered that when I read your comment):

http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/