Bugzilla – Bug 1049226
spurious 'No space left on device' on LXCs created by machinectl
Last modified: 2019-07-11 11:14:28 UTC
Created attachment 732805 [details] reproducer Hi, From time to time, when trying to use 'machinectl' to create an LXC container, I get 'No space left on device' errors. The filesytem has plenty of disk space, and so does the /var/lib/images subvolume. I am not sure if this is a systemd-machined issue or a btrfs one so feel free to adjust component and title if necessary. vagrant@vagrant-openSUSE-Leap:~> sudo btrfs fi show Label: none uuid: e433f7ff-86be-4a03-9aea-6316f9124243 Total devices 1 FS bytes used 3.91GiB devid 1 size 38.54GiB used 4.88GiB path /dev/sda2 vagrant@vagrant-openSUSE-Leap:~> sudo btrfs fi df -h /var/lib/machines/ Data, single: total=4.01GiB, used=3.76GiB System, DUP: total=64.00MiB, used=16.00KiB Metadata, DUP: total=384.00MiB, used=150.72MiB GlobalReserve, single: total=16.00MiB, used=0.00B I have attached a script which can reproduce the problem. It usually happens when you provision a Leap 42.2 machine using vagrant So in order to reproduce just do vagrant init opensuse/openSUSE-42.2-x86_64 vagrant up vagrant ssh cd /vagrant ./foo.sh The following error should happen after 2-3 attempts ==> opensuse422: (113/343) Installing: perl-5.18.2-5.6.x86_64 [. ==> opensuse422: . ==> opensuse422: . ==> opensuse422: . ==> opensuse422: . ==> opensuse422: . ==> opensuse422: . ==> opensuse422: . ==> opensuse422: . ==> opensuse422: . ==> opensuse422: . ==> opensuse422: . ==> opensuse422: error] ==> opensuse422: Abort, retry, ignore? [a/r/i] (a): a ==> opensuse422: Installation of perl-5.18.2-5.6.x86_64 failed: ==> opensuse422: Error: Subprocess failed. Error: RPM failed: error: unpacking of archive failed on file /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/linux/dv b/dmx.ph: cpio: rename failed - No space left on device ==> opensuse422: error: perl-5.18.2-5.6.x86_64: install failed ==> opensuse422: Warning: %posttrans scripts skipped while aborting: ==> opensuse422: cronie-1.4.11-58.25.x86_64.rpm ==> opensuse422: Problem occurred during or after installation or removal of packages: ==> opensuse422: Installation aborted by user ==> opensuse422: ==> opensuse422: Please see the above error message for a hint. Let me know if you need me to provide more information.
What kernel release were you using?
(In reply to Jeff Mahoney from comment #1) > What kernel release were you using? It's 4.4.74-18.20-default
Do you need me to provide any more information? This is still an issue when running various openstack tests.
This is also reproducible on 42.3. here is the related log from Leap 42.3 zypper -n dup -l zypper -n in -l ca-certificates git-core openssh python-devel python python-xml r sync sudo wget tar which Installation of python-devel-2.7.13-26.1.x86_64 failed: Error: Subprocess failed. Error: RPM failed: error: unpacking of archive failed on file /usr/lib64/python2.7/test/fork_wait.py: cpio: rename failed - No space left on device error: python-devel-2.7.13-26.1.x86_64: install failed Problem occurred during or after installation or removal of packages: Installation aborted by user Please see the above error message for a hint. The disk size and usage is about the same with the 42.2 case
I believe I might have a tentative patch, worth testing, what would be the best way to give it to your - schedule a patched kernel build on OBS or just send you the patch ?
(In reply to Nikolay Borisov from comment #5) > I believe I might have a tentative patch, worth testing, what would be the > best way to give it to your - schedule a patched kernel build on OBS or just > send you the patch ? A published rpm file in OBS would be more convenient to me. Thank you
(In reply to Markos Chandras from comment #6) > (In reply to Nikolay Borisov from comment #5) > > I believe I might have a tentative patch, worth testing, what would be the > > best way to give it to your - schedule a patched kernel build on OBS or just > > send you the patch ? > > A published rpm file in OBS would be more convenient to me. Thank you I've kicked a build of the patched kernel here https://build.opensuse.org/project/show/home:nikbor:OpenSuse42.3:bsc1049226
(In reply to Nikolay Borisov from comment #7) > (In reply to Markos Chandras from comment #6) > > (In reply to Nikolay Borisov from comment #5) > > > I believe I might have a tentative patch, worth testing, what would be the > > > best way to give it to your - schedule a patched kernel build on OBS or just > > > send you the patch ? > > > > A published rpm file in OBS would be more convenient to me. Thank you > > I've kicked a build of the patched kernel here > https://build.opensuse.org/project/show/home:nikbor:OpenSuse42.3:bsc1049226 The patch I was given seems to work for me on 42.2 and 42.3. I did quite a few runs over the weekend and I was not able to see this problem. So either this patch is good or I just got lucky.
openSUSE-SU-2017:2384-1: An update that solves two vulnerabilities and has 58 fixes is now available. Category: security (important) Bug References: 1005776,1015342,1020645,1020657,1030850,1031717,1031784,1034048,1037838,1040813,1042847,1047487,1047989,1048155,1048228,1048325,1048327,1048356,1048501,1048912,1048934,1049226,1049272,1049291,1049336,1050211,1050742,1051790,1052093,1052094,1052095,1052384,1052580,1052888,1053117,1053309,1053472,1053627,1053629,1053633,1053681,1053685,1053802,1053915,1053919,1054082,1054084,1055013,1055096,1055272,1055290,1055359,1055709,1055896,1055935,1055963,1056185,1056588,1056827,969756 CVE References: CVE-2017-12134,CVE-2017-14051 Sources used: openSUSE Leap 42.3 (src): kernel-debug-4.4.85-22.1, kernel-default-4.4.85-22.1, kernel-docs-4.4.85-22.3, kernel-obs-build-4.4.85-22.1, kernel-obs-qa-4.4.85-22.1, kernel-source-4.4.85-22.1, kernel-syms-4.4.85-22.1, kernel-vanilla-4.4.85-22.1
openSUSE-SU-2017:2495-1: An update that solves 5 vulnerabilities and has 32 fixes is now available. Category: security (important) Bug References: 1012829,1020645,1020657,1021424,1022743,1024405,1030850,1031717,1031784,1034048,1038583,1047487,1048155,1048893,1048934,1049226,1049580,1051790,1052580,1052888,1053117,1053802,1053915,1053919,1054084,1055013,1055096,1055359,1056261,1056588,1056827,1056982,1057015,1057389,1058116,971975,981309 CVE References: CVE-2017-1000251,CVE-2017-11472,CVE-2017-12134,CVE-2017-14051,CVE-2017-14106 Sources used: openSUSE Leap 42.2 (src): kernel-debug-4.4.87-18.29.1, kernel-default-4.4.87-18.29.1, kernel-docs-4.4.87-18.29.2, kernel-obs-build-4.4.87-18.29.1, kernel-obs-qa-4.4.87-18.29.1, kernel-source-4.4.87-18.29.1, kernel-syms-4.4.87-18.29.1, kernel-vanilla-4.4.87-18.29.1
SUSE-SU-2017:2847-1: An update that solves 11 vulnerabilities and has 170 fixes is now available. Category: security (important) Bug References: 1004527,1005776,1005778,1005780,1005781,1012382,1012829,1015342,1015343,1019675,1019680,1019695,1019699,1020412,1020645,1020657,1020989,1021424,1022595,1022604,1022743,1022912,1022967,1024346,1024373,1024405,1025461,1030850,1031717,1031784,1032150,1034048,1034075,1035479,1036060,1036215,1036737,1037579,1037838,1037890,1038583,1040813,1042847,1043598,1044503,1046529,1047238,1047487,1047989,1048155,1048228,1048325,1048327,1048356,1048501,1048893,1048912,1048934,1049226,1049272,1049291,1049336,1049361,1049580,1050471,1050742,1051790,1051987,1052093,1052094,1052095,1052360,1052384,1052580,1052593,1052888,1053043,1053309,1053472,1053627,1053629,1053633,1053681,1053685,1053802,1053915,1053919,1054082,1054084,1054654,1055013,1055096,1055272,1055290,1055359,1055493,1055567,1055709,1055755,1055896,1055935,1055963,1056061,1056185,1056230,1056261,1056427,1056587,1056588,1056596,1056686,1056827,1056849,1056982,1057015,1057031,1057035,1057038,1057047,1057067,1057383,1057498,1057849,1058038,1058116,1058135,1058410,1058507,1058512,1058550,1059051,1059465,1059500,1059863,1060197,1060229,1060249,1060400,1060985,1061017,1061046,1061064,1061067,1061172,1061451,1061721,1061775,1061831,1061872,1062279,1062520,1062962,1063102,1063349,1063460,1063475,1063479,1063501,1063509,1063520,1063570,1063667,1063671,1063695,1064064,1064206,1064388,1064436,963575,964944,966170,966172,966186,966191,966316,966318,969476,969477,969756,971975,981309 CVE References: CVE-2017-1000252,CVE-2017-11472,CVE-2017-12134,CVE-2017-12153,CVE-2017-12154,CVE-2017-13080,CVE-2017-14051,CVE-2017-14106,CVE-2017-14489,CVE-2017-15265,CVE-2017-15649 Sources used: SUSE Linux Enterprise Workstation Extension 12-SP3 (src): kernel-default-4.4.92-6.18.1 SUSE Linux Enterprise Software Development Kit 12-SP3 (src): kernel-docs-4.4.92-6.18.3, kernel-obs-build-4.4.92-6.18.1 SUSE Linux Enterprise Server 12-SP3 (src): kernel-default-4.4.92-6.18.1, kernel-source-4.4.92-6.18.1, kernel-syms-4.4.92-6.18.1 SUSE Linux Enterprise Live Patching 12-SP3 (src): kgraft-patch-SLE12-SP3_Update_4-1-4.3 SUSE Linux Enterprise High Availability 12-SP3 (src): kernel-default-4.4.92-6.18.1 SUSE Linux Enterprise Desktop 12-SP3 (src): kernel-default-4.4.92-6.18.1, kernel-source-4.4.92-6.18.1, kernel-syms-4.4.92-6.18.1
SUSE-SU-2017:2869-1: An update that solves 16 vulnerabilities and has 120 fixes is now available. Category: security (important) Bug References: 1006180,1011913,1012382,1012829,1013887,1019151,1020645,1020657,1021424,1022476,1022743,1022967,1023175,1024405,1028173,1028286,1029693,1030552,1030850,1031515,1031717,1031784,1033587,1034048,1034075,1034762,1036303,1036632,1037344,1037404,1037994,1038078,1038583,1038616,1038792,1039915,1040307,1040351,1041958,1042286,1042314,1042422,1042778,1043652,1044112,1044636,1045154,1045563,1045922,1046682,1046821,1046985,1047027,1047048,1047096,1047118,1047121,1047152,1047277,1047343,1047354,1047487,1047651,1047653,1047670,1048155,1048221,1048317,1048891,1048893,1048914,1048934,1049226,1049483,1049486,1049580,1049603,1049645,1049882,1050061,1050188,1051022,1051059,1051239,1051399,1051478,1051479,1051556,1051663,1051790,1052049,1052223,1052533,1052580,1052593,1052709,1052773,1052794,1052888,1053117,1053802,1053915,1053919,1054084,1055013,1055096,1055359,1055493,1055755,1055896,1056261,1056588,1056827,1056982,1057015,1058038,1058116,1058410,1058507,1059051,1059465,1060197,1061017,1061046,1061064,1061067,1061172,1061831,1061872,1063667,1064206,1064388,964063,971975,974215,981309 CVE References: CVE-2017-1000252,CVE-2017-10810,CVE-2017-11472,CVE-2017-11473,CVE-2017-12134,CVE-2017-12153,CVE-2017-12154,CVE-2017-13080,CVE-2017-14051,CVE-2017-14106,CVE-2017-14489,CVE-2017-15649,CVE-2017-7518,CVE-2017-7541,CVE-2017-7542,CVE-2017-8831 Sources used: SUSE Linux Enterprise Workstation Extension 12-SP2 (src): kernel-default-4.4.90-92.45.1 SUSE Linux Enterprise Software Development Kit 12-SP2 (src): kernel-docs-4.4.90-92.45.3, kernel-obs-build-4.4.90-92.45.1 SUSE Linux Enterprise Server for Raspberry Pi 12-SP2 (src): kernel-default-4.4.90-92.45.1, kernel-source-4.4.90-92.45.1, kernel-syms-4.4.90-92.45.1 SUSE Linux Enterprise Server 12-SP2 (src): kernel-default-4.4.90-92.45.1, kernel-source-4.4.90-92.45.1, kernel-syms-4.4.90-92.45.1 SUSE Linux Enterprise Live Patching 12 (src): kgraft-patch-SLE12-SP2_Update_14-1-2.4 SUSE Linux Enterprise High Availability 12-SP2 (src): kernel-default-4.4.90-92.45.1 SUSE Linux Enterprise Desktop 12-SP2 (src): kernel-default-4.4.90-92.45.1, kernel-source-4.4.90-92.45.1, kernel-syms-4.4.90-92.45.1 SUSE Container as a Service Platform ALL (src): kernel-default-4.4.90-92.45.1 OpenStack Cloud Magnum Orchestration 7 (src): kernel-default-4.4.90-92.45.1
SUSE-SU-2017:2956-1: An update that solves 17 vulnerabilities and has 113 fixes is now available. Category: security (important) Bug References: 1005917,1006180,1011913,1012382,1012829,1013887,1018419,1019151,1020645,1020657,1020685,1021424,1022476,1022743,1023175,1024405,1028173,1028286,1028819,1029693,1030552,1030850,1031515,1031717,1031784,1033587,1034048,1034075,1034762,1036303,1036632,1037344,1037404,1037994,1038078,1038583,1038616,1038792,1038846,1038847,1039354,1039915,1040307,1040351,1041958,1042286,1042314,1042422,1042778,1043652,1044112,1044636,1045154,1045563,1045922,1046682,1046821,1046985,1047027,1047048,1047096,1047118,1047121,1047152,1047277,1047343,1047354,1047487,1047651,1047653,1047670,1048155,1048221,1048317,1048891,1048893,1048914,1048934,1049226,1049483,1049486,1049580,1049603,1049645,1049882,1050061,1050188,1051022,1051059,1051239,1051399,1051478,1051479,1051556,1051663,1051790,1052049,1052223,1052311,1052365,1052533,1052580,1052709,1052773,1052794,1052888,1053117,1053802,1053915,1054084,1055013,1055096,1055359,1056261,1056588,1056827,1056982,1057015,1057389,1058038,1058116,1058507,963619,964063,964944,971975,974215,981309,988784,993890 CVE References: CVE-2017-1000111,CVE-2017-1000112,CVE-2017-1000251,CVE-2017-1000252,CVE-2017-1000365,CVE-2017-10810,CVE-2017-11472,CVE-2017-11473,CVE-2017-12134,CVE-2017-12154,CVE-2017-14051,CVE-2017-14106,CVE-2017-7518,CVE-2017-7533,CVE-2017-7541,CVE-2017-7542,CVE-2017-8831 Sources used: SUSE Linux Enterprise Real Time Extension 12-SP2 (src): kernel-rt-4.4.88-18.1, kernel-rt_debug-4.4.88-18.1, kernel-source-rt-4.4.88-18.1, kernel-syms-rt-4.4.88-18.1
I guess this is OK to close it now?
This is automated batch bugzilla cleanup. The openSUSE 42.3 changed to end-of-life (EOL [1]) status. As such it is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of openSUSE (At this moment openSUSE Leap 15.1, 15.0 and Tumbleweed) please feel free to reopen this bug against that version (!you must update the "Version" component in the bug fields, do not just reopen please), or alternatively create a new ticket. Thank you for reporting this bug and we are sorry it could not be fixed during the lifetime of the release. [1] https://en.opensuse.org/Lifetime