[RFC] Allow sync only work on specified systems #2601

cxfcxf · 2021-03-18T03:50:50Z

Problem: if number of systems a single cobbler instance manage gets over couple of thousands, the cobbler sync will take very long time to finish (we usually see a sync takes over 10 mins)

Observation: the process of writing pxe config files for every systems to /tftpboot take 90% of the sync time

Solution: in here, i am trying to introduce an option after regular cobbler sync as cobbler sync --systems a.b.c,a.x.c and API will result a call as sync_systems(systems), if no --systems get passed after cobbler sync, it will default to run regular cobbler sync

the function/command name can be changed, or we can make it a standalone command with options

please be note that i haven't test PR code
this is just a preliminary idea and code change, if we can agree on this change, i can make up all the piece including unit test

SchoolGuy · 2021-03-18T05:47:16Z

This is exactly what we need to circumvent our current performance bottleneck. So yes please go forward with this PR. If you can't finish it, I will take over. If you need help, ping me! Thanks for tackling this!

cxfcxf · 2021-03-18T06:03:41Z

This is exactly what we need to circumvent our current performance bottleneck. So yes please go forward with this PR. If you can't finish it, I will take over. If you need help, ping me! Thanks for tackling this!

thanks, i will make sure everything get tracked and test through out

codecov · 2021-03-18T06:10:48Z

Codecov Report

Merging #2601 (1250fe9) into master (9f997bf) will increase coverage by 0.15%.
The diff coverage is 30.55%.

@@            Coverage Diff             @@
##           master    #2601      +/-   ##
==========================================
+ Coverage   28.10%   28.25%   +0.15%     
==========================================
  Files          84       84              
  Lines       11690    11752      +62     
==========================================
+ Hits         3285     3321      +36     
- Misses       8405     8431      +26

Impacted Files	Coverage Δ
cobbler/actions/sync.py	`25.53% <7.69%> (-2.57%)`	⬇️
cobbler/remote.py	`19.37% <11.11%> (+0.01%)`	⬆️
cobbler/modules/managers/in_tftpd.py	`30.92% <11.76%> (+3.76%)`	⬆️
cobbler/cli.py	`62.45% <76.92%> (+0.39%)`	⬆️
cobbler/api.py	`41.23% <100.00%> (+0.68%)`	⬆️
cobbler/utils.py	`54.99% <0.00%> (+0.17%)`	⬆️
cobbler/manager.py	`85.71% <0.00%> (+42.85%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f997bf...1250fe9. Read the comment docs.

SchoolGuy · 2021-03-18T07:15:04Z

@watologo1 You definitely want to help out here please! Test it at least since this will spare us of a lot of time for Orthos 2!

cxfcxf · 2021-03-18T19:17:20Z

@SchoolGuy
i just added one test case and doc, the lint issue was actually due to i reused some old code from other functions you already have, i think they might be appropriated stay that way? or you can change them
A regular full sync

(venv) [root@centos-s-1vcpu-1gb-sfo3-01 cobbler]# python3 bin/cobbler sync
task started: 2021-03-18_191855_sync
task started (id=Sync, time=Thu Mar 18 19:18:55 2021)
running python triggers from /var/lib/cobbler/triggers/task/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/sync/pre/*
shell triggers finished successfully
running pre-sync triggers
cleaning trees
removing: /var/www/cobbler/images/CentOS7.4-x86_64
removing: /var/www/cobbler/images/testdistroedit
removing: /var/lib/tftpboot/pxelinux.cfg/01-7b-ee-9d-b2-5e-d5
removing: /var/lib/tftpboot/pxelinux.cfg/01-7b-ee-9d-b2-5e-d6
removing: /var/lib/tftpboot/pxelinux.cfg/default
removing: /var/lib/tftpboot/grub/images
removing: /var/lib/tftpboot/grub/system
removing: /var/lib/tftpboot/grub/system_link
removing: /var/lib/tftpboot/grub/grub.cfg
removing: /var/lib/tftpboot/grub/local_efi.cfg
removing: /var/lib/tftpboot/grub/local_legacy.cfg
removing: /var/lib/tftpboot/grub/local_powerpc-ieee1275.cfg
removing: /var/lib/tftpboot/grub/x86_64_menu_items.cfg
removing: /var/lib/tftpboot/images/CentOS7.4-x86_64
removing: /var/lib/tftpboot/images/testdistroedit
copying bootloaders
running: ['rsync', '-rpt', '--copy-links', '--exclude=.cobbler_postun_cleanup', '/var/lib/cobbler/loaders/', '/var/lib/tftpboot']
received on stdout:
received on stderr:
running: ['rsync', '-rpt', '--copy-links', '--exclude=README.grubconfig', '/var/lib/cobbler/grub_config/', '/var/lib/tftpboot']
received on stdout:
received on stderr:
copying distros to tftpboot
copying files for distro: CentOS7.4-x86_64
trying hardlink /var/www/cobbler/distro_mirror/CentOS7.4-x86_64/images/pxeboot/vmlinuz -> /var/lib/tftpboot/images/CentOS7.4-x86_64/vmlinuz
trying hardlink /var/www/cobbler/distro_mirror/CentOS7.4-x86_64/images/pxeboot/initrd.img -> /var/lib/tftpboot/images/CentOS7.4-x86_64/initrd.img
copying files for distro: testdistroedit
running: /usr/bin/sha1sum /var/log/cobbler/cobbler.log
received on stdout: 54bf36f727db6a6efac3b8c2dd85b4cbc0688eb2  /var/log/cobbler/cobbler.log

received on stderr:
trying to create cache file /var/lib/tftpboot/images/.link_cache/54bf36f727db6a6efac3b8c2dd85b4cbc0688eb2
copying: /var/log/cobbler/cobbler.log -> /var/lib/tftpboot/images/.link_cache/54bf36f727db6a6efac3b8c2dd85b4cbc0688eb2
trying cachelink /var/log/cobbler/cobbler.log -> /var/lib/tftpboot/images/.link_cache/54bf36f727db6a6efac3b8c2dd85b4cbc0688eb2 -> /var/lib/tftpboot/images/testdistroedit/cobbler.log
trying cachelink /var/log/cobbler/cobbler.log -> /var/lib/tftpboot/images/.link_cache/54bf36f727db6a6efac3b8c2dd85b4cbc0688eb2 -> /var/lib/tftpboot/images/testdistroedit/cobbler.log
copying: /var/log/cobbler/cobbler.log -> /var/lib/tftpboot/images/testdistroedit/cobbler.log
copying images
generating PXE configuration files
generating: /var/lib/tftpboot/pxelinux.cfg/01-7b-ee-9d-b2-5e-d5
generating: /var/lib/tftpboot/grub/system/7b:ee:9d:b2:5e:d5
generating: /var/lib/tftpboot/pxelinux.cfg/01-7b-ee-9d-b2-5e-d6
generating: /var/lib/tftpboot/grub/system/7b:ee:9d:b2:5e:d6
generating PXE menu structure
copying files for distro: CentOS7.4-x86_64
trying hardlink /var/www/cobbler/distro_mirror/CentOS7.4-x86_64/images/pxeboot/vmlinuz -> /var/www/cobbler/images/CentOS7.4-x86_64/vmlinuz
trying hardlink /var/www/cobbler/distro_mirror/CentOS7.4-x86_64/images/pxeboot/initrd.img -> /var/www/cobbler/images/CentOS7.4-x86_64/initrd.img
Writing template files for CentOS7.4-x86_64
copying files for distro: testdistroedit
trying symlink /var/log/cobbler/cobbler.log -> /var/www/cobbler/images/testdistroedit/cobbler.log
removing: /var/www/cobbler/images/testdistroedit/cobbler.log
trying symlink /var/log/cobbler/cobbler.log -> /var/www/cobbler/images/testdistroedit/cobbler.log
Writing template files for testdistroedit
processing boot_files for distro: CentOS7.4-x86_64
processing boot_files for distro: testdistroedit
cleaning link caches
running: find /var/lib/tftpboot/images/.link_cache -maxdepth 1 -type f -links 1 -exec rm -f '{}' ';'
received on stdout:
received on stderr:
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
shell triggers finished successfully
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running python trigger cobbler.modules.managers.genders
running shell triggers from /var/lib/cobbler/triggers/change/*
shell triggers finished successfully
*** TASK COMPLETE ***

A systems sync would like this (only one systems actually exists a.b.com and c.d.com does not exist)

(venv) [root@centos-s-1vcpu-1gb-sfo3-01 cobbler]# python3 bin/cobbler sync --system a.b.com,c.d.com
task started: 2021-03-18_191400_syncsystems
task started (id=Syncsystems, time=Thu Mar 18 19:14:00 2021)
running python triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
shell triggers finished successfully
running pre-sync triggers
did not find any system named c.d.com  <---
generating PXE configuration files
generating: /var/lib/tftpboot/pxelinux.cfg/01-7b-ee-9d-b2-5e-d5
generating: /var/lib/tftpboot/grub/system/7b:ee:9d:b2:5e:d5
generating PXE menu structure
cleaning link caches
running: find /var/lib/tftpboot/images/.link_cache -maxdepth 1 -type f -links 1 -exec rm -f '{}' ';'
received on stdout:
received on stderr:
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
shell triggers finished successfully
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running python trigger cobbler.modules.managers.genders
running shell triggers from /var/lib/cobbler/triggers/change/*
shell triggers finished successfully
*** TASK COMPLETE ***

i m seeing some more unit test can be added, but not super familiar with how you guys does the unit test here.

(venv) [root@centos-s-1vcpu-1gb-sfo3-01 tests]# pytest cli/cobbler_cli_direct_test.py
================================================= test session starts ==================================================
platform linux -- Python 3.6.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /root/cobbler, configfile: pytest.ini
collected 17 items

cli/cobbler_cli_direct_test.py ...........s...s.                                                                 [100%]

============================================ 15 passed, 2 skipped in 21.17s ============================================

please let me know if you need more case in.

hbokh

Mainly touched some spelling and consistency issues.
I haven't tested this myself and can't judge the code in a professional way.

docs/cobbler.rst

cobbler/actions/sync.py

cobbler/api.py

cobbler/cli.py

cobbler/modules/managers/in_tftpd.py

cobbler/remote.py

docs/cobbler.rst

cxfcxf · 2021-03-19T16:40:59Z

Mainly touched some spelling and consistency issues.
I haven't tested this myself and can't judge the code in a professional way.

thanks for checking my grammar.
addressed most of them, the bits i left there are those i m trying to keep thing consistent for them, if you look at other function already in there (usually just a function below), they are using the same kind of grammar (word) since i copied those from them.
an example will be background_syncsystems
if you look at this:
https://github.com/cobbler/cobbler/blob/master/cobbler/remote.py#L205
it also uses Not known what this parameter does.

same as the case i use x as a variable, cause its consistent with x the function use i copied from just below.
like in here:
https://github.com/cobbler/cobbler/blob/master/cobbler/modules/managers/in_tftpd.py#L181

i am happy to change them if its necessary, but i suggest we keep them the same, cause they are already like a standard there?
or we need to re-word all of them in all the functions, which might not be related to this issue i am trying to solve.

SchoolGuy · 2021-03-22T14:24:01Z

So although not being a formal review of your PR, I so far like what I see. Codacy complaints we will need to solve on a different PR, so yes you may ignore that.

Regarding the tests... I believe that this will be trick 8000 y. Either you do it like me and dig into pytest and how we do it or we leave this open until @nodeg or I have time to write tests for you. Merging this without tests would be unwise in my eyes, we have enough bugs.

@watologo1 maybe you could do some manual testing for this PR in case you have the time please? I don't have that time currently, since it is the final weeks of my apprenticeship as you know...

watologo1 · 2021-04-06T20:39:15Z

@cxfcxf
Hi, I have a similar/related patch which I now already adopted to latest logging cleanups from Enno (which touched some lines in nearly all files...).
#2610
It would be great if you could have a look and take a review if possible.
I introduce:
cobbler sync --dhcp
and
cobbler sync --dns

Let's discuss what the best way is to get this all together...
I had a look at the tftp syncing and wanted to clean up there a bit as well..., but I did not dare in the end.
Still it would be great if complexity is reduced there are some point of time and IMO our patches are the right way to get there...
What do you think?

cxfcxf · 2021-04-07T05:58:12Z

@cxfcxf
Hi, I have a similar/related patch which I now already adopted to latest logging cleanups from Enno (which touched some lines in nearly all files...).
#2610
It would be great if you could have a look and take a review if possible.
I introduce:
cobbler sync --dhcp
and
cobbler sync --dns
Let's discuss what the best way is to get this all together...
I had a look at the tftp syncing and wanted to clean up there a bit as well..., but I did not dare in the end.
Still it would be great if complexity is reduced there are some point of time and IMO our patches are the right way to get there...
What do you think?

totally, in my (company) daily use case, we have the cobbler manage all the bits of dns/dhcp/tftp
so for us, the changes of tftp, dns and dhcp configs are actually binding to a system record change.
I tried to solve the time a sync take for us since each of our cobbler manages thousands of servers
then the logic for me becomes if you change a system record, it will only write to tftp (pxeboot file) for that system, then regenerate dhcp/dns config file ( those configs usually are a single file unlike pxeboot file) which would save a lot of time compares to totally rewrite of all pxeboot files for thousands of servers

there was actually another way (make write on file multithreading) open ton of file handles and write all of them in the same time, this is another choice, but requires more code to handle threading.(prob not a good route thus not picked)

wondering if you agree on above logic?

if we agree, we can have all of them (i can rebase my change on top of yours with some modification of code)

cobbler sync --dns
cobbler sync --dhcp
cobbler sync --system a.b.c  (this will also do dns and dhcp)

SchoolGuy · 2021-04-09T12:08:39Z

@cxfcxf We will rebase this for you and write tests for you. Thank you for your efforts! This will circumvent the performance issues greatly!

This will refactor cobbler sync by moving everything from litesync.py into sync.py. Furthermore `make_tftpboot()` was made private and renamed to `__create_tftpboot_dirs()` since it does only create the required directories. This is a preliminary work for cobbler#2601.

This will 6D40 refactor cobbler sync by moving everything from litesync.py into sync.py. Furthermore `make_tftpboot()` was made private and renamed to `__create_tftpboot_dirs()` since it does only create the required directories. This is a preliminary work for cobbler#2601.

nodeg · 2021-04-27T06:18:15Z

Rebase done. Still todo before merging:

Testing
Extend the unit tests from @SchoolGuy

cobbler/api.py

nodeg · 2021-05-03T10:00:02Z

I did some testing:

$ cobbler version
Cobbler 3.2.1
  source: 87d60a6d, Mon May 3 11:11:42 2021 +0200
  build time: Mon May  3 09:25:23 2021

$ cobbler system list
   dom_test
   dom_test2

$ tree -L 3 .
.
├── boot
├── etc
├── grub
│   ├── images -> ../images
│   ├── system
│   ├── system_link
│   └── x86_64_menu_items.cfg
├── images
│   └── CentOS-Stream-x86_64
│       ├── initrd.img
│       └── vmlinuz
├── images2
├── ppc
├── pxelinux.cfg
│   └── default
└── s390x

12 directories, 4 files

$ cobbler system edit --name=dom_test --mac-address=random

$ tree -L 3 .
.
├── boot
├── etc
├── grub
│   ├── images -> ../images
│   ├── system
│   │   └── 00:16:3e:0e:ec:13
│   ├── system_link
│   │   └── dom_test -> ../system/00:16:3e:0e:ec:13
│   └── x86_64_menu_items.cfg
├── images
│   └── CentOS-Stream-x86_64
│       ├── initrd.img
│       └── vmlinuz
├── images2
├── ppc
├── pxelinux.cfg
│   ├── 01-00-16-3e-0e-ec-13
│   └── default
└── s390x

12 directories, 7 files

$ cobbler system report
Name                           : dom_test
(...)
Profile                        : CentOS-Stream:x86_64
(...)
MAC Address                    : 00:16:3e:0e:ec:13
(...)

After generati 9E88 ng a random MAC address, I manually changed a file and did a cobbler sync --system=dom_test to see if my changes will get overwritten.

$ cat pxelinux.cfg/01-00-16-3e-0e-ec-13 
default linux
prompt 0
timeout 1
label linux
        kernel /images/CentOS-Stream-x86_64/vmlinuz
        ipappend 2
        append initrd=/images/CentOS-Stream-x86_64/initrd.img  kssendmac inst.ks=http://127.0.0.1/cblr/svc/op/autoinstall/system/dom_test

$ sudo vi pxelinux.cfg/01-00-16-3e-0e-ec-13

$ cat pxelinux.cfg/01-00-16-3e-0e-ec-13 
default linux_dom_test
prompt 0
timeout 4
label linux_dom_test
        kernel /images/CentOS-Stream-x86_64/vmlinuz
        ipappend 2
        append initrd=/images/CentOS-Stream-x86_64/initrd.img  kssendmac inst.ks=http://127.0.0.1/cblr/svc/op/autoinstall/system/dom_test

$ cobbler sync --systems=dom_test
task started: 2021-05-03_114354_syncsystems
task started (id=Syncsystems, time=Mon May  3 11:43:54 2021)
running python triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
shell triggers finished successfully
sync_systems
running pre-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/sync/pre/*
shell triggers finished successfully
generating PXE configuration files
generating: /srv/tftpboot/pxelinux.cfg/01-00-16-3e-0e-ec-13
generating: /srv/tftpboot/grub/system/00:16:3e:0e:ec:13
removing: /srv/tftpboot/grub/system_link/dom_test
generating PXE menu structure
mkdir: /srv/tftpboot/pxelinux.cfg
cleaning link caches
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
shell triggers finished successfully
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running python trigger cobbler.modules.managers.genders
running shell triggers from /var/lib/cobbler/triggers/change/*
shell triggers finished successfully
*** TASK COMPLETE ***


$ cat pxelinux.cfg/01-00-16-3e-0e-ec-13 
default linux
prompt 0
timeout 1
label linux
        kernel /images/CentOS-Stream-x86_64/vmlinuz
        ipappend 2
        append initrd=/images/CentOS-Stream-x86_64/initrd.img  kssendmac inst.ks=http://127.0.0.1/cblr/svc/op/autoinstall/system/dom_test

My changes got sucesssfully overwritten.
Another test I did was the complete removal of the tftpboot folder and do a systems sync

$ rm -rf /srv/tftpboot

cobbler sync --systems=dom_test
task started: 2021-05-03_114549_syncsystems
task started (id=Syncsystems, time=Mon May  3 11:45:49 2021)
running python triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
shell triggers finished successfully
sync_systems
mkdir: /srv/tftpboot/pxelinux.cfg
mkdir: /srv/tftpboot/grub
mkdir: /srv/tftpboot/images
mkdir: /srv/tftpboot/ppc
mkdir: /srv/tftpboot/etc
running pre-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/sync/pre/*
shell triggers finished successfully
generating PXE configuration files
generating: /srv/tftpboot/pxelinux.cfg/01-00-16-3e-0e-ec-13
generating: /srv/tftpboot/grub/system/00:16:3e:0e:ec:13
Got "[Errno 2] No such file or directory: '/srv/tftpboot/grub/system/00:16:3e:0e:ec:13'" while trying to write /srv/tftpboot/grub/system/00:16:3e:0e:ec:13
Trying to write /srv/tftpboot/grub/system/00:16:3e:0e:ec:13 again after some delay.
Exception occurred: <class 'FileNotFoundError'>
Exception value: [Errno 2] No such file or directory: '/srv/tftpboot/grub/system/00:16:3e:0e:ec:13'
Exception Info:
!!! TASK FAILED !!!

$ tree -L 4 tftpboot/
tftpboot/
├── etc
├── grub
│   └── images -> ../images
├── images
├── ppc
└── pxelinux.cfg
    └── 01-00-16-3e-0e-ec-13

6 directories, 1 file

The systems sync worked correctly because in this case a full sync has to be done beforehand. After doing a full sync the systems sync works correctly.

$ cobbler sync
task started: 2021-05-03_114639_sync
task started (id=Sync, time=Mon May  3 11:46:39 2021)
running python triggers from /var/lib/cobbler/triggers/task/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/sync/pre/*
shell triggers finished successfully
syncing all
running pre-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/sync/pre/*
shell triggers finished successfully
cleaning trees
removing: /srv/www/cobbler/images/CentOS-Stream-x86_64
removing: /srv/tftpboot/pxelinux.cfg/01-00-16-3e-0e-ec-13
removing: /srv/tftpboot/grub/images
copying bootloaders
running: ['rsync', '-rpt', '--copy-links', '--exclude=.cobbler_postun_cleanup', '/var/lib/cobbler/loaders/', '/srv/tftpboot']
received on stdout: 
received on stderr: 
running: ['rsync', '-rpt', '--copy-links', '--exclude=README.grubconfig', '/var/lib/cobbler/grub_config/', '/srv/tftpboot']
received on stdout: 
received on stderr: 
copying distros to tftpboot
copying files for distro: CentOS-Stream-x86_64
mkdir: /srv/tftpboot/images/CentOS-Stream-x86_64
trying hardlink /srv/www/cobbler/distro_mirror/CentOS-Stream/images/pxeboot/vmlinuz -> /srv/tftpboot/images/CentOS-Stream-x86_64/vmlinuz
trying hardlink /srv/www/cobbler/distro_mirror/CentOS-Stream/images/pxeboot/initrd.img -> /srv/tftpboot/images/CentOS-Stream-x86_64/initrd.img
copying images
generating PXE configuration files
generating: /srv/tftpboot/pxelinux.cfg/01-00-16-3e-0e-ec-13
generating: /srv/tftpboot/grub/system/00:16:3e:0e:ec:13
generating PXE menu structure
mkdir: /srv/tftpboot/pxelinux.cfg
copying files for distro: CentOS-Stream-x86_64
mkdir: /srv/www/cobbler/images/CentOS-Stream-x86_64
trying hardlink /srv/www/cobbler/distro_mirror/CentOS-Stream/images/pxeboot/vmlinuz -> /srv/www/cobbler/images/CentOS-Stream-x86_64/vmlinuz
trying hardlink /srv/www/cobbler/distro_mirror/CentOS-Stream/images/pxeboot/initrd.img -> /srv/www/cobbler/images/CentOS-Stream-x86_64/initrd.img
Writing template files for CentOS-Stream-x86_64
mkdir: /srv/tftpboot/pxelinux.cfg
processing boot_files for distro: CentOS-Stream-x86_64
cleaning link caches
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
shell triggers finished successfully
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running python trigger cobbler.modules.managers.genders
running shell triggers from /var/lib/cobbler/triggers/change/*
shell triggers finished successfully
*** TASK COMPLETE ***

$ tree -L 4 tftpboot/
tftpboot/
├── etc
├── grub
│   ├── grub.cfg
│   ├── images -> ../images
│   ├── local_efi.cfg
│   ├── local_legacy.cfg
│   ├── local_powerpc-ieee1275.cfg
│   ├── system
│   │   └── 00:16:3e:0e:ec:13
│   ├── system_link
│   │   └── dom_test -> ../system/00:16:3e:0e:ec:13
│   └── x86_64_menu_items.cfg
├── grub.cfg
├── images
│   └── CentOS-Stream-x86_64
│       ├── initrd.img
│       └── vmlinuz
├── ppc
└── pxelinux.cfg
    ├── 01-00-16-3e-0e-ec-13
    └── default

9 directories, 12 files

$ cobbler sync --systems=dom_test
task started: 2021-05-03_114704_syncsystems
task started (id=Syncsystems, time=Mon May  3 11:47:04 2021)
running python triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
shell triggers finished successfully
sync_systems
running pre-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/sync/pre/*
shell triggers finished successfully
generating PXE configuration files
generating: /srv/tftpboot/pxelinux.cfg/01-00-16-3e-0e-ec-13
generating: /srv/tftpboot/grub/system/00:16:3e:0e:ec:13
removing: /srv/tftpboot/grub/system_link/dom_test
generating PXE menu structure
mkdir: /srv/tftpboot/pxelinux.cfg
cleaning link caches
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
shell triggers finished successfully
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running python trigger cobbler.modules.managers.genders
running shell triggers from /var/lib/cobbler/triggers/change/*
shell triggers finished successfully
*** TASK COMPLETE ***

@cxfcxf Would it be possible for you to to some real life testing with the newly introduced changes? Otherwise I think we are good to go for this PR to be merged. The rest of the unit tests, especially for methods which get called by your introduced changes, will be done in the future in another PR.

SchoolGuy · 2021-05-03T10:50:01Z

There are two things left to get this merged:

CI Unbreak it: See https://github.com/cobbler/cobbler/pull/2601/checks?check_run_id=2490888272#step:7:46
Docs: Document that a full sync is required to have a successful partial sync.

nodeg · 2021-05-03T11:50:18Z

@cxfcxf From @SchoolGuy and my perspective this PR should be fine for merging now but we still want to wait for some feedback from you.

nodeg · 2021-05-04T09:26:51Z

One last thing I will do is a rebase to clean up the git history.

@cxfcxf

If managing a vast amount of systems via a single Cobbler instance the amount of time a `cobbler sync` call needs to synchronize everything is quite a lot. @cxfcxf observed that writing PXE configuration files for every system to the tftpboot location takes up to 90% of the time of a `cobbler sync`. So he introduced the option to synchronize only specified systems with `cobbler sync --systems=system1,system2`. This also adds some basic unit tests and unit test skeletons which has to be completet in the future.

cxfcxf · 2021-05-04T13:55:16Z

@cxfcxf From @SchoolGuy and my perspective this PR should be fine for merging now but we still want to wait for some feedback from you.

PR looks good to me, thanks for point out the additional deserialize() and writing to all the test cases

cxfcxf · 2021-05-04T13:58:54Z

I did some testing:

$ cobbler version
Cobbler 3.2.1
  source: 87d60a6d, Mon May 3 11:11:42 2021 +0200
  build time: Mon May  3 09:25:23 2021

$ cobbler system list
   dom_test
   dom_test2

$ tree -L 3 .
.
├── boot
├── etc
├── grub
│   ├── images -> ../images
│   ├── system
│   ├── system_link
│   └── x86_64_menu_items.cfg
├── images
│   └── CentOS-Stream-x86_64
│       ├── initrd.img
│       └── vmlinuz
├── images2
├── ppc
├── pxelinux.cfg
│   └── default
└── s390x

12 directories, 4 files

$ cobbler system edit --name=dom_test --mac-address=random

$ tree -L 3 .
.
├── boot
├── etc
├── grub
│   ├── images -> ../images
│   ├── system
│   │   └── 00:16:3e:0e:ec:13
│   ├── system_link
│   │   └── dom_test -> ../system/00:16:3e:0e:ec:13
│   └── x86_64_menu_items.cfg
├── images
│   └── CentOS-Stream-x86_64
│       ├── initrd.img
│       └── vmlinuz
├── images2
├── ppc
├── pxelinux.cfg
│   ├── 01-00-16-3e-0e-ec-13
│   └── default
└── s390x

12 directories, 7 files

$ cobbler system report
Name                           : dom_test
(...)
Profile                        : CentOS-Stream:x86_64
(...)
MAC Address                    : 00:16:3e:0e:ec:13
(...)

After generating a random MAC address, I manually changed a file and did a cobbler sync --system=dom_test to see if my changes will get overwritten.

$ cat pxelinux.cfg/01-00-16-3e-0e-ec-13 
default linux
prompt 0
timeout 1
label linux
        kernel /images/CentOS-Stream-x86_64/vmlinuz
        ipappend 2
        append initrd=/images/CentOS-Stream-x86_64/initrd.img  kssendmac inst.ks=http://127.0.0.1/cblr/svc/op/autoinstall/system/dom_test

$ sudo vi pxelinux.cfg/01-00-16-3e-0e-ec-13

$ cat pxelinux.cfg/01-00-16-3e-0e-ec-13 
default linux_dom_test
prompt 0
timeout 4
label linux_dom_test
        kernel /images/CentOS-Stream-x86_64/vmlinuz
        ipappend 2
        append initrd=/images/CentOS-Stream-x86_64/initrd.img  kssendmac inst.ks=http://127.0.0.1/cblr/svc/op/autoinstall/system/dom_test

$ cobbler sync --systems=dom_test
task started: 2021-05-03_114354_syncsystems
task started (id=Syncsystems, time=Mon May  3 11:43:54 2021)
running python triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
shell triggers finished successfully
sync_systems
running pre-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/sync/pre/*
shell triggers finished successfully
generating PXE configuration files
generating: /srv/tftpboot/pxelinux.cfg/01-00-16-3e-0e-ec-13
generating: /srv/tftpboot/grub/system/00:16:3e:0e:ec:13
removing: /srv/tftpboot/grub/system_link/dom_test
generating PXE menu structure
mkdir: /srv/tftpboot/pxelinux.cfg
cleaning link caches
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
shell triggers finished successfully
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running python trigger cobbler.modules.managers.genders
running shell triggers from /var/lib/cobbler/triggers/change/*
shell triggers finished successfully
*** TASK COMPLETE ***


$ cat pxelinux.cfg/01-00-16-3e-0e-ec-13 
default linux
prompt 0
timeout 1
label linux
        kernel /images/CentOS-Stream-x86_64/vmlinuz
        ipappend 2
        append initrd=/images/CentOS-Stream-x86_64/initrd.img  kssendmac inst.ks=http://127.0.0.1/cblr/svc/op/autoinstall/system/dom_test

My changes got sucesssfully overwritten.
Another test I did was the complete removal of the tftpboot folder and do a systems sync

$ rm -rf /srv/tftpboot

cobbler sync --systems=dom_test
task started: 2021-05-03_114549_syncsystems
task started (id=Syncsystems, time=Mon May  3 11:45:49 2021)
running python triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
shell triggers finished successfully
sync_systems
mkdir: /srv/tftpboot/pxelinux.cfg
mkdir: /srv/tftpboot/grub
mkdir: /srv/tftpboot/images
mkdir: /srv/tftpboot/ppc
mkdir: /srv/tftpboot/etc
running pre-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/sync/pre/*
shell triggers finished successfully
generating PXE configuration files
generating: /srv/tftpboot/pxelinux.cfg/01-00-16-3e-0e-ec-13
generating: /srv/tftpboot/grub/system/00:16:3e:0e:ec:13
Got "[Errno 2] No such file or directory: '/srv/tftpboot/grub/system/00:16:3e:0e:ec:13'" while trying to write /srv/tftpboot/grub/system/00:16:3e:0e:ec:13
Trying to write /srv/tftpboot/grub/system/00:16:3e:0e:ec:13 again after some delay.
Exception occurred: <class 'FileNotFoundError'>
Exception value: [Errno 2] No such file or directory: '/srv/tftpboot/grub/system/00:16:3e:0e:ec:13'
Exception Info:
!!! TASK FAILED !!!

$ tree -L 4 tftpboot/
tftpboot/
├── etc
├── grub
│   └── images -> ../images
├── images
├── ppc
└── pxelinux.cfg
    └── 01-00-16-3e-0e-ec-13

6 directories, 1 file

The systems sync worked correctly because in this case a full sync has to be done beforehand. After doing a full sync the systems sync works correctly.

$ cobbler sync
task started: 2021-05-03_114639_sync
task started (id=Sync, time=Mon May  3 11:46:39 2021)
running python triggers from /var/lib/cobbler/triggers/task/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/sync/pre/*
shell triggers finished successfully
syncing all
running pre-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/sync/pre/*
shell triggers finished successfully
cleaning trees
removing: /srv/www/cobbler/images/CentOS-Stream-x86_64
removing: /srv/tftpboot/pxelinux.cfg/01-00-16-3e-0e-ec-13
removing: /srv/tftpboot/grub/images
copying bootloaders
running: ['rsync', '-rpt', '--copy-links', '--exclude=.cobbler_postun_cleanup', '/var/lib/cobbler/loaders/', '/srv/tftpboot']
received on stdout: 
received on stderr: 
running: ['rsync', '-rpt', '--copy-links', '--exclude=README.grubconfig', '/var/lib/cobbler/grub_config/', '/srv/tftpboot']
received on stdout: 
received on stderr: 
copying distros to tftpboot
copying files for distro: CentOS-Stream-x86_64
mkdir: /srv/tftpboot/images/CentOS-Stream-x86_64
trying hardlink /srv/www/cobbler/distro_mirror/CentOS-Stream/images/pxeboot/vmlinuz -> /srv/tftpboot/images/CentOS-Stream-x86_64/vmlinuz
trying hardlink /srv/www/cobbler/distro_mirror/CentOS-Stream/images/pxeboot/initrd.img -> /srv/tftpboot/images/CentOS-Stream-x86_64/initrd.img
copying images
generating PXE configuration files
generating: /srv/tftpboot/pxelinux.cfg/01-00-16-3e-0e-ec-13
generating: /srv/tftpboot/grub/system/00:16:3e:0e:ec:13
generating PXE menu structure
mkdir: /srv/tftpboot/pxelinux.cfg
copying files for distro: CentOS-Stream-x86_64
mkdir: /srv/www/cobbler/images/CentOS-Stream-x86_64
trying hardlink /srv/www/cobbler/distro_mirror/CentOS-Stream/images/pxeboot/vmlinuz -> /srv/www/cobbler/images/CentOS-Stream-x86_64/vmlinuz
trying hardlink /srv/www/cobbler/distro_mirror/CentOS-Stream/images/pxeboot/initrd.img -> /srv/www/cobbler/images/CentOS-Stream-x86_64/initrd.img
Writing template files for CentOS-Stream-x86_64
mkdir: /srv/tftpboot/pxelinux.cfg
processing boot_files for distro: CentOS-Stream-x86_64
cleaning link caches
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
shell triggers finished successfully
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running python trigger cobbler.modules.managers.genders
running shell triggers from /var/lib/cobbler/triggers/change/*
shell triggers finished successfully
*** TASK COMPLETE ***

$ tree -L 4 tftpboot/
tftpboot/
├── etc
├── grub
│   ├── grub.cfg
│   ├── images -> ../images
│   ├── local_efi.cfg
│   ├── local_legacy.cfg
│   ├── local_powerpc-ieee1275.cfg
│   ├── system
│   │   └── 00:16:3e:0e:ec:13
│   ├── system_link
│   │   └── dom_test -> ../system/00:16:3e:0e:ec:13
│   └── x86_64_menu_items.cfg
├── grub.cfg
├── images
│   └── CentOS-Stream-x86_64
│       ├── initrd.img
│       └── vmlinuz
├── ppc
└── pxelinux.cfg
    ├── 01-00-16-3e-0e-ec-13
    └── default

9 directories, 12 files

$ cobbler sync --systems=dom_test
task started: 2021-05-03_114704_syncsystems
task started (id=Syncsystems, time=Mon May  3 11:47:04 2021)
running python triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/syncsystems/pre/*
shell triggers finished successfully
sync_systems
running pre-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/sync/pre/*
shell triggers finished successfully
generating PXE configuration files
generating: /srv/tftpboot/pxelinux.cfg/01-00-16-3e-0e-ec-13
generating: /srv/tftpboot/grub/system/00:16:3e:0e:ec:13
removing: /srv/tftpboot/grub/system_link/dom_test
generating PXE menu structure
mkdir: /srv/tftpboot/pxelinux.cfg
cleaning link caches
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
shell triggers finished successfully
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running python trigger cobbler.modules.managers.genders
running shell triggers from /var/lib/cobbler/triggers/change/*
shell triggers finished successfully
*** TASK COMPLETE ***

@cxfcxf Would it be possible for you to to some real life testing with the newly introduced changes? Otherwise I think we are good to go for this PR to be merged. The rest of the unit tests, especially for methods which get called by your introduced changes, will be done in the future in another PR.

i can do as much as you, we are running on cobbler 2.8, i did some test on our test lab, with only the code change i put up here writing in python 2.7 since in production we are still running on python2.7, we would like to upgrade python to 3.x and upgrade cobbler as well, thus the PR,

nodeg · 2021-05-04T14:00:57Z

PR looks good to me, thanks for point out the additional deserialize() and writing to all the test cases

Your welcome and thank you very much for your contribution and this PR!

nodeg

LGTM. Waiting for your feedback @SchoolGuy.

SchoolGuy

LGTM

cxfcxf changed the title ~~WIP: allow sync only work on specified systems~~ RFC: allow sync only work on specified systems Mar 18, 2021

SchoolGuy requested a review from a team March 18, 2021 05:42

SchoolGuy added this to the v3.3.0 milestone Mar 18, 2021

SchoolGuy requested a review from watologo1 March 18, 2021 07:15

hbokh reviewed Mar 19, 2021

View reviewed changes

SchoolGuy self-requested a review March 23, 2021 17:57

SchoolGuy changed the title ~~RFC: allow sync only work on specified systems~~ [RFC] Allow sync only work on specified systems Mar 23, 2021

nodeg added API main Not a release but referring to the Git main branch Usability labels Apr 7, 2021

nodeg mentioned this pull request Apr 9, 2021

Use a base class ManagerModule for all modules/manager/*.py module cl… #2610

Merged

nodeg force-pushed the sync_systems branch from 70b9e66 to 56fcba1 Compare April 9, 2021 13:20

nodeg mentioned this pull request Apr 14, 2021

Refactor cobbler sync #2615

Merged

nodeg force-pushed the sync_systems branch from f42909a to fd52a7f Compare April 26, 2021 12:27

nodeg reviewed Apr 30, 2021

View reviewed changes

cobbler/api.py Outdated Show resolved Hide resolved

nodeg force-pushed the sync_systems branch from 87d60a6 to 53e0d75 Compare May 3, 2021 11:32

nodeg added 3 commits May 4, 2021 11:37

fix: Line length

8eb4f64

fix: Logger usage

1250fe9

nodeg force-pushed the sync_systems branch from 9b78d23 to 1250fe9 Compare May 4, 2021 09:57

nodeg self-requested a review May 4, 2021 14:01

nodeg approved these changes May 4, 2021

View reviewed changes

SchoolGuy approved these changes May 4, 2021

View reviewed changes

SchoolGuy merged commit f8faec3 into cobbler:master May 4, 2021

Uh oh!

[RFC] Allow sync only work on specified systems #2601

[RFC] Allow sync only work on specified systems #2601

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!