[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Page MenuHomePhabricator
Create Task
Maniphest T299876

Upgrade database backup sources and dbprov* hosts to Bullseye + MariaDB 10.4
Closed, ResolvedPublic
Actions

Description

The database backup sources can be upgraded to Bullseye + MariaDB 10.4
This is not a blocker for upgrading the rest of hosts (including the master) on a section, given that the MariaDB version doesn't change.

backup sources:

  • db1102
  • db1116
  • db1139
  • db1140
  • db1145
  • db1150
  • db1171
  • db2097
  • db2098
  • db2099
  • db2100
  • db2101
  • db2139
  • db2141

mediabackupstemp and backup tests:

  • db1176
  • db2151
  • db1133
  • db2102

Adding the database provisioning hosts, which has to be coordinated with the above and technically also have a mariadb package (even if no mariadb server is -normally- running there) to run xtrabackup:

  • dbprov1001
  • dbprov1002
  • dbprov1003
  • dbprov2001
  • dbprov2002
  • dbprov2003

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2022, 7:47 AM
Marostegui mentioned this in T299479: Upgrade s6 to Bullseye.Jan 24 2022, 7:48 AM
• jcrespo added a project: database-backups.Jan 24 2022, 8:03 AM
• jcrespo triaged this task as Low priority.Jan 24 2022, 8:15 AM
LSobanski subscribed.Jan 24 2022, 9:32 AM
Marostegui mentioned this in T300099: Upgrade x1 to Bullseye.Jan 26 2022, 6:37 AM
Marostegui mentioned this in T300473: Upgrade s5 to Bullseye.Jan 31 2022, 6:47 AM
Ladsgroup mentioned this in T300510: Upgrade s2 to Bullseye.Jan 31 2022, 2:12 PM
Marostegui mentioned this in T300600: Upgrade s3 to Bullseye.Feb 1 2022, 8:03 AM
Marostegui mentioned this in T301653: Upgrade s7 to Bullseye.Feb 14 2022, 6:43 AM
Ladsgroup mentioned this in T302185: Upgrade s8 to Bullseye.Feb 21 2022, 1:35 AM
Ladsgroup mentioned this in T302363: Upgrade s7 to bullseye.Feb 23 2022, 4:23 AM
Ladsgroup mentioned this in T302950: Upgrade s4 to bullseye.Mar 3 2022, 3:01 AM
Marostegui mentioned this in T303171: Upgrade s1 to Bullseye.Mar 7 2022, 12:06 PM
• jcrespo claimed this task.Apr 5 2022, 9:52 AM
• jcrespo added a project: Goal.
• jcrespo renamed this task from Upgrade database backup sources to Bullseye + MariaDB 10.4 to Upgrade database backup sources and dbprov* hosts to Bullseye + MariaDB 10.4.Apr 5 2022, 9:55 AM
• jcrespo updated the task description. (Show Details)
• jcrespo raised the priority of this task from Low to High.Apr 5 2022, 9:59 AM
ops-monitoring-bot added a comment.Apr 6 2022, 4:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host db2097.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 6 2022, 5:25 PM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host db2097.codfw.wmnet with OS bullseye completed:

  • db2097 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204061657_jynus_1125810_db2097.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 7 2022, 8:17 AM
ops-monitoring-bot added a comment.Apr 7 2022, 9:00 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host db2098.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2022, 9:35 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host db2098.codfw.wmnet with OS bullseye completed:

  • db2098 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204070900_jynus_1239719_db2098.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 7 2022, 9:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host db1102.eqiad.wmnet with OS bullseye

• jcrespo updated the task description. (Show Details)Apr 7 2022, 9:44 AM
ops-monitoring-bot added a comment.Apr 7 2022, 9:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host db2099.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2022, 10:08 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host db1102.eqiad.wmnet with OS bullseye completed:

  • db1102 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204070943_jynus_124518_db1102.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 7 2022, 10:10 AM
ops-monitoring-bot added a comment.Apr 7 2022, 10:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host db1116.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2022, 10:27 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host db2099.codfw.wmnet with OS bullseye completed:

  • db2099 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204070958_jynus_1248336_db2099.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 7 2022, 10:28 AM
ops-monitoring-bot added a comment.Apr 7 2022, 10:41 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host db1116.eqiad.wmnet with OS bullseye completed:

  • db1116 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071016_jynus_158868_db1116.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 7 2022, 10:44 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host db2100.codfw.wmnet with OS bullseye

• jcrespo updated the task description. (Show Details)Apr 7 2022, 10:46 AM
ops-monitoring-bot added a comment.Apr 7 2022, 10:51 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host db1139.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2022, 11:12 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host db2100.codfw.wmnet with OS bullseye completed:

  • db2100 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071043_jynus_1255928_db2100.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 7 2022, 11:14 AM
ops-monitoring-bot added a comment.Apr 7 2022, 11:17 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host db1139.eqiad.wmnet with OS bullseye completed:

  • db1139 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071050_jynus_191796_db1139.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 7 2022, 11:17 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host db2101.codfw.wmnet with OS bullseye

• jcrespo updated the task description. (Show Details)Apr 7 2022, 11:20 AM
ops-monitoring-bot added a comment.Apr 7 2022, 11:23 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host db1140.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2022, 11:46 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host db2101.codfw.wmnet with OS bullseye completed:

  • db2101 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071117_jynus_1261319_db2101.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 7 2022, 11:49 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host db1140.eqiad.wmnet with OS bullseye completed:

  • db1140 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071123_jynus_222060_db1140.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 7 2022, 11:57 AM
• jcrespo updated the task description. (Show Details)Apr 7 2022, 12:35 PM
ops-monitoring-bot added a comment.Apr 7 2022, 12:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host db1145.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2022, 12:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host db2139.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2022, 1:14 PM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host db1145.eqiad.wmnet with OS bullseye completed:

  • db1145 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071244_jynus_300240_db1145.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 7 2022, 1:21 PM
ops-monitoring-bot added a comment.Apr 7 2022, 1:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host db1150.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2022, 1:29 PM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host db2139.codfw.wmnet with OS bullseye completed:

  • db2139 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071255_jynus_1273237_db2139.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 7 2022, 1:40 PM
ops-monitoring-bot added a comment.Apr 7 2022, 1:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host db2141.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2022, 1:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host db1150.eqiad.wmnet with OS bullseye completed:

  • db1150 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071324_jynus_339639_db1150.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 7 2022, 1:55 PM
ops-monitoring-bot added a comment.Apr 7 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host db2141.codfw.wmnet with OS bullseye completed:

  • db2141 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071349_jynus_1281164_db2141.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 7 2022, 2:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host db1171.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2022, 3:14 PM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host db1171.eqiad.wmnet with OS bullseye completed:

  • db1171 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071444_jynus_419522_db1171.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 7 2022, 3:19 PM
• jcrespo updated the task description. (Show Details)Apr 7 2022, 3:47 PM
• jcrespo updated the task description. (Show Details)Apr 7 2022, 4:46 PM
ops-monitoring-bot added a comment.Apr 8 2022, 7:31 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host db1176.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2022, 7:36 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host dbprov2001.codfw.wmnet with OS bullseye

• jcrespo updated the task description. (Show Details)Apr 8 2022, 7:59 AM
ops-monitoring-bot added a comment.Apr 8 2022, 7:59 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host db1176.eqiad.wmnet with OS bullseye completed:

  • db1176 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080730_jynus_672443_db1176.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 8 2022, 8:01 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host db2151.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2022, 8:06 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host dbprov2001.codfw.wmnet with OS bullseye completed:

  • dbprov2001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080736_jynus_1407339_dbprov2001.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 8 2022, 8:10 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T08:10:11Z] <jynus> restart db1133 T299876

• jcrespo updated the task description. (Show Details)Apr 8 2022, 8:13 AM
ops-monitoring-bot added a comment.Apr 8 2022, 8:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host dbprov1001.eqiad.wmnet with OS bullseye

• jcrespo updated the task description. (Show Details)Apr 8 2022, 8:23 AM
ops-monitoring-bot added a comment.Apr 8 2022, 8:29 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host db2102.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2022, 8:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host db2151.codfw.wmnet with OS bullseye completed:

  • db2151 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080801_jynus_1410340_db2151.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 8 2022, 8:33 AM
ops-monitoring-bot added a comment.Apr 8 2022, 8:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host dbprov2002.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2022, 8:49 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host dbprov1001.eqiad.wmnet with OS bullseye completed:

  • dbprov1001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080821_jynus_683565_dbprov1001.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 8 2022, 8:51 AM
ops-monitoring-bot added a comment.Apr 8 2022, 8:57 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host db2102.codfw.wmnet with OS bullseye completed:

  • db2102 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080829_jynus_1415186_db2102.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 8 2022, 9:00 AM
ops-monitoring-bot added a comment.Apr 8 2022, 9:08 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host dbprov2002.codfw.wmnet with OS bullseye completed:

  • dbprov2002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080840_jynus_1417809_dbprov2002.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 8 2022, 9:08 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host dbprov1002.eqiad.wmnet with OS bullseye

• jcrespo updated the task description. (Show Details)Apr 8 2022, 9:10 AM
ops-monitoring-bot added a comment.Apr 8 2022, 9:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host dbprov2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2022, 9:35 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host dbprov1002.eqiad.wmnet with OS bullseye completed:

  • dbprov1002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080908_jynus_691929_dbprov1002.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 8 2022, 9:37 AM
ops-monitoring-bot added a comment.Apr 8 2022, 9:47 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host dbprov2003.codfw.wmnet with OS bullseye completed:

  • dbprov2003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080916_jynus_1424965_dbprov2003.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 8 2022, 9:47 AM
ops-monitoring-bot added a comment.Apr 8 2022, 9:48 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin1001 for host dbprov1003.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2022, 10:15 AM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin1001 for host dbprov1003.eqiad.wmnet with OS bullseye completed:

  • dbprov1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080947_jynus_700837_dbprov1003.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• jcrespo updated the task description. (Show Details)Apr 8 2022, 10:15 AM
• jcrespo added a comment.Apr 8 2022, 10:33 AM

Reimages completed- I had to fix some permissions that were wrong after reimage.

I will now schedule a couple of backups to make sure things are working as expected and then close the ticket.

• jcrespo closed this task as Resolved.Apr 8 2022, 12:59 PM

Backups ran successfully, resolving.

• jcrespo mentioned this in T298585: Upgrade WMF database-and-backup-related hosts to bullseye.Apr 8 2022, 1:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits