[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Page MenuHomePhabricator
Paste P71498

Masterwork From Distant Lands
ActivePublic

Authored by ProdPasteBot on Tue, Dec 3, 1:41 PM.
Tags
None
Referenced Files
F57775114: Masterwork From Distant Lands
Tue, Dec 3, 1:41 PM
Subscribers
None
arnaudb@cumin1002:~ $ test-cookbook -c 1063167 sre.mysql.restart_sanitarium --test
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 32
INFO:__main__:Setting up Cookbooks change 1063167 patch set 32 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/arnaudb/cookbooks_testing/cookbooks-1063167 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/arnaudb/cookbooks_testing/cookbooks-1063167 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/changes/67/1063167/32
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/67/1063167/32 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/arnaudb/cookbooks_testing/cookbooks-1063167 rev-parse --verify change-1063167-32
8ff9a91ec32461db6cd3650048252e8dc84312ef
INFO:__main__:Switching to already existing branch change-1063167-32
INFO:__main__:Executing command /usr/bin/git -C /home/arnaudb/cookbooks_testing/cookbooks-1063167 checkout change-1063167-32
Already on 'change-1063167-32'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/arnaudb/cookbooks_testing/config.yaml sre.mysql.restart_sanitarium --test
INFO:__main__:==================================================
==> Restart all instances on 1 host(s)
Type "go" to proceed or "abort" to interrupt the execution
> go
User input is: "go"
Acquired lock for key /spicerack/locks/cookbooks/sre.mysql.restart_sanitarium: {'concurrency': 20, 'created': '2024-12-03 13:40:43.290001', 'owner': 'arnaudb@cumin1002 [1961663]', 'ttl': 1800}
START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
Will handle 1 instances
Will stop replication on db2230.codfw.wmnet -
Will restart mysql on db2230.codfw.wmnet -
----- OUTPUT of '/usr/bin/systemc... mariadb.service' -----
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:04<00:00, 4.30s/hosts]
FAIL | | 0% (0/1) [00:04<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: '/usr/bin/systemc... mariadb.service'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Will start replication on db2230.codfw.wmnet -
Will wait for replication on db2230.codfw.wmnet -
Released lock for key /spicerack/locks/cookbooks/sre.mysql.restart_sanitarium: {'concurrency': 20, 'created': '2024-12-03 13:40:43.290001', 'owner': 'arnaudb@cumin1002 [1961663]', 'ttl': 1800}
END (PASS) - Cookbook sre.mysql.restart_sanitarium (exit_code=0) Restart a pool of Sanitarium MariaDB instances and/or hosts.
arnaudb@cumin1002:~ $ test-cookbook -c 1063167 sre.mysql.restart_sanitarium --dc codfw
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 32
INFO:__main__:Setting up Cookbooks change 1063167 patch set 32 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/arnaudb/cookbooks_testing/cookbooks-1063167 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/arnaudb/cookbooks_testing/cookbooks-1063167 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/changes/67/1063167/32
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/67/1063167/32 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/arnaudb/cookbooks_testing/cookbooks-1063167 rev-parse --verify change-1063167-32
8ff9a91ec32461db6cd3650048252e8dc84312ef
INFO:__main__:Switching to already existing branch change-1063167-32
INFO:__main__:Executing command /usr/bin/git -C /home/arnaudb/cookbooks_testing/cookbooks-1063167 checkout change-1063167-32
Already on 'change-1063167-32'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/arnaudb/cookbooks_testing/config.yaml sre.mysql.restart_sanitarium --dc codfw
INFO:__main__:==================================================
==> Restart all instances on 2 host(s)
Type "go" to proceed or "abort" to interrupt the execution
> go
User input is: "go"
Acquired lock for key /spicerack/locks/cookbooks/sre.mysql.restart_sanitarium: {'concurrency': 20, 'created': '2024-12-03 13:40:59.896154', 'owner': 'arnaudb@cumin1002 [1961727]', 'ttl': 1800}
START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
Will handle 4 instances
Will stop replication on db2186.codfw.wmnet - s8
Exception raised while executing cookbook sre.mysql.restart_sanitarium:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 613, in connect
sock = socket.create_connection(
File "/usr/lib/python3.9/socket.py", line 843, in create_connection
raise err
File "/usr/lib/python3.9/socket.py", line 831, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 251, in _run
raw_ret = runner.run()
File "/home/arnaudb/cookbooks_testing/cookbooks/cookbooks/sre/mysql/restart_sanitarium.py", line 70, in run
self._rolling_restart(self.mysql.get_dbs(str(host)))
File "/home/arnaudb/cookbooks_testing/cookbooks/cookbooks/sre/mysql/restart_sanitarium.py", line 77, in _rolling_restart
instance.stop_slave()
File "/usr/lib/python3/dist-packages/spicerack/mysql.py", line 472, in stop_slave
self.execute("STOP SLAVE")
File "/usr/lib/python3/dist-packages/spicerack/mysql.py", line 318, in execute
with self.cursor(**kwargs) as (_connection, cursor):
File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
return next(self.gen)
File "/usr/lib/python3/dist-packages/spicerack/mysql.py", line 253, in cursor
with self._mysql.connect(**kwargs) as connection:
File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
return next(self.gen)
File "/usr/lib/python3/dist-packages/spicerack/mysql.py", line 152, in connect
conn = Connection(**params)
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 353, in __init__
self.connect()
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 664, in connect
raise exc
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'db2186.codfw.wmnet' (timed out)")
Released lock for key /spicerack/locks/cookbooks/sre.mysql.restart_sanitarium: {'concurrency': 20, 'created': '2024-12-03 13:40:59.896154', 'owner': 'arnaudb@cumin1002 [1961727]', 'ttl': 1800}
END (FAIL) - Cookbook sre.mysql.restart_sanitarium (exit_code=99) Restart a pool of Sanitarium MariaDB instances and/or hosts.
arnaudb@cumin1002:~ $

Event Timeline

ProdPasteBot changed the title of this paste from untitled to Masterwork From Distant Lands.