feat(dre): qualification via command #649

NikolaMilosa · 2024-07-20T02:29:47Z

Previously qualification used to be run with python scripts that were sitting in a different repository and were hard to maintain. With this addition we are a couple of steps closer to achieving better solution for qualification.

Reasons why this is good:

Less code duplication
All in one place and technology
Allows similar / better running.

To run everything you need a:

Network (any network apart from mainnet which is not allowed by the tool)
A neuron (with majority so its proposals get adopted instantly)
Prometheus instance used for scraping the metrics of xnet and workload tests
Deployment name (staging, ...) again used for the ic label.

Since this command is rarely meant to be used manually it provides a lot of options and to run it one could use a following script:

#!/bin/bash

# To obtain all these url's one can spin up a farm testnet from ic repo
# Goto the root and run ./gitlab-ci/container/container-run.sh
# Create from there run:
# ict testnet create --lifetime-mins 180 --from-ic-config-path <(cat <<EOF
# {
#  "subnets": [
#    {
#      "subnet_type": "application",
#      "num_nodes": 4
#    },
#    {
#      "subnet_type": "application",
#      "num_nodes": 4
#    },
#    {
#      "subnet_type": "system",
#      "num_nodes": 4
#    }
#  ],
#  "num_unassigned_nodes": 0,
#  "initial_version": "7dee90107a88b836fc72e78993913988f4f73ca2"
# }
# EOF
# )

# NOTE! Farm testnets need some time to deploy and when they are done you HAVE TO KEEP
# THE TERMINAL SESSION going and not exit the command 

NNS_URL_1="http://[system-subnet-ipv6-1]:8080"
NNS_URL_2="http://[system-subnet-ipv6-2]:8080"
NNS_URL_3="http://[system-subnet-ipv6-3]:8080"
NNS_URL_4="http://[system-subnet-ipv6-4]:8080"
PROM_URL="http://prometheus.from-config--<fill-in>.testnet.farm.dfinity.systems/api/v1/query"
DEPLOYMENT_NAME="from-config--<fill-in>"
# On farm testnets this is the neuron with the most voting power so you can leave it as is
TEST_NEURON_ID=449479075714955186 
# To obtain the identity.pem you can create a simple rust binary, 
# add ic-nervous-system-common-test-keys = { workspace = true } as a dependency
# and write a simple function in main that contains => println!("{}", TEST_NEURON_1_OWNER_KEYPAIR.to_pem());
# and store it to the following path
TEST_NEURON_KEY="$HOME/.config/dfx/identity/test_neuron_1/identity.pem"
rm -rf ~/.cache/ic-registry-cache/configured-testnet

# At the moment of time of writing this, cargo on linux machines doesn't compile because of 
# canister code being in the same workspace as other tools so use bazel
bazel run //rs/cli:dre -- --yes --network configured-testnet \
   --nns-urls $NNS_URL_1 --nns-urls $NNS_URL_2 --nns-urls $NNS_URL_3 \
   --nns-urls $NNS_URL_4 --neuron-id $TEST_NEURON_ID \ 
   --private-key-pem $TEST_NEURON_KEY  qualify execute \
   --version a3831c87440df4821b435050c8a8fcb3745d86f6 \ 
   --prometheus-endpoint $PROM_URL --deployment-name $DEPLOYMENT_NAME

The idea is that this gives us the driving power of qualifying on any network (apart from mainnet) and there would be one more tool which is meant to automate the things that script right now does. That tool will run some more checks as well and will call dre as a library instead of having to call it as a separate binary which gives us less code duplication again and compile time errors if something in the cli args changes.

The ETA I've had when testing this across multiple runs was around 58 mins which equates to a run on staging (give or take).

One more feature the tool has is the ability to specify the steps. This means that only those steps will be executed (if used with the execute command since list only prints the plan)
One can run:

dre qualify list --step-range 1..5

The output looks like this:

bazel run //rs/cli:dre -- qualify list --step-range 1..2 
INFO: Invocation ID: 01c269a0-ae7a-4a47-869c-458b21493117
INFO: Analyzed target //rs/cli:dre (26 packages loaded, 188 targets configured).
INFO: Found 1 target...
Target //rs/cli:dre up-to-date:
  bazel-bin/rs/cli/dre
INFO: Elapsed time: 0.301s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
 INFO  dre > Running version 0.4.3-5b3ca130-dirty
| Index | Will run | Name                                 | Help                                                                                     |
| ..... | ........ | .................................... | ........................................................................................ |
|   0   |  false   | ensure_blessed_revision              | This step runs the check to determine if all versions are blessed                        |
|   1   |   true   | update_deployment_canisters          | This step ensures that deployment canisters match the version of nns deployment canister |
|   2   |   true   | upgrade_application_subnet_version   | This step upgrade all the application subnets to the desired version                     |
|   3   |  false   | upgrade_system_subnet_version        | This step upgrade all the system subnets to the desired version                          |
|   4   |  false   | upgrade_unassigned_nodes_version     | This step upgrade all the unassigned nodes to the desired version                        |
|   5   |  false   | workload_test                        | This step runs the workload test on one app subnet.                                      |
|   6   |  false   | xnet_test                            | This step runs xnet test on a subnet                                                     |
|   7   |  false   | retire_blessed_versions              | Ensure that versions are retired                                                         |
|   8   |  false   | ensure_blessed_revision              | This step runs the check to determine if all versions are blessed                        |
|   9   |  false   | downgrade_application_subnet_version | This step downgrade all the application subnets to the desired version                   |
|  10   |  false   | downgrade_system_subnet_version      | This step downgrade all the system subnets to the desired version                        |
|  11   |  false   | downgrade_unassigned_nodes_version   | This step downgrade all the unassigned nodes to the desired version                      |

I will attach a log of a successful qualification (which is quite long here as well).
output.log

rs/cli/src/ctx.rs

rs/cli/Cargo.toml

rs/cli/src/qualification/mod.rs

rs/cli/src/qualification/run_workload_test.rs

NikolaMilosa added 30 commits July 17, 2024 18:25

structuring the command

bbe552e

adding simple tabular util

2a66834

implementing starting of execution

4313fef

started work on ensuring blessed versions

6406608

adding ic admin call

36b16c8

adding creating proposal

752c24e

ensuring blessed replica versions

9f7d98b

Merge branch 'main' into nm-qualification-via-dre-command

8ce8f8d

finishing first step

66c1cea

implementing upgrading

719f06c

fixing operator on non mainnnet networks

c3392ed

linting

37e1534

upgrade subnet rule

9d5c7b0

linting

76fbb83

fixing erroring on registry errors

< 8000 div class="pr-1 flex-shrink-0" style="width: 16px;">

dfa4840

prettier print

ab90d94

implementing unelecting

91b9980

linting

682e508

adding exponential backoff for placing proposals

14c2047

adding simple filter for steps

148eec2

adding listing

fba3abf

adding xnet test

3104591

refactoring

1fa1ee2

fixing xnet test

125ef6b

implementing xnet test

05e6a14

fixing clippy

fdfab7a

adding command

a4852d4

prettier print

79dc7b7

implementing xnet test

4a20778

Merge branch 'main' into nm-qualification-via-dre-command

2597063

NikolaMilosa requested a review from a team as a code owner July 20, 2024 02:29

sa-github-api added 3 commits July 22, 2024 11:09

Branch was auto-updated.

a7e4ae2

Branch was auto-updated.

d448e37

Branch was auto-updated.

59e6a88