8000 CI: New Manual workflow for testing 2 builds by bbpetukhov · Pull Request #744 · bloomberg/blazingmq · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

CI: New Manual workflow for testing 2 builds #744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 115 commits into
base: main
Choose a base branch
from
Open

CI: New Manual workflow for testing 2 builds #744

wants to merge 115 commits into from

Conversation

bbpetukhov
Copy link
Collaborator
@bbpetukhov bbpetukhov commented May 12, 2025 •
edited
Loading

Before accepting this PR, it's possible for run the new workflow from here: https://github.com/bbpetukhov/blazingmq/actions/workflows/manual.yaml (only collaborators of the fork)

Describe your changes
Added a new workflow to be run manually. It builds 2 versions of BlazingMQ broker binaries and runs a new batch of integration tests which covers broker binary upgrade via stop/restart.

The build job is extracted into a separed file and reused between existing Tests workflow and a new manually run one.
In case of UTs cmake "all.t" target generated huge size of build artifacts (~1GB), so the upload is skipped and tests are run right away as a step in the build job.

image

Git refs for builds 1 and 2 can be specified via input params.
Integration tests sources are downloaded from the selected Branch ("Use workflow from").

Switch from cache to Artifacts
The build_ubuntu workflow migrated from storing build in cache to uploading as artifact. According to the documentation cache is ideal for storing results for later sharing them between workflow runs. Existing "Dependencies" workflow uploads built deps into cache to be shared in other build workflows -- this is a solid application for cache.
However for sharing data between jobs in a single workflow runs it's advised to store them as artifacts, because upload/download operations are synchronous (once data is uploaded, it's immediately available to be downloaded in the next job). Meanwhile cache save/restore operations are asynchronous, thus the results may not be immediately available for restore in the next job. Also due to security restrictions experiments showed that it's impossible to restore from 2 caches in one job which is necessary to get 2 versions of broker.
Total storage for artifacts is unlimited for public repositories.
Making the builds publicly available for download does not seem like a vulnerability, because since the repo is public, similar workflow can be easily simulated in another public fork.

However the downside of artifacts is that being a plain .zip archive it drops execute permissions for broker and bmq-tool binaries. Execute permissions have to be restored after artifact download.

Testing performed

  • Existing Tests workflow passed, i.e. build, UTs and ITs passed
  • New workflow ITs passed. Made sure that the binary upgrade from ref1 to ref2 indeed happens and logged.
    There are 2 new tests running cluster of 3 nodes.
    The first test starts the cluster with all nodes running version 1, then cluster is stopped, all nodes upgrade to version 2 and restarts again.
    The second test upgrades nodes one by one every time restarting the whole cluster.

Additional context

  • Added new "multi3" fixture cluster with odd number of nodes for Integration tests.
  • Clean up of redundant code in failures check logic
  • For both ref1 and ref2 build results are stored in cache and restored next time the manual workflow is initiated for the same refs.
  • In Integration tests cluster leader can be preset via BLAZINGMQ_LEADER_NAME env var (CI: Can preset leader via env var BLAZINGMQ_LEADER_NAME for integration tests #693)

bbpetukhov added 30 commits April 29, 2025 12:40
@bbpetukhov
Can checkout 2 refs
0486786 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Refactor manual ci
dfa7952 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Can run ITs for 2 broker versions
fa35a1b 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
fix manual ci
ba83d72 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Fix build; fix run UTs
a072147 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Fix ITs in manual ci
9e10a81 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Add cache_key_suffix
7d88fa7 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Cancel build if cache exists
7e69525 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Fix more typos
b25e133 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Do not use global variables
59132b4 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
CI Manual fail-on-cache-miss: true
b38e548 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Save build to artifact instead of cache
918724a 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Fix ITs run script location
59ffde8 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Add some ls cmds for testsing
842a065 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
do not save to aftifacts /opt/bb/include
dcc90c9 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Add chmod +x for broker
fd14240 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Add +x for all .tsk
edc9725 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Fix path in download-artifact include
487fd49 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Can preset leader via env var BLAZINGMQ_LEADER_NAME
60629d0 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Refactor manual workflow; fix prometheus build job
1974f2b 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Refactor manual wf
1ec3a9f 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
fix cache restore
507efe2 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Fix fuzz tests
4bd649e 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Load build from cache for fuzztest
8c06d38 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Can cleaun up artifacts after manual run
cbe39fa 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Run cleanup always
61a360f 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Fix artifacts cleanup
255d6e8 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Customize build_ubuntu workflow
37a9f9b 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Update passing cache_key
c18b839 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Swtich from cache to artifacts
94a7a80 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov bbpetukhov force-pushed the manual-ci branch 2 times, most recently from d2706a1 to 938c9ec Compare June 5, 2025 10:52
@bbpetukhov
Fix artifact name collision for when same ref is used for both builds…
d8825c2 
… -- add new "name" parameter to the build ubuntu workflow and use it as part of artifact_key

Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
bbpetukhov added 5 commits June 5, 2025 16:19
@bbpetukhov
For new ITs it is enough to specify BLAZINGMQ_BROKER for old broker v…
939c577 
…erions, no need to specify path per node

Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Merge branch 'main' into manual-ci
3ef77aa
@bbpetukhov
Do not check exist code on cluster shutdown in legacy mode
b3e7f83 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Disable parallel pytest execution for multi 7-node cluster fixture
0e4fb90 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Increased timeout for 7-node cluster running on CI 4-cpu machine
75506bf 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
New redeploy test - restart node one by one
c7dce91 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Lint format
7c07369 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Merge branch 'main' into manual-ci
bf5cf79
@bbpetukhov
Fix stop_some_nodes method
ae8c98a 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Fix redeploy tests -- do not check for leader ready when replica is u…
c890091 
…pgraded

Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Merge branch 'bloomberg:main' into manual-ci
78b0530
@bbpetukhov
Use correct broker entity in redeploy IT
2418a76 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Restrict leader change in whole cluster restart test
fc6bd52 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Refactor cluster python class: remove redundant functionality to mana…
799fbed 
…ge individual nodes

Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov bbpetukhov requested a review from kaikulimu June 24, 2025 14:49
kaikulimu
kaikulimu reviewed Jun 24, 2025
View reviewed changes
@bbpetukhov
Merge branch 'main' into manual-ci
32eb383 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Fix merge conflicts
d65a577 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Remove reverse_proxy for multi7 cluster it fixture
b9ed2b0 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
@bbpetukhov
Merge branch 'main' into manual-ci
bf5db2d 
Signed-off-by: Dmitrii Petukhov <dpetukhov@bloomberg.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaikulimu kaikulimu kaikulimu requested changes

Copilot code review Copilot Copilot left review comments

Requested changes must be addressed to merge this pull request.

Assignees

@bbpetukhov bbpetukhov

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bbpetukhov @kaikulimu
0