build: add s390x, ARM to CI #58676
Labels
A-build-system
B-unsupported-arch
Non-x86_64 architectures: PPC, MIPS, etc
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-dev-inf
X-nostale
Marks an issue/pr that should be ignored by the stale bot
Comments
|
related to #55307 |
|
gentle ping, any updates for this issue? Thanks : ). |
|
Any updates on this issue? |
|
Hey all, here's the current state of things:
BTW: for all of the above, any changes are only going to be implemented for the new build system (Bazel). Let me know if there are any questions about any of this. |
|
@rickystewart Thank you for the update this really helps!! |
craig bot
pushed a commit
that referenced
this issue
Apr 29, 2022
80141: rowexec: use streamer for lookup joins with no ordering r=yuzefovich a=yuzefovich **row: fix the usage of Get requests with the streamer in some cases** Previously, `TxnKVStreamer` processed the Get responses under the assumption that `nil` `enqueueKeys` argument was provided. In such a scenario, `EnqueueKeysSatisfied` contained an ordinal into the original spans slice that is needed to process the Get response correctly. We will shortly expand the usage of the Streamer to use non-nil `enqueueKeys` argument which breaks that assumption. We go around it by exporting `Result.Position` value which tracks exactly the information we need. Release note: None **rowexec: use streamer for lookup joins with no ordering** This commit expands the usage of the Streamer to lookup joins when ordering doesn't have to be maintained. We can do so easily since the join reader span generators de-duplicate spans, so the Streamer will only be asked to process unique requests. In theory, we should be able to do the same for the cases when ordering needs to be maintained, but `InOrder` mode of the Streamer currently has known bugs / limitations when a single request can result in multiple rows - this wasn't an issue for the index joins since those are guaranteed to get a single row for each request. Also, we want to push the de-duplication logic from the join reader into the Streamer (in order to inform the behavior of the yet-to-be-added cache in the Streamer), but that will come after addressing the known `InOrder` bugs mentioned above. Some adjustments were needed: - In order to improve the behavior with low `workmem` limit (in particular, `regional_by_row` CCL logic test file was erroring out) of the join reader we now make the batch size hint at most 1/12 of the limit while giving the streamer the same three quarters - this allows for other in-memory state of the join reader itself to have some space. - One lookup join unit test is designed with the non-streamer code path in mind, so we disable the usage of the streamer there. - `lookup_join_trace` execbuilder file needed to filter out messages produced by the range cache. Out of curiosity, I decided to do a quick comparison of TPCH numbers with the Streamer `off` (first column) and `on` (second column) (negative numbers indicate the latency reduction, positive - latency increase): ``` Q1: 3.78s 3.28s -13.36% Q2: 0.19s 0.19s 2.11% Q3: 1.72s 1.48s -13.89% Q4: 1.57s 1.25s -20.54% Q5: 2.74s 2.49s -9.06% Q6: 6.36s 8.19s 28.74% Q7: 6.97s 7.35s 5.42% Q8: 1.00s 1.09s 8.68% Q9: 6.52s 6.93s 6.32% Q10: 1.93s 1.63s -15.60% Q11: 0.52s 0.57s 9.06% Q12: 7.16s 8.89s 24.17% Q13: 1.17s 1.13s -3.43% Q14: 0.57s 0.66s 13.91% Q15: 4.83s 5.53s 14.34% Q16: 0.81s 0.78s -4.08% Q17: 0.48s 0.48s 0.84% Q18: 5.61s 5.33s -4.96% Q19: 3.86s 3.41s -11.83% Q20: 12.00s 14.37s 19.79% Q21: 9.33s 7.24s -22.38% Q22: 0.57s 0.55s -4.03% ``` This mostly confirms our expectations - lookup joins should get faster since we now parallelize the lookups. Index joins already get parallel lookups in the old code path, so we have some regressions. Notably, Q6 and Q12 spend most of the time performing index joins and Q20 is dominated by an index join and a parallelizable (when lookup columns form a key) lookup joins. Addresses: #54680 Release note: None **sql: don't sort spans redundantly when using the streamer** This commit makes it so that the lookup and index joins don't sort the spans when the ordering doesn't have to be maintained and the Streamer API is used - the Streamer itself performs the sort of spans within single-range batches, so the caller's sort is redundant. Release note: None 80698: Added s390x support to Bazel's toolchain r=rickystewart a=dandotimujahid This PR is in reference to the issue #58676 It will enable s390x binaries to be built using Bazel's toolchain 80745: opt: break memo's dependence on builtins r=mgartner a=mgartner This commit breaks the `memo` packages dependence on the `builtins` package so that they can be compiled concurrently. Optimizer packages are no longer on the critical path, for now. Informs #79357 Release note: None 80778: dev: make top-level dev runnable from anywhere r=irfansharif a=irfansharif This should obviate the need to run dev from the top-level checkout; as long as it's in your path or have an alias to the script, you should be able to run dev wherever you are. It'll save you the need for the leading './' when invoking things. Release note: None Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com> Co-authored-by: dandotimujahid <Mujahid.Dandoti@ibm.com> Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com> Co-authored-by: irfan sharif <irfanmahmoudsharif@gmail.com>
This was referenced Aug 15, 2022
craig bot
pushed a commit
that referenced
this issue
Nov 21, 2022
92269: bazel: Add s390x support for libkrb5 cross-compilation r=rickystewart a=vivkong This PR is in reference to #58676 This change will allow cross-compiling s390x binary on Intel 64bit. I was able to run `bazel build pkg/cmd/cockroach --config with_ui --config crosslinuxs390x //pkg/cmd/cockroach-short //pkg/cmd/cockroach //pkg/cmd/cockroach-sql //pkg/cmd/cockroach-oss //c-deps:libgeos //pkg/cmd/roachprod` and got the `Build completed successfully` message. Will CockroachDB consider adding s390x to CI and publishing experimental binaries for s390x? Thanks. Co-authored-by: Vivian Kong <vivkong@ca.ibm.com>
|
We have marked this issue as stale because it has been inactive for |
690E
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
See previous discussion in #55617.
Investigate whether s390x and ARM can be added to our list of cross-compiled targets from CI. Also: is automated testing for new targets feasible?
Jira issue: CRDB-3371
The text was updated successfully, but these errors were encountered: