Forward-merge branch-25.06 into branch-25.08 #897
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eters in CAGRA index parameters and other changes (rapidsai#831) The PR includes code changes for the following: - Automation of Panama bindings generation using jextract. - Adding the ability to configure IVF_PQ index and search parameters via the cuvs-java API (to adapt with the following [underlying changes](rapidsai@18a3d90)). - Updating HNSW example to show the above. - Updating the readme files. - Simplifying logging in examples. - Bumping up the maven-javadoc-plugin version. - Updating and consolidating gitignore file. - Removing unused imports etc. Please note that the existing Panama classes are being deleted because they were manually created and managed. With the new cleaner approach, this will not be needed anymore. Now these binding classes will be generated at build time and so no need to be in the codebase. Authors: - Vivek Narang (https://github.com/narangvivek10) - Ishan Chattopadhyaya (https://github.com/chatman) Approvers: - rhdong (https://github.com/rhdong) - James Lamb (https://github.com/jameslamb) URL: rapidsai#831
…#865) Contributes to rapidsai/build-planning#135 Follow-up to rapidsai#662 While reviewing rapidsai#805 and rapidsai#831, I found myself suggesting things manually that I know `shellcheck` would have caught automatically. To prevent that for reviewers in the future, this proposes running `shellcheck` on **all** shell scripts in the repo, not just those in the `ci/` directory. Other changes: * updates `rapids-dependency-file-generator` to its latest version (1.18.1) * consolidates duplicate entries for https://github.com/pre-commit/pre-commit-hooks in `.pre-commit-config.yaml` Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Ben Frederickson (https://github.com/benfred) URL: rapidsai#865
### Issue
Original code (below) generated serial assembly and used strictly-ordered `fadda` instruction on ARM with gcc & clang. That resulted in suboptimal performance.
```c++
for (size_t k = 0; k < dim; k++) {
distance += DC::template eval<DistanceT>(query[k], row[k]);
}
```
### Proposed solution
This PR provides euclidean distance optimized with partial vector sum (below), that helps vectorization but loses strcictly-ordered compliance.
```c++
template <typename DC, typename DistanceT, typename DataT>
DistanceT euclidean_distance_squared_generic(DataT const* a, DataT const* b, size_t n) {
size_t constexpr max_vreg_len = 512 / (8 * sizeof(DistanceT));
// max_vreg_len is a power of two
size_t n_rounded = n & (0xFFFFFFFF ^ (max_vreg_len - 1));
DistanceT distance[max_vreg_len] = {0};
for (size_t i = 0; i < n_rounded; i += max_vreg_len) {
for (size_t j = 0; j < max_vreg_len; ++j) {
distance[j] += DC::template eval<DistanceT>(a[i + j], b[i + j]);
}
}
for (size_t i = n_rounded; i < n; ++i) {
distance[i] += DC::template eval<DistanceT>(a[i], b[i]);
}
for (size_t i = 1; i < max_vreg_len; ++i) {
distance[0] += distance[i];
}
return distance[0];
}
```
In addition, it has an implementation with NEON intrinsics which provides further speedup on certain test cases (can be removed if arch-specific code is undesired).
### Results
Authors:
- Anna Verner (https://github.com/anstellaire)
- Tamas Bela Feher (https://github.com/tfeher)
Approvers:
- Tamas Bela Feher (https://github.com/tfeher)
URL: rapidsai#689
….py (rapidsai#818) When `configs/algos/*.yaml` has string configurations, they will be converted into "label" column in exported CSV files. But the exported CSV files only have "label" column for build configuration. search configuration is missing. Before this patch, the search label is overriden by build label in exported CSV files. After this patch, CSV files have "label" column as build label and "search_label" column as search label. Authors: - Yinzuo Jiang (https://github.com/jiangyinzuo) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: rapidsai#818
Closes rapidsai#649 Authors: - Tarang Jain (https://github.com/tarang-jain) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: rapidsai#782
Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Jake Awe (https://github.com/AyodeAwe) - James Lamb (https://github.com/jameslamb) URL: rapidsai#872
Contributes to rapidsai/build-planning#120 This PR adds support for Python 3.13. ## Notes for Reviewers This is part of ongoing work to add Python 3.13 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.13, from rapidsai/shared-workflows#268. A follow-up PR will revert back to pointing at the `branch-25.06` branch of `shared-workflows` once all RAPIDS repos have added Python 3.13 support. ### This will fail until all dependencies have been updated to Python 3.13 CI here is expected to fail until all of this project's upstream dependencies support Python 3.13. This can be merged whenever all CI jobs are passing. Authors: - Gil Forsyth (https://github.com/gforsyth) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#874
This PR add support for half dtype for HNSW in C++, C and python, as well as some tests with it. I had to modify a bit the HNSW patch in order to enable computation on half data types. I also added the support of inner-product distance for int8/uint8 data type. Authors: - Micka (https://github.com/lowener) - Corey J. Nolet (https://github.com/cjnolet) - Divye Gala (https://github.com/divyegala) Approvers: - Divye Gala (https://github.com/divyegala) URL: rapidsai#813
librmm will ship a shared library component in 25.06 (xref: rapidsai/rmm#1779). This PR updates `auditwheel` calls to exclude `librmm.so`. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Gil Forsyth (https://github.com/gforsyth) URL: rapidsai#878
…tion (rapidsai#861) parallel_mode determines how queries are parallelized with OpenMP in FAISS CPU IVF implementation: - 0 Split over queries. - 1 Parallelize over inverted lists. - 2 Parallelize over both queries and inverted lists. - 3 Split over queries with finer granularity. Authors: - Artem M. Chirkin (https://github.com/achirkin) Approvers: - Tamas Bela Feher (https://github.com/tfeher) URL: rapidsai#861
Closing rapidsai#849 Many parts of NN Descent is written on top of the assumption that smaller distance is closer. Therefore, using the current trick (using -(dot product) values) to make it consistent with other metrics to be sorted in ascending order, then negating the value at the end should be the neatest solution. Authors: - Jinsol Park (https://github.com/jinsolp) Approvers: - Ben Frederickson (https://github.com/benfred) URL: rapidsai#859
Closes rapidsai#858 Authors: - Ben Frederickson (https://github.com/benfred) Approvers: - Micka (https://github.com/lowener) URL: rapidsai#880
This PR adds changes for Java CI. Some scripts modified here also appear in [PR rapidsai#831](rapidsai#831). Once 831 is merged, I’ll rebase and make sure everything stays consistent. Authors: - rhdong (https://github.com/rhdong) - Vivek Narang (https://github.com/narangvivek10) - James Lamb (https://github.com/jameslamb) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - James Lamb (https://github.com/jameslamb) - Ray Douglass (https://github.com/raydouglass) URL: rapidsai#805
Related to rapidsai#841. The IVF-PQ build metric is not properly initialized in the case of InnerProduct, so I am adding here correct initialization on the C layer, as well as some checks on the C++ side so that the CAGRA metric match the knn metric. Authors: - Micka (https://github.com/lowener) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Tarang Jain (https://github.com/tarang-jain) - Ben Frederickson (https://github.com/benfred) URL: rapidsai#862
Authors: - Ben Frederickson (https://github.com/benfred) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: rapidsai#881
This PR brings in RBC implementation from RAFT while also reducing the number of templates that are instantiated by moving the following templates to runtime parameters: 1. `dims` 2. booleans 3. distance functor ## Notes for Reviewers ### Benefits of these changes Allows for a reduction in cuML binary sizes, once cuML switches from RAFT's implementation to this one. See: rapidsai/cuml#6626 (comment) Authors: - Corey J. Nolet (https://github.com/cjnolet) - Divye Gala (https://github.com/divyegala) Approvers: - Divye Gala (https://github.com/divyegala) - James Lamb (https://github.com/jameslamb) URL: rapidsai#218
…only modify update-version.sh (rapidsai#875) While reviewing rapidsai#805, I found an issue with this project's `update-version.sh`... it refers to a script `examples/cmake/thirdparty/fetch_rapids.cmake` which no long exists (removed in rapidsai#824). This proposes the following: * removing that reference from `update-version.sh` * skipping most CI on PRs that only modify `update-version.sh` ## Notes for Reviewers `ci/release/update-version.sh` is standardized (same filepath, same usage) across almost all RAPIDS repos. I cannot think of a situation where a PR that only changes that file would need to have any CI re-run (other than linting, e.g. for `shellcheck`). If folks agree, I'll roll out a change like that more broadly across RAPIDS. Authors: - James Lamb (https://github.com/jameslamb) - Ben Frederickson (https://github.com/benfred) Approvers: - Ben Frederickson (https://github.com/benfred) - Corey J. Nolet (https://github.com/cjnolet) - Gil Forsyth (https://github.com/gforsyth) URL: rapidsai#875
This quotes `head_rev` to ensure that commits with leading zeros in the git SHA include those zeros in the output package name. xref: rapidsai/build-planning#176 Authors: - Bradley Dice (https://github.com/bdice) Approvers: - James Lamb (https://github.com/jameslamb) URL: rapidsai#892
Follow-up to rapidsai#834 Contributes to rapidsai/build-infra#237 Uses GitHub Actions artifact store, instead of `downloads.rapids.ai`, to download `libcuvs` conda artifacts in CI. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#893
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes forward merger xref #864