8000 [BUG] failing geneformer tutorial geneformer_cellxgene_tutorial.ipynb · Issue #778 · NVIDIA/bionemo-framework · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[BUG] failing geneformer tutorial geneformer_cellxgene_tutorial.ipynb #778

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dorotat-nv opened this issue Mar 24, 2025 · 0 comments
Open
Labels
bug Something isn't working

Comments

@dorotat-nv
Copy link
Collaborator

BioNeMo Framework Version

32be470

Bug Description

The notebook

(752 durations < 30s hidden.  Use -vv to show these durations.)
=========================== short test summary info ============================
FAILED docs/docs/user-guide/examples/bionemo-geneformer/geneformer_cellxgene_tutorial.ipynb::Cell 13
FAILED docs/docs/user-guide/examples/bionemo-geneformer/geneformer_cellxgene_tutorial.ipynb::Cell 14
FAILED docs/docs/user-guide/examples/bionemo-geneformer/geneformer_cellxgene_tutorial.ipynb::Cell 15
FAILED docs/docs/user-guide/examples/bionemo-geneformer/geneformer_cellxgene_tutorial.ipynb::Cell 16
================== 4 failed, 257 passed in 3297.18s (0:54:57) ==================
fails to be executed in GitHub CI limited to L4 GPU nodes

See log
https://github.com/NVIDIA/bionemo-framework/actions/runs/14003818760/job/39215127375

Make sure that this PR passes with INCLUDE_NOTEBOKS_TESTS

Steps to Reproduce

  1. run in the docker container
pytest -v --nbval-lax docs/docs/user-guide/examples/bionemo-geneformer/geneformer_cellxgene_tutorial.ipynb

Error Messages and Logs

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[1], line 5
      1 # Load inference results with torch load
      2 import torch
----> 5 inference_results = torch.load(tutorial_output_inference_pickle)
      6 print(inference_results.keys())
      8 # print(len(inference_results), adata.shape, inference_results[0].keys())

File /usr/local/lib/python3.12/dist-packages/torch/serialization.py:1415, in load(f, map_location, pickle_module, weights_only, mmap, **pickle_load_args)
   1412 if "encoding" not in pickle_load_args.keys():
   1413     pickle_load_args["encoding"] = "utf-8"
-> 1415 with _open_file_like(f,"rb") as opened_file:
   1416     if _is_zipfile(opened_file):
   1417         # The zipfile reader is going to advance the current file position.
   1418         # If we want to actually tail call to torch.jit.load, we need to
   1419         # reset back to the original position.
   1420         orig_position = opened_file.tell()

File /usr/local/lib/python3.12/dist-packages/torch/serialization.py:736, in _open_file_like(name_or_buffer, mode)
    734 def _open_file_like(name_or_buffer, mode):
    735     if _is_path(name_or_buffer):
--> 736         return _open_file(name_or_buffer,mode)
    737     else:
    738         if "w" in mode:

File /usr/local/lib/python3.12/dist-packages/torch/serialization.py:717, in _open_file.__init__(self, name, mode)
    716 def __init__(self, name, mode):
--> 717     super().__init__(open(name,mode))

FileNotFoundError: [Errno 2] No such file or directory: '/workspace/bionemo2/data/singlecell_tutorial/inference_output/predictions__rank_0.pt'

_ docs/docs/user-guide/examples/bionemo-geneformer/geneformer_cellxgene_tutorial.ipynb::Cell 14 _
Notebook cell execution failed
Cell 14: Cell execution caused an exception

Input:
import umap


reducer = umap.UMAP()
embedding = reducer.fit_transform(inference_results["embeddings"].float())

Traceback:

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[1], line 5
      1 import umap
      4 reducer = umap.UMAP()
----> 5 embedding = reducer.fit_transform(inference_results["embeddings"].float())

NameError: name 'inference_results' is not defined

_ docs/docs/user-guide/examples/bionemo-geneformer/geneformer_cellxgene_tutorial.ipynb::Cell 15 _
Notebook cell execution failed
Cell 15: Cell execution caused an exception

Input:
print("embedding.shape: ", embedding.shape)
print("adata_test.obs.shape[0]: ", adata_test.obs.shape[0])
assert adata_test.obs.shape[0] == inference_results["embeddings"].shape[0]

Docker Image

No response

System Information

GPU Details:

  • GPU Model: NVIDIA L4 (CI runner)

Additional Context

No response

@dorotat-nv dorotat-nv added the bug Something isn't working label Mar 24, 2025
github-merge-queue bot pushed a commit that referenced this issue Mar 24, 2025
…e_tutorial.ipynb (#779)

### Description
Skipping execution of this notebook in GitHub CI due execution error

`pytest -v --nbval-lax -p no:python
--ignore="docs/docs/user-guide/examples/bionemo-geneformer/geneformer_cellxgene_tutorial.ipynb"
docs/ sub-packages/`

Issue: #778

### Type of changes
<!-- Mark the relevant option with an [x] -->

- [x]  Bug fix (non-breaking change which fixes an issue)
- [ ]  New feature (non-breaking change which adds functionality)
- [ ]  Refactor
- [ ]  Documentation update
- [ ]  Other (please describe):

### CI Pipeline Configuration
Configure CI behavior by applying the relevant labels:

-
[SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci)
- Skip all continuous integration tests
-
[INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests)
- Execute notebook validation tests in pytest
-
[INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests)
- Execute tests labelled as slow in pytest for extensive testing

> [!NOTE]
> By default, the notebooks validation tests are skipped unless
explicitly enabled.

#### Authorizing CI Runs

We use
[copy-pr-bot](https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/#automation)
to manage authorization of CI
runs on NVIDIA's compute resources.

* If a pull request is opened by a trusted user and contains only
trusted changes, the pull request's code will
automatically be copied to a pull-request/ prefixed branch in the source
repository (e.g. pull-request/123)
* If a pull request is opened by an untrusted user or contains untrusted
changes, an NVIDIA org member must leave an
`/ok to test` comment on the pull request to trigger CI. This will need
to be done for each new commit.

### Usage
<!--- How does a user interact with the changed code -->
```python
TODO: Add code snippet
```

### Pre-submit Checklist
<!--- Ensure all items are completed before submitting -->

 - [ ] I have tested these changes locally
 - [ ] I have updated the documentation accordingly
 - [ ] I have added/updated tests as needed
 - [ ] All existing tests pass successfully
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant
0