8000 Training model on doclaynet dataset using multi-gpu · Issue #222 · deepdoctection/deepdoctection · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Training model on doclaynet dataset using multi-gpu #222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Shravan-Ganji opened this issue Sep 5, 2023 · 2 comments
Open

Training model on doclaynet dataset using multi-gpu #222

Shravan-Ganji opened this issue Sep 5, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@Shravan-Ganji
Copy link
Shravan-Ganji commented Sep 5, 2023 •
edited
Loading

Bug 💥
I am Trying to train the model on doclaynet dataset using multiple gpu, but facing error as CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Screenshot 2023-09-05 195455

I have changed the code in d2_frcnn_train.py as follows:

def main(num_gpus,path_config_yaml,dataset_train,path_weights,config_overwrite,log_dir,build_train_config,dataset_val,build_val_config,metric_name,metric,pipeline_component_name):
launch(train_d2_faster_rcnn,
num_gpus,
1,
0,
"auto",
args=(path_config_yaml,
dataset_train,
path_weights,
config_overwrite,
log_dir,
build_train_config,
dataset_val,
build_val_config,
metric_name,
metric,
pipeline_component_name),)

where i am passing these parameters from Datasets_and_Eval.ipynb
multi_gpu_FRCNN

Expected behavior 🧮
Reduce the computational time by some fraction

Screenshots 🖼
If possible, please add a screenshot of the error message, if possible

Desktop (please complete the following information, if any other than the one in the install requirements):

  • OS: [e.g. iOS]
    ubuntu 20.04
    cuda 11.3
    torch 1.12.1 cuda enabled

Additional context 🧬
its working fine on the single gpu, but i want to train it on multiple gpus

everything has been modified accordingly to the changes like init.py across train and deepdoctection folder
@JaMe76
Copy link
Contributor
JaMe76 commented Sep 5, 2023

Thanks for reporting.

Does this error occur at the first iteration?

Or can you complete one training loop?

My first suggestion is there might be something wrong with the training data but I do not really evidence for that ...

@Shravan-Ganji
Copy link
Author

Yes, its occurring at the first iteration.
The data for the doclaynet has been downloaded from the link provided from datasets_and_eval.ipynb
Using this data, I am able to train it on single GPU and it has been successfully trained.

@JaMe76 JaMe76 added the bug Something isn't working label Sep 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JaMe76 @Shravan-Ganji
0