Replace biopython with faster dnaio, buffer writes to reduce memory spikes during renaming #62
Conversation
|
Warning Newer version of the nf-core template is available. Your pipeline is using an old version of the nf-core template: 3.0.2. For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation. |
|
Thanks a lot for sharing your improvement! The speed of this step is a bottleneck, indeed. |
|
Since I’ve been tagged: dnaio supports reading concatenated/multiblock gzip files. |
Great, thanks, so adding this improvement doesn't need to be alongside the old implementation. Unfortunately all that failing tests are from the outdated nf-core template, see #60 |
|
Thanks @d4straub, I mentioned the possibility of regression out of caution, not because I'm aware of issues. Used with custom config specifying memory limits, this PR was the first version that completed successfully for all of my test datasets in a single execution without retries or crashes. If there is appetite to merge, I could also look into applying prerequisite nf-core template updates, but likely not for another week or two. |
|
It would be very nice to have that improvement in the pipeline. So yes, if you find the time it would be great to update template & merge the PR (to dev though, not master, at least as first step). |
This PR contains necessary modifications to read renaming in order to use detaxizer with large datasets (50-100GB compressed) on a machine with 128GB of RAM. Pipeline execution time is also dramatically reduced. Please do not feel obliged to merge, but I wanted to share my changes in case they help someone.
process_lowsince memory usage is now boundedTest profile runs successfully with conda and docker.
Should you consider merging, I will update the changelog etc. accordingly. These modifications appear to be working well for me, but it's possible renaming has regressed e.g. wrt blocked gzip support.