New bash files for Autometa workflow #281
Conversation
Removed identifier
|
There was a problem hiding this comment.
In each workflow's current state, required flags are not required and the script attempts to run each command even when no flags are supplied. There are also some commands that may cause unintended side-effects that should be removed.
Each of these workflows is mostly copy/paste with the exception of the coverage table generation sub-workflow. Do you think it would be more appropriate to mention coverage in the filenames? I'm not certain whether this is necessary, but curious to hear your thoughts.
Co-authored-by: Evan Rees <25933122+WiscEvan@users.noreply.github.com>
Added features: - User can choose between calculating coverage from spades headers or from builtin paired-end coverage calcs. - Checks that all required flags are present - Checks that correct conda env is activated -Confirms that ncbi files can be found. If not provides command to download files but does not actually download the db for the user. - Camel case has also been converted to snake case throughout. - Various steps are also now sent to the std out file to track and clarify where things may go awry for a user. - Fiddling with automata config files removed Example usage: For Spades coverage: sbatch autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Spades -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v spades For paired-end cov: sbatch autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired -f /home/sam/test/10.1.fastq.gz -r /home/sam/test/10.2.fastq.gz Tests run to confirm errors pop up as expected: #Return error that incorrect option provided bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v Paired -f /home/sam/test/10.1.fastq.gz -r /home/sam/test/10.2.fastq.gz #Return error that fwd reads not provided bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired #Return error that rev reads not provided bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired -f /home/sam/test/10.1.fastq.gz #Return error the fwd read file doesn't exist bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired -f /home/sam/test/10fastq.gz -r /home/sam/test/10.2.fastq.gz #Return error the rev read file doesn't exist bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired -f /home/sam/test/10.1.fastq.gz -r /home/sam/test/10.r.fastq.gz #When in conda base env, return that env is incorrect bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired -f /home/sam/test/10.1.fastq.gz -r /home/sam/test/10.2.fastq.gz All tests were repeated for spades coverage calc version. Similarly, the script was run using metagenomic sequence data from a Lagria beetle and successfully generated bins in both "spades" and "paired" mode.
There was a problem hiding this comment.
Added features:
- User can choose between calculating coverage from spades headers or from builtin paired-end coverage calcs.
- Checks that all required flags are present
- Checks that correct conda env is activated
-Confirms that ncbi diamond-formattednr.dmndfile can be found. If not provides command to download files but does not actually download the db for the user. - Camel case has also been converted to snake case throughout.
- Various steps are also now sent to the std out file to track and clarify where things may go awry for a user.
- Fiddling with automata config files removed
Example usage:
For Spades coverage:
sbatch autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Spades -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v spades
For paired-end cov:
sbatch autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired -f /home/sam/test/10.1.fastq.gz -r /home/sam/test/10.2.fastq.gz
Tests run to confirm errors pop up as expected:
#Return error that incorrect option provided
bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v Paired -f /home/sam/test/10.1.fastq.gz -r /home/sam/test/10.2.fastq.gz
#Return error that fwd reads not provided
bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired
#Return error that rev reads not provided
bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired -f /home/sam/test/10.1.fastq.gz
#Return error the fwd read file doesn't exist
bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired -f /home/sam/test/10fastq.gz -r /home/sam/test/10.2.fastq.gz
#Return error the rev read file doesn't exist
bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired -f /home/sam/test/10.1.fastq.gz -r /home/sam/test/10.r.fastq.gz
#When in conda base env, return that env is incorrect
bash autometa_flagged.sh -o /home/sam/test -a /home/sam/test/10_scaffolds.fasta -s 10_Paired_end -c 16 -n /media/bigdrive1/Databases/autometa_databases -m /home/sam/Tools/Autometa/autometa/databases/markers -l 1000 -v paired -f /home/sam/test/10.1.fastq.gz -r /home/sam/test/10.2.fastq.gz
All tests were repeated for spades coverage calc version.
Similarly, the script was run using metagenomic sequence data from a Lagria beetle and successfully generated bins in both "spades" and "paired" mode.
| if [ -f "$ncbi/nr.dmnd" ]; then | ||
| echo "NCBI files are present where specified, moving on..." | ||
| else | ||
| echo "NCBI files not found where specified. Please run autometa-update-databases --update-ncbi to download" | ||
| exit | ||
| fi |
There was a problem hiding this comment.
nodes.dmp, names.dmp, merged.dmp, delnodes.dmp (from taxdump.tar.gz) as well as prot.accession2taxid.gz are also used but not checked here. Should the echo statement only mention nr.dmnd or should this also check whether these other files exist?
There was a problem hiding this comment.
I actually didn't know I needed to check for those. It was just a quick way to check if the diamond database was there, like a quick diagnostic - not meant to be a comprehensive check. Can add them in if you think necessary though?
Co-authored-by: Evan Rees <25933122+WiscEvan@users.noreply.github.com>
…ow if I missed anything :)
There was a problem hiding this comment.
A few minor nitpicks regarding variables and help text. I encountered one bug and some code was a bit out of order. I think most is covered with the code suggestions. I can try to get around to following up on the suggested edits. I think there are a few more entrypoints that may be echoed using the set -x then { set +x; } 2>/dev/null syntax. Examples are available in the suggestions. This is looking great!
| echo "-n Path to NCBI databases directory" | ||
| echo "-m Path to marker files directory" | ||
| echo "-l Contig length cutoff in base pairs. (NOTE: Change this according to the metagenome assembly's N50)" | ||
| echo "-v Coverage calculation. Options are 'spades' or 'paired'. If you assembled your metagenome with spades, choose the spades option. If you used a different assembler, please use the 'paired' option for the coverage calculation. Paired reads used for assembly are required as input" |
There was a problem hiding this comment.
Can this help text be broken up across multiple lines or made more concise?
- 🐛 Fix kmers check prior to `autometa-binning`
- 📝🐚 echo autometa entrypoints using `set -x` and `{ set +x; } 2>/dev/null` syntax
- 🔥 Remove ORFs generation check
- prepend `set -e` to cause immediate exit of the workflow for non-zero exit code
- 📝🐚 Update FileNotFound errors to also emit the corresponding parameter
- 📝 Replace `densne` under choices for `embed_method` with `densmap`
- 🎨🐚 Move inputs echo to follow parameter checks
📝 Clarify parameter flag when user does not provide correct input
There was a problem hiding this comment.
Looks good. After merge, this will need to be updated in PR #284
🐚 Add feature: New batch file scripts for documentation
One for when spades was used for the assembly, one where coverage is calculated using bwa/kart/samtools and one that uses the built in autometa coverages option with paired-end data. The spades one has been tested and ran successfully on the KwanLab server. A second test is ongoing on the agrp server in SA with a different, much larger dataset.
Note: For autometa_bwa_kart.sh to work, bwa kart and samtools will need to be added to the autometa conda env set up.
If you are happy with the scripts, I can create a QuickStart guide much as I did for the nextflow workflow.
PR checklist