truncation scripts

Make a lists of files you want to process, and call it all.txt.

Then run following in bash:
while read fna;
do read qual;
echo "Running $fna";
time truncate_fasta_qual_files.py -f $fna -q $qual -b 220 -o filtered/;
echo "";
done < all.txt

I hope this works.

quality scores script

In a terminal, cd to folder with all the converted .qual files.
Get a list of all .qual files in that folder by running:
ls *.qual > quallist.txt

Analyse the quality of every .qual file in that list by running:
while read line;
do time quality_scores_plot.py -q $line -o quality_histograms/$line/;
done < quallist.txt

bash Scripts for Lab

Goal: make a bash script that runs convert_fastaqual_fastq.py on all the files in a folder.

Methods: First, we needs a list of all the files we want to run a script on.
Open a terminal, and use cd to move into the folder you want.

Then run:
ls > list.txt
This takes the results of ls and stores it all in list.txt.

Open list.txt in a text editor. If you don't want to run the qiime script on certain file, remove them from list.txt. Make sure there is one file name on each line and no extra or blank lines anywhere in the file.

Save it! Now copy and paste the following into that terminal you still have open:

while read line;
do time convert_fastaqual_fastq.py -f $line -o Output_Fastq -c fastq_to_fastaqual;
done < list.txt

This runs the script convert_fastaqual_fastq.py on each file in list.txt with all the given flags.

Here is the same code generalized to run other scripts:
while read line;
do time <copy and paste your script here, using $line as the input>;
done < list.txt

Qiime install disk

Goal: make a quick way to install Qiime.

Methods: the following files were burned to a single DVD.

• Qiime 1.6.0 .vdi.gz (zipped Qiime image for the VM)
• VirtualBox (VM software for both Mac OSX and Windows)
• 7-zip (archiving/compression software for Windows)
• A README.txt describing the files.

The included README files is as follows:

The two VirtualBox files are installers for a Virtual Machine (VM) that will allow us to easily run Qiime. There is an installer for Windows and another for Mac OSX. (Using Linux? Talk to me.)

QIIME-1.6.0-amd64.vdi.gz is a compressed archive we will run using the VM. It contains all of the Qiime software libraries installed on the operating system Ubuntu 12.04 LTS. We will need to extract this archive and save to our computer before it can be used. It will take up to 30 GB of hard drive space when fully extracted.

7z920-x64.msi is a Windows installer for 7-zip. This program will allow us to extract the very large archive mentioned above. (It will also let us compress and extract other archives, which can save a lot of space.)


This was quickly created by Colin Brislawn on 2/24/2013 for the Lamendella Lab at Juniata College.

Result: DVD was used by the class to install the Qiime VM. Perhaps a compression method that has faster decompression (LZMA2? LZO?) could be used next time. Addition copies of the DVD would also make the process much faster.

Normalizing reads

Can Qiime analyze paired end reads?
Some parts of the workflow can handle it, but we will analysis from a single end because it that workflow has been established and used before.

When are the sequesces of the 515F or 806R primers?
In our sequences, will we see these primers or their reverse compliments?

In 16S, is the regions of high variability closer to the 515F or 806R primer?

Method to normalize reads:
Our data has already been quality filtered. Because we do not have to worry about quality, we will use the following process because it uses existing qiime scripts (and knowledge Erin's expert knowledge).
Use convert_fastaqual_fastq.py to get .fna and .qual files from the .fastq we have.

convert_fastaqual_fastq.py -f file_you_want_converted.fastq -o output_directory -c fastq_to_fastaqual

To run the files in bulk I added all the files names to a file called test.txt and can the following bash script overnight:

while read line
do time convert_fastaqual_fastq.py -f $line -o R1_fasta -c fastq_to_fastaqual
done < list.txt

I tried running this from an internal hard drive, but it was not faster at all.

I can truncate_fasta_qual_files.py on the resulting .fna and .qual files, using the following script.

while read line; do echo -e "\n\n running $line"; time truncate_fasta_qual_files.py -f $line.fna -q $line.qual -b 150 -o fasta_filtered/; done < files.txt

This runs the truncation script on every file listed in files.txt, which looks like this:

ALXM_S15_L001_R1_001
BCWL_S16_L001_R1_001
CaRM_S17_L001_R1_001
CroRM_S18_L001_R1_001
...

All files are now the right length for the next stage in the pipeline!

A Quick and Dirty search for Bottlenecks

2 hours of work over 3 hours

Objective: find and remove bottlenecks from a system running qiime 
Method: check resource use in the running system. Bottlenecks occur when one resource is the 'limiting reagent' in the speed of a program.

1. Understand the system
   In this case the top part of the running stack was basically:
   
   • Qiime script written in python
   • running in the command line in Ubuntu 12.04 64-bit
   • in an Oracle VM
   • on a server running Windows Server 2008 R2E 64-bit
   There are other components of the stack that are not listed here, but these are the ones more available for us to look at.
2. Look where you can
   We can't easily look at everything (CPU cycles, Python). What we can't see, we can't optimize. We are only going to worry about what we can see and change. From the list above, I choose to look at resource use in Ubuntu and Windows because both OSs have good, built-in resource monitors.
   
   System Monitor in Ubuntu
   Ubuntu is towards the top of the stack, so I'll look at it first.
   
   
   1. Click on Dash Home in the top right corner of the screen.
   2. Search for 'monitor' and open it. (you can right-click and 'lock to launcher too')
   3. It shows use of CPU, Memory (and swap), and network. You can also see number of CPU cores and RAM and all running processes.
   
   
   Task Manager in Windows
   This offers a quick comparison of what's going on in Windows and Ubuntu.
   
   1. Right click on the taskbar. Click 'Start Task Manager'.
   2. This shows CPU, Memory, network, and processes. You can also see number of CPU cores and RAM and all running applications, processes, and services.
   Resource Monitor in Windows (as an Admin)
   This is like Task Manager, but a lot more detailed.
   
   1. Click start and search 'resource'
   2. CPU use is divided into processes and services and threads of each.
      RAM is divided by process.
      Disk reads and writes are shown by process. If Disk Access is the bottleneck, this is the quickest way to find it.
   
   
3. Identify resources that are usually fully used...
   ... and then see if you can increase that resource.

Conclusion: CPU appeared to be the bottleneck. Ubuntu was running on a single core as the VM was assigned only 1 core.

Actions Taken: I closed the script (ctl+c in terminal) and shut down the VM. In the VM's settings, I gave the VM a total of 15 cores and 16 Gb of ram. This improved speed tremendously (~1 order of magnitude). In retrospect, I should have given it more ram, as more was available and the 16 Gb was quickly used up. However, the CPUs were also maxed, so ram may not have been the primary bottleneck.

Cleaning

I'm scheduled to clean the lab this week and I have never done this before. I dropped by before dinner and Erin showed me the basics.
I tidied up the lab, loaded up one box of pipette tips, readied to autoclave some prepared pipette tips.
With the help of another Aaron, I autoclaved several boxes of tips and returned them to the lab.