A Quick and Dirty search for Bottlenecks

2 hours of work over 3 hours

Objective: find and remove bottlenecks from a system running qiime 
Method: check resource use in the running system. Bottlenecks occur when one resource is the 'limiting reagent' in the speed of a program.

1. Understand the system
   In this case the top part of the running stack was basically:
   
   • Qiime script written in python
   • running in the command line in Ubuntu 12.04 64-bit
   • in an Oracle VM
   • on a server running Windows Server 2008 R2E 64-bit
   There are other components of the stack that are not listed here, but these are the ones more available for us to look at.
2. Look where you can
   We can't easily look at everything (CPU cycles, Python). What we can't see, we can't optimize. We are only going to worry about what we can see and change. From the list above, I choose to look at resource use in Ubuntu and Windows because both OSs have good, built-in resource monitors.
   
   System Monitor in Ubuntu
   Ubuntu is towards the top of the stack, so I'll look at it first.
   
   
   1. Click on Dash Home in the top right corner of the screen.
   2. Search for 'monitor' and open it. (you can right-click and 'lock to launcher too')
   3. It shows use of CPU, Memory (and swap), and network. You can also see number of CPU cores and RAM and all running processes.
   
   
   Task Manager in Windows
   This offers a quick comparison of what's going on in Windows and Ubuntu.
   
   1. Right click on the taskbar. Click 'Start Task Manager'.
   2. This shows CPU, Memory, network, and processes. You can also see number of CPU cores and RAM and all running applications, processes, and services.
   Resource Monitor in Windows (as an Admin)
   This is like Task Manager, but a lot more detailed.
   
   1. Click start and search 'resource'
   2. CPU use is divided into processes and services and threads of each.
      RAM is divided by process.
      Disk reads and writes are shown by process. If Disk Access is the bottleneck, this is the quickest way to find it.
   
   
3. Identify resources that are usually fully used...
   ... and then see if you can increase that resource.

Conclusion: CPU appeared to be the bottleneck. Ubuntu was running on a single core as the VM was assigned only 1 core.

Actions Taken: I closed the script (ctl+c in terminal) and shut down the VM. In the VM's settings, I gave the VM a total of 15 cores and 16 Gb of ram. This improved speed tremendously (~1 order of magnitude). In retrospect, I should have given it more ram, as more was available and the 16 Gb was quickly used up. However, the CPUs were also maxed, so ram may not have been the primary bottleneck.