Data Analysis Tools

Scripts

My current research involves processing a lot of data: one run of my model generates about 10 terabytes of data. This is unwieldy, and would be impossible to work with without some automation. I use Python to analyze netCDF files, and I use bash scripts to generate and run batches of files. My model simulates a three dimensional rectangular ocean, which I divide into smaller patches for each vertical level. Depending on the quantity I'm calculating, I use a script that loops over each patch at a single vertical level, or each vertical level over a single patch. I've linked some files below that are generic versions of my script generators, for both ways of dividing the domain.

Running scriptgenerator.sh will make copies of template_pythonscript.py and template_submissionscript.sh, modify them using the information specified in scriptgenerator.sh, and then submit them to the queue of the system you're using. I run my research on Compute Canada resources; I've used Guillimin (from Calcul Quebec) and Orca (from SHARCNET). Guillimin uses a straightforward Torque/MOAB system, while SHARCNET uses their own system which is not standardized. You'll need to alter all the scripts with your own data / calculations, but there are comments throughout to tell you what you need.

  • A Zip file with all the scripts for both horizontal and vertical slicing.
  • scriptgenerator.sh: Run this file to create a Python file for each vertical level, and submit it to your job queue.
  • template_pythonscript.py: This file is copied for each vertical level, with variables replaced with quantities as specified in scriptgenerator.sh
  • template_qsubsubmission.sh: The standard submission file for TORQUE systems (anything that uses qsub commands).
  • template_sharcnetsubmission.sh: My submission file for SHARCNET.
Colourmaps and Colour Schemes

These are some useful colour schemes for matlab and gnuplot plotting.