Data Analysis Tools


My current research involves processing a lot of data: one run of my model generates about 10 terabytes of data. This is unwieldy, and would be impossible to work with without some automation. I use Python to analyze netCDF files, and I use bash scripts to generate and run batches of files. My model simulates a three dimensional rectangular ocean, which I divide into smaller patches for each vertical level. Depending on the quantity I'm calculating, I use a script that loops over each patch at a single vertical level, or each vertical level over a single patch. I've linked some files below that are generic versions of my script generators, for both ways of dividing the domain.

Running will make copies of and, modify them using the information specified in, and then submit them to the queue of the system you're using. I run my research on Compute Canada resources; I've used Guillimin (from Calcul Quebec) and Orca (from SHARCNET). Guillimin uses a straightforward Torque/MOAB system, while SHARCNET uses their own system which is not standardized. You'll need to alter all the scripts with your own data / calculations, but there are comments throughout to tell you what you need.

  • A Zip file with all the scripts for both horizontal and vertical slicing.
  • Run this file to create a Python file for each vertical level, and submit it to your job queue.
  • This file is copied for each vertical level, with variables replaced with quantities as specified in
  • The standard submission file for TORQUE systems (anything that uses qsub commands).
  • My submission file for SHARCNET.
Colourmaps and Colour Schemes

These are some useful colour schemes for matlab and gnuplot plotting.