Added numpy+scipy and mapping notebooks

Author: Ben Mather

CourseContent/Notebooks/Mapping/0-Preliminaries.ipynb

@@ -0,0 +1,185 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#  Data storage\n",
+    "\n",
+    "Python provides file read write and object serialisation / reconstruction (python pickle module). `numpy` provides methods for storing and retrieving structured arrays quickly and efficiently (including data compression). `scipy` provides some helper functions for common file formats such as `netcdf` and `matlab` etc etc.\n",
+    "\n",
+    "Sometimes data are hard won - and reformatting them into easily retrieved files can be a lifesaver."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "from scipy.io import netcdf"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## text-based data files\n",
+    "\n",
+    "Completely portable and human readable, text-based formats are common outputs from simple scripted programs, web searches, program logs etc. Reading and writing them is trivial, and it is easy to append information to a file. The only problem is that the conversion can be extremely slow. \n",
+    "\n",
+    "This example is taken from our mapping exercise and shows the benefit of converting data to binary formats.\n",
+    "\n",
+    "---\n",
+    "\n",
+    "```python\n",
+    "\n",
+    "# Seafloor age data and global image - data from Earthbyters\n",
+    "\n",
+    "# The data come as ascii lon / lat / age tuples with NaN for no data. \n",
+    "# This can be loaded with ...\n",
+    "\n",
+    "age = np.loadtxt(\"global_age_data.3.6.xyz\")\n",
+    "age_data = age.reshape(1801,3601,3)  # I looked at the data and figured out what numbers to use\n",
+    "age_img  = age_data[:,:,2]\n",
+    "\n",
+    "# But this is super slow, so I have just stored the Age data on the grid (1801 x 3601) which we can reconstruct easily\n",
+    "\n",
+    "datasize = (1801, 3601, 3)\n",
+    "age_data = np.empty(datasize)\n",
+    "\n",
+    "ages = np.load(\"global_age_data.3.6.z.npz\")[\"ageData\"]\n",
+    "\n",
+    "lats = np.linspace(90, -90, datasize[0])\n",
+    "lons = np.linspace(-180.0,180.0, datasize[1])\n",
+    "\n",
+    "arrlons,arrlats = np.meshgrid(lons, lats)\n",
+    "\n",
+    "age_data[...,0] = arrlons[...]\n",
+    "age_data[...,1] = arrlats[...]\n",
+    "age_data[...,2] = ages[...]\n",
+    "```\n",
+    "\n",
+    "\n",
+    "---\n",
+    "\n",
+    "The timing comparison is astonishing\n",
+    "\n",
+    "On my laptop the numpy binary file is about a million times faster to read. I cut out the lat/lon values from this file to save some space, but this would add, at most, a factor of 3 to the npz timing. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "28.5 s ± 104 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%timeit\n",
+    "\n",
+    "gadtxt = np.loadtxt(\"../../Data/Resources/global_age_data.3.6.xyz\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "35.7 µs ± 130 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%timeit\n",
+    "\n",
+    "gadnpy = np.load(\"../../Data/Resources/global_age_data.3.6.z.npz\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## netcdf\n",
+    "\n",
+    "Many earth-science datasets are stored in the `netcdf` format. `scipy` provides a reader for this format"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "OrderedDict([('lat', 161), ('lon', 360)])\n",
+      "OrderedDict([('ve', <scipy.io.netcdf.netcdf_variable object at 0x7f34817cf940>),\n",
+      "             ('vn', <scipy.io.netcdf.netcdf_variable object at 0x7f34817cf908>),\n",
+      "             ('lat',\n",
+      "              <scipy.io.netcdf.netcdf_variable object at 0x7f34817cf9e8>),\n",
+      "             ('lon',\n",
+      "              <scipy.io.netcdf.netcdf_variable object at 0x7f34817cfa90>)])\n",
+      "(161,)\n",
+      "(360,)\n",
+      "(360, 161)\n",
+      "(360, 161)\n"
+     ]
+    }
+   ],
+   "source": [
+    "nf = netcdf.netcdf_file(filename=\"../../Data/Reference/velocity_AU.nc\")\n",
+    "\n",
+    "from pprint import pprint # pretty printer for python objects\n",
+    "\n",
+    "pprint( nf.dimensions )\n",
+    "pprint( nf.variables )\n",
+    "\n",
+    "print (nf.variables[\"lat\"].data.shape)\n",
+    "print (nf.variables[\"lon\"].data.shape)\n",
+    "print (nf.variables['ve'].data.shape)\n",
+    "print (nf.variables['vn'].data.shape)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}