Reading_a_JWST_ASDF_file.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline and Introduction\n",
    "\n",
    "- Examining the structure\n",
    "- Opening an ASDF file\n",
    "- Summarizing the structure of the file\n",
    "- Accessing the contents\n",
    "- Searching for content\n",
    "\n",
    "### Basic organization of ASDF files\n",
    "\n",
    "Understanding the basic organizational structures of an ASDF file will help in navigating the contents of such files. In brief, the organization consists of an arbitrary nesting of the equivalent of Python lists and dictionaries. As a result, getting to any particular item involves either indexing by number when a list is encountered (YAML parlance is an \"array\") or by a key when a dictionary is encountered (YAML parlance is \"map\"). Furthermore, if any item is preceded by a YAML \"tag\", Python generally knows (if the appropriate software is installed) how to turn it into a corresponding Python object. The most common instance of this case is for ndarray or table data. If a tag for a ndarray is encountered, Python will turn this into a numpy array automatically (though perhaps not until you wish to access the array). But many other kinds of tags and corresponding Python objects will be encountered.\n",
    "\n",
    "### Reading a JWST ASDF file\n",
    "\n",
    "Unbeknownst to many, while the JWST pipeline normally works with FITS files, it has been designed to work with ASDF files as well (though there is currently a small change needed to enable this). Turning JWST data products into ASDF files is easy, and our first example will be with such a file. First some preliminaries such as doing the needed imports and references to the data files used for this tutorial (they must be downloaded, but this notebook handles that)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import asdf\n",
    "from astropy.utils.data import download_file\n",
    "import matplotlib\n",
    "matplotlib.rcParams['figure.figsize'] = (10, 10)\n",
    "from matplotlib import pyplot as plt\n",
    "file1_url_1 = 'https://data.science.stsci.edu/redirect/ASDF/asdf_tut_1.asdf'\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a large file and may take a while to download\n",
    "filepath = download_file(file1_url_1)\n",
    "# Wait until download completes (when 'In [*]' changes the  '* to a number')\n",
    "# Ignore warnings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This may generate a couple warnings\n",
    "af = asdf.open(filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quick look at contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `info()` method is extremely useful for examining the contents of an ASDF file. There is a command line version as well discussed later. But first we'll do a little data access. Note all the items labeled as `NDArrayTypes`. As you might guess, these are all images so we will access the main science image, which has the name \"data\". First thing to know is that when you open ASDF files, images are not loaded into memory when it is opened, but information about that image (or array) is. When you actually access the data itself, it will be loaded. We will do that now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look at the science data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sci = af.tree['data']\n",
    "sci.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now display the image\n",
    "plt.imshow(sci, vmin=0, vmax=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data set is a simulated image of 47 Tuc. Given that JWST has not been launched yet, most of the current data files are not very interesting to view. Let's examine the contents of the file a bit more using the optional arguments for the info() method. This [link](https://asdf.readthedocs.io/en/2.7.1/api/asdf.info.html#asdf.info) shows all the options, but we will show a few examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to show all items\n",
    "af.info(max_rows=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To disable the printing of values\n",
    "af.info(show_values=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing metadata\n",
    "\n",
    "Most metadata for JWST files is under the toplevel attribute meta. There is a lot of information there, and some if it is more complicated than a simple value. Some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta = af.tree['meta']\n",
    "meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple value:\n",
    "meta['filename']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Collection of values as a dict\n",
    "meta['exposure']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta['exposure']['integration_time']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What about comments?\n",
    "\n",
    "The short answer is that there are not any in the file. We will be working on tools to extract the equivalent information from the associated schemas that define what attributes and values are expected in a JWST file, but there isn't one yet. Generally speaking one can look at the corresponding schema to find out definitions and units for values, but that won't be covered in this tutorial. For Roman we expect to use quantities for many values, which have explicit units. But JWST had to mirror the values in FITS keyword/value cards so quantities are not generally used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### About ASDF Terminology \n",
    "\n",
    "The structure of an ASDF file is essentially a tree. Each node may take on one of four general characteristics:\n",
    "\n",
    "1. It may be a simple value, e.g., a string, number, or boolean value.\n",
    "2. It may be a list of values, in which case each of these values is indexed by number\n",
    "3. It may be a dictionary of values, in which case each of these values is indexed by the dictionary key.\n",
    "4. A reference to a value elsewhere in the file, either a simple value, list, or dict\n",
    "\n",
    "So branching at nodes is either done by lists or dictionaries. Simple values are endpoints of a branch. When tags are used, a whole subtree may be turned into a corresponding Python object, which in Python is the endpoint of a branch. To illustrate, in the YAML header, an array is characterized by the number of items that defined the array and where the data are located. But when reading this YAML header into Python, all that information is used to create a numpy array and information about that array is now embedded in the numpy array itself (e.g., its shape, type and other attributes). A \"path\" to a node is essentially the sequence of indices needed to locate it. In the case above, the science data is only one level down, and its path is ['data']. In the original YAML header, there would be deeper levels, but they have all been used to create the numpy array. Much of the metadata for this data set is in the attribute labeled 'meta'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching for Content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To show only where array data is located\n",
    "af.search(type='NDArrayType')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find any WCS-related items\n",
    "sr = af.search('wcs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dir(sr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(sr.nodes)\n",
    "sr = af.search('author')\n",
    "sr.nodes[0] = 'Bozo'\n",
    "sr = af.search('author')\n",
    "sr.nodes[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(af.tree['meta']['wcs'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the WCS\n",
    "\n",
    "This is one of those tags that tells the Python ASDF library to convert this node and all below it into a GWCS object. By using this, you can convert pixel coordinates into sky coordinates.\n",
    "\n",
    "For example, if you wish to see where pixel 100, 200 is on the sky, or the converse, what pixel a sky coordinate lands in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wcs = af.tree['meta']['wcs']\n",
    "wcs(100,200) # both ra and dec are in degrees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wcs.invert(5.9, -72) # return pixel coordinates corresponding to this RA, Dec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# what is the output coordinate frame and units?\n",
    "wcs.output_frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Command line tools\n",
    "\n",
    "There is a command line tool that provides a number of utilities for use with ASDF files. Most relevant for this tutorial are the info. For example, you can get information on an ASDF file by typing at the shell level:\n",
    "```\n",
    "asdftool info <filename>\n",
    "```\n",
    "\n",
    "This variant of the asdftool command accepts the following options corresponding to the optional keyword options for the info method:\n",
    "```\n",
    "--max-rows MAX_ROWS  maximum number of lines\n",
    "--max-cols MAX_COLS  maximum length of line\n",
    "--show-values\n",
    "--no-show-values\n",
    "```\n",
    "\n",
    "For example:\n",
    "\n",
    "```\n",
    "asdftool info --max-rows 100 myfile.asdf\n",
    "```\n",
    "\n",
    "More advanced use of asdftool will be discussed in other tutorials. The documentation for it can be obtained by typing `asdftool --help` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "\n",
    "Find all values that contain the string `crds`. (The documentation on search is available at: https://asdf.readthedocs.io/en/stable/asdf/features.html#searching-the-asdf-tree). Then obtain the value for the filter offset from the tree itself (not copy it from the search display) and save in a variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2\n",
    "\n",
    "Find the world coordinates of pixel 532, 602."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3\n",
    "What are the values at that pixel in the science, dq, and error arrays? Take care with the array indices."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}