V0.6.x

.gitignore

@@ -10,3 +10,4 @@ hexrd.egg-info
 *.so
 *.*~
 *#
+.#*

README.md

@@ -1,4 +1,51 @@
-HEXRD provides a collection of resources for analysis of x-ray diffraction
-data, especially high-energy x-ray diffraction. HEXRD is comprised of a
-library and API for writing scripts, a command line interface, and an
-interactive graphical user interface.
+HEXRD
+=====
+
+HEXRD provides a collection of resources for analysis of X-ray diffraction
+data, including powder diffraction, Laue diffraction, and monochromatic rotation series (_i.e._, 3DXRD/HEDM).
+HEXRD is comprised of a library and API for writing scripts, a command line interface, and an
+interactive graphical user interface (though this is not up to date in python2.7).
+
+Note that this is a _legacy_ repo with minimal maintenance; the canonical HEXRD repos can now be found at https://github.com/HEXRD/hexrd.
+
+It is recomended that you use the conda package manager for your python environment (available from either [here](https://docs.conda.io/en/latest/miniconda.html) or [here](https://www.anaconda.com/products/individual), with the former being a smaller, more barebones install).
+
+Building
+--------
+You can skip this if you find a build of the desired version at my [my anaconda cloud](https://anaconda.org/joelvbernier/hexrd) page, which I update periodically.  Otherwise, the recommended method is via `conda-build`.  If you installed Miniconda, you will have to first install `conda-build` in your base env: `conda install conda-build`.  Otherwise, using conda 4.8.3 (from Miniconda3 or Anaconda3) the best procedure is as follows:
+- go to wherever you keep your git repos, _e.g._, `cd ~/Documents/GitHub`
+- if you have the repo all ready, update it with a fetch and pull `fit fetch -av; git pull'
+- otherwise, clone the hexrd repo: `git clone https://github.com/joelvbernier/hexrd.git`
+- `cd hexrd`
+- checkout the v0.6.x branch: `git checkout v0.6.x`
+- make an empty env with python2.7 and numpy: `conda create --name hexrd_0.6 -c anaconda -c conda-forge python=2 numpy`
+- activate your new env: `conda activate hexrd_0.6`
+- install fabio from [here](https://github.com/joelvbernier/fabio.git)
+  - cd into wherever you keep your git repos, _e.g._, `cd ~/Documents/GitHub`
+  - clone repo: `git clone https://github.com/joelvbernier/fabio.git`
+  - grab the python 2.7 compatible branch: `git checkout py27_compat`
+  - `cd fabio`
+  - `pip install ./`
+- build hexrd from the conda recipe: `conda build conda.recipe/ --python=2 -c anaconda -c conda-forge`
+
+Installing
+----------
+You can check [my anaconda cloud](https://anaconda.org/joelvbernier/hexrd) for prebuillt versions; if you ffind one for your platform, then simply execute
+- `conda install hexrd=0.6 -c joelvbernier`
+
+Otherwise, you can install from a local build as follows:
+- `conda install hexrd=0.6 --use-local -c anaconda -c conda-forge`
+
+Running
+-------
+The function libraries lend themselves to scripts for your vaired purposes, but there is a CLI for the ff-HEDM workflow, namely indexing, `hexrd find-orientations`, and grain parameter refinement, `hexrd fit-grains`.  More documentation to come.
+
+Additional Packages
+-------------------
+It is highly recommended to install the `fast-histogram` package for the indexing:
+
+- `pip install fast-histogram`
+
+And is you want spyder, the default channel is broken for python2.7.  Use the following:
+
+- `conda install spyder=3 jupyter_client=5.3.4 -c anaconda -c conda-forge`

conda.recipe/bld.bat

@@ -1,5 +1,6 @@
-git describe --tags --dirty > %SRC_DIR%/__conda_version__.txt
-%PYTHON% %RECIPE_DIR%/format_version.py %SRC_DIR%/__conda_version__.txt
+REM !!! need to replace for proper versioning under setuptools???
+REM git describe --tags --dirty > %SRC_DIR%/__conda_version__.txt
+REM %PYTHON% %RECIPE_DIR%/format_version.py %SRC_DIR%/__conda_version__.txt
 
 rmdir build /s /q
 

conda.recipe/build.sh

@@ -1,5 +1,6 @@
-git describe --tags --dirty > $SRC_DIR/__conda_version__.txt
-$PYTHON $RECIPE_DIR/format_version.py $SRC_DIR/__conda_version__.txt
+# !!! need to replace for proper versioning under setuptools???
+#git describe --tags --dirty > $SRC_DIR/__conda_version__.txt
+#$PYTHON $RECIPE_DIR/format_version.py $SRC_DIR/__conda_version__.txt
 
 rm -rf build
 

conda.recipe/meta.yaml

@@ -1,12 +1,14 @@
 package:
   name: hexrd
-  version: master
+  version: {{ environ.get('GIT_DESCRIBE_TAG', '')[1:] }}
 
 source:
-  git_url: https://github.com/joelvbernier/hexrd.git
-  git_tag: master # edit to point to specific branch or tag
+  # git_url: https://github.com/joelvbernier/hexrd.git
+  # git_tag: v0.6.x
+  git_url: ../
 
 build:
+  number: {{ environ.get('GIT_DESCRIBE_NUMBER', 0) }}
   detect_binary_files_with_prefix: true
   osx_is_app: yes
   entry_points:
@@ -18,39 +20,39 @@ app:
 
 requirements:
   build:
-    # - nomkl # in case MKL is broken on Linux
+    - numba
     - numpy
     - python
     - setuptools
   run:
-    - dill
+    - h5py
     - matplotlib
-    # - nomkl # in case MKL is broken on Linux
     - numba
     - numpy
-    - progressbar >=2.3
+    - psutil
+    - progressbar
     - python
     - python.app # [osx]
     - pyyaml
-    - qtconsole
+    - scikit-image
     - scikit-learn
     - scipy
     - wxpython
 
 test:
   imports:
     - hexrd
-  commands:
-    - hexrd -V
-    - hexrd -h
-    - hexrd help
-    - hexrd find-orientations -h
-    - hexrd help find-orientations
-    - hexrd fit-grains -h
-    - hexrd help fit-grains
-    - hexrd gui -h
-    - hexrd help gui
-    - hexrd test
+   # commands:
+   #   - hexrd -V
+   #   - hexrd -h
+   #   - hexrd help
+   #   - hexrd find-orientations -h
+   #   - hexrd help find-orientations
+   #   - hexrd fit-grains -h
+   #   - hexrd help fit-grains
+   #   - hexrd gui -h
+   #   - hexrd help gui
+   #   - hexrd test
 
 about:
   license: LGPL

docs/build.rst

@@ -0,0 +1,112 @@
+HEXRD Build Instructions
+------------------------
+
+The preferred method for building the HEXRD package is via the conda
+recipe located in ``<hexrd root>/conda.recipe``
+
+Requirements
+------------
+The following tools are needed to build the package::
+
+    conda
+    conda-build
+
+With `Anaconda <https://store.continuum.io/cshop/anaconda/>`_-based Python
+environments, you should be able to run::
+
+    conda build conda.recipe/
+
+Building
+--------
+
+First, the dependencies for building an environment to run hexrd::
+
+    - cython
+    - fabio <pip>
+    - h5py
+    - matplotlib
+    - numba
+    - numpy
+    - progressbar >=2.3
+    - python
+    - pyyaml
+    - setuptools
+    - scikit-image
+    - scikit-learn
+    - scipy
+    - wxpython
+
+If you will be running scripts of you own, I also strongly suggest adding spyder::
+
+    - spyder
+
+For example, to buid an environment to run hexrd v0.6.x, do the following::
+
+    conda create --name hexrd_0.6 cython h5py matplotlib numba numpy python=2.7 pyyaml setuptools scikit-image scikit-learn scipy spyder
+    conda install -c anaconda --name hexrd_0.6 wxpython
+    conda install -c anaconda --name hexrd_0.6 progressbar
+    conda activate hexrd_0.6
+
+
+Then install using setuptools::
+
+    python setup.py install
+
+Note, you will have to install fabio in the same environment using ``setup.py`` as well.
+The procedure for building/installing with conda-build is as follows (*this is curently broken*)
+
+First, update conda and conda-build::
+
+    conda update conda
+    conda update conda-build
+
+Second, using ``conda-build``, purge previous builds (recommended,
+not strictly required)::
+
+    conda build purge
+
+In the event that you have previously run either
+``python setup.py develop`` OR ``python setup.py install``, then first run
+either::
+
+    python setup.py develop --uninstall
+
+or::
+
+    python setup.py install --record files.txt
+    cat files.txt | xargs rm -rf
+
+depending on how it was installed using ``distutils``.  This will
+remove any old builds/links.
+
+Note that the "nuclear option" for removing hexrd is as follows::
+
+    rm -rf <anaconda root>/lib/python2.7/site-packages/hexrd*
+    rm <anaconda root>/bin/hexrd*
+
+If you have installed ``hexrd`` in a specific conda environment, then
+be sure to use the proper path to ``lib/`` under the root anaconda directory.
+
+Next, run ``conda-build``::
+
+    conda build conda.recipe/ --no-test
+
+Note that the ``--no-test`` flag supresses running the internal tests
+until they are fixed (stay tuned...)
+
+Installation
+------------
+
+Findally, run ``conda install`` using the local package::
+
+    conda install hexrd=0.6 --use-local
+
+Conda should echo the proper version number package in the package
+install list, which includes all dependencies.
+
+At this point, a check in a fresh terminal (outside the root hexrd
+directory) and run::
+
+    hexrd --verison
+
+It should currently read ``hexrd 0.6.5``

docs/source/dev/imageseries.rst

@@ -0,0 +1,17 @@
+imageseries package
+===============
+The *imageseries* package provides a standard API for accessing image-based data sets.  The primary tool in the package is the ImageSeries class. It's interface is analagous to a list of images with  associated image metadata. The number of images is given by the len() function. Properties are defined for image shape (shape), data type (dtype) and metadata (metadata). Individual images are accessed by standard subscripting (e.g. image[i]).
+
+The package contains interfaces for loading (load) and saving (save) imageseries. Images can be loaded in three formats: 'array', 'hdf5' and 'frame-cache'. The 'array' format takes the images from a 3D numpy array. With 'hdf5', images are stored in hdf5 file and accessed on demand. The 'frame-cache' is a list of sparse matrices, useful for thresholded images. An imageseries can be saved in 'hdf5' or 'frame-cache' format.
+
+The imageseries package also contains a  module for modifying the images (process). The process module provides the ProcessedImageSeries class, which takes a given imageseries and produces a new one by modifying the images. It has certain built-in image operations including transposition, flipping, dark subtraction and restriction to a subset.
+
+
+Metadata
+----------------
+
+The metadata property is generically a dictionary. The actual contents depends on the application. For common hexrd applications in which the specimen is rotated while being exposed to x-rays, the metadata has an 'omega' key with an associated value being an nx2 numpy array where n is the number of frames and the two associated values give the omega (rotation) range for that frame.
+
+Reader Refactor
+-------------
+While the imageseries package is in itself indpendent of hexrd, it was used as the basis of a refactoring of the reader classes originally found in the detector module. The main reader class was ReadGE. In the refactored code, the reader classes are now in their own module, image_io, but imported into detector to preserve the interface. The image_io module contains a generic OmegaImageSeries class for working with imageseries having omega metadata. The refactored ReadGE class simply uses the OmegaImageSeries class to provide the same methods as the old class. New code should use the OmegaImageSeries (or the standard ImageSeries) class directly.

docs/source/dev/index.rst

@@ -9,3 +9,4 @@ Contents:
 
    getting_started
    releases
+   imageseries

hexrd/cacheframes.py

@@ -64,8 +64,10 @@ def load_frames(reader, cfg, show_progress=False):
     # TODO: this should be updated to read only the frames requested in cfg
     # either the images start, step, stop, or based on omega start, step, stop
     start = time.time()
-
-    n_frames = reader.getNFrames()
+    if cfg.image_series.images.stop is not None:
+        n_frames = cfg.image_series.images.stop
+    else:
+        n_frames = reader.getNFrames()
     logger.info("reading %d frames of data, storing values > %.1f", n_frames, cfg.fit_grains.threshold)
     if show_progress:
         widgets = [Bar('>'), ' ', ETA(), ' ', ReverseBar('<')]
@@ -90,6 +92,7 @@ def load_frames(reader, cfg, show_progress=False):
     return reader
 
 def cache_frames(reader, cfg, show_progress=False, overwrite=True):
+    start = time.time()
     cache_file = os.path.join(cfg.analysis_dir, 'frame_cache.npz')
     # load the data
     reader = load_frames(reader, cfg, show_progress)
@@ -101,7 +104,6 @@ def cache_frames(reader, cfg, show_progress=False, overwrite=True):
         arrs['%d_data' % i] = coo.data
         arrs['%d_row' % i] = coo.row
         arrs['%d_col' % i] = coo.col
-    start = time.time()
     np.savez_compressed(cache_file, **arrs)
     elapsed = time.time()-start
     logger.info('wrote %d frames to cache in %g seconds', len(reader[0]), elapsed)
@@ -121,12 +123,23 @@ def get_frames(reader, cfg, show_progress=False, force=False, clean=False):
 
     # temporary catch if reader is None; i.e. raw data not here but cache is
     # ...NEED TO FIX THIS WHEN AXING OLD READER CLASS!
+    # the stop is treated as total number of frames read, which is inconsistent with
+    # how the start value is used, which specifies empty frames to skip at the start
+    # of each image.  What a fucking mess!
     if reader is not None:
-        n_frames = reader.getNFrames()
+        # override
+        if cfg.image_series.images.stop is not None:
+            n_frames = cfg.image_series.images.stop
+        else:
+            n_frames = reader.getNFrames()
     else:
-        n_frames = int( (cfg.image_series.omega.stop \
-                         - cfg.image_series.omega.start) \
-                         / float(cfg.image_series.omega.step) )
+        # still need a fix here for when you have cache only...
+        if cfg.image_series.images.stop is not None:
+            n_frames = cfg.image_series.images.stop
+        else:
+            n_frames = int( (cfg.image_series.omega.stop \
+                            - cfg.image_series.omega.start) \
+                            / float(cfg.image_series.omega.step) )
     logger.info("reading %d frames from cache", n_frames)
 
     with np.load(cache_file) as npz: