Update docs

Author: darafferty

.gitignore

@@ -66,3 +66,15 @@ Session.vim
 .idea/*
 *.iml
 
+
+doc/factor.aux
+
+doc/factor.fdb_latexmk
+
+doc/factor.fls
+
+doc/factor.out
+
+doc/factor.pdf
+
+doc/factor.synctex.gz

bin/runfactor

@@ -31,10 +31,10 @@ from factor.operations.facet_ops import *
 
 if __name__=='__main__':
     import optparse
-    opt = optparse.OptionParser(usage='%prog [-v|-q] parset [default: factor.parset] \n',
+    opt = optparse.OptionParser(usage='%prog [-v|-q] parset [default: factor.parset]',
             version='%%prog %s' % (factor._version.__version__))
-    opt.add_option('-q', help='Quiet', action='store_true', default=False)
-    opt.add_option('-v', help='Verbose', action='store_true', default=False)
+    opt.add_option('-q', help='enable quiet mode', action='store_true', default=False)
+    opt.add_option('-v', help='enable verbose mode', action='store_true', default=False)
     (options, args) = opt.parse_args()
 
     if len(args) not in [0, 1]:

doc/factor.tex

@@ -34,12 +34,10 @@
 Factor is a Python package that allows for the production of wide-field HBA
 LOFAR images using the facet calibration scheme described in van Weeren et al.\
 (2015). Note that Factor is still beta software. Please report bugs to
-[email protected].
-
-To initialize your environment for Factor, users on CEP2 and CEP3 should run the
-following commands:
+{\tt [email protected]}. To initialize your environment for Factor,
+users on CEP2 and CEP3 should run the following commands:
 \begin{verbatim}
-use LofIm
+use Lofar
 source ~rafferty/init_factor
 use Pythonlibs
 \end{verbatim}
@@ -51,11 +49,13 @@ \subsection{Usage}
 Factor is run from the command-line with the {\tt runfactor} executable as
 follows:
 \begin{verbatim}
-Usage: runfactor <parset>
+Usage: runfactor [-v|-q] parset [default: factor.parset]
+
 Options:
   --version   show program's version number and exit
   -h, --help  show this help message and exit
-  -v          Verbose
+  -q          enable quiet mode
+  -v          enable verbose mode
 \end{verbatim}
 The parset specifies the parameters of the run. These are described in the next
 sections.
@@ -161,12 +161,25 @@ \subsection{Parset}
 param2 = 123
 \end{verbatim}
 
+
+%-----------------------------------------------------------
+\subsection{Data Preparation}
+\label{factor:data}
+
+The input data must be have the average amplitude scale set and average clock
+offsets removed. Futhermore, the LOFAR beam towards the phase center should be
+removed. The data should then undergo direction-independent, phase-only self
+calibration, and the resulting solutions must be provided to Factor. If desired,
+all sources can be subtracted before Factor is run, thereby allowing one to skip
+the initial subtraction operation. These steps can be automated using the facet
+calibration preparation pipeline.
+
 %-----------------------------------------------------------
 \subsection{Directions}
-\label{factor:directions
+\label{factor:directions}
 
 The directions for which DDE calibration is performed can be selected
-automatically by factor or specified in a file. The format of the file is as
+automatically in Factor or specified in a file. The format of the file is as
 follows:
 
 \begin{verbatim}
@@ -176,15 +189,13 @@ \subsection{Directions}
 #
 # Columns are defined as follows (in [] are optional parameters with their default values):
 # Name (str), RA (float), DEC (float), [regionfile (str, default='')],
-# [multiscale (bool, default=True)], [solint_amp (int, default=600)],
+# [multiscale (bool, default=False)], [solint_amp (int, default=600)],
 # [solint_ph (int, default=5)], [make_final_image (bool, default=True)],
 # [cal_radius (float arcmin, default=3.0)]
-
 calibrator_1, 111.11, 22.22, , , , , ,
 calibrator_2, 222.22, 33.33, reg/cal2.reg, True, 600, 5, False, 4.5
 calibrator_3, 333.33, 44.44, , True, 600, 5, ,
 \end{verbatim}
-
 This file should be specified in the parset file as the {\tt directions\_file}
 parameter.
 
@@ -197,7 +208,9 @@ \subsection{Directions}
 the specified flux and size limits ({\tt flux\_min\_Jy} and {\tt
 size\_max\_arcmin}, respectively) and nearby patches are merged (specified by
 {\tt separation\_max\_arcmin}). The resulting patches are then sorted by
-apparent flux at the lowest available frequency (from bright to faint).
+apparent flux at the lowest available frequency (from bright to faint), and the
+final {\tt directions\_file} is written to the file {\tt
+factor\_directions.txt} in the working directory.
 
 %Additionally, the direction centers used for tessellation can be automatically
 %adjusted to avoid cases in which known sources (from the initial CC skymodels)
@@ -211,39 +224,46 @@ \subsection{Directions}
 with the groupings parameter. Note that more than one node is required to
 process directions in parallel (as little benefit will result from running
 multiple directions in parallel on a single node). The groupings parameter
-specifies the the grouping level (the number of directions to process
+specifies the grouping level (the number of directions to process
 simultaneously) and the total number of directions at each grouping level. For
 example, {\tt groupings = 1:5, 4:0} means two groupings are used, with the first
-5 directions put into groups of one (i.e., each direction is processed in
+5 directions put into groups of one (i.e., the directions are processed in
 series) and the rest of the directions divided into groups of 4 (i.e., 4
 directions processed in parallel). Shuffling is done between neighboring groups
 to achieve the largest minimum separation between directions in the groups. This
-sorting attempts to minimize the effects that any artifacts from the
-simultaneously processed directions have on one another.
+sorting attempts to minimize the effects that any artifacts from one direction
+might have on the other simultaneously processed directions.
 
 
 %-----------------------------------------------------------
 \subsection{Parallelization}
 \label{factor:parallel}
 
 Factor has been designed to run as much as possible in a parallel manner. To
-this end, support is present for distributing the reduction over nodes of a
-compute cluster and over the processors of a single computer.
+this end, support is present for distributing the reduction over the processors
+and nodes of a compute cluster. To use Factor on a single computer, no setup is
+necessary beyond specifying the maximum number of CPUs to use. To use Factor on
+multiple nodes of a compute cluster, the required setup depends on the cluster
+layout (distributed vs. shared file system) and scheduling system (if any), as
+described in the following sections.
+
+\subsubsection{Cluster without a scheduler}
+For use on a cluster that does not use a scheduler to assign nodes to the user, such as CEP2 and CEP3,
+you must provide a LOFAR pipeline cluster description file that tells Factor which nodes are
+available. An example of such a file is shown below.
 
-To use Factor on a single computer, no setup is necessary beyond specifying the
-maximum number of CPUs to use.
-
-To use factor on multiple nodes of a compute cluster, the required setup depends
-on the cluster layout (distributed vs. shared file system) and scheduling system
-(if any).
+\begin{verbatim}
+ClusterName = MyCEP3
+Compute.Nodes = [lof021, lof022]
+\end{verbatim}
 
-\subsubsection{Torque / PBS scheduler}
-For use on a compute cluster that uses torque and PBS, set {\tt clusterdesc =
+\subsubsection{Cluster with Torque / PBS }
+For use on a cluster that uses torque and PBS, you can set {\tt clusterdesc =
 PBS}. Factor will then automatically determine the nodes for which you have a
 PBS reservation and use them. Note that you must ask for all the nodes you need
-in a single PBS script (e.g., with {\tt #PBS -l nodes=6:ppn=6}), so that all
-nodes are available for the full Factor run. An example PBS script is shown
-below.
+in a single PBS script (e.g., to use 6 nodes, with {\tt \#PBS -l
+nodes=6:ppn=6}), so that all nodes are available for the full Factor run. An
+example PBS script is shown below.
 
 \begin{verbatim}
 #!/bin/bash
@@ -252,16 +272,15 @@ \subsubsection{Torque / PBS scheduler}
 #PBS -l nodes=6:ppn=6
 #PBS -j oe
 #PBS -o output-$PBS_JOBNAME-$PBS_JOBID
-
 cd $PBS_O_WORKDIR
 source /home/sttf201/init-lofar.sh
 runfactor factor.parset
 \end{verbatim}
 
-\subsubsection{Shared file system}
+\subsubsection{Cluster with a shared file system}
 Factor works by default on a compute cluster with a shared file system.
 
-\subsubsection{Distributed file system}
+\subsubsection{Cluster with a distributed file system}
 For use on a cluster with a distributed file system (such as CEP2 and CEP3), set
 {\tt distribute = True}. Factor will then synchronize files between the
 available nodes. Note that all data will be mirrored on each node, so one must
@@ -317,14 +336,20 @@ \subsection{Structure}
 \item[FacetImage] The direction-dependent calibration is applied, and the
 bands are concatenated and averaged. The entire facet is then imaged, producing
 a final model of the facet sources.
-\item[FacetSubAll] The final model is subtracted, producing improved ``source-
-free'' datasets.
+\item[FacetSubAll] The final model is subtracted, producing improved
+``source-free'' datasets.
 \end{description}
 
 The facet operations are repeated for each direction. Some operations, such as
-the FacetSelfcal operation, can run on multiple directions simultaneously.
-
-After each action finishes, the saved state is updated, allowing for resumption
+the FacetSelfcal operation, can run on multiple directions simultaneously. After
+each action finishes, the saved state is updated, allowing for resumption
 of steps at the granularity of the actions in a given operation.
 
+The results of the processing are stored in subdirectories in the working
+directory by type (e.g. {\tt images} and {\tt logs}). Inside these
+subdirectories, results are organized by operation, action, band, and/or
+direction. For example, for a direction named ``dir1'', the images produced
+during self calibration are stored in {{\tt images/FacetSelfcal/MakeImage/dir1/}.
+
+
 \end{document}