Skip to content

Home

Jump to bottom
Adam Wright edited this page Aug 19, 2019 · 11 revisions

Welcome to the wtgpipeline wiki!

In order to get cluster lensing masses, you're going to have to work your way through the ten steps:

  1. Preprocessing/ISR/Brighter-Fatter Correction
  2. Distribute Sets and Mask
  3. Directories and Headers changed. Cross-talk Correction applied.
  4. Initial Astrometry
  5. Illumination Correction
  6. Final astrometry
  7. Coadding
  8. Coadd Masking
  9. Photometric Measurement and Calibration
  10. Photometric Redshifts

Steps 11 (measure CC masses) and 12 (measure p(z) masses) are on the gravitas repo: https://github.com/arwright3/gravitas

These are the image processing steps that come before lensing measurements are made:

  1. preprocessing
do_Subaru_preprocess_10_2_adam.sh do_Subaru_preprocess_10_2_simple.sh do_Subaru_preprocess_10_3_adam.sh do_Subaru_preprocess_10_3_simple.sh do_Subaru_preprocess_adam.sh do_Subaru_preprocess_config8.sh do_Subaru_preprocess_notes.sh do_Subaru_preprocess_template.sh

if including the BFcorr, then you have to use process_science_4channels_eclipse_para-noos_BFCTcorr.sh instead!!!
if you haven't yet done distribute sets, then do this:./distribute_sets_subaru.sh ${SUBARUDIR} ${run}_${filter}/SCIENCE ${ending} 1000 ${SUBARUDIR}/SUBARU.list
  1. distribute sets and mask
adam_do_masking_master.sh (old versions: distribute_sets_subaru.sh and do_masking.sh OR instructions.sh)
  1. Directories and headers changed. Cross-talk correction applied.
directories changed from $filter_$run to $filter AND setup header files (for each ${filter}) AND do the cross-talk correction
./adam_do_linking_filter_dirs.sh ${cluster} ${filter_run_pairs}
./adam_do_update_headers.sh ${cluster} ${filter_run_pairs}

CTcorr uses: adam_CTcorr_make_images_para.sh adam_CTcorr_run_correction_para.py
Made batch wrapper for them using the combo of these:

    * adam_TEMPLATE_switcher.py
    * adam_CTcorr_allOCF-TEMPLATE.sh

a.) change directory structure
ln -s /nfs/slac/g/ki/ki18/anja/SUBARU/${cluster}/${filter}_*/SCIENCE/SUPA*.fits /nfs/slac/g/ki/ki18/anja/SUBARU/${cluster}/${filter}/SCIENCE/
ln -s /nfs/slac/g/ki/ki18/anja/SUBARU/${cluster}/${filter}_*/WEIGHTS/SUPA*.fits /nfs/slac/g/ki/ki18/anja/SUBARU/${cluster}/${filter}/WEIGHTS/
b.) setup header files (for each ${filter})
./update_config_header.sh /nfs/slac/g/ki/ki18/anja/SUBARU/${cluster}/${filter}/SCIENCE SUBARU ${cluster}
c.) do the cross talk correction for 10_3 ONLY!
  1. initial astrometry
have `mode=astrom` in do_Subaru_register_4batch.sh
./do_Subaru_register_4batch.sh ${cluster} 2MASS astrom "W-J-B W-C-RC W-S-Z+"
OR if in SDSS footprint: SDSS DR10 Finding Chart Tool at skyserver.sdss.org
./do_Subaru_register_4batch.sh ${cluster} SDSS-R6 astrom "W-J-B W-J-V W-C-RC W-C-IC W-S-Z+" 
after astrometry call check the output plots in /nfs/slac/g/ki/ki18/anja/SUBARU/${cluster}/${filter}/SCIENCE/astrom_scamp_2MASS/plots
  1. illumination correction ( see Evernote)
before running simple_ic.py, I need to have OBJNAME in all images (and consistent in all images). Might as well do the same for OBJECT and MYOBJ too.
I also should rename PPRUN to PPRUN0 and have filter_run be the pattern for PPRUN
ipython develop_simple_ic/simple_ic.py
  1. final astrometry
have `mode=photom` in do_Subaru_register_4batch.sh
./do_Subaru_register_4batch.sh ${cluster} 2MASS photom "W-J-B W-C-RC W-S-Z+"
OR if in SDSS footprint: SDSS DR10 Finding Chart Tool at skyserver.sdss.org
./do_Subaru_register_4batch.sh ${cluster} SDSS-R6 photom "W-J-B W-J-V W-C-RC W-C-IC W-S-Z+"
check the output plots in /nfs/slac/g/ki/ki18/anja/SUBARU/${cluster}/${filter}/SCIENCE/astrom_photom_scamp_2MASS/plots

  1. Coadding

  2. Coadd Masking

  3. Photometry

  4. Photo-Zs

Once lensing measurements have been completed, there are other important steps like the ones which use the COSMOS catalog as a calibration tool:

Other pages might be helpful to understand what's going on in the code. E.g. https://github.com/deapplegate/wtgpipeline/wiki/Code-Comment-Syntax

Everything you need to know in order to run the Weighing the Giants pipeline

In case anyone ever needs to be able to run the Weighing the Giants cluster lensing mass pipeline, it's almost all on this github repo and the gravitas repo (https://github.com/arwright3/gravitas). Here are some other pieces of the pipeline that are called in this code, but aren't included in the repo. You can email me if you need them:

  • useful archive checker: telarchive-1.8.2
  • Brighter-Fatter correction from Augustine: SoftForAdam and filter_adam
  • sdfred Subaru pipeline (ctcorr is short for cross-talk corrector): sdfred20130924_mf2 , ctcorr20140102.tar.gz, and sdfredSH
  • Some useful sextractor stuff: sextractimtools
  • BigMacs pre-Panstarrs: big-macs-calibrate

If there are problems, or questions about which versions of certain scripts to use, or why something isn't working, then there are two other places to look for answers for collaborators with access to the computers at SLAC:

  1. Adam's bonnpipeline directory: ~awright/bonnpipeline
  2. Doug's bonnpipeline directory: ~dapple/nfs/pipeline/bonnpipeline

Everything in wtgpipeline came directly from one of these two script collections.

Here are some further pipeline-specific resources:

Clone this wiki locally