The entire CodingDonky repo is going to be removed soon. It serves only as a testing ground for Alex Newberry. Everything will be cleaned up and ported over to the ActiveBrainAtlas repo. Some files I will need to be extra careful with as I have altered them significantly.
Notebooks that have been altered:
- preprocess_cshl_data_v2_neurotrace.ipynb
- preprocess_ucsd_data_v2.ipynb
- brightness_correction.ipynb
- distributed_utilities.py
- utilities2015.py
MD662&661-F1-2017.06.02-17.07.55_MD662_1_0001 <STACK_NAME>----<STACK_NAME_2>_-
- STACK_NAME:
- AA:
RAW: All *_raw.j2 are stored in Bucket mousebrainatlas-rawdata (as well _lossy files that are typically unused).DATA: Bucket mousebrainatlas-data contains ALL other files.
Naming conventions are the each file has a unique filename, FILENAME. Every different transform of the file will have the naming convention FILE_SUFFIX where SUFFIX is an abbreviated description of the file's state.
DATA files will always be in *.tif form unless otherwise stated. One important exception is that the RAW files given to us are typically in the format *_lossless.jp2.
Examples:
[FILENAME STEM] = MD662&661-F1-2017.06.02-17.07.55_MD662_1_0001
- raw_input =
MD662&661-F1-2017.06.02-17.07.55_MD662_1_0001_lossless.jp2 - output_1 =
MD662&661-F1-2017.06.02-17.07.55_MD662_1_0001_raw_Ntb.tif - output_2 =
MD662&661-F1-2017.06.02-17.07.55_MD662_1_0001_thumbnail_Ntb.tif
STACKNAME = the unique identifier for every stack. Example: MD662
folder naming conventions: STACKNAME_SUFFIX/
Preprocessing Steps:
- raw (.jp2) -> raw_Ntb (.tif): extract_a single channel
- raw_Ntb -> thumbnail_Ntb: rescale
- thumbnail_Ntb -> thumbnail_NtbNormalized: normalize_intensity
- Compute transforms using thumbnail_NtbNormalized: align + compose
- Supply prep1_thumbnail_mask :
- prep1_thumbnail_mask -> thumbnail_mask: warp
- raw_Ntb -> raw_NtbNormalizedAdaptiveInvertedGamma: brightness_correction
- Compute prep5 (alignedWithMargin) cropping box based on prep1_thumbnail_mask
- raw_NtbNormalizedAdaptiveInvertedGamma -> prep5_raw_NtbNormalizedAdaptiveInvertedGamma: align + crop
- thumbnail_NtbNormalized -> prep5_thumbnail_NtbNormalized: align + crop
- prep5_raw_NtbNormalizedAdaptiveInvertedGamma -> prep5_thumbnail_NtbNormalizedAdaptiveInvertedGamma: rescale
- Specify prep2 (alignedBrainstemCrop) cropping box
- prep5_raw_NtbNormalizedAdaptiveInvertedGamma -> prep2_raw_NtbNormalizedAdaptiveInvertedGamma: crop
- prep2_raw_NtbNormalizedAdaptiveInvertedGamma -> prep2_raw_NtbNormalizedAdaptiveInvertedGammaJpeg: compress_jpeg
- Doc: documentation
- src: code
-
- Running Guide made by AlexN to assist with running the code [INC]
-
[Historical] Yuncong User Guide
- User Guide made by Yuncong to assist with running the code
-
[Historical/Incomplete] Dev Guide
- Incomplete to-be guide of entire running process
-
[Historical] Registration Steps
- Guide for Registration with list of relevant scripts
-
[Historical] Old README
- Yuncong's old README
- File Transfer
- Guide for transferring files to/from AWS S3 and Birdstore
- AWS Instructions
- AWS instruction manual
- [Incomplete] S3 File Organization
- Naming conventions in S3 storage [INC]
- Stack Directories
- Description of every stack, all relevant information
- Bayesian Parameters
- Parameters and associated uncertainty in Atlas building process
- Uncertainty
- Z-score and Hessians used to quantify uncertainty