Complete Preprocessing MATLAB Validation/Repro

[garrettwrong]

In the course of attempting to reproduce legacy MATLAB results I found that the Python code was not reproducing MATLAB preprocessed data. The biggest offenders were phase_flip, where our formula was incorrect, and whitening, where we were constructing an entirely different filter.

For both of those, I found the corresponding preprocessing method in MATLAB that consumes and returns an MRC stack. The Python methods were debugged/validated until close results were achieved. The comparisons were demonstrated in a straightforward notebook saving the invocations required to produce results in both languages. The preprocessed data using all Python methods was then usable for my purposes, but I am not sure it is totally reproducing the MATLAB preprocessed data beyond what I documented (ie it was just close enough to move on to bigger problems).

This process needs to be continued for the remaining preprocessing methods.

• phase_flip
• whiten (legacy_whiten)
• downsampling
• background normalization
• contrast inversion

Ideally I'd like to see the default invocation reproduce the MATLAB results unless we have very good reason not too. Should that case arise, I still want a legacy wrapper that produces that result. For each method, I want to see a small notebook documenting the calls in MATLAB, and demonstrating the equivalence/closeness. This will be a good opportunity to improve tests for these methods. This notebook should be saved in any related PRs.

When all methods are completed, the entire set of 5 methods, including data files and repros, should be collected and archived for reference.

[garrettwrong]

For this issue I just want to run all the preprocess methods chained together in Python and compare with MATLAB to see how close we are getting now that each method has become allclose. Assuming that is the case we can save off the generating scripts and results on a couple datasets as a future reference set for ASPIRE-Python.

[garrettwrong]

Recapping the work last week comparing the preprocessing with MATLAB.

We have significantly different CTF filters. The past attempt to reconcile them to within 1% was not good enough, and I will be recreating the MATLAB filter directly from scratch.

We're missing an entire high level preprocessing method (crop) that the Python code does not need, but is required to reproduce the MATLAB code. Their methods only support odd images. It turns out that portions of the MATLAB code and the "workflow" alter data by applying cropping silently (making no corrections for centering/offsets etc). This destroys any chance of directly comparing the workflow preprocessing results (after phase flip) 1:1 unless replicated.

Once the hidden cropping was applied, the only difference was minor, in normalize background. The "outofcore" method's defaults were take for Python, but the "workflow" tool is overriding them. We should take the workflow defaults, but allow users to enter the other values if they want.

I will put together separate issues and PR making the code changes.

[garrettwrong]

To close this issue I will create a preprocessing gallery example that parallels and catalogs the MATLAB workflow prompts. This will be sent to Yoel for his verifcation.

[garrettwrong]

#1323 brought in a demonstration of the legacy preprocessing. I was able to achieve a per image relative error on the order of single precision for the ribosome dataset at 129 and 179 pixel. Yoel's been emailed. Closing.