Use tempdir during execution

[alecandido]

New implementation of #152, adapting some content from there.

• propagate changes to the rest of the package
• fix unit tests
• fix unit test on py3.8
• fix log-less isolated benchmarks
• check still to accomplish the goal of Output metadata #162
• Reorganize CLI #171 (review)
• Reorganize CLI #171 (comment)
• CVE-2007-4559 Patch #154 +

  eko/src/eko/output/struct.py

  Lines 622 to 634 in 2077372

  	shutil.unpack_archive(path, target, "tar")
  	# load necessary assets
  	theory = yaml.safe_load((target / THEORYFILE).read_text())
  	operator = yaml.safe_load((target / OPERATORFILE).read_text())
  	eko = cls.detached(theory, operator, path=path, working_dir=target)
  	logger.info(f"Operator loaded from path '{path}' into '{target}'")
  	return eko
  	def write_tar(self):
  	"""Write the operator back to disk."""
  	shutil.make_archive(self.path.stem, "tar", self.working_dir)

• import and test old tar output
• scope num_flavs_ref and num_flavs_max_as in Couplings structure
• prevent operations on a closed EKO
• prevent modifications on a read-only EKO
• more Pythonic lambergrid name Use tempdir during execution #172 (comment)
• test metadata presence Use tempdir during execution #172 (comment)
• remove defaults Use tempdir during execution #172 (comment)
• change num_flavs_max_as to a less redundant one Use tempdir during execution #172 (comment)
• update docstring Use tempdir during execution #172 (comment)
• fix tutorials Use tempdir during execution #172 (review)
• check $\alpha_s$ Use tempdir during execution #172 (comment)
• recover 100% coverage

[codecov-commenter]

Codecov Report

Merging #172 (d15e1d1) into cli (c32d054) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Additional details and impacted files

@@           Coverage Diff           @@
##              cli     #172   +/-   ##
=======================================
  Coverage   99.43%   99.43%           
=======================================
  Files         100      100           
  Lines        4586     4587    +1     
=======================================
+ Hits         4560     4561    +1     
  Misses         26       26           

Flag	Coverage Δ
isobench	53.65% <100.00%> (+0.01%)	⬆️
unittests	99.43% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/eko/__init__.py	100.00% <100.00%> (ø)

[alecandido]

Ok, I completed the massive rework of the structure, now I have to fix all tests and polish everything.

I will add the to-do list in the original message.

[alecandido]

Ok, now this implements also the changes that were scheduled for #160.

I'm at the level of fixing tests. I'm not sure if I should target coverage in this PR though, I fear it'll become too big...

[alecandido]

I still have 38 tests failing, mostly for AttributeErrors, that are generated by the reshuffling.

At this point, features should be all available, and this PR should also close #114 and addresses part of #157

[alecandido]

At this point, I decided that it is easiest to merge just this one, closing #171 #162 #161

This branch contains all of them, and it is already big enough that it doesn't change so much, including the other ones.
Moreover, a lot of changes there are partially overwritten here.

[alecandido]

15 + 8 tests still failing, I'm getting there...

[alecandido]

Since now we have open (with dir available) and closed (just a path) operators, a question arises.

When we compute an EKO, should we return an open one (and leave the responsibility to close it to a user that never opened it), or a closed one (and the user has to load it to use)?

[alecandido]

Last 3 unit tests missing, but for them I will take slightly more, since I want to briefly rework the legacy output.

In particular, I will drop load_yaml and dump_yaml (never used yaml format in production, so much better to get rid of one extra unused format) and dump_tar, since we will never produce new EKO in the old format.
Instead, I will make sure load_tar is actually working with the new struct, since this is the basic ingredient for the converter to the new format.

[felixhekhorn]

first batch

[felixhekhorn]

Second batch before attempting the big ones

[felixhekhorn]

I had a look to everything and with one read it seems fine ... depending on the urgency we can postpone

• documentation is still scarse
• tutorials are of course completely broken

to be definitely addressed before merging:

• recover LHA benchmark

[alecandido]

Looks like now I'm in trouble with python versions: tests are passing on 3.10, but I used some syntax that was not available in 3.8.
Most of them I can easily fix, let's hope all...

[alecandido]

Tests are failing on py3.8 for a not very clear reason, I need further investigation, but passing on py3.9 and py3.10.
The motivations are actually only two:

ValueError: negative dimensions are not allowed
TypeError: 'numpy.float64' object cannot be interpreted as an integer

All coming from the same place: deserialization. But it is mystery why it fails only on that Python version, it would be useful to have py3.8 instance to debug, maybe I will do inside a container.

I will try to debug first isolated benchmarks, they seem to be easier to solve, and failing also on recent versions :)

[alecandido]

Just missing coverage, that I would postpone to a later PR.

I'm ready to merge, and we are just 3 commits away from the 200 target :)

[felixhekhorn]

Just missing coverage, that I would postpone to a later PR.

Fine - then let's do it (of course you have the pleasure of the big green button 🙃 ). However, afterwards, we have to actually put some effort to cover the remaining 140 lines ...

I'm ready to merge, and we are just 3 commits away from the 200 target :)

and with d05ca84 there are only two left ... (the improved test fails indeed without the accompanying fix)

[felixhekhorn]

The moment we merge

• @giacomomagni has to take care of N3LO ad variations #148 (mostly orthogonal, but still non-zero impact)
• @niclaurenti has to take care of Implement QED exact solution #135 (mostly orthogonal, but still non-zero impact)
• @andreab1997 should try to get pineko back
• among the three of us @alecandido @giacomomagni to fix LO polarised anomalous dimension #170 for the meeting on Thursday

[andreab1997]

The moment we merge

• @giacomomagni has to take care of N3LO ad variations #148 (mostly orthogonal, but still non-zero impact)
• @niclaurenti has to take care of Implement QED exact solution #135 (mostly orthogonal, but still non-zero impact)
• @andreab1997 should try to get pineko back
• among the three of us @alecandido @giacomomagni to fix LO polarised anomalous dimension #170 for the meeting on Thursday

Yes, I will do that. BTW remember to revert all the alpha_raw stuff here (as we agreed, but we could also open another PR of course).

[alecandido]

I would just wait for @giacomomagni confirmation, and then merge.

Yes, I will do that. BTW remember to revert all the alpha_raw stuff here (as we agreed, but we could also open another PR of course).

@andreab1997 open a dedicated issue for the time being, and right after merging we will start the (hopefully small) PR

[giacomomagni]

Okay green light from my side.
We might need to updated something minor also in Yadism?

[alecandido]

We might need to updated something minor also in Yadism?

Eventually, yes (at least for computing predictions without PineAPPL).
For the time being, yadism is just running with an older version of EKO (essentially for interpolation, that is almost unchanged), so you might need to environments for grids and FkTables computation, but it would work.

[giacomomagni]

Eventually, yes (at least for computing predictions without PineAPPL). For the time being, yadism is just running with an older version of EKO (essentially for interpolation, that is almost unchanged), so you might need to environments for grids and FkTables computation, but it would work.

If I recall correctly we have already introduced eko=0.11 in yadism see here. I can take care of it.

[alecandido]

If I recall correctly we have already introduced eko=0.11 in yadism see here. I can take care of it.

You're right, and thanks for the offering. I would suggest to pretend it is working, and as soon as anyone find an issue, we will get back to you (or any of us) and ask to make it compatible.

[felixhekhorn]

just to say that for sure we need to do some more adjustments for eko v0.12 e.g. here