Hey @perrette -- quick question; do I understand correctly that the cache decorator (implemented in config.py) intentionally caches just the last remote lookup?
In further looking at #93, would it make sense to to try to append to this file, so you can have a bunch of the last calls cached?
I have no intuition about the size here -- I made this observation in looking at the behavior when I do recursive extraction (#92) on 100-1000 files, and I see the cache size drops sometimes, which I don't expect if we are only adding to it.
I can guard against this by setting an upper bound on the cache and making is user adjustable in the config....
Let me know?
Hey @perrette -- quick question; do I understand correctly that the cache decorator (implemented in
config.py) intentionally caches just the last remote lookup?In further looking at #93, would it make sense to to try to append to this file, so you can have a bunch of the last calls cached?
I have no intuition about the size here -- I made this observation in looking at the behavior when I do recursive extraction (#92) on 100-1000 files, and I see the cache size drops sometimes, which I don't expect if we are only adding to it.
I can guard against this by setting an upper bound on the cache and making is user adjustable in the config....
Let me know?