Comments

[martindurant]

I haven't looked at the code yet, but I am glad that this demonstration is here!
I have a few things to say up front:

• the comparison of on-disk size should be made versus highly Zstd-compressed JSON, since the parquet data is compressed internally, and Zstd adds little overhead when reading. Also, make sure you have ujson installed for fair comparison, as it's much faster than standard json, and referenceFS will use it when it can.
• loading time and file size can probably both be improved by setting offset/size to 0 rather than NULL where there ie embedded data; this may also be true if loading the string columns as string[pyarrow] (NB: you don't specify the parquet engine to use). That column type is not yet supported in fastparquet, but would be far more efficient in terms of memory usage.
• you mention the future with lazy loading and partitioning, and I think these will be the killer features. Remember, we could structure the parquet data files to contain only specific subsets of key prefixes, or even use the path names of the data files to encode part of the keys. This is necessary as reference sets will easily become too big to simply fit into memory; but of course then we need to have an extra indirection and cache parts of the dataframe.
• do we have a benchmark of df.loc versus dict[item]? What about if the df is sorted on the index? Is pandas actually the most efficient data structure for this?

How much of an ask is it to upstream this into referenceFS? Should it be a different implementation when it shares everything except the reference lookups?

[d70-t]

Thanks for the feedback!
I can provide only a few more data points for now:

• I've had ujson installed already (although I didn't think of it before), so it's already the "fast" baseline.
• using default zstd compression (i.e. zstd infile -o outfile) gets the JSON down from 6.8 GB -> 110 MB 👍 (parquet was 360MB)
• using zstd compression within parquet (instead of snappy, the pandas-default) gets the parquet down from 360 MB -> 31 MB 👍 👍
• I was assuming sorted dataframes, I think looking up by key (and in particular key-ranges for ls) doesn't work if it isn't sorted anyways.

I currently can't estimate how much effort it would be to merge it with the current referenceFS. I didn't pay much attention at how the differrent remote protocols are handled in referenceFS and didn't really care if anything except getting the mapper works in preffs. However, I needed support for concatenation in my use-case, so I wanted to have that one in. But yes, it's not very far away and if df.loc is similar to dict[item], we could probably use pandas as the common backend. However, if we plan to go towards lazy loading (I don't know how to do that), we might have to switch once more, so probably we'd want to have something a bit more generic...

[martindurant]

However, I needed support for concatenation in my use-case

It's a fine point. But also one of those things where the complexity of including the feature could plausibly affect performance. Perhaps there can be a global option when we don't ever have multiple entries for any key.

We would need to agree on how this is represented in the JSON standard, if at all. A handy tool to convert between the two types would be good.