Data ontology preparation agent

[eugeneboms]

Proposal

Implement an agent (more likely as a collection of skills) that takes several high-dimensional CSVs and makes them suitable for Fabric Ontology upload.

Justification

Throwing an arbitrary unprocessed CSV file into Fabric Ontology often results in slow processing and inferior query responses due to the following problems commonly present in real-life data:

• Too high dimensionality (above ~30 columns things become extremely slow)
• Data redundancy, whereas multiple columns of nearly the same meaning confuse the ontology builder and inference
• Data noise, gaps, errors, acronyms resulting in additional errors

We've observed that situation during one of the hackathons with a customer. Their (semi-synthetic, anonymized) data in several CSV files featured over 100 dimensions and high redundancy. It took over 24 hours to create the ontology, and the resulting agent often had troubles with relatively simple questions.

Details

The agent would need to perform the following:

1. Clean up common errors like missing data.
2. Acronyms expansion, whenever possible.
3. Dimensionality reduction via elimination of columns that are:
   3.1. Easily derived from other columns
   3.2. Easily predictable from other columns (when numeric)
   3.3. Blank, uniform, or with very few bits of information.

Challenges

• Data elimination is risky.
• Finding all "predictable" columns from a data with N columns is, in general, an O(N^2) task and should be done carefully to avoid a perf bottleneck.
• The overall workflow is complex and needs to be effectively split into multiple skills.

Current status

A local early POC on my machine, seems to be working, with admittingly very limited testing.

Open questions

1. Do we think something like this would be useful?
2. Does anybody already have it, or some parts of it? (To avoid duplication of work)
3. Anyone interested to cooperate?