Granite layout hierarchy awareness

[tbutler-p]

Hi,
For documents that don't have clear table structures, text elements for key-value pars, are enclosed in loc_xxx tags (bounding boxes). For example, converted medical records can contain key-value pairs like Patient name: and John Smith. When these are enclosed in loc tags (in json or doctags), the key-value relation is lost. For the most simple usecase, it would be great if you can give Granite 4 a doctags file, and ask it to return the patient name, date of birth and initial diagnosis. Currently, Granite 4 is not able to understand the layout hierarchy of text elements enclosed in bounding boxes. Are there any plans to:

• Train Granite on doctags format
• Develop a method to add layout hierarchy awareness to vector store embedding and retrieval, so it can be used with RAG setups

This would help with QA on document datasets that are currently not possible. Example questions that currently won't work in above usecase are:

• Give me the date of birth of the 100 youngest patients diagnosed with xxx
• Give me the results of x for patients with blood type y

The above examples won't work if the medical records are from different systems from different hospitals, and the documents have a mixture of tables and bounding boxes in their doctags document. For bounding boxes, the key-value relation is lost. I think the biggest challenge here might be to identify the key-value pair relation to begin with. The value may be to the right of the key, but it can also be below (or other orders for other languages).