[TO-DO] Investigate and create a dataset on HuggingFace to store our testing receipts

[Sing-Li]

This is a desirable best practice originating from the "Perfect AI Docs Assistant" project https://github.com/RocketChat/google-summer-of-code/blob/main/google-summer-of-code-2025.md#-perfect-ai-docs-assistant-app

The dataset should include receipts (perhaps in base64 format) that we will use in our unit tests. It should be organized in a manner that is easy for our unit tests to consume.

In the future perhaps we can even experiment with the use of the dataset to fine-tune/post-train a rc-receipts-friendly vision model.

[moyrsd]

Hi, I have created a dataset hugging face dataset. Do you think it will work.

[Sing-Li]

Thanks @moyrsd ! I don't know, can you try to use HuggingFace Typescript/Javascript API and see if you can access them one at a time.

Please post some sample code of access in your reply.

[moyrsd]

Hi, when I try to hit the api it gives the following error TooBigContentError , hugging face does not support api access for this large dataset, the max limit is around 200 mb. I am thinking to reduce dataset, and include one more column which contains item and price. Please let me know if I should do this or I should look for some way to compress base64 images .

async function getDatasetRow() {
  const response = await fetch(
    `https://datasets-server.huggingface.co/rows?dataset=moyrsd%2Frocketchat_receipts_dataset&config=default&split=train&offset=0&length=1`
  );
  const dataset = await response.json(); 
  console.log(dataset);
}

getDatasetRow();

Output

{
  error: 'Rows from parquet row groups are too big to be read: 704.03 MiB (max=286.10 MiB)'
}```

[Sing-Li]

@moyrsd good work! But please realize that if the row is larger than 286,000,000 bytes each in base64 <-- it will be totally useless for us. Since I don't think any LLM has context that is 286,000,000 yet.

The solution is to create another dataset that contains rows (each well under 286mb, llama3.2 11b vision can only take 125k max image up to 1120 x 1120 in size) : reducing your original scanned image size to what is acceptable to the LLM model and THEN base64 encoded

[moyrsd]

This dataset is formated according to the expected promt output in our app. I have tested hugging face api calls, it works. I have also tested using the called data as a input in llama-3.2-vision, it works. Now, I can work on unit tests

[Sing-Li]

@moyrsd have you really tested with all the rows? Some of them are larger than the context of llama-3.2-vision, how can it have worked?

[moyrsd]

Hi, I had tested fot only the first one. The complete test is as follows. In output, there were some HF 504(Bad gateway error) and Llama 402 error (my token limit is reached ).

API_KEY = "Your API KEY";
PROMT = "Describe the contents of this receipt image";
async function testHFandVision(rowIndex) {
  try {
    const hfResponse = await fetch(
      `https://datasets-server.huggingface.co/rows?dataset=moyrsd%2Frocketchat_receipts_dataset&config=default&split=train&offset=${rowIndex}&length=1`
    );

    if (!hfResponse.ok) {
      console.log(`Row ${rowIndex}: HF failed - HTTP ${hfResponse.status}`);
      return;
    }

    const hfData = await hfResponse.json();
    const rowData = hfData.rows[0]?.row;

    if (!rowData?.image_base64) {
      console.log(`Row ${rowIndex}: Missing image data`);
      return;
    }

    const visionRequest = {
      model: "meta-llama/Llama-3.2-11B-Vision-Instruct",
      messages: [
        {
          role: "user",
          content: [
            {
              type: "image_url",
              image_url: {
                url: `data:image/jpeg;base64,${rowData.image_base64}`,
              },
            },
            {
              type: "text",
              text: PROMT,
            },
          ],
        },
      ],
    };

    const visionResponse = await fetch(
      "https://api.deepinfra.com/v1/openai/chat/completions",
      {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Authorization: `Bearer ${API_KEY}`,
        },
        body: JSON.stringify(visionRequest),
      }
    );

    if (!visionResponse.ok) {
      console.log(
        `Row ${rowIndex}: Vision failed - HTTP ${visionResponse.status}`
      );
      return;
    }

    const visionData = await visionResponse.json();
    console.log(
      `Row ${rowIndex}: Both APIs succeeded - Vision response: ${visionData.choices[0].message.content.substring(
        0,
        50
      )}...`
    );
  } catch (error) {
    console.error(`Row ${rowIndex}: Error - ${error.message}`);
  }
}


const TOTAL_ROWS = 40;
for (let i = 0; i < TOTAL_ROWS; i++) {
  testHFandVision(i);
}

Output

# node test.js
st3.js
Row 22: HF failed - HTTP 502
Row 33: HF failed - HTTP 502
Row 14: HF failed - HTTP 502
Row 1: HF failed - HTTP 502
Row 16: Both APIs succeeded - Vision response: This receipt, from Ted Heng Stationery & Books, di...
Row 12: Both APIs succeeded - Vision response: The receipt is from Old Town White Coffee in Sri R...
Row 13: Both APIs succeeded - Vision response: The receipt is from Uroko Japanese Cuisine Sdn Bhd...
Row 38: Both APIs succeeded - Vision response: **Receipt Details**

* **Business Name:** PASARAYA...
Row 21: Both APIs succeeded - Vision response: The receipt in the image is from Pasar Mini Jin Se...
Row 24: Both APIs succeeded - Vision response: The image depicts a receipt from Bar Wang Rice, a ...
Row 31: Both APIs succeeded - Vision response: The receipt is for a purchase from Bens Independen...
Row 36: Both APIs succeeded - Vision response: The receipt image displays a bill from Restaurant ...
Row 28: Both APIs succeeded - Vision response: The receipt is for one item, SR 10510000061.5 Vege...
Row 10: Both APIs succeeded - Vision response: This parking receipt is from Secure Parking Corpor...
Row 6: Both APIs succeeded - Vision response: The receipt shows a total of RM 7.95, with a GST r...
Row 25: Both APIs succeeded - Vision response: This receipt is from Yong Tat Hardware Trading, lo...
Row 32: Both APIs succeeded - Vision response: This receipt is for a purchase made at the 99 Spee...
Row 23: Both APIs succeeded - Vision response: This receipt is from Tony Roma\'s restaurant in Joh...
Row 7: Both APIs succeeded - Vision response: **Receipt Details**

* **Business Name:** SWC Ente...
Row 29: Both APIs succeeded - Vision response: The receipt in this image shows a purchase from a ...
Row 27: Both APIs succeeded - Vision response: **Receipt Details**

* **Company Name:** Unihakka ...
Row 17: Both APIs succeeded - Vision response: This receipt is from SWC Enterprise Sdn Bhd, locat...
Row 5: Both APIs succeeded - Vision response: **Receipt Details**

* **Company Name:** Cross Cha...
Row 15: Both APIs succeeded - Vision response: **Receipt Details**

* **Store Name:** Teo Heng St...
Row 26: Both APIs succeeded - Vision response: The receipt is for a purchase made at Triple Six P...
Row 35: Both APIs succeeded - Vision response: **Receipt Details**

* **Store Name:** Pasaraya Bo...
Row 2: Both APIs succeeded - Vision response: The receipt is from Petron BKT Lanjan SB, located ...
Row 11: Both APIs succeeded - Vision response: The receipt is from Lightroom Gallery Sdn Bhd and ...
Row 4: Both APIs succeeded - Vision response: **Receipt Details**

* **Company Name:** Sin Lianh...
Row 19: Both APIs succeeded - Vision response: **Receipt Details**

* **Invoice Number:** DR18030...
Row 37: Both APIs succeeded - Vision response: The receipt is from \"PASARAYA BORONG PINTAR SDN BH...
Row 8: Both APIs succeeded - Vision response: The receipt features a light gray background with ...
Row 39: Both APIs succeeded - Vision response: The receipt is from the Popular Book Company (M) S...
Row 18: Both APIs succeeded - Vision response: **Receipt Details**

* **Business Name:** F&P Phar...
Row 20: Vision failed - HTTP 402
Row 3: Both APIs succeeded - Vision response: **Receipt Details**

* **Store Information:**
        + N...
Row 0: Vision failed - HTTP 402
Row 9: Vision failed - HTTP 402
Row 30: Both APIs succeeded - Vision response: The receipt is for Ben\'s Independent Grocer (B.I.G...
Row 34: Vision failed - HTTP 402

[Sing-Li]

@moyrsd Right! Basically it didn't work.

Don't Google or Kaggle your dataset. It is useless for this purpose.

Use receipts collected from our community and then prep the data so the base64 is SMALL.

[moyrsd]

Ok I will remove images with larger base64 and add new images sourced from community.

[moyrsd]

I have updated the dataset with images sourced from internet, the lenght is less than 113k. I have tested all of them.

const API_KEY = "Your API KEY";
const PROMPT = "Describe the contents of this receipt image";

async function processAllRows() {
  try {
    const hfResponse = await fetch(
      "https://datasets-server.huggingface.co/rows?" +
        new URLSearchParams({
          dataset: "moyrsd/rocketchat_receipts_dataset",
          config: "default",
          split: "train",
          offset: 0,
          length: 20,
        }),
      {
        headers: {
          Accept: "application/json",
          "User-Agent": "MyApp/1.0",
        },
      }
    );

    if (!hfResponse.ok) {
      console.log(`HF failed - HTTP ${hfResponse.status}`);
      return;
    } else {
      console.log(`HF successfully retrived - HTTP`);
    }

    const hfData = await hfResponse.json();
    const allRows = hfData.rows;

    for (const [index, row] of allRows.entries()) {
      try {
        const rowData = row?.row;
        if (!rowData?.image_base64) {
          console.log(`Row ${index}: Missing image data`);
          continue;
        }

        const visionResponse = await fetch(
          "https://api.deepinfra.com/v1/openai/chat/completions",
          {
            method: "POST",
            headers: {
              "Content-Type": "application/json",
              Authorization: `Bearer ${API_KEY}`,
              Accept: "application/json",
            },
            body: JSON.stringify({
              model: "meta-llama/Llama-3.2-11B-Vision-Instruct",
              messages: [
                {
                  role: "user",
                  content: [
                    {
                      type: "image_url",
                      image_url: {
                        url: `data:image/jpeg;base64,${rowData.image_base64}`,
                      },
                    },
                    { type: "text", text: PROMPT },
                  ],
                },
              ],
            }),
          }
        );

        if (!visionResponse.ok) {
          console.log(
            `Row ${index}: Vision failed - HTTP ${visionResponse.status}`
          );
          continue;
        }

        const visionData = await visionResponse.json();
        console.log(
          `Row ${index}: Success - ${visionData.choices[0].message.content.substring(
            0,
            50
          )}...`
        );
      } catch (error) {
        console.error(`Row ${index}: Error - ${error.message}`);
      }
    }
  } catch (error) {
    console.error(`Global Error: ${error.message}`);
  }
}

processAllRows();

Output

# node test3.js
HF successfully retrived - HTTP
Row 0: Success - **Receipt Details**

* **Company Name:** Western E...
Row 1: Success - The receipt image displays a receipt from Walmart ...
Row 2: Success - **Receipt Details**

* **Business Name:** Western ...
Row 3: Success - The receipt is from a restaurant named "Restoran H...
Row 4: Success - The receipt is for a purchase of one PC of KF Mode...
Row 5: Success - This image depicts a receipt for a purchase from I...
Row 6: Success - The receipt is from "tan woon yann" and lists the ...
Row 7: Success - The receipt provided is for a single item of a "EL...
Row 8: Success - The receipt is for a purchase made at a store in M...
Row 9: Success - The receipt is for "tan chay yee" and is from "ABC...
Row 10: Success - The receipt in the image is from S.H.H. Motor, a c...
Row 11: Success - This receipt, dated 18/03/18, is from Shell ISNI P...
Row 12: Success - This receipt image is from GL Handicraft & Tailori...
Row 13: Success - This receipt is from Lian Hing Stationery Sdn Bhd,...
Row 14: Success - **Receipt Details**

* **Company Name:** Western E...
Row 15: Success - The receipt is for a purchase from Western Eastern...
Row 16: Success - The receipt displays the following information:

*...
Row 17: Success - The receipt is from Amtech Electrical Supplies, a ...
Row 18: Success - The receipt is for GL Handicraft & Tailoring, loca...
Row 19: Success - The receipt is for Lijan Hing Stationery Sdn Bhd, ...

[Sing-Li]

Good work! @moyrsd For our "prompt-library unit test", we will need only 10 rows.

What we'll need to do is run the same prompt over the 10 rows (each representing a different category / layout / region of receipts) and IF WE GET 7 out of 10 or higher - we consider that test case to be successful.

Please remove all rows that are not sourced from our community and leave 10 rows all community receipts.

Then please come over to the PR and collaborate to get this test in.

#13