Converting MuPDF coordinates to pdf-lib/pdf.js coordinates

[lbirkert]

Since the update fixed the memory issue I've decided to hop back to this library for analysis (not rendering yet, since I'm still having issues with rendering speed and also with bold fonts, but I might come back to those).

Now I am facing another issue with MuPDF. It uses a different coordinate system compared to pdf-lib/pdf.js. At first I thought it might just be the Y-Axis, which is flipped (for MuPDF, top = 0, bottom = big value <-> pdf.js) but I tried converting the bounding box rects I get from toStructuredText (using getWords) by using this formula: pdfLibY = pageHeight - muPdfY, but this yields a weird offset, which is not hardcodeable, since it seems to depend on the font-size (and maybe the font itself as well?).

for the sake of completeness, here is the function I use for converting the rects (note that I also have to flip rect[1] and rect[3], since the maximum value flips as well.)

const viewport = page.getBounds();
const convrect = (rect: Rect) => {
    [rect[3], rect[1]] = [
        viewport[3] - rect[1],
        viewport[3] - rect[3]
    ];
    return rect;
};

// convrect([0, 0, 0, 100]) yields [0, viewport[3] - 100, 0, viewport[3]]

[ccxvii]

I don't know exactly which coordinate space pdf.js uses, but if it's the same as the PDF userspace you can convert from the mupdf unified coordinate space (top left at 0,0, +y down, 72dpi) you can use the transform matrix from PDFPage.getTransform()

[MrVoshel]

Maybe this is related to my issue. I cross checked the values I get in the browser with a program called Sumatra PDF (also uses top left as the origin) just to make sure I wasn't getting the wrong values.

When I try to insert text a slight offset is also added, not sure if I'm using it wrong, since I'm using the top left of the text I'm targeting as the origin (and I can't find if the docs mention that's the case for the text coordinates too).

2025-03-14.00-22-44-00.00.02.083-00.00.37.500.mp4

[lbirkert]

I don't know exactly which coordinate space pdf.js uses, but if it's the same as the PDF userspace you can convert from the mupdf unified coordinate space (top left at 0,0, +y down, 72dpi) you can use the transform matrix from PDFPage.getTransform()

The problem is not, that there's no such transformation. I could hardcode an offset to either fix the chords being overwritten OR the title. But not both. I believe that .toStructuredText yields invalid coordinates, as the offset seems to depend on text size/and or style.

In the picture I have sent you can see thise quite clearly. I am redacting text by drawing a white rectangle over the bounding rects I get from getWords.

For the title this bounding rect is ABOVE what it should be. I'd have to use an offset < 0 in the transformation to shift these coordinates down in pdf-lib space.

For the chords this bounding rect is BELOW what it should be. I'd have to use an offset > 0 in the transformation to shift these coordinates up in the pdf-lib space.

As I don't have a uniform offset, this transformation is non-trivialy doable by just looking at the X and Y coordinates, as it depends on font size/style as well.

I hope I could clarify some things.

[lbirkert]

Maybe this is related to my issue. I cross checked the values I get in the browser with a program called Sumatra PDF (also uses top left as the origin) just to make sure I wasn't getting the wrong values.

When I try to insert text a slight offset is also added, not sure if I'm using it wrong, since I'm using the top left of the text I'm targeting as the origin (and I can't find if the docs mention that's the case for the text coordinates too).
2025-03-14.00-22-44-00.00.02.083-00.00.37.500.mp4

You're trying to use the inserText method in MuPDF I am guessing, and you're using mouse coordinates and try to translate these to MuPDF space?

This could be a related issue, as maybe inserText compensates for this offset previously created by toStructuredText. This would explain the additional compensation offset you are facing.

@ccxvii @jamie-lemon do you guys know whether it is possible to draw a plain rectangle using MuPDF (with no redaction text detection and whatnot. I know that blackbox-annotations exist, do they use the exact rectangle I give them, or do they somehow find the closest word and map to this?) - We could debug this using the following method then:

1. getWords from a PDF with multiple font sizes and styles
2. draw rectangles using MuPDF at the returned bounding box coordinates (no coordinate space transformation needed)
3. See, whether the word's are perfectly covered (apart from ascenders/descenders) and therefore conclude whether the coordinates returned by getWords/toStructuredText are accurate

[jamie-lemon]

@lbirkert

@ccxvii @jamie-lemon do you guys know whether it is possible to draw a plain rectangle using MuPDF (with no redaction text detection and whatnot. I know that blackbox-annotations exist, do they use the exact rectangle I give them, or do they somehow find the closest word and map to this?) - We could debug this using the following method then

I think we could just draw "square" annotations with e.g.

let annotation = page.createAnnotation("Square")
annotation.setRect([0, 0, 300, 300])
annotation.setColor([1, 0, 1])
annotation.setInteriorColor([0.5, 0, 0])
annotation.setOpacity(0.75)

( I set the border to magenta just so it would stand out more, and the opacity to 75% so that you could still see content underneath the annotation )

[lbirkert]

@lbirkert

@ccxvii @jamie-lemon do you guys know whether it is possible to draw a plain rectangle using MuPDF (with no redaction text detection and whatnot. I know that blackbox-annotations exist, do they use the exact rectangle I give them, or do they somehow find the closest word and map to this?) - We could debug this using the following method then

I think we could just draw "square" annotations with e.g.

let annotation = page.createAnnotation("Square")
annotation.setRect([0, 0, 300, 300])
annotation.setColor([1, 0, 1])
annotation.setInteriorColor([0.5, 0, 0])
annotation.setOpacity(0.75)

( I set the border to magenta just so it would stand out more, and the opacity to 75% so that you could still see content underneath the annotation )

this is the result directly using the coordinates without conversion. There is definetly something wrong with the coordinates returned by toStructuredText().

[jamie-lemon]

@ccxvii What do you think is going on here? Something to do with the page boxes maybe?

[ccxvii]

Please share the file for investigation. There are many possible reasons the bounding boxes can be off.

[lbirkert]

Please share the file for investigation. There are many possible reasons the bounding boxes can be off.

I've sent the file via mail at tor@ccxvii.net as they are copyright protected and I am not allowed to share them publicly.

[ccxvii]

The file looks fine, and the coordinates also look fine when using this test script, so I suspect the error must be somewhere in how you use the coordinates:

import * as fs from "fs"
import * as mupdf from "mupdf"
var doc = mupdf.Document.openDocument(fs.readFileSync("oceans.pdf"), "application/pdf")
var page = doc.loadPage(0)
page.toStructuredText().walk({
        beginLine(bbox, wmode, dir) {
                var annot = page.createAnnotation("Square")
                annot.setRect(bbox)
        }
})
page.update()
fs.writeFileSync("out.pdf", doc.saveToBuffer().asUint8Array())

[lbirkert]

The file looks fine, and the coordinates also look fine when using this test script, so I suspect the error must be somewhere in how you use the coordinates:

import * as fs from "fs"
import * as mupdf from "mupdf"
var doc = mupdf.Document.openDocument(fs.readFileSync("oceans.pdf"), "application/pdf")
var page = doc.loadPage(0)
page.toStructuredText().walk({
        beginLine(bbox, wmode, dir) {
                var annot = page.createAnnotation("Square")
                annot.setRect(bbox)
        }
})
page.update()
fs.writeFileSync("out.pdf", doc.saveToBuffer().asUint8Array())

getWords uses the glyph rect received by onChar (I think that was its name). I'll try to debug some more at home to get a minimal reproducible example working.

[ccxvii]

Okay, I have tried the example file with the experimental mupdf/mupdfjs API as well (using getWords) and it gives reasonable rectangles for me:

import * as fs from "fs"
import * as mupdfjs from "mupdf/mupdfjs"
var doc = mupdfjs.PDFDocument.openDocument(fs.readFileSync("oceans.pdf"), "application/pdf")
var page = new mupdfjs.PDFPage(doc, 0)
for (let word of page.getWords()) {
        page.addAnnotation("Square", {
                x: word.rect[0],
                y: word.rect[1],
                width: word.rect[2] - word.rect[0],
                height: word.rect[3] - word.rect[1]
        })
}
page.update()
fs.writeFileSync("out.pdf", doc.saveToBuffer().asUint8Array())

[lbirkert]

Okay, I have tried the example file with the experimental mupdf/mupdfjs API as well (using getWords) and it gives reasonable rectangles for me:

import * as fs from "fs"
import * as mupdfjs from "mupdf/mupdfjs"
var doc = mupdfjs.PDFDocument.openDocument(fs.readFileSync("oceans.pdf"), "application/pdf")
var page = new mupdfjs.PDFPage(doc, 0)
for (let word of page.getWords()) {
        page.addAnnotation("Square", {
                x: word.rect[0],
                y: word.rect[1],
                width: word.rect[2] - word.rect[0],
                height: word.rect[3] - word.rect[1]
        })
}
page.update()
fs.writeFileSync("out.pdf", doc.saveToBuffer().asUint8Array())

Then it must be either the web PDF viewer or the web version of MuPDF.js

[jamie-lemon]

@lbirkert What do you mean by "the web version of MuPDF.js" ?