`autolink_bare_uris` when converting from a man page

[acuteenvy]

Describe your proposed improvement and the problem it solves.
Man pages sometimes contain links to websites. When the man page is converted to html, pandoc does not create hyperlinks in the resulting html.

input.1:

.TH "TEST" "1" "2025-01-25"  "test" "test manual"
.SH HEADING
Pandoc's website: https://pandoc.org

$ pandoc --to html input.1
<h1>HEADING</h1>
<p>Pandoc's website: https://pandoc.org</p>

Would it be possible for the autolink_bare_uris extension to work with the man reader?

Describe alternatives you've considered.
Workaround: first convert man to markdown and then markdown to html.

$ pandoc --to markdown input.1 | pandoc --from markdown+autolink_bare_uris --to html
<h1 id="heading">HEADING</h1>
<p>Pandoc's website: <a href="https://pandoc.org"
class="uri">https://pandoc.org</a></p>

[silby]

I've been meditating about this off and on and I think that there are philosophical issues that cut against using the autolink_bare_uris extension with man in this way.

In the case of Markdown, the idea of an "extension" to the language arises pretty naturally because a lot of things that Pandoc supports are extensions to Markdown.pl and commonmark. Users might need to enable or disable various extensions to treat inputs in the way they are meant to be treated by their original authors, who maybe wrote them for different software supporting an exotic batch of markdown-features. Among these extensions is autolink_bare_uris, which indeed is not a feature of Markdown.pl, and most importantly is on by default for Github-flavored markdown.

Outside of the case of Markdown, don't seem to be crystal clear lines about what functionality belongs in an extension, as opposed to a regular reader or writer option. And arguably it's less desirable to extend input formats that, unlike markdown, have either documented standards or canonical implementations. In the status quo, the only input formats supporting the autolink_bare_uris extension are Markdown variants. (And the Markdown cells in ipynb Jupyter notebooks.)

I think the relevant contrast is that the man language as documented and implemented elsewhere never treats a bare URI-looking thing as a link. A GNU extension to heirloom man supports link markup (cf. groff_man(7) and mandoc's man(7).) So there's not some important implementation of man that Pandoc would become more compatible with by supporting autolink_bare_uris.

The functionality you want could be implemented as a Lua filter, using topdown traversal to avoid filtering URIs that are marked up as links.

[acuteenvy]

the man language as documented and implemented elsewhere never treats a bare URI-looking thing as a link.

man itself indeed doesn't, but terminal emulators do, and man pages are most often viewed in a terminal, where bare links are clickable. And all the online man page websites also automatically treat bare URIs as links, so I thought that would be desirable, even though the man specification doesn't say anything about it.

Anyway, if this is not something Pandoc should support in your opinion, could you please share how exactly can this be implemented as a Lua filter?

[silby]

And all the online man page websites also automatically treat bare URIs as links

Can you link to an example where the source manual page isn't marked up with .UR/.UE but the renderer is autolinking?

Note that neither groff(1) nor mandoc(1) format your example in the issue with active links when using -Thtml:

$ groff -man -Thtml input.1 | grep -B 1 pandoc.org
<p style="margin-left:9%; margin-top: 1em">Pandoc&rsquo;s
website: https://pandoc.org</p>
$ mandoc -man -Thtml input.1 | grep pandoc.org
<p class="Pp">Pandoc's website: https://pandoc.org</p>

A lua filter could look something like this:

traverse = 'topdown'

function Link(el)
    -- avoid creating nested links
    return el, false
end

function is_uri(text)
    -- TODO exercise for the reader
end

function Str(el)
    if is_uri(el.text) then
        return pandoc.Link(el.text, el.text)
    else
        return el
    end
end

[acuteenvy]

Can you link to an example where the source manual page isn't marked up with .UR/.UE but the renderer is autolinking?

Sure:
https://git-scm.com/docs/git
or
https://manned.org/git

https://git-scm.com/docs (at the end of DESCRIPTION) is hyperlinked, even though groff and mandoc don't do it:

$ zcat /usr/share/man/man1/git.1.gz | groff -man -Thtml | grep git-scm
<b><font color="#0000FF">https://git-scm.com/docs</font></b><font color="#000000">.</font></p>

$ zcat /usr/share/man/man1/git.1.gz | mandoc -man -Thtml | grep git-scm
<b>https://git-scm.com/docs</b>.</p>

[silby]

In the case of git-scm.org, from what I can tell the HTML documentation doesn't go through groff or mandoc but is converted to docbook by asciidoctor and then to HTML using some XSLT package. manned.org is more interesting; it uses mandoc to generate the ASCII backspace encoded (i.e. terminal) rendering of the manual and then heuristically tries to identify links, see here.

It does this badly though, compare the rendering of curl(1) on manned.org to man.archlinux.org. In the source manual, those examples are marked up using (non-semantic) roff .nf requests to enter "no-fill"/literal text mode. I think archmanweb's postprocessing does it slightly better, tho it's still postprocessing mandoc's HTML output with regex, which as we know is a sin.

Anyway the details of how those websites do this don't imply that Pandoc should never do this but I don't think that it makes sense to add the feature specifically to the man reader. There's other non-markdown formats where bare URLs aren't autolinked, such as Textile, djot, tikiwiki, which could reasonably also get this feature request, and autolinking would have to be incorporated into and maintained in each of those parsers in distinct ways. I think the correct place for this feature in Pandoc's model is as a filter.

Pandoc doesn't currently have a notion of a "built-in filter", nor does it ship with any Lua filters by default. If it did, it could perhaps ship an autolink.lua that works with any input format. But I assume if jgm wanted to maintain a curated collection of Lua filters that ship with Pandoc he would've started by now; tarleb and others have https://github.com/pandoc-ext as an auxiliary project.

Whether we added autolinks to man alone or as some sort of blessed filter, we'd have to decide which autolinks. Consider this example input:

$ cat ex
https://example.org

telnet:127.0.0.1

fake:lol
$ pandoc -r rst -w html ex
<p><a href="https://example.org">https://example.org</a></p>
<p><a href="telnet:10.0.0.1">telnet:10.0.0.1</a></p>
<p>fake:lol</p>
$ pandoc -r gfm -w html ex
<p><a href="https://example.org">https://example.org</a></p>
<p>telnet:10.0.0.1</p>
<p>fake:lol</p>

(cf. the GFM spec and the RST reference)

Should one or the other of these rules apply to man input? Some third option we make up? I don't think either choice is self-evident especially in the absence of a specification to conform to. And whichever choice would be the wrong choice for some user, and we'd either have to add a knob to allow the user to choose or those users would be back to having to provide their own filter.

My personal point of view (as a person who has used Pandoc a lot and contributed a few things in the past year but who is not jgm) is that a Pandoc reader should create the best possible representation of the input document in Pandoc's AST, in conformance with the specification and reference implementations for the input format, and stop there. Novel extensions to the semantics of input formats that aren't Markdown should generally stay out of scope, even in the case of extensions with a certain straightforward appeal like this one. The scope is plenty big as it is!

[acuteenvy]

Should one or the other of these rules apply to man input? Some third option we make up? I don't think either choice is self-evident especially in the absence of a specification to conform to. And whichever choice would be the wrong choice for some user, and we'd either have to add a knob to allow the user to choose or those users would be back to having to provide their own filter.

That makes sense, thanks for explaining. Some users might indeed want different autolinking behavior.

The Lua filter you posted doesn't work though, it gets stuck in an infinite loop when a link is encountered. I had to change return pandoc.Link(el.text, el.text) to return pandoc.Link(el.text, el.text), false or set traverse = "typewise" to make it work.

[silby]

I should’ve mentioned it was written but not tested lol. Glad you got it to work.