Add a "no UTF-8 stripping URL" option

[marcanuy]

I am working with Chinese content (using UTF-8), while most of the time it generates the right url, sometimes it strips certain Chinese characters from URL.

Some examples of these characters are:

• 〇
• ○
• 〡
• 〤
• 〢
• ⺮
• 〣

When generating a page for each character, i.e.: example.com/post/〇 it generates empty paths example.com/post// .

Steps

To reproduce the bug, add

slug: "foo〇○〡〤〢⺮〣21三bar"

in the front matter of any page Hugo will generate the following stripped path:

http://localhost:1313/post/foo21三bar/` 

removing 〇○〡〤〢⺮〣.

*Tested with latest Hugo release: Hugo Static Site Generator v0.30.2 linux/amd64 BuildDate: 2017-10-19T08:34:27-03:00, SO: 4.10.0-37-generic #41-Ubuntu SMP Fri Oct 6 20:20:37 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Ubuntu 17.04 *

(x-post: stackoverflow.com, forum)

[bep]

@marcanuy I'm reopening this. I quoted you a part of the comment describing the current behaviour. I'm sure the original motivation for this "unicode sanitize" was good and founded in file system support or something (that function precedes my time on Hugo).

So, we cannot just change that behaviour, that would break lots of sites. But we could consider adding some "no URL sanitize whatsoever" option.

[marcanuy]

Great, a configuration flag to avoid it would be really helpful, especially for SEO purposes.

[biodranik]

Am I right that this issue is about the same IRI/IRL (International Resource Identifier/Locator) support as this forum topic https://discourse.gohugo.io/t/bug-feature-hugo-wrong-support-non-acsii-symbols-in-url/8375 and closed issue #3039?

It would be great to avoid converting valid UTF-8 IRI into percent-encoded URIs at least for two reasons:

1. User-friendly links in non-ASCII, non-English or multilingual sites (though it also depends on a browser).
2. Readable diffs of generated HTML files in git commits and easier code changes review/support/debug. It's impossible to visually decode and understand links in HTML like this: href="/%D0%BA%D0%BE%D0%BD%D1%82%D0%B0%D0%BA%D1%82%D1%8B/">

And probably better SEO too.

A simple option like EnableIRI (false by default) would be great!

[rinetd]

期待　unicode sanitize
or add function urldecode transform %e5%a5%bd to 好