Clarify the use of UTF-8 characters in bucket / key names [JIRA: PMT-86]

[kesslerm]

Due to the difference the protocol buffer and HTTP protocol handlers deal with characters that are not allowed unencoded in HTTP URLs, one needs to be careful when using e.g. UTF-8 for bucket / key names.

[Basho-JIRA]

Charlie suggests asking John Daily because he knows the most about language encoding.

[posted via JIRA by Lauren Rother]

[Basho-JIRA]

Discussed with John this morning. He will look into it.

[posted via JIRA by Lauren Rother]

[macintux]

So.

For the most part, this is an annoying but workable non-issue for anyone who understands character encoding. (Which, of course, is virtually nobody, but all you really have to understand is "use UTF-8" and hand-wave your way from there.)

• Even though we typically think of the names of bucket types, buckets and keys as ASCII strings, they are actually binary data
• The names could be strings encoded as a Unicode standard such as UTF-8, but do not have to be (but see the note below on riak-admin)
• Protocol buffers allow binaries to be encoded natively, but each client library will handle it differently depending on the programming language facilities
• The HTTP interface allows bucket type, bucket & key names using hex encoding. Presumably PHP, the only user-facing client library that uses the HTTP interface, has some mechanism to make this easy.
• The riak-admin command converts any string passed on the command line into UTF-8. riak-admin is the sole interface for managing bucket types in KV, so this is a particularly important caveat.

[macintux]

Worth noting: the Java client uses UTF-16 because Java, but the data is still interpreted as UTF-8 by Riak. I'd have to dig to figure out which layer is making the assumption that UTF-8 is the standard encoding.