Investigate use of binary protocol instead of JSON for data transfer

[heartsucker]

A disadvantage of JSON is that in order to verify the metadata, it has to be parsed first. It would be better to use a binary protocol that lets us treat the signed field as raw bytes until after we verify it. Then, do a second round of parsing on the trusted data. For example:

message SignedRootMetadata {
  repeated Signature signature = 1;
  bytes signed = 2;   
}

message Signature {
    bytes keyid = 1;
    bytes sig = 2;
}

Followed by

let signed_root = parse_signed_root(bytes_from_network)?;
let root = if verify(signed_root) {
    parse_root(signed_root.signed)?
} else {
    return Err("oh dear")
}
// do stuff with root

Protobufs were used in the example here, but they may not be deterministic, so something like ASN.1 / DER would need to be used.

This is motivated by my dislike of parsers being allowed to work on any more untrusted data than necessary.

[tarcieri]

For what it's worth, I've been working on a "Merkelized Protobufs" format which can be bidirectionally transcoded to (T)JSON while preserving the same content hashes:

https://github.com/zcred/zser

DER would certainly be more in line with IETF SOP.

[heartsucker]

Interesting stuff. I think this ticket is a long ways off since making everything work in JSON is a pain enough as it is.

[heartsucker]

Depends on #73

[heartsucker]

I'm not implementing anything yet, but I had to do a huge rewrite because the current API backed me into a corner. The rewrite includes support for non-JSON formats, and implementation should be trivial now that Serialize and Deserialize are correctly implemented. Work tracked in rewrite branch.