Protocol upgrade for new features

[PeterSurda]

Rationale:

The protocol needs to be upgraded to allow for things like forward secrecy and anonymity,, compressed pubkeys. Issues mentioning forward secrecy: #454, #833, #1015 forward anonymity #563.

Furthermore, the current use of pubkey/getpubkey objects has some drawbacks, as pointed out by @gmaxwell (see thread e.g. here: https://www.reddit.com/r/bitmessage/comments/g2u4a5/how_bitmessage_keeps_your_anonymity/fnr5zry/ ). He proposes including the public key inside the address.

Summary of pubkey/getpubkey mechanism drawbacks:

• using pubkey/getpubkey prevents (or at least make difficult and non-standard) a fully passive recipient
• the sender needs to wait until they receive a pubkey, which often causes confusion for senders unfamiliar with the inner workings of the bitmessage
• the sender may end up unnecessarily creating a getpubkey before synced. This wastes PoW
• upon creation of a new address, the creator needs to do PoW, which may be a waste and may expose the creator
• more resource usage is worse for mobile devices

Summary of advantages of pubkey/getpubkey mechanism:

• allows to change some metadata (behavior bitfield and difficulty) without changing address. However, reacting to changed pubkey metadata currently isn't implemented very well and at least would need some design work on fixing
• easier to publish prekeys (e.g. for PFS). However, this isn't implemented yet anyway
• address is shorter. With some optimisation, the new format for an identifier may be cut down to max. 39 bytes, while the old one is 26.
• allows to use a separate key for signing and encryption. However this probably isn't necessary

There are some other minor issues, e.g. there is currently no signalling of support of extended encoding in the protocol, the hashing algorithm used for ECDSA signatures (SHA1 vs SHA256). The behavior bitfield is in my opinion unnecessarily long (4 bytes).

Proposal

I think it makes sense to modify the protocol (new message and broadcast versions) to:

1. obsolete old standards, like SHA1, old encodings (i.e. simple and trivial, I think ignore encoding can be still kept optionally for future use in onion routing).
2. fix issues like compressed pubkeys (pyelliptic now supports them)
3. remove pubkey/getpubkey
4. allow for easy adding of new features (e.g. there can be a section reserved for PFS which can be ignored until PFS is actually implemented).

Pubkeys can still be added in the future, for example as a helper for PFS. A node could refresh pubkey objects if the collection of prekeys (if we're using something like X3DH) or other data that is used for deriving session keys expires or are used up. Without PFS they probably aren't necessary, and even with PFS, the mechanism could allow reuse of long term key for the initial contact only.

PFS will most likely be implemented later, as the exact method has to be chosen, and also a suitable library, as I'm not sure OpenSSL is sufficient for any existing PFS mechanism.

A more detailed specification will follow later. I'm collecting feedback now.

[PeterSurda]

@mirrorwish @DanielKrawisz @Erkan-Yilmaz @AyrA @Atheros1

[zciendor]

When considering a protocol update, you may want to have a look at libp2p and their protocol concepts for versioning, negotiation and addressing.

there is currently no signalling of support of extended encoding in the protocol

You may want to look at libp2p's Peer-Ids and multibase prefixes

1. obsolete old standards, like SHA1, old encodings (i.e. simple and trivial, I think ignore encoding can be still kept optionally for future use in onion routing).

You may want to have a look at multihash encoding

2. fix issues like compressed pubkeys (pyelliptic now supports them)

pyelliptic is officially deprecated: yann2192/pyelliptic#50 (comment)

[PeterSurda]

When considering a protocol update, you may want to have a look at libp2p and their protocol concepts for versioning, negotiation and addressing.

Unforunately this would be just a change for no good reason and break backwards compatibility as far as I can see. I did a research a couple of years ago what to replace the threaded network model with and decided to go with asyncore. Bitmesasge protocol has versioning as well as negotiation of features during the handshake. The concept of addressing as it's described in the link doesn't seem to make any sense for bitmessage.

You need to consider also that the priority of Bitmesasge is anonymity, and some concepts used by other protocols need to be avoided for this reason.

You may want to look at libp2p's Peer-Ids and multibase prefixes.

Peer ID breaks anonymity. Multibase prefix wouldn't affect extended encoding because it happens at a different layer.

You may want to have a look at multihash encoding.

A couple of years ago I analysed available encoding libraries, collected feedback and decided to go with msgpack. I see no reason to change it now again.

Again, the issue with extended encoding signalling happens on another layer (recipient rather than node), plus at the time when you're sending the message, the easier options won't always provide you with the signal at the time of sending. I think that only a version bump for message/broadcast, or if bitfields are moved to the address would solve this correctly. An address version needs the communicating parties to notify of the change and there is currently no provision in the protocol to do that, so using it just for the signalling of extended encoding I find too rash.

pyelliptic is officially deprecated: yann2192/pyelliptic#50 (comment)

PyBitmessage uses its own fork which has updates for different versions of OpenSSL, blind signatures and so on.