Multi WFE nonce service design planning

Currently each WFE has it's own internal nonce service with which it verifies requests from clients. Because it is internal to the WFE whenever we run more than one instance clients may present nonces that they received from one WFE to another which has no idea that the nonce is valid, causing errors.

Currently we run two WFEs, one in each DC, but we would like to be able to run multiple instances in order to spread load out. The best option is to split the nonce service out from the WFE (or at least the nonce pool) so each instance knows about all current nonces.

For the initial solution to this problem we shouldn't worry about solving this problem across DCs since we currently implement sticky sessions per IP.

In order to implement a solution we'll need some fast data-store that can hold the nonce pool and counters (and store state across Boulder restarts) as well as a small standalone service that can handle new nonce and validity check requests. The small service is required for the generation/validation since some state is required for both these operations that cannot be shared across multiple WFEs.

The simplest approach would be to implement a gRPC server which exposes the methods NewNonce and ValidateNonce and talks to the fast data-store itself instead of leaving any of this logic in the WFEs.

For the data-store the best option is probably Redis, which is fast enough for this purpose and provides a few persistence modes. It's replication mode may allow us to also use it across multiple DCs, although this would require a more flexible design of the nonce services.