Question about precision?

[ThanhThuy2908]

I am very interested in your project. But I have a question. Why do you choose the precision = 100? And the reason why you get conv_bias * (precision**2) while dense1_bias * (precision **5) and dense2_bias * (precision **11)? I do not find out the answer on your paper.
I am looking forward to hearing from you soon.
Thank you very much and have a nice weekend.

[armandofln]

Sorry I missed your question all this time, I didn't notice. Just a quick disclamer: this is an old project, and there's a lot of room for improvement, especially on the computational part of the neural network.

To answer your question, precision is an arbitrary parameter. The higher the parameter is, the lower the loss of accuracy of the neural network, when converted to work with only integer numbers. But on the other hand it can't be too big. As I wrote in the paper in section 2.2, it is necessary that during the whole inference no computation exceeds t/2.

Now that I think about it, maybe this constraint is a bit too restrictive. It is possible that in reality it's enough to require that during the encrypted inference phase q/2 is not exceeded, and only after decryption took place is necessary that no number must exceed t/2. However, this is just a speculation that came to my mind now, and I'm not sure about it, it should be tested.

As for the choice of precision = 100, it was chosen only for the simple fact that in this way the first two decimal places after the decimal point are maintained. For example, if we have a value equal to x = 0.9725985, using a precision equal to 100 and rounding to the closest integer we get 97, which, if divided again by the precision, gives us a value equal to 0.97, which is very close to the original value, with an error equal to only 0.0025985.

But all this is not strictly necessary. Any number large enough to guarantee good accuracy, but small enough not to exceed t/2 during the inference phase, is enough.

[armandofln]

As for the second part of your question, you're right to ask. This is actually something I didn't explain in the paper. Basically, while running the inference phase, the numbers keep accumulating multiple precision factors, due to multiplication. Suppose, for example, we have three values: an input, a weight, and a bias.

x = 3, w = 2, b = 4

If we want to calculate a single neuron, without an activation function, we get the following result:

xw + b = 32 + 4 = 10

Now suppose we have to introduce a multiplicative factor called precision, and we set it in this way: p = 10. If we also multiply the bias by the same factor, we would not be able to obtain the correct value:

X = xp = x10 = 30
W = xp = w10 = 20
B = xp = b10 = 40

XW + B = 3020 + 40 = 640

So the calculation has been distorted: we have not obtained a number from which we can reconstruct the true result (10). What is happening is that, when we multiply X*W, we are also multiplying the precision twice. So the bias, to remain of the same order of magnitude as the result of the multiplication, must be multiplied by the square of the precision:

X = xp = x10 = 30
W = xp = w10 = 20
B = xp^2 = b10^2 = 400

XW + B = 3020 + 400 = 1000

And from 1000 we reconstruct the exact result:

1000/p^2 = 1000/100 = 10

In slightly more rigorous terms:

X = xp
W = wp
B = b*p^2

XW + B = xpwp + bp^2 =
= (xw + b)*p^2

(XW + B)/p^2 = xw + b

So every time there's a multiplication, we are adding a p factor to the current computed numbers. And every time there's a square function, we're doubling the number of p factors in our computed numbers. And thus those exponents in the biases conversion. This must be done regardless of what NN we want to run: the plain one (infere_plain.py), or the one that works with encrypted numbers (infere_enc.py)