Consolidate privacy/ and differential_privacy/.

Author: Xin Pan

differential_privacy/per_example_gradients/__init__.py renamed to WORKSPACE

@@ -1,12 +1,9 @@
 <font size=4><b>Deep Learning with Differential Privacy</b></font>
 
-Authors:
-Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang
-
 Open Sourced By: Xin Pan ([email protected], github: panyx0718)
 
 
-<Introduction>
+###Introduction for dp_sgd/README.md
 
 Machine learning techniques based on neural networks are achieving remarkable 
 results in a wide variety of domains. Often, the training of models requires 
@@ -21,73 +18,14 @@ manageable cost in software complexity, training efficiency, and model quality.
 paper: https://arxiv.org/abs/1607.00133
 
 
-<b>Requirements:</b>
-
-1. Tensorflow 0.10.0 (master branch)
-
-Note: r0.11 might experience some problems
-
-2. Bazel 0.3.1
-
-3. Download MNIST data
-
-TODO(xpan): Complete the link:
-[train](http://download.tensorflow.org/models/)
-[test](http://download.tensorflow.org/models/)
-
-Alternatively, download the tfrecord format MNIST from:
-https://github.com/panyx0718/models/tree/master/slim
-
-<b>How to run:</b>
-
-```shell
-# Clone the codes under differential_privacy.
-# Create an empty WORKSPACE file.
-# Download the data to the data/ directory.
-ls -R
-.:
-data  differential_privacy  WORKSPACE
-
-./data:
-mnist_test.tfrecord  mnist_train.tfrecord
-
-./differential_privacy:
-dp_mnist  dp_optimizer  __init__.py  per_example_gradients  README.md
-
-./differential_privacy/dp_mnist:
-BUILD  dp_mnist.py
-
-./differential_privacy/dp_optimizer:
-accountant.py  BUILD  dp_optimizer.py  dp_pca.py  sanitizer.py  utils.py
-
-./differential_privacy/per_example_gradients:
-BUILD  __init__.py  per_example_gradients.py
-
-# Build the codes.
-bazel build -c opt differential_privacy/...
+###Introduction for multiple_teachers/README.md
 
-# Run the mnist differntial privacy training codes.
-bazel-bin/differential_privacy/dp_mnist/dp_mnist \
-    --training_data_path=data/mnist_train.tfrecord \
-    --eval_data_path=data/mnist_test.tfrecord \
-    --save_path=/tmp/mnist_dir
+This repository contains code to create a setup for learning privacy-preserving 
+student models by transferring knowledge from an ensemble of teachers trained 
+on disjoint subsets of the data for which privacy guarantees are to be provided.
 
-...
-step: 1
-step: 2
-...
-step: 9
-spent privacy: eps 0.1250 delta 0.72709
-spent privacy: eps 0.2500 delta 0.24708
-spent privacy: eps 0.5000 delta 0.0029139
-spent privacy: eps 1.0000 delta 6.494e-10
-spent privacy: eps 2.0000 delta 8.2242e-24
-spent privacy: eps 4.0000 delta 1.319e-51
-spent privacy: eps 8.0000 delta 3.3927e-107
-train_accuracy: 0.53
-eval_accuracy: 0.53
-...
+Knowledge acquired by teachers is transferred to the student in a differentially
+private manner by noisily aggregating the teacher decisions before feeding them
+to the student during training.
 
-ls /tmp/mnist_dir/
-checkpoint  ckpt  ckpt.meta  results-0.json
-```
+paper: https://arxiv.org/abs/1610.05755

differential_privacy/README.md

@@ -0,0 +1,107 @@
+<font size=4><b>Deep Learning with Differential Privacy</b></font>
+
+Authors:
+Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang
+
+Open Sourced By: Xin Pan ([email protected], github: panyx0718)
+
+
+<Introduction>
+
+Machine learning techniques based on neural networks are achieving remarkable 
+results in a wide variety of domains. Often, the training of models requires 
+large, representative datasets, which may be crowdsourced and contain sensitive 
+information. The models should not expose private information in these datasets. 
+Addressing this goal, we develop new algorithmic techniques for learning and a 
+refined analysis of privacy costs within the framework of differential privacy. 
+Our implementation and experiments demonstrate that we can train deep neural 
+networks with non-convex objectives, under a modest privacy budget, and at a 
+manageable cost in software complexity, training efficiency, and model quality.
+
+paper: https://arxiv.org/abs/1607.00133
+
+
+<b>Requirements:</b>
+
+1. Tensorflow 0.10.0 (master branch)
+
+Note: r0.11 might experience some problems
+
+2. Bazel 0.3.1
+
+3. Download MNIST data
+
+TODO(xpan): Complete the link:
+[train](http://download.tensorflow.org/models/)
+[test](http://download.tensorflow.org/models/)
+
+Alternatively, download the tfrecord format MNIST from:
+https://github.com/panyx0718/models/tree/master/slim
+
+<b>How to run:</b>
+
+```shell
+# Clone the codes under differential_privacy.
+# Create an empty WORKSPACE file.
+# Download the data to the data/ directory.
+
+# List the codes.
+ls -R differential_privacy/
+differential_privacy/:
+dp_sgd  __init__.py  privacy_accountant  README.md
+
+differential_privacy/dp_sgd:
+dp_mnist  dp_optimizer  per_example_gradients  README.md
+
+differential_privacy/dp_sgd/dp_mnist:
+BUILD  dp_mnist.py
+
+differential_privacy/dp_sgd/dp_optimizer:
+BUILD  dp_optimizer.py  dp_pca.py  sanitizer.py  utils.py
+
+differential_privacy/dp_sgd/per_example_gradients:
+BUILD  per_example_gradients.py
+
+differential_privacy/privacy_accountant:
+python  tf
+
+differential_privacy/privacy_accountant/python:
+BUILD  gaussian_moments.py
+
+differential_privacy/privacy_accountant/tf:
+accountant.py  accountant_test.py  BUILD
+
+# List the data.
+ls -R data/
+
+./data:
+mnist_test.tfrecord  mnist_train.tfrecord
+
+# Build the codes.
+bazel build -c opt differential_privacy/...
+
+# Run the mnist differntial privacy training codes.
+bazel-bin/differential_privacy/dp_sgd/dp_mnist/dp_mnist \
+    --training_data_path=data/mnist_train.tfrecord \
+    --eval_data_path=data/mnist_test.tfrecord \
+    --save_path=/tmp/mnist_dir
+
+...
+step: 1
+step: 2
+...
+step: 9
+spent privacy: eps 0.1250 delta 0.72709
+spent privacy: eps 0.2500 delta 0.24708
+spent privacy: eps 0.5000 delta 0.0029139
+spent privacy: eps 1.0000 delta 6.494e-10
+spent privacy: eps 2.0000 delta 8.2242e-24
+spent privacy: eps 4.0000 delta 1.319e-51
+spent privacy: eps 8.0000 delta 3.3927e-107
+train_accuracy: 0.53
+eval_accuracy: 0.53
+...
+
+ls /tmp/mnist_dir/
+checkpoint  ckpt  ckpt.meta  results-0.json
+```

differential_privacy/dp_sgd/README.md

@@ -17,8 +17,8 @@ py_binary(
         "dp_mnist.py",
     ],
     deps = [
-        "//differential_privacy/dp_optimizer",
-        "//differential_privacy/dp_optimizer:dp_pca",
-        "//differential_privacy/dp_optimizer:utils",
+        "//differential_privacy/dp_sgd/dp_optimizer",
+        "//differential_privacy/dp_sgd/dp_optimizer:dp_pca",
+        "//differential_privacy/dp_sgd/dp_optimizer:utils",
     ],
 )

differential_privacy/dp_mnist/BUILD renamed to differential_privacy/dp_sgd/dp_mnist/BUILD

@@ -24,11 +24,11 @@
 import numpy as np
 import tensorflow as tf
 
-from differential_privacy.dp_optimizer import accountant
-from differential_privacy.dp_optimizer import dp_optimizer
-from differential_privacy.dp_optimizer import dp_pca
-from differential_privacy.dp_optimizer import sanitizer
-from differential_privacy.dp_optimizer import utils
+from differential_privacy.dp_sgd.dp_optimizer import dp_optimizer
+from differential_privacy.dp_sgd.dp_optimizer import dp_pca
+from differential_privacy.dp_sgd.dp_optimizer import sanitizer
+from differential_privacy.dp_sgd.dp_optimizer import utils
+from differential_privacy.privacy_accountant.tf import accountant
 
 # parameters for the training
 tf.flags.DEFINE_integer("batch_size", 600,

differential_privacy/dp_mnist/dp_mnist.py renamed to differential_privacy/dp_sgd/dp_mnist/dp_mnist.py

@@ -29,26 +29,16 @@ py_library(
     ],
 )
 
-py_library(
-    name = "accountant",
-    srcs = [
-        "accountant.py",
-    ],
-    deps = [
-        ":utils",
-    ],
-)
-
 py_library(
     name = "dp_optimizer",
     srcs = [
         "dp_optimizer.py",
         "sanitizer.py",
     ],
     deps = [
-        ":accountant",
         ":utils",
-        "//differential_privacy/per_example_gradients",
+        "//differential_privacy/dp_sgd/per_example_gradients",
+        "//differential_privacy/privacy_accountant/tf:accountant",
     ],
 )
 

differential_privacy/dp_optimizer/BUILD renamed to differential_privacy/dp_sgd/dp_optimizer/BUILD

@@ -19,8 +19,8 @@
 
 import tensorflow as tf
 
-from differential_privacy.dp_optimizer import utils
-from differential_privacy.per_example_gradients import per_example_gradients
+from differential_privacy.dp_sgd.dp_optimizer import utils
+from differential_privacy.dp_sgd.per_example_gradients import per_example_gradients
 
 
 class DPGradientDescentOptimizer(tf.train.GradientDescentOptimizer):

differential_privacy/dp_optimizer/dp_optimizer.py renamed to differential_privacy/dp_sgd/dp_optimizer/dp_optimizer.py

@@ -15,10 +15,9 @@
 
 """Differentially private optimizers.
 """
-
 import tensorflow as tf
 
-from differential_privacy.dp_optimizer import sanitizer as san
+from differential_privacy.dp_sgd.dp_optimizer import sanitizer as san
 
 
 def ComputeDPPrincipalProjection(data, projection_dims,

differential_privacy/dp_optimizer/dp_pca.py renamed to differential_privacy/dp_sgd/dp_optimizer/dp_pca.py

@@ -26,7 +26,7 @@
 
 import tensorflow as tf
 
-from differential_privacy.dp_optimizer import utils
+from differential_privacy.dp_sgd.dp_optimizer import utils
 
 
 ClipOption = collections.namedtuple("ClipOption",

differential_privacy/dp_optimizer/sanitizer.py renamed to differential_privacy/dp_sgd/dp_optimizer/sanitizer.py

@@ -58,9 +58,10 @@ def _ListUnion(list_1, list_2):
 
 
 def Interface(ys, xs):
-  """
-  Returns a dict mapping each element of xs to any of its consumers that are
-  indirectly consumed by ys.
+  """Maps xs to consumers.
+
+    Returns a dict mapping each element of xs to any of its consumers that are
+    indirectly consumed by ys.
 
   Args:
     ys: The outputs
@@ -181,10 +182,10 @@ def __call__(self, x, z_grads):
     idx = list(self.op.inputs).index(x)
     assert idx != -1
     assert len(z_grads) == len(self.op.outputs)
-    assert idx == 1 # We expect weights to be arg 1
+    assert idx == 1  # We expect weights to be arg 1
     # We don't expect anyone to per-example differentiate with repsect
     # to anything other than the weights.
-    x, w = self.op.inputs
+    x, _ = self.op.inputs
     z_grads, = z_grads
     x_expanded = tf.expand_dims(x, 2)
     z_grads_expanded = tf.expand_dims(z_grads, 1)
@@ -305,7 +306,7 @@ def __call__(self, x, z_grads):
     assert idx == 1 # We expect biases to be arg 1
     # We don't expect anyone to per-example differentiate with respect
     # to anything other than the biases.
-    x, b = self.op.inputs
+    x, _ = self.op.inputs
     z_grads, = z_grads
     return z_grads
 
@@ -336,10 +337,10 @@ def PerExampleGradients(ys, xs, grad_ys=None, name="gradients",
     merged_interface = _ListUnion(merged_interface, interface[x])
   # Differentiate with respect to the interface
   interface_gradients = tf.gradients(ys, merged_interface, grad_ys=grad_ys,
-                                    name=name,
-                                    colocate_gradients_with_ops=
-                                    colocate_gradients_with_ops,
-                                    gate_gradients=gate_gradients)
+                                     name=name,
+                                     colocate_gradients_with_ops=
+                                     colocate_gradients_with_ops,
+                                     gate_gradients=gate_gradients)
   grad_dict = OrderedDict(zip(merged_interface, interface_gradients))
   # Build the per-example gradients with respect to the xs
   if colocate_gradients_with_ops: