SPU 0.2.x (secretflow#20)

* Sync SPU 0.2.x

Author: anakinxc

.bazelrc

@@ -6,7 +6,6 @@ build --enable_platform_specific_config
 
 build --cxxopt=-std=c++17
 build --host_cxxopt=-std=c++17
-build --linkopt -fvisibility=hidden
 
 
 test --keep_going
@@ -19,6 +18,11 @@ build:macos --features=-supports_dynamic_linker
 build:macos --cxxopt -Wno-deprecated-enum-enum-conversion
 build:macos --cxxopt -Wno-deprecated-anon-enum-enum-conversion
 build:macos --macos_minimum_os=11.0
+build:macos --host_macos_minimum_os=11.0
+
+# static link libstdc++ & libgcc on Linux
+build:linux --action_env=BAZEL_LINKOPTS=-static-libstdc++:-static-libgcc
+build:linux --action_env=BAZEL_LINKLIBS=-l%:libstdc++.a:-l%:libgcc.a
 
 build:asan --strip=never
 build:asan --copt -fno-sanitize-recover=all

.bazelversion

@@ -1 +1 @@
-5.2.0
+5.1.1

BUILD.bazel

@@ -0,0 +1,13 @@
+# Copyright 2022 Ant Group Co., Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

CHANGELOG.md

@@ -9,6 +9,24 @@
 ## staging
 > please add your unrelease change here.
 
+## 20220905
+- [SPU] 0.2.4 release
+- [bugfix] Fix Tensorflow example
+- [miscs] Performance improvements
+
+## 20220822
+- [SPU] 0.2.1 release
+- [Feature] Add compiler cache
+- [Feature] Support bc22-pcg psi
+- [API] Refactor psi api
+- [miscs] Various performance improvements
+
+## 20220727
+- [SPU] 0.1.1 release
+- [API] Added rsqrt support
+- [bugfix] Fixed some crashes and wrong answers
+
+
 ## 20220527
 - [SPU] 0.1.0 release
 - [API] Add exp/log approximation configurations.

CONTRIBUTING.md

@@ -1,4 +1,4 @@
-# Contributing
+# Contribution guidelines
 
 ## Contributor License Agreement
 
@@ -31,6 +31,90 @@ The compiler portion of the project follows [MLIR style](https://mlir.llvm.org/g
 * Developer must write unit-test (line coverage must be greater than 80%), tests should be deterministic.
 * Read awesome [Abseil Tips](https://abseil.io/tips/)
 
+## Build
+
+### Prerequisite
+
+#### Docker
+```sh
+## start container
+docker run -d -it --name spu-gcc11-anolis-dev-$(whoami) \
+         --mount type=bind,source="$(pwd)",target=/home/admin/dev/ \
+         -w /home/admin/dev \
+         --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
+         --cap-add=NET_ADMIN \
+         --privileged=true \
+         secretflow/secretflow-gcc11-anolis-dev:latest
+
+# attach to build container
+docker exec -it spu-gcc11-anolis-dev-$(whoami) bash
+```
+
+#### Linux
+
+```sh
+Install gcc>=11.2, cmake>=3.18, ninja, nasm>=2.15, python==3.8, bazel==5.1.1
+
+python3 -m pip install -r requirements.txt
+```
+
+#### macOS
+
+```sh
+# macOS >= 11, Xcode >=13.0
+
+# Install Xcode
+https://apps.apple.com/us/app/xcode/id497799835?mt=12
+
+# Select Xcode toolchain version
+sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
+
+# Install homebrew
+https://brew.sh/
+
+# Install dependencies
+brew install bazel cmake ninja nasm
+
+# Extra setup step for Apple Silicon users
+conda install grpcio
+
+# Install python dependencies
+pip install -r requirements.txt
+```
+
+### Build & UnitTest
+
+``` sh
+
+# build as debug
+bazel build //... -c dbg
+
+# build as release
+bazel build //... -c opt
+
+# test
+bazel test //...
+
+# [optional] build & test with ASAN or UBSAN, for macOS users please use configs with macOS prefix
+bazel test //... --config=[macos-]asan
+bazel test //... --config=[macos-]ubsan
+```
+
+### Bazel build options
+
+- `--define gperf=on` enable gperf
+- `--define tracelog=on` enable link trace log.
+
+### Build docs
+
+```sh
+# prerequisite
+pip install -U docs/requirements.txt
+
+cd docs & make html  # html docs will be in docs/_build/html
+```
+
+
 ## Release cycle
 
 SPU recommends users "live-at-head" like [abseil-cpp](https://github.com/abseil/abseil-cpp), just like abseil, spu also provide Long Term Support Releases to which we backport fixes for severe bugs.

INSTALLATION.md

@@ -0,0 +1,39 @@
+# Installation Guidelines
+
+## Environment
+
+### Linux
+
+SPU has been tested with the following settings:
+
+- Anolis OS 8.4 or later
+- python3.8
+- 8c16g
+
+### MacOS
+
+We have conducted some successful preliminary testings on macOS Monterey 12.4 with Intel processors and Apple Silicon.
+
+### Docker Image
+
+Please check [official Docker image](https://registry.hub.docker.com/r/secretflow/secretflow-gcc11-anolis-dev).
+
+
+## Binaries
+
+### From PyPI
+
+You could install SPU via the [official PyPI package](https://pypi.org/project/spu/)
+
+```bash
+pip install spu
+```
+
+
+### From Source
+
+At the root of repo, run
+
+```bash
+sh build_and_packaging.sh -i
+```

README.md

@@ -4,99 +4,21 @@
 
 SPU (Secure Processing Unit) aims to be a `provable`, `measurable` secure computation device, which provides computation ability while keeping your private data protected.
 
-## Project status
+SPU could be treated as a programmable device, it's not designed to be used directly. Normally we use SecretFlow framework, which use SPU as the underline secure computing device.
 
 Currently, we mainly focus on `provable` security. It contains a secure runtime that evaluates [XLA](https://www.tensorflow.org/xla/operation_semantics)-like tensor operations, which use [MPC](https://en.wikipedia.org/wiki/Secure_multi-party_computation) as the underline evaluation engine to protect privacy information.
 
-## Contents
-- [Documentation](https://secretflow.readthedocs.io)
-- [Roadmap](TBD)
-- [Build and test](#Build)
-- [FAQ](#FAQ)
-
-## Build
-
-### Prerequisite
-
-#### Docker
-```sh
-## start container
-docker run -d -it --name spu-gcc11-anolis-dev-$(whoami) \
-         --mount type=bind,source="$(pwd)",target=/home/admin/dev/ \
-         -w /home/admin/dev \
-         --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
-         --cap-add=NET_ADMIN \
-         --privileged=true \
-         registry.hub.docker.com/secretflow/spu-gcc11-anolis-dev:latest
-
-# attach to build container
-docker exec -it spu-gcc11-anolis-dev-$(whoami) bash
-```
-
-#### Linux
-
-```sh
-Install gcc>=11.2, cmake>=3.18, ninja, nasm>=2.15, python==3.8, bazel==5.1.1
-
-python3 -m pip install -r docker/requirements.txt
-```
-
-#### macOS
-
-```sh
-# macOS >= 11, Xcode >=13.0
-
-# Install Xcode
-https://apps.apple.com/us/app/xcode/id497799835?mt=12
-
-# Select Xcode toolchain version
-sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
-
-# Install homebrew
-https://brew.sh/
-
-# Install dependencies
-brew install bazel cmake ninja nasm
-
-# Install python dependencies
-python3 -m pip install -r docker/requirements.txt
-```
-
-### Build & UnitTest
-
-``` sh
-
-# build as debug
-bazel build //... -c dbg
-
-# build as release
-bazel build //... -c opt
-
-# test
-bazel test //...
-
-# [optional] build & test with ASAN or UBSAN, for macOS users please use configs with macOS prefix
-bazel test //... --config=[macos-]asan
-bazel test //... --config=[macos-]ubsan
-```
-
-
-### Build docs
-
-```sh
-# prerequisite
-pip install -U sphinx
+SPU python package also contains a simple distributed module to demo SPU usage, but it's **NOT designed for production** due to system security and performance concerns, please **DO NOT** use it directly in production.
 
-cd docs & make html  # html docs will be in docs/_build/html
-```
+## Contribution Guidelines
 
-## FAQ
+If you would like to contribute to SPU, please check [Contribution guidelines](CONTRIBUTING.md).
 
-> How can I use SPU?
+This documentation also contains instructions for [build and testing](CONTRIBUTING.md#build).
 
-SPU could be treated as a programmable device, it's not designed to be used directly. Normally we use SecretFlow framework, which use SPU as the underline secure computing device.
+## Installation Guidelines
 
-SPU python package also contains a simple distributed module to demo SPU usage, but it's **NOT designed for production** due to system security and performance concerns, please **DO NOT** use it directly in production.
+Please follow [Installation Guidelines](INSTALLATION.md) to install SPU.
 
 ## Acknowledgement
 

WORKSPACE

@@ -6,6 +6,8 @@ spu_deps()
 
 #
 # yasl
+# Warning: SPU relies on yasl to bring in common 3p libraries.
+#          Please make sure yasl_deps are called right after spu_deps.
 #
 load("@yasl//bazel:repositories.bzl", "yasl_deps")
 
@@ -55,7 +57,7 @@ load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
 # https://github.com/hedronvision/bazel-compile-commands-extractor
 git_repository(
     name = "hedron_compile_commands",
-    commit = "30149bdadf7b1d0b0367592bfb4864318202765d",
+    commit = "57046dba8d60f819887ea81933ed90f8e23a458a",
     remote = "https://github.com/hedronvision/bazel-compile-commands-extractor.git",
     shallow_since = "1644967664 -0800",
 )

bazel/BUILD.bazel

@@ -1,3 +1,17 @@
+# Copyright 2022 Ant Group Co., Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 package(default_visibility = ["//visibility:public"])
 
 config_setting(

bazel/easyloggingpp.BUILD

@@ -1,3 +1,17 @@
+# Copyright 2022 Ant Group Co., Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 load("@spulib//bazel:spu.bzl", "spu_cmake_external")
 
 package(default_visibility = ["//visibility:public"])