[facemesh] Add model. (tensorflow#413)

Author: annxingyuan

.gitignore

@@ -14,3 +14,5 @@ clone/
 .rpt2_cache/
 
 *.pyc
+
+*.tgz

blazeface/scripts/publish-demo.sh

@@ -17,4 +17,5 @@
 cd demo
 rm -rf dist
 yarn build
-gsutil rsync -J -d -r dist/ gs://tfjs-models/demos/blazeface
+gsutil -m rm -r gs://tfjs-models/demos/blazeface/*
+gsutil -m cp -Z -r dist/ gs://tfjs-models/demos/blazeface

facemesh/.npmignore

@@ -0,0 +1,20 @@
+.DS_Store
+.yalc/
+.vscode/
+.rpt2_cache/
+demo/
+scripts/
+src/
+coverage/
+node_modules/
+karma.conf.js
+*.tgz
+.travis.yml
+.npmignore
+tslint.json
+yarn.lock
+yalc.lock
+cloudbuild.yml
+dist/*_test.js
+dist/*_test.js.map
+dist/test_util*

facemesh/README.md

@@ -0,0 +1,131 @@
+# MediaPipe Facemesh
+
+MediaPipe Facemesh is a lightweight machine learning pipeline predicting 486 3D facial landmarks to infer the approximate surface geometry of a human face ([paper](https://arxiv.org/pdf/1907.06724.pdf)).
+
+<img src="demo.gif" alt="demo" style="width: 640px;"/>
+
+More background information about the model, as well as its performance characteristics on different datasets, can be found here: [https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view](https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view)
+
+The model is designed for front-facing cameras on mobile devices, where faces in view tend to occupy a relatively large fraction of the canvas. MediaPipe Facemesh may struggle to identify far-away faces.
+
+Check out our [demo](https://storage.googleapis.com/tfjs-models/demos/facemesh/index.html), which uses the model to detect facial landmarks in a live video stream.
+
+This model is also available as part of [MediaPipe](https://github.com/google/mediapipe/tree/master/mediapipe/models), a
+framework for building multimodal applied ML pipelines.
+
+## Installation
+
+Using `yarn`:
+
+    $ yarn add @tensorflow-models/facemesh
+
+Using `npm`:
+
+    $ npm install @tensorflow-models/facemesh
+
+Note that this package specifies `@tensorflow/tfjs-core` and `@tensorflow/tfjs-converter` as peer dependencies, so they will also need to be installed.
+
+## Usage
+
+To import in npm:
+
+```js
+import * as facemesh from '@tensorflow-models/facemesh';
+```
+
+or as a standalone script tag:
+
+```html
+<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-core"></script>
+<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-converter"></script>
+<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/facemesh"></script>
+```
+
+Then:
+
+```js
+
+async function main() {
+  // Load the MediaPipe facemesh model.
+  const model = await facemesh.load();
+
+  // Pass in a video stream (or an image, canvas, or 3D tensor) to obtain an
+  // array of detected faces from the MediaPipe graph.
+  const predictions = await model.estimateFaces(document.querySelector("video"));
+
+  if (predictions.length > 0) {
+    /*
+    `predictions` is an array of objects describing each detected face, for example:
+
+    [
+      {
+        faceInViewConfidence: 1, // The probability of a face being present.
+        boundingBox: { // The bounding box surrounding the face.
+          topLeft: [232.28, 145.26],
+          bottomRight: [449.75, 308.36],
+        },
+        mesh: [ // The 3D coordinates of each facial landmark.
+          [92.07, 119.49, -17.54],
+          [91.97, 102.52, -30.54],
+          ...
+        ],
+        scaledMesh: [ // The 3D coordinates of each facial landmark, normalized.
+          [322.32, 297.58, -17.54],
+          [322.18, 263.95, -30.54]
+        ],
+        annotations: { // Semantic groupings of the `scaledMesh` coordinates.
+          silhouette: [
+            [326.19, 124.72, -3.82],
+            [351.06, 126.30, -3.00],
+            ...
+          ],
+          ...
+        }
+      }
+    ]
+    */
+
+    for (let i = 0; i < predictions.length; i++) {
+      const keypoints = predictions[i].scaledMesh;
+
+      // Log facial keypoints.
+      for (let i = 0; i < keypoints.length; i++) {
+        const [x, y, z] = keypoints[i];
+
+        console.log(`Keypoint ${i}: [${x}, ${y}, ${z}]`);
+      }
+    }
+  }
+}
+
+main();
+
+```
+
+#### Parameters for facemesh.load()
+
+`facemesh.load()` takes a configuration object with the following properties:
+
+* **maxContinuousChecks** - How many frames to go without running the bounding box detector. Only relevant if maxFaces > 1. Defaults to 5.
+
+* **detectionConfidence** - Threshold for discarding a prediction. Defaults to 0.9.
+
+* **maxFaces** - The maximum number of faces detected in the input. Should be set to the minimum number for performance. Defaults to 10.
+
+* **iouThreshold** - A float representing the threshold for deciding whether boxes overlap too much in non-maximum suppression. Must be between [0, 1]. Defaults to 0.3.
+
+* **scoreThreshold** - A threshold for deciding when to remove boxes based on score in non-maximum suppression. Defaults to 0.75.
+
+#### Parameters for model.estimateFace()
+
+* **input** - The image to classify. Can be a tensor, DOM element image, video, or canvas.
+
+* **returnTensors** - (defaults to `false`) Whether to return tensors as opposed to values.
+
+* **flipHorizontal** - Whether to flip/mirror the facial keypoints horizontally. Should be true for videos that are flipped by default (e.g. webcams).
+
+#### Keypoints
+
+Here is map of the keypoints:
+
+<img src="mesh_map.jpg" alt="keypoints_map" style="width: 500px; height: 500px">

facemesh/demo.gif

@@ -0,0 +1,23 @@
+{
+  "presets": [
+    [
+      "env",
+      {
+        "esmodules": false,
+        "targets": {
+          "browsers": [
+            "> 3%"
+          ]
+        }
+      }
+    ]
+  ],
+  "plugins": [
+    [
+      "transform-runtime",
+      {
+        "polyfill": false
+      }
+    ]
+  ]
+}

facemesh/demo/.babelrc

@@ -0,0 +1,66 @@
+# MediaPipe Facemesh demo
+
+## Contents
+
+The MediaPipe Facemesh demo shows how to use the MediaPipe Facemesh model to estimate keypoints on a face.
+
+## Setup
+
+cd into the demos folder:
+
+```sh
+cd facemesh/demos
+```
+
+Install dependencies and prepare the build directory:
+
+```sh
+yarn
+```
+
+To watch files for changes, and launch a dev server:
+
+```sh
+yarn watch
+```
+
+## If you are developing facemesh locally, and want to test the changes in the demos
+
+Cd into the facemesh folder:
+```sh
+cd facemesh
+```
+
+Install dependencies:
+```sh
+yarn
+```
+
+Publish facemesh locally:
+```sh
+yarn build && yarn yalc publish
+```
+
+Cd into the demos and install dependencies:
+
+```sh
+cd demos
+yarn
+```
+
+Link the local facemesh to the demos:
+```sh
+yarn yalc link @tensorflow-models/facemesh
+```
+
+Start the dev demo server:
+```sh
+yarn watch
+```
+
+To get future updates from the facemesh source code:
+```
+# cd up into the facemesh directory
+cd ../
+yarn build && yarn yalc push
+```

facemesh/demo/README.md

@@ -0,0 +1,60 @@
+<!-- Copyright 2020 Google LLC. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================-->
+
+<head>
+  <!-- Load three.js -->
+<script src="https://cdn.jsdelivr.net/npm/[email protected]/build/three.min.js"></script>
+<!-- Load scatter-gl.js -->
+<script src="https://cdn.jsdelivr.net/npm/[email protected]/lib/scatter-gl.min.js"></script>
+<style>
+  .canvas-wrapper, #scatter-gl-container {
+    display: inline-block;
+    vertical-align: top;
+  }
+
+  #scatter-gl-container {
+    border: solid 1px black;
+    position: relative;
+  }
+
+  /* center the canvas within its wrapper */
+  #scatter-gl-container canvas {
+    transform: translate3d(-50%, -50%, 0);
+    left: 50%;
+    top: 50%;
+    position: absolute;
+  }
+</style>
+</head>
+<body>
+  <div id="main">
+    <div class="container">
+      <div class="canvas-wrapper">
+        <canvas id="output"></canvas>
+        <video id="video" playsinline style="
+          -webkit-transform: scaleX(-1);
+          transform: scaleX(-1);
+          visibility: hidden;
+          width: auto;
+          height: auto;
+          ">
+        </video>
+      </div>
+      <div id="scatter-gl-container"></div>
+    </div>
+  </div>
+</body>
+<script src="https://cdnjs.cloudflare.com/ajax/libs/dat-gui/0.7.6/dat.gui.min.js"></script>
+<script src="index.js"></script>