[handpose] Add model. (tensorflow#414)

Author: annxingyuan

blazeface/README.md

@@ -1,6 +1,6 @@
 # Blazeface detector
 
-[Blazeface](https://arxiv.org/abs/1907.05047) is a lightweight model that detects faces in images. Blazeface makes use of the [Single Shot Detector](https://arxiv.org/abs/1512.02325) architecture with a custom encoder. The model may serve as a first step for face-related computer vision applications, such as facial keypoint recognition.
+[Blazeface](https://arxiv.org/abs/1907.05047) is a lightweight model that detects faces in images. Blazeface makes use of a modified [Single Shot Detector](https://arxiv.org/abs/1512.02325) architecture with a custom encoder. The model may serve as a first step for face-related computer vision applications, such as facial keypoint recognition.
 
 <img src="demo/demo.gif" alt="demo" style="width: 437px;"/>
 

handpose/.npmignore

@@ -0,0 +1,20 @@
+.DS_Store
+.yalc/
+.vscode/
+.rpt2_cache/
+demo/
+scripts/
+src/
+coverage/
+node_modules/
+karma.conf.js
+*.tgz
+.travis.yml
+.npmignore
+tslint.json
+yarn.lock
+yalc.lock
+cloudbuild.yml
+dist/*_test.js
+dist/*_test.js.map
+dist/test_util*

handpose/README.md

@@ -0,0 +1,113 @@
+# MediaPipe Handpose
+
+MediaPipe Handpose is a lightweight ML pipeline consisting of two models: A palm detector and a hand-skeleton finger tracking model. It predicts 21 2D hand keypoints per detected hand. For more details, please read our Google AI [blogpost](https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html).
+
+<img src="demo/demo.gif" alt="demo" style="width:640px" />
+
+Given an input, the model predicts whether it contains a hand. If so, the model returns coordinates for the bounding box around the hand, as well as 21 keypoints within the hand, outlining the location of each finger joint and the palm.
+
+More background information about the model, as well as its performance characteristics on different datasets, can be found here: [https://drive.google.com/file/d/1sv4sSb9BSNVZhLzxXJ0jBv9DqD-4jnAz/view](https://drive.google.com/file/d/1sv4sSb9BSNVZhLzxXJ0jBv9DqD-4jnAz/view)
+
+Check out our [demo](https://storage.googleapis.com/tfjs-models/demos/handpose/index.html), which uses the model to detect hand landmarks in a live video stream.
+
+This model is also available as part of [MediaPipe](https://hand.mediapipe.dev/), a framework for building multimodal applied ML pipelines.
+
+# Performance
+
+MediaPipe Handpose consists of ~12MB of weights, and is well-suited for real time inference across a variety of devices (40 FPS on a 2018 MacBook Pro, 35 FPS on an iPhone11, 6 FPS on a Pixel3).
+
+## Installation
+
+Using `yarn`:
+
+    $ yarn add @tensorflow-models/handpose
+
+Using `npm`:
+
+    $ npm install @tensorflow-models/handpose
+
+Note that this package specifies `@tensorflow/tfjs-core` and `@tensorflow/tfjs-converter` as peer dependencies, so they will also need to be installed.
+
+## Usage
+
+To import in npm:
+
+```js
+import * as handpose from '@tensorflow-models/handpose';
+```
+
+or as a standalone script tag:
+
+```html
+<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-core"></script>
+<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-converter"></script>
+<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/handpose"></script>
+```
+
+Then:
+
+```js
+async function main() {
+  // Load the MediaPipe handpose model.
+  const model = await handpose.load();
+  // Pass in a video stream (or an image, canvas, or 3D tensor) to obtain a
+  // hand prediction from the MediaPipe graph.
+  const predictions = await model.estimateHands(document.querySelector("video"));
+  if (predictions.length > 0) {
+    /*
+    `predictions` is an array of objects describing each detected hand, for example:
+    [
+      {
+        handInViewConfidence: 1, // The probability of a hand being present.
+        boundingBox: { // The bounding box surrounding the hand.
+          topLeft: [162.91, -17.42],
+          bottomRight: [548.56, 368.23],
+        },
+        landmarks: [ // The 3D coordinates of each hand landmark.
+          [472.52, 298.59, 0.00],
+          [412.80, 315.64, -6.18],
+          ...
+        ],
+        annotations: { // Semantic groupings of the `landmarks` coordinates.
+          thumb: [
+            [412.80, 315.64, -6.18]
+            [350.02, 298.38, -7.14],
+            ...
+          ],
+          ...
+        }
+      }
+    ]
+    */
+
+    for (let i = 0; i < predictions.length; i++) {
+      const keypoints = predictions[i].landmarks;
+
+      // Log hand keypoints.
+      for (let i = 0; i < keypoints.length; i++) {
+        const [x, y, z] = keypoints[i];
+        console.log(`Keypoint ${i}: [${x}, ${y}, ${z}]`);
+      }
+    }
+  }
+}
+main();
+```
+
+#### Parameters for handpose.load()
+
+`handpose.load()` takes a configuration object with the following properties:
+
+* **maxContinuousChecks** - How many frames to go without running the bounding box detector. Defaults to infinity. Set to a lower value if you want a safety net in case the mesh detector produces consistently flawed predictions.
+
+* **detectionConfidence** - Threshold for discarding a prediction. Defaults to 0.8.
+
+* **iouThreshold** - A float representing the threshold for deciding whether boxes overlap too much in non-maximum suppression. Must be between [0, 1]. Defaults to 0.3.
+
+* **scoreThreshold** - A threshold for deciding when to remove boxes based on score in non-maximum suppression. Defaults to 0.75.
+
+#### Parameters for handpose.estimateHands()
+
+* **input** - The image to classify. Can be a tensor, DOM element image, video, or canvas.
+
+* **flipHorizontal** - Whether to flip/mirror the facial keypoints horizontally. Should be true for videos that are flipped by default (e.g. webcams).

handpose/cloudbuild.yml

@@ -0,0 +1,49 @@
+
+steps:
+
+# Install common dependencies.
+- name: 'node:10'
+  id: 'yarn-common'
+  entrypoint: 'yarn'
+  args: ['install']
+
+# Install handpose dependencies.
+- name: 'node:10'
+  dir: 'handpose'
+  entrypoint: 'yarn'
+  id: 'yarn'
+  args: ['install']
+  waitFor: ['yarn-common']
+
+# Lint.
+- name: 'node:10'
+  dir: 'handpose'
+  entrypoint: 'yarn'
+  id: 'lint'
+  args: ['lint']
+  waitFor: ['yarn']
+
+# Build.
+- name: 'node:10'
+  dir: 'handpose'
+  entrypoint: 'yarn'
+  id: 'build'
+  args: ['build']
+  waitFor: ['yarn']
+
+# Run tests.
+- name: 'node:10'
+  dir: 'handpose'
+  entrypoint: 'yarn'
+  id: 'test'
+  args: ['test']
+  waitFor: ['yarn']
+
+# General configuration
+timeout: 1800s
+logsBucket: 'gs://tfjs-build-logs'
+substitutions:
+  _NIGHTLY: ''
+options:
+  logStreamingOption: 'STREAM_ON'
+  substitution_option: 'ALLOW_LOOSE'

handpose/demo/.babelrc

@@ -0,0 +1,23 @@
+{
+  "presets": [
+    [
+      "env",
+      {
+        "esmodules": false,
+        "targets": {
+          "browsers": [
+            "> 3%"
+          ]
+        }
+      }
+    ]
+  ],
+  "plugins": [
+    [
+      "transform-runtime",
+      {
+        "polyfill": false
+      }
+    ]
+  ]
+}

handpose/demo/README.md

@@ -0,0 +1,66 @@
+# Handpose demo
+
+## Contents
+
+This demo shows how to use the handpose model to detect hands in a video stream.
+
+## Setup
+
+cd into the demos folder:
+
+```sh
+cd handpose/demos
+```
+
+Install dependencies and prepare the build directory:
+
+```sh
+yarn
+```
+
+To watch files for changes, and launch a dev server:
+
+```sh
+yarn watch
+```
+
+## If you are developing handpose locally, and want to test the changes in the demos
+
+Cd into the handpose folder:
+```sh
+cd handpose
+```
+
+Install dependencies:
+```sh
+yarn
+```
+
+Publish handpose locally:
+```sh
+yarn build && yarn yalc publish
+```
+
+Cd into the demos and install dependencies:
+
+```sh
+cd demos
+yarn
+```
+
+Link the local handpose to the demos:
+```sh
+yarn yalc link @tensorflow-models/handpose
+```
+
+Start the dev demo server:
+```sh
+yarn watch
+```
+
+To get future updates from the handpose source code:
+```
+# cd up into the handpose directory
+cd ../
+yarn build && yarn yalc push
+```

handpose/demo/demo.gif

@@ -0,0 +1,50 @@
+<!-- Copyright 2020 Google LLC. All Rights Reserved.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================-->
+
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1.0, user-scalable=no">
+<!-- Load three.js -->
+<script src="https://cdn.jsdelivr.net/npm/[email protected]/build/three.min.js"></script>
+<!-- Load scatter-gl.js -->
+<!-- TODO(annxingyuan): Submit a PR to scatter-gl that allows polylines to update in the render loop. -->
+<script src="https://storage.googleapis.com/learnjs-data/handtrack_staging/scatter-gl.js"></script>
+<!-- <script src="https://cdn.jsdelivr.net/npm/[email protected]/lib/scatter-gl.min.js"></script> -->
+<style>
+  #canvas-wrapper {
+    position: relative;
+  }
+  #canvas-wrapper, #scatter-gl-container {
+    display: inline-block;
+    vertical-align: top;
+  }
+</style>
+<script src="https://cdn.jsdelivr.net/npm/[email protected]/build/stats.min.js"></script>
+
+<body>
+  <div id="info" style='display:none'></div>
+  <div id="predictions"></div>
+  <div id="canvas-wrapper">
+    <canvas id="output" style=""></canvas>
+    <video id="video" playsinline style="
+    -webkit-transform: scaleX(-1);
+    transform: scaleX(-1);
+    visibility: hidden;
+    width: auto;
+    height: auto;
+    position: absolute;
+    ">
+  </video>
+  </div>
+  <div id="scatter-gl-container"></div>
+  <script src="https://cdnjs.cloudflare.com/ajax/libs/dat-gui/0.7.6/dat.gui.min.js"></script>
+  <script src="./index.js"></script>
+</body>