multiprocessing iterator for images

Author: jonasteuwen

00-Image-iterator-multiprocessing.ipynb

@@ -0,0 +1,172 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "# Image iterators using the multiprocessing module\n",
+    "*Jonas Teuwen*"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this notebook we show how to combine the multiprocessing module in Python with iterators.\n",
+    "\n",
+    "This can be useful in deep learning scripts when you, for instance, want to write an iterator which extracts and augments patches from your image on-the-fly to feed to your convolutional neural network (CNN).\n",
+    "\n",
+    "We simulate the following problem:\n",
+    "- You have a list of images, in this case represented by filenames in `self.images`.\n",
+    "- We continuously load an image, and put these into a queue, ready to read.\n",
+    "- Next should give your next image, in this case we only output the filename.\n",
+    "\n",
+    "Each process will load one of the images, but as the processes are now separated we need some way to track which images have already been passed and which ones have not. To implement such a counter we can use shared memory in python. The multiprocessing `RawValue` implements such a `ctype` which allows multiple processes to read from the same variable. There is one problem: race conditions. It might happen that the counter value has not been updated yet and then the next process will output the same image. To handle this we use `Lock` to lock the counter when we either read or write to it to prevent such conditions.\n",
+    "\n",
+    "Note that your output will not necessarily be in the same order as your input as some of the processes processes might complete faster. For CNNs with stochastic gradient decent this is definitely no problem as shuffling improves the result. Check out the Stochastic Gradient Descent Tricks (Sec. 4) at http://cilvr.cs.nyu.edu/diglib/lsml/bottou-sgd-tricks-2012.pdf. \n",
+    "\n",
+    "For more information check https://docs.python.org/2/library/multiprocessing.html."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "from multiprocessing import Process, RawValue, Lock, Queue"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "class CounterMulti(object):\n",
+    "    def __init__(self, val=0):\n",
+    "        self.val = RawValue('i', val)\n",
+    "        self.lock = Lock()\n",
+    "\n",
+    "    def incr(self):\n",
+    "        with self.lock:\n",
+    "            self.val.value += 1\n",
+    "\n",
+    "    def value(self):\n",
+    "        with self.lock:\n",
+    "            return self.val.value\n",
+    "\n",
+    "class MultiImageIter(object):\n",
+    "    def __init__(self):\n",
+    "        self.images = ['image_{}.jpg'.format(i) for i in range(10)]\n",
+    "        self.n_images = len(self.images)\n",
+    "        \n",
+    "        # We define a cursor to track if we have gone through the whole list already\n",
+    "        self.cursor = 0\n",
+    "        self.q = Queue(maxsize=2)\n",
+    "        \n",
+    "        self.counter = CounterMulti(0)\n",
+    "        self.procs = [Process(target=self.writer, args=(self.counter,)) for i in range(4)]\n",
+    "\n",
+    "        for p in self.procs: \n",
+    "            p.deamon = True\n",
+    "            p.start()\n",
+    "    \n",
+    "    def writer(self, counter):\n",
+    "        while True:\n",
+    "            if counter.value() < self.n_images:\n",
+    "                next_value = self.images[counter.value()]\n",
+    "                counter.incr()\n",
+    "                self.q.put(next_value)\n",
+    "            \n",
+    "    def next(self):\n",
+    "        if self.cursor < self.n_images:\n",
+    "            self.cursor += 1\n",
+    "            image = self.q.get()\n",
+    "            # Here you can do stuff with your image\n",
+    "            return image\n",
+    "        else:\n",
+    "            raise StopIteration()\n",
+    "        "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "s = MultiImageIter()\n",
+    "vals = []\n",
+    "for i in range(10):\n",
+    "    vals.append(s.next())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['image_1.jpg',\n",
+       " 'image_2.jpg',\n",
+       " 'image_3.jpg',\n",
+       " 'image_0.jpg',\n",
+       " 'image_5.jpg',\n",
+       " 'image_6.jpg',\n",
+       " 'image_4.jpg',\n",
+       " 'image_7.jpg',\n",
+       " 'image_9.jpg',\n",
+       " 'image_8.jpg']"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "vals"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

README.md

@@ -1,2 +1,7 @@
 # Python-examples
-Python examples
+
+Repository with Python examples.
+
+List
+----
+00: Image iterator which uses the `multiprocessing` module of Python to prefetch images.