satori-distributed-deeplearning.html

<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>Distributed Deep Learning &mdash; MIT Satori User Documentation  documentation</title>
  

  
  
  
  
    <link rel="canonical" href="https://researchcomputing.mit.edu/satori-distributed-deeplearning.html"/>
  

  
  <script type="text/javascript" src="_static/js/modernizr.min.js"></script>
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
        <script type="text/javascript" src="_static/jquery.js"></script>
        <script type="text/javascript" src="_static/underscore.js"></script>
        <script type="text/javascript" src="_static/doctools.js"></script>
        <script type="text/javascript" src="_static/language_data.js"></script>
    
    <script type="text/javascript" src="_static/js/theme.js"></script>

    

  
  <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="IBM Large Model Support (LMS)" href="satori-large-model-support.html" />
    <link rel="prev" title="IBM Watson Machine Learning Community Edition (WMLCE)" href="satori-ai-frameworks.html" /> 
</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

          
            <a href="index.html" class="icon icon-home"> MIT Satori User Documentation
          

          
          </a>

          
            
            
          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="satori-basics.html">Satori Basics</a><ul>
<li class="toctree-l2"><a class="reference internal" href="satori-basics.html#what-is-satori">What is Satori?</a></li>
<li class="toctree-l2"><a class="reference internal" href="satori-basics.html#how-can-i-get-an-account">How can I get an account?</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="satori-ssh.html">Satori Login</a></li>
<li class="toctree-l1"><a class="reference internal" href="satori-training.html">Training for faster onboarding in the system HW and SW architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="satori-workload-manager.html">Running your AI training jobs on Satori</a><ul>
<li class="toctree-l2"><a class="reference internal" href="satori-workload-manager.html#interactive-jobs">Interactive Jobs</a></li>
<li class="toctree-l2"><a class="reference internal" href="satori-workload-manager.html#batch-scripts">Batch Scripts</a><ul>
<li class="toctree-l3"><a class="reference internal" href="satori-workload-manager.html#job-states">Job States</a></li>
<li class="toctree-l3"><a class="reference internal" href="satori-workload-manager.html#monitoring-jobs">Monitoring Jobs</a></li>
<li class="toctree-l3"><a class="reference internal" href="satori-workload-manager.html#scheduling-policy">Scheduling Policy</a></li>
<li class="toctree-l3"><a class="reference internal" href="satori-workload-manager.html#batch-queue-policy">Batch Queue Policy</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="satori-troubleshooting.html">Troubleshooting</a></li>
<li class="toctree-l1"><a class="reference internal" href="satori-ai-frameworks.html">IBM Watson Machine Learning Community Edition (WMLCE)</a><ul>
<li class="toctree-l2"><a class="reference internal" href="satori-ai-frameworks.html#install-anaconda">[1] Install Anaconda</a></li>
<li class="toctree-l2"><a class="reference internal" href="satori-ai-frameworks.html#wmlce-setting-up-the-software-repository">[2] WMLCE: Setting up the software repository</a></li>
<li class="toctree-l2"><a class="reference internal" href="satori-ai-frameworks.html#wmlce-creating-and-activate-conda-environments-recommended">[3] WMLCE: Creating and activate conda environments (recommended)</a></li>
<li class="toctree-l2"><a class="reference internal" href="satori-ai-frameworks.html#wmlce-installing-all-frameworks-at-the-same-time">[4] WMLCE: Installing all frameworks at the same time</a></li>
<li class="toctree-l2"><a class="reference internal" href="satori-ai-frameworks.html#wmlce-testing-ml-dl-frameworks-pytorch-tensorflow-etc-installation">[5] WMLCE: Testing ML/DL frameworks (Pytorch, TensorFlow etc) installation</a><ul>
<li class="toctree-l3"><a class="reference internal" href="satori-ai-frameworks.html#controlling-wmlce-release-packages">Controlling WMLCE release packages</a></li>
<li class="toctree-l3"><a class="reference internal" href="satori-ai-frameworks.html#additional-conda-channels">Additional conda channels</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="satori-ai-frameworks.html#the-wml-ce-supplementary-channel-is-available-at-https-anaconda-org-powerai">The WML CE Supplementary channel is available at: https://anaconda.org/powerai/.</a></li>
<li class="toctree-l2"><a class="reference internal" href="satori-ai-frameworks.html#the-wml-ce-early-access-channel-is-available-at-https-public-dhe-ibm-com-ibmdl-export-pub-software-server-ibm-ai-conda-early-access">The WML CE Early Access channel is available at: https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/.</a></li>
</ul>
</li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Distributed Deep Learning</a></li>
<li class="toctree-l1"><a class="reference internal" href="satori-large-model-support.html">IBM Large Model Support (LMS)</a></li>
<li class="toctree-l1"><a class="reference internal" href="lsf-templates/satori-lsf-ml-examples.html">Example machine learning LSF jobs</a><ul>
<li class="toctree-l2"><a class="reference internal" href="lsf-templates/satori-lsf-ml-examples.html#a-single-node-4-gpu-keras-example">A single node, 4 GPU Keras example</a></li>
<li class="toctree-l2"><a class="reference internal" href="lsf-templates/satori-lsf-ml-examples.html#a-single-node-4-gpu-caffe-example">A single node, 4 GPU Caffe example</a></li>
<li class="toctree-l2"><a class="reference internal" href="lsf-templates/satori-lsf-ml-examples.html#a-multi-node-pytorch-example">A multi-node, pytorch example</a></li>
<li class="toctree-l2"><a class="reference internal" href="lsf-templates/satori-lsf-ml-examples.html#a-multi-node-pytorch-example-with-the-horovod-conda-environment">A multi-node, pytorch example with the horovod conda environment</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="satori-howto-videos.html">Satori Howto Video Sessions</a><ul>
<li class="toctree-l2"><a class="reference internal" href="satori-howto-videos.html#installing-wmcle-on-satori">Installing WMCLE on Satori</a></li>
<li class="toctree-l2"><a class="reference internal" href="satori-howto-videos.html#pytorch-with-ddl-on-satori">Pytorch with DDL on Satori</a></li>
<li class="toctree-l2"><a class="reference internal" href="satori-howto-videos.html#tensorflow-with-ddl-on-satori">Tensorflow with DDL on Satori</a></li>
<li class="toctree-l2"><a class="reference internal" href="satori-howto-videos.html#jupyterlab-with-ssh-tunnel-on-satori">Jupyterlab with SSH Tunnel on Satori</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="satori-doc-examples-contributing.html">Contributing documentation and examples</a></li>
</ul>

            
          
        </div>
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="index.html">MIT Satori User Documentation</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content style-external-links">
        
          















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="index.html">Docs</a> &raquo;</li>
        
      <li>Distributed Deep Learning</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
            
            
              <a href="https://github.com/mit-satori/getting-started/blob/master/satori-distributed-deeplearning.rst" class="fa fa-github"> Edit on GitHub</a>
            
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <div class="section" id="distributed-deep-learning">
<h1>Distributed Deep Learning<a class="headerlink" href="#distributed-deep-learning" title="Permalink to this headline">¶</a></h1>
<p>The following tools are libraries, which provide the communication
functions necessary to perform distributed training. Primarily allReduce
and broadcast functions.</p>
<ul class="simple">
<li>IBM Spectrum MPI: Classic tool for distributed computing. Still
commonly used for distributed deep learning.</li>
<li>NVIDIA NCCL: Nvidia’s gpu-to-gpu communication library. Since NCCL2,
between-node communication is supported.</li>
<li>IBM DDL: Provides a topology-aware all-Reduce. Capable of optimally
dividing communication across hierarchies of fabrics. Utilizes
different communication protocols at different hierarchies. When
WMLCE is installed all related frameworks are comming with IBM DDL
support, you don’t have to compile additional software packages, only
to modify your training scripts to make use of the need distributed
deep learning APIs.</li>
</ul>
<p>Integrations into deep learning frameworks to enable distributed
training is using common communication libraries such as:</p>
<ul class="simple">
<li>TensorFlow Distribution Strategies. Native Tensorflow distribution
methods.</li>
<li>IBM DDL. Provides integrations into common frameworks, including a
Tensorflow operator that integrates IBM DDL with Tensorflow and
similar for Pytorch.</li>
<li>Horovod [Sergeev et al.&nbsp;2018]. Provides integration libraries into
common frameworks which enable distributed training with common
communication libraries, including. IBM DDL can be used as backend
for Horovod implementation.</li>
</ul>
<p>IBM DDL - Documentation and Tutorial:</p>
<ol class="arabic simple">
<li>IBM <a class="reference external" href="https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_ddltf_tutorial.html" target="_blank">DDL integration with
TensorFlow/Keras</a></li>
<li>IBM <a class="reference external" href="https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_ddlpytorch_tutorial.html" target="_blank">DDL integration with
Pytorch</a></li>
<li>IBM <a class="reference external" href="https://developer.ibm.com/linuxonpower/2018/08/24/distributed-deep-learning-horovod-powerai-ddl/" target="_blank">DDL integration with
Horovod</a></li>
<li>IBM <a class="reference external" href="https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_ddlapi.html" target="_blank">DDL
APIs</a>
for a better integration</li>
</ol>
<p>Examples:</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/IBM/powerai/tree/master/examples/tensorflow_large_model_support/v2" target="_blank">Keras/TensorFlow</a></li>
<li>Pytorch</li>
</ul>
<p>How to get Horovod with DDL? follow bellow instructions (optional 0 - 2 if you have already install WMLCE):</p>
<ol class="arabic simple" start="0">
<li>Add ppc64le conda channel for WMLCE</li>
</ol>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>conda config --prepend channels <span class="se">\</span>
https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
</pre></div>
</div>
<ol class="arabic simple">
<li>Create Conda Virtual Environment</li>
</ol>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>conda create --name horovod <span class="nv">python</span><span class="o">=</span><span class="m">3</span>.6
</pre></div>
</div>
<ol class="arabic simple" start="2">
<li>Install WMLCE (TF, Pytorch, DDL etc)</li>
</ol>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>conda install powerai
</pre></div>
</div>
<ol class="arabic simple" start="3">
<li>Install the packages to build Horovod</li>
</ol>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>conda install gxx_linux-ppc64le<span class="o">=</span><span class="m">7</span>.3.0 cffi cudatoolkit-dev
</pre></div>
</div>
<ol class="arabic simple" start="4">
<li>Install Horovod with DDL backend</li>
</ol>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nv">HOROVOD_CUDA_HOME</span><span class="o">=</span><span class="nv">$CONDA_PREFIX</span> <span class="nv">HOROVOD_GPU_ALLREDUCE</span><span class="o">=</span>DDL pip install horovod --no-cache-dir
</pre></div>
</div>
<p>Original IBM DDL paper, can be found at this URL:
<a class="reference external" href="https://arxiv.org/pdf/1708.02188.pdf" target="_blank">https://arxiv.org/pdf/1708.02188.pdf</a></p>
</div>


           </div>
           
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="satori-large-model-support.html" class="btn btn-neutral float-right" title="IBM Large Model Support (LMS)" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
      
      
        <a href="satori-ai-frameworks.html" class="btn btn-neutral float-left" title="IBM Watson Machine Learning Community Edition (WMLCE)" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
      
    </div>
  

  <hr/>

  <div role="contentinfo">
    <p>
        &copy; Copyright 2019, MIT Satori Project

    </p>
  </div>
  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  


  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>