Related work

• mutil, shift
  • Paul Kolano
  • Lustre-aware versions of GNU coreutils
  • multi-thread and multi-node
  • LUG11 talk
  • Transparent Optimization of Parallel File System I/O via Standard System Tool Enhancement. May 2013
  • High Performance Multi-Node File Copies and Checksums for Clustered File Systems. Nov 2010
  • Advanced Features of the Shift Automated File Transfer Tool
  • High Performance Reliable File Transfers Using Automatic Many-to-Many Parallelization Aug 2012
• SPDCP and PLTAR - MPI parallel copy and tar commands
  • page 14 in slides
  • spdcp broken links
  • A first look at scalable I/O in Linux commands 2008
• fsmr
  • fsmr is for running a mapreduce algorithm where the inputs are every entity in a filesystem hierarchy. It is a combination of two excellent pieces of software -- libcircle/libdftw (for the map) and libmrmpi (for the reduce). It is a distributed algorithm using MPI and intended to run on clusters with storage that scales well under such applications.
• e2scan
• ne2scan, fsfind
  • [Nick Cardo] (http://www.nersc.gov/about/nersc-staff/computational-systems-group/nick-cardo/)
  • Reaping the Benefits of MetaData LUG'10.
  • Monitoring Tools for Large Scale Systems Galen Shipman 2010
  • Improving Management of Large Lustre File Systems David Dillow LUG'11
• libcircle + {dcp, dftw, dgrep, purger, mpiwalk}
  • distributed queue and gossip-type load balancing to implement parallel copy, file walk, and grep
• Asynchronous Dynamic Load Balancing
• pftool
  • implements parallel list, copy, and compare using master/worker
  • optimizations for PLFS files
• robinhood
  • caches file system metadata in MySQL database so fast queries can replace some directory walks
  • specialized for Lustre
  • not parallel but utility we may integrate with
  • could use better parallel tools for initial scan and processing of file lists
• xCAT
  • parallel tools for cloud system management