README.Install

Installing HPCToolkit with Spack
================================

1 Introduction
2 Prerequisites
3 Spack Notation
4 Clone Spack and HPCToolkit
5 Config.yaml and Install Roots
6 Packages.yaml
6.1 External Packages
6.2 Micro-Architecture Targets
7 Compilers and compilers.yaml
8 Spack Install
9 Manual Install
10 Advanced Options
10.1 CUDA
10.2 ROCM
10.3 MPI
10.4 PAPI vs Perfmon
11 Platform Specific Notes
11.1 Blue Gene
11.2 Cray
12 HPCToolkit GUI Interface (Hpcviewer)
12.1 Spack Install
12.2 Manual Install
13 Building a New Compiler
13.1 Using the New Compiler
13.2 Bootstrapping Environment Modules
14 Spack Mirrors
15 Common Problems
15.1 Unable to fetch tar file
15.2 New releases break the build
15.3 Failure to load modules
    

1 Introduction
==============

These notes describe how to build and install HPCToolkit and hpcviewer
and their prerequisites with Spack.  HPCToolkit proper (hpcrun,
hpcstruct and hpcprof) is used to measure and analyze an application's
performance and then produce a database for hpcviewer.  HPCToolkit is
supported on the following platforms.  IBM Blue Gene is no longer
supported.

  1. Linux (64-bit) on x86_64, little-endian powerpc (power8 and 9) and
     ARM (aarch64).  Big endian powerpc is no longer supported.

  2. Cray XT/XE/XK/XC on x86_64 and Compute Node Linux.

We provide binary distributions for hpcviewer and hpctraceviewer on
Linux (x86_64, ppc64/le and aarch64), Windows and MacOS. HPCToolkit
databases are platform-independent and it is common to run hpcrun on one
machine and then view the results on another machine.

   We build HPCToolkit and its prerequisite libraries from source.
HPCToolkit has some 15-20 prerequisites and we now use spack to build
them.  It is possible to use spack to install all of hpctoolkit or build
just the prerequisites and then build hpctoolkit with the traditional
'configure ; make ; make install' method from autotools.  Developers
will probably want to run 'configure' and 'make' manually, but both
methods are supported.

   Note: the old method of using hpctoolkit-externals to build the
prerequisite libraries is now superseded by spack and is no longer
supported.

   These notes are written mostly from the view of using spack to build
hpctoolkit and its dependencies.  If you are a more experienced spack
user, especially if you want to use spack to build hpctoolkit plus
several other packages, then you will want to adapt these directions to
your own needs.

   Spack documentation is available at:

     <https://spack.readthedocs.io/en/latest/index.html>

The current status of using Spack for HPCToolkit is at:

     <http://hpctoolkit.org/spack-issues.html>

Last revised: July 13, 2021.

2 Prerequisites
===============

Building HPCToolkit requires the following prerequisites.

  1. The GNU gcc and g++ compilers version 7.3 or later.  On systems
     with older compilers, you can use spack to build a later version of
     gcc.

  2. GNU glibc version 2.16 or later.  Note: Red Hat 6.x uses glibc 2.12
     which is too old.

  3. Basic build tools: make, ld, ar, objcopy, nm, etc, and shell
     utilities: bash, sed, awk, grep, etc.  Most Linux systems have
     these tools, or else you couldn't compile anything.

  4. Cmake version 3.4 or later, perl version 5.x, and python version
     3.4 or later.  On systems that are missing these tools or have
     versions that are too old, you can use spack to build a later
     version.

  5. Git and curl for downloading files.

  6. (optional) Environment (TCL) or LUA (Lmod) modules, if you want to
     make HPCToolkit available as a module.  Again, spack can install
     these packages if they are missing from your system.

Hpcviewer and hpctraceviewer require Java version 8 or 11.  Starting
with version 2021.01, the viewers now need Java 11 or later (but not
15).  Up through 2020.12, the viewers need Java 8 (not later).  On
Linux, the viewers also require GTK+ version 3.20 or later starting with
2021.01 and GTK+ version 2.x up through 2020.12.

3 Spack Notation
================

Spack uses a special notation for specifying the version, variants,
compilers and dependencies when describing how to build a package.  This
combination of version, variants, etc is called a 'spec' and is used
both on the command line and in config files.

  1. '@' specifies the package version.  'spack info <package>' shows
     the available versions and variants for a package.  In most cases,
     spaces are optional between elements of a spec.  For example:

          boost@1.66.0    dyninst @10.1.0    hpctoolkit @master

  2. '+', '-', '~' specify boolean (on/off) variants.  Note: '-' (dash)
     and '~' (tilde) both mean 'off'.  Use dash after a space and tilde
     after a non-space.  For example:

          elfutils+bzip2~nls    elfutils +bzip2 -nls    elfutils@0.176 +bzip2~nls

  3. 'name=value' specifies a non-boolean variant, for example:

          dyninst+openmp build_type=RelWithDebInfo    xerces-c@3.2.2 transcoder=iconv

  4. '%' specifies the build compiler and its version, for example:

          hpctoolkit@master %gcc@7.3.0

  5. 'cflags', 'cxxflags', 'fflags', 'cppflags', 'ldflags' and 'ldlibs'
     are special name/value variants for compiler flags.  These are
     normally not needed, but if you do need to add a flag to the build,
     then one example might be:

          amg2013 cflags='-O2 -mavx512pf'

  6. '^' represents a dependency spec.  The spec for a dependency
     package is a full spec and may include its own version, variants,
     etc.  For example:

          hpctoolkit@master ^dyninst@10.1.0+openmp

  7. 'arch', 'platform', 'os' and 'target' are special options for the
     system architecture and machine type.  Platform is normally
     'linux', or else 'cray' (or even 'darwin').  OS is the Linux
     distribution, 'rhel7', 'sles15', etc, and target is the machine
     type, 'x86_64', 'ppc64le', etc.  Arch is a triple of platform, os
     and target separated by dashes.

     Normally, a system has only one arch type and you don't need to
     specify this.  However, for systems with separate front and
     back-end types, the default is the back end.  For example, if you
     wanted to build python for the front end on Cray, then you might
     use something like this.

          python@3.7.4 arch=cray-sles15-x86_64

     Also, now that spack has implemented microarchitecture targets
     (haswell, ivybridge, etc), you can use 'target' to build for a
     generic x86_64 or a specific CPU type.  For example:

          amg2013 target=x86_64    lulesh target=ivybridge

When writing a spec (for 'spack install' or other), spack will fully
resolve all possible choices for the package and all of its dependencies
and create a unique hash value for that exact configuration.  This
process is called 'concretization.'  To see how spack would concretize a
spec, use 'spack spec'.

     spack spec hpctoolkit@master ^binutils@2.34 ^boost@1.72.0

     <https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies>

4 Clone Spack and HPCToolkit
============================

Spack is available via git clone from GitHub.  This includes the core
spack machinery and recipes for building more than 5,000 packages (and
growing).  You should also clone HPCToolkit for the 'packages.yaml' file
which is used to configure the spack build.

     git clone https://github.com/spack/spack.git
     git clone https://github.com/hpctoolkit/hpctoolkit.git

After cloning, add the 'spack/bin' directory to your PATH, or else
source the spack 'setup-env' script.

     (bash)   .  /path/to/spack/share/spack/setup-env.sh
     (csh)    setenv SPACK_ROOT /path/to/spack/root
              source $SPACK_ROOT/share/spack/setup-env.csh

It suffices to add 'spack/bin' to your PATH (or even symlink the spack
launch script).  Sourcing the 'setup-env' script adds extra support for
modules built by spack.

5 Config.yaml and Install Roots
===============================

'config.yaml' is the top-level spack config file.  This specifies the
directory layout for installed files and the top-level spack parameters.
There are two fields that you normally want to set, especially if you
want to install packages and modules for hpctoolkit in a public
directory.  For a module to be available, both the 'install_tree' and
'module_roots' directories must be accessible.

   By default, spack installs packages inside the spack repository at
'spack/opt/spack'.  To use another location, set the 'root' field under
'install_tree' in 'config.yaml'.

     config:
       install_tree:
         root: /path/to/top-level/install/directory

By default, spack installs module files inside the spack repository at
'spack/share/spack'.  The syntax for resetting this is changing.  In the
old syntax, you put the top-level directory in 'config.yaml' and enable
the module type(s) in 'modules.yaml'.

     config:
       module_roots:
         # normally, need only one of these
         tcl:  /path/to/top-level/tcl-module/directory
         lmod: /path/to/top-level/lmod-module/directory

     modules:
       enable:
        - tcl  (or lmod)

In the new syntax, the paths go in 'modules.yaml', and there is one
extra step in the hierarchy ('default:').

     modules:
       default:
         roots:
           # normally, need only one of these
           tcl:  /path/to/top-level/tcl-module/directory
           lmod: /path/to/top-level/lmod-module/directory
         enable:
          - tcl  (or lmod)

There are two other fields that you may want to set for your local
system.

  1. 'connect_timeout' - some download sites, especially sourceforge are
     often slow to connect.  If you find that connections are timing
     out, then increase this time to 30 or 60 seconds (default is 10
     seconds).

  2. 'build_jobs' - by default, spack uses all available hardware
     threads for parallel make, up to a limit of 16.  If you want to use
     a different number, then set this.

There are also parameters for the locations of the build directories,
the cache of downloaded tar files, etc, which you may wish to set.

   The default 'config.yaml' file is in the spack repository at
'spack/etc/spack/defaults'.  The simplest solution is to copy this file
one directory up and then edit the copy (don't edit the default file
directly).

     cd spack/etc/spack
     cp defaults/config.yaml .
     vi config.yaml

Alternatively, you could put this file in a separate directory, outside
of the spack repository and then use '-C dir' on the spack command line.
(The '-C' option goes before the spack command name.)

     spack -C dir install ...

Note: if you put 'config.yaml' in 'spack/etc/spack', then it will apply
to every spack command for that repository (and you won't forget).
Putting it in a separate directory is more flexible because you can
support multiple configurations from the same repository.  But then you
must use '-C dir' with every spack command or else you will get
inconsistent results.

   You can view the current configuration with 'spack config'.

     spack [-C dir] config get config

See the spack docs on 'Configuration Files' and 'Basic Settings'.

     <https://spack.readthedocs.io/en/latest/configuration.html>
     <https://spack.readthedocs.io/en/latest/config_yaml.html>

6 Packages.yaml
===============

The 'packages.yaml' file specifies the versions and variants for the
packages that spack installs and serves as a common reference point for
HPCToolkit's prerequisites.  This file also specifies the paths or
modules for system build tools (cmake, python, etc) to avoid rebuilding
them.  Put this file in the same directory as 'config.yaml'.  A sample
'packages.yaml' file is available in the 'spack' directory of the
hpctoolkit repository.

   There are two main sections to 'packages.yaml'.  The first specifies
the versions and variants for hpctoolkit's prereqs.  By default, spack
will choose the latest version of each package (plus any constraints
from hpctoolkit's 'package.py' file).  In most cases, this will work,
but not always.  If you need to specify a different version or variant,
then set this in 'packages.yaml'.

   Note: the versions and variants specified in hpctoolkit's
'package.py' file are hard constraints and should not be changed.
Variants in 'packages.yaml' are preferences that may be modified for
your local system.  (But don't report a bug until you have first tried
the versions from 'packages.yaml' that we supply.)

   There are at least two packages with a variant that you may need to
change depending on your system.  But always check the current
'packages.yaml' file to see if any more have been added.

  1. 'binutils' - avoid versions 2.35 and 2.35.1, they have a bug that
     spews errors from hpcprof.  Use 2.34 until 2.36 is available.

  2. 'intel-tbb' - for very old Intel or AMD systems that don't support
     transactional memory, change '+tm' to '~tm'.  (This option has no
     effect on non-x86 systems.)

6.1 External Packages
---------------------

The other sections in 'packages.yaml' specify paths or modules for other
packages and system build tools.  Building hpctoolkit's prerequisites
requires cmake 3.4 or later, perl 5.x and python 3.4 or later.  There
are three ways to satisfy these requirements: a system installed version
(eg, /usr), a pre-built module or build from scratch.

   By default, spack will rebuild these from scratch, even if your local
version is perfectly fine.  If you already have an installed version and
prefer to use that instead, then you can specify this in
'packages.yaml'.

   For example, this entry says that cmake 3.7.2 is available from
module 'CMake/3.7.2' and that spack should use this instead of building
its own copy.  You may use more than one spec per package and more than
one module per spec, if needed.  'buildable: False' is optional and
means that spack must find a matching external spec or else fail the
build.

     cmake:
       externals:
       - spec: cmake@3.7.2
         modules:
         - CMake/3.7.2
       buildable: False

This example says that perl v5.16.3 is installed at '/usr/bin/perl'.
Note that the 'prefix' entry is the parent directory of 'bin', not the
bin directory itself.

     perl:
       externals:
       - spec: perl@5.16.3
         prefix: /usr
       buildable: False

Note that you may specify more than one spec for a package.  For
example, your system might have both python2 and python3.

     python:
       externals:
       - spec: python@3.6.5
         prefix: /usr
       - spec: python@2.7.15
         prefix: /usr
       buildable: False

6.2 Micro-Architecture Targets
------------------------------

Spack implements a hierarchy of micro-architecture targets, where
'target' is a specific architecture (eg, haswell, ivybridge) instead of
a generic family (x86_64).  This allows the compiler to optimize code
for the specific target.

   You will notice this choice in two main places: the 'spack spec' and
the path for the install directory.  For example, 'linux-rhel7-x86_64'
might become 'linux-rhel7-broadwell'.  You can use 'spack arch' to see
the list of generic families and micro-architecture targets.

     spack arch --known-targets

If you prefer a generic install, you can use the 'target' option to
specify a generic family instead of a micro-architecture target.  This
would be useful for a shared install that needs to work across multiple
machines with different micro-arch types.  For example:

     spack install hpctoolkit ... target=x86_64

If you want to use this target for all spack builds, then you can
specify this in 'packages.yaml'.  The generic families are: x86_64,
ppc64, ppc64le and aarch64.  For example:

     packages:
       all:
         target: [x86_64]

See the spack docs on 'Build Customization' and 'Specs and
Dependencies'.

     <https://spack.readthedocs.io/en/latest/build_settings.html>
     <https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies>

7 Compilers and compilers.yaml
==============================

Building HPCToolkit requires GNU gcc/g++ version 7.3 or later.  By
default, spack uses the latest available version of gcc, but you can
specify a different compiler, if one is available.

   Spack uses a separate file, 'compilers.yaml' to store information
about available compilers.  This file is normally in your home directory
at '~/.spack/platform' where 'platform' is normally 'linux' (or else
'cray' or 'bgq').

   The first time you use spack, or after adding a new compiler, you
should run 'spack compiler find' to have spack search your system for
available compilers.  If a compiler is provided as a module, then you
should load the module before running 'find'.  Normally, you only need
to run 'find' once, unless you want to add or delete a compiler.  You
can also run 'spack compiler list' and 'spack compiler info' to see what
compilers spack knows about.

   For example, on one power8 system running RedHat 7.3, /usr/bin/gcc is
version 4.8.5, but gcc 8.3.0 is available as module 'GCC/8.3.0'.

     module load GCC/8.3.0

     spack compiler find
     ==> Added 2 new compilers to /home/krentel/.spack/linux/compilers.yaml
         gcc@8.3.0  gcc@4.8.5
     ==> Compilers are defined in the following files:
         /home/krentel/.spack/linux/compilers.yaml

     spack compiler list
     ==> Available compilers
     -- gcc rhel7-ppc64le --------------------------------------------
     gcc@8.3.0  gcc@4.8.5

     spack compiler info gcc@8.3
     gcc@8.3.0:
         paths:
             cc = /opt/apps/software/Core/GCCcore/8.3.0/bin/gcc
             cxx = /opt/apps/software/Core/GCCcore/8.3.0/bin/g++
             f77 = /opt/apps/software/Core/GCCcore/8.3.0/bin/gfortran
             fc = /opt/apps/software/Core/GCCcore/8.3.0/bin/gfortran
         modules  = ['GCC/8.3.0']
         operating system  = rhel7

Note: for compilers from modules, spack does not fill in the 'modules:'
field in the 'compilers.yaml' file.  You need to do this manually.  In
the above example, after running 'find', I edited 'compilers.yaml' to
add 'GCC/8.3.0' to the 'modules:' field as below.  This is important to
how spack manipulates the build environment.

     - compiler:
         modules: [GCC/8.3.0]
         operating_system: rhel7
         spec: gcc@8.3.0
         ...

Spack uses '%' syntax to specify the build compiler and '@' syntax to
specify the version.  For example, suppose you had gcc versions 7.3.1,
8.3.0 and 10.2.0 available and you wanted to use 8.3.0.  You could write
this as:

     spack install package %gcc@8.3.0

See the spack docs on 'Compiler Configuration'.

     <https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration>

8 Spack Install
===============

First, set up your 'config.yaml', 'packages.yaml' and 'compilers.yaml'
files as above and edit them for your system.  You can see how spack
will build hpctoolkit with 'spack spec'.

     spack spec hpctoolkit

Then, the "one button" method uses spack to install everything.

     spack install hpctoolkit

Tip: Spack fetch is somewhat fragile and often has transient problems
downloading files.  You can use 'spack fetch -D' to pre-fetch all of the
tar files and resolve any downloading problems before starting the full
install.

     spack fetch -D hpctoolkit

9 Manual Install
================

The manual method uses spack to build hpctoolkit's prerequisites and
then uses the traditional autotools 'configure && make && make install'
to install hpctoolkit.  This method is primarily for developers who want
to compile hpctoolkit, edit the source code and recompile, etc.  This
method is also useful if you need some configure option that is not
available through spack.

   First, use spack to build hpctoolkit's prerequisites as above.  You
can either build some version of hpctoolkit as before (which will pull
in the prerequisites), or else use '--only dependencies' to avoid
building hpctoolkit itself.

     spack install --only dependencies hpctoolkit

Then, configure and build hpctoolkit as follows.  Hpctoolkit uses
automake and so allows for parallel make.

     configure  \
        --prefix=/path/to/hpctoolkit/install/prefix  \
        --with-spack=/path/to/spack/install_tree/linux-fedora26-x86_64/gcc-7.3.1  \
        ...
     make -j <num>
     make install

The argument to '--with-spack' should be the directory containing all of
the individual package directories, normally two directories down from
the top-level 'install_tree' and named by the platform and compiler.
This option replaces the old '--with-externals'.  The following are
other options that may be useful.  For the full list of options, see
'configure -h'.

  1. '--enable-all-static' - build hpcprof-mpi statically linked for the
     compute nodes.

  2. '--enable-bgq' - configure for Blue Gene/Q back end.

  3. '--enable-develop' - compile with optimization turned off for
     debugging.

  4. '--with-package=path' - specify the install prefix for some
     prerequisite package, mostly for developers who want to use a
     custom, non-spack version of some package.

  5. 'MPICXX=compiler' - specify the MPI C++ compiler for hpcprof-mpi,
     may be a compiler name or full path.

Note: if your spack install tree has multiple versions or variants for
the same package, then '--with-spack' will select the one with the most
recent directory time stamp (and issue a warning).  If this is not what
you want, then you will need to specify the correct version with a
'--with-package' option.

10 Advanced Options
===================

10.1 CUDA
---------

Beginning with the 2020.03.01 version, HPCToolkit now supports profiling
CUDA binaries (nVidia only).  For best results, use CUDA version 10.1 or
later and Dyninst 10.1 or later.  Note: in addition to a CUDA
installation, you also need the CUDA system drivers installed.  This
normally requires root access and is outside the scope of spack.

   For a spack install with CUDA, use the '+cuda' variant.

     spack install hpctoolkit +cuda

For a manual install, either download and install CUDA or use an
existing module, and then use the '--with-cuda' configure option.

     configure  \
        --prefix=/path/to/hpctoolkit/install/prefix  \
        --with-spack=/path/to/spack/install/dir   \
        --with-cuda=/path/to/cuda/install/prefix  \
        ...

If you installed CUDA with spack in the same directory as the rest of
the prerequisites, then the '--with-spack' option should find it
automatically (but check the summary at the end of the configure
output).  If you are using CUDA from a separate system module, then you
will need the '--with-cuda' option.

10.2 ROCM
---------

Beginning with version 2021.02.01, HPCToolkit now supports profiling AMD
GPU binaries through the HIP/ROCM interface.  This requires ROCM 3.10 or
later and also requires matching the HPCToolkit ROCM version to the
version the application uses.

   The spack build with ROCM support requires a 2-step build.  First,
build the ROCM prerequisites (hip, rocm-dbgapi and roctracer-dev), and
second, specify them as spack externals packages and install hpctoolkit
with '+rocm'.

   The easiest way is to install the AMD package for ROCM in '/opt'.
Normally, this requires root access.  For example, suppose ROCM 4.0.0 is
installed in '/opt/rocm-4.0.0'.  In this case, edit 'packages.yaml' or
'spack.yaml' to specify the three spack packages.

     packages:
       hip:
         externals:
         - spec: hip@4.0.0
           prefix: /opt/rocm-4.0.0/hip

       rocm-dbgapi:
         externals:
         - spec: rocm-dbgapi@4.0.0
           prefix: /opt/rocm-4.0.0

       roctracer-dev:
         externals:
         - spec: roctracer-dev@4.0.0
           prefix: /opt/rocm-4.0.0

Then, build HPCToolkit with the '+rocm' variant.

     spack install hpctoolkit +rocm ...

Note: the hip prefix requires the hip subdirectory, but the other two
use the top-level '/opt/rocm' prefix.  This is somewhat fluid and may
change by system.

   If '/opt/rocm' is not available or is missing some of the pieces,
then build the missing packages with spack and again add them to
'packages.yaml'.  Note: you must build the packages one at a time as
separate roots.  This will necessarily build some packages twice.  For
example:

     hip:
       externals:
       - spec: hip@4.0.0
         prefix: /home/krentel/spack/linux-rhel8-x86_64/gcc-8.3.1/hip-4.0.0-qzauqkhwrfmpa6j37cpzysfpgipvhkmj

Note: for now, you must use the 2-step build and specify the ROCM
packages as spack externals, even if building them with spack.  There
are two conflicts in the 'spack spec' for an all-in-one build.
Roctracer-dev requires python 2, but other packages require python 3,
and rocm-dbgapi requires libelf but dyninst requires elfutils.  Writing
the packages as externals works around this problem.

   For developers building with autotools, use the following configure
options.  If '/opt/rocm' is available, then use just the '--with-rocm'
option.  Otherwise, use the three other options.

     --with-rocm=/opt/rocm    (for all-in-one /opt/rocm)

     --with-rocm-hip=/path/to/hip/install
     --with-rocm-dbgapi=/path/to/rocm-dbgapi/install
     --with-rocm-tracer=/path/to/roctracer-dev/install

It is allowed to mix the all-in-one option with the individual packages.
The rule is that the specific overrides the general.

10.3 MPI
--------

HPCToolkit always supports profiling MPI applications.  For hpctoolkit,
the spack variant '+mpi' is for building hpcprof-mpi, the MPI version of
hpcprof.  If you want to build hpcprof-mpi, then you need to supply an
installation of MPI.

   Normally, for systems with compute nodes, you should use an existing
MPI module that was built for the correct interconnect for your system
and add this to 'packages.yaml'.  The MPI module should be built with
the same version of GNU gcc/g++ used to build hpctoolkit (to keep the
C++ libraries in sync).

10.4 PAPI vs Perfmon
--------------------

HPCToolkit can access the Hardware Performance Counters with either PAPI
or Perfmon (libpfm4).  By default, the hpctoolkit package uses perfmon.
If you want to use PAPI instead, then build hpctoolkit with '+papi'.
However, you can't use both due to a potential conflict in their header
files.

   PAPI runs on top of the perfmon library, but PAPI uses its own,
internal copy of perfmon.  Prior to version 5.6.0, PAPI did not install
the perfmon header files, so it was impossible to access the perfmon
events through PAPI.

   However, starting with version 5.6.0, PAPI now installs both the
perfmon library and its header files.  Hpctoolkit configure will
automatically detect this, so if you build hpctoolkit with a recent
enough version of PAPI, then both the PAPI and perfmon interfaces will
be available.

11 Platform Specific Notes
==========================

11.1 Blue Gene
--------------

Support for Blue Gene/Q is now deprecated and versions 2020.03.01 and
later will no longer build.  If you want to install HPCToolkit on Blue
Gene, then use spack version 2019.12.28 or git commit 0d91fb34edb2 with
gcc 4.8.x and follow the directions below.  These directions will also
work (without Blue Gene) for Red Hat 6.x (glibc 2.12 is too old to
support current hpctoolkit).

   Blue Gene systems are being phased out, without a next generation
replacement, but are still supported by hpctoolkit (for now).  Blue Gene
normally comes with a Red Hat Enterprise Linux (RHEL) 6.x front end and
the GNU gcc/g++ 4.4 compilers with are too old to support the latest
version of hpctoolkit.

   Spack supports two architecture types on Blue Gene.  The front-end
arch is 'bgq-rhel6-ppc64' and the back-end arch is 'bgq-cnk-ppc64'.
Normally, it would matter whether we built for the front end or back
end.  But for the GNU compilers on Blue Gene, both compilers are
/usr/bin/gcc.  So, we just build for the default back end.

   There are two ways to build hpctoolkit for Blue Gene, depending on
the version of the compiler and Dyninst.  The simple method uses the
default GCC 4.4 compiler and Dyninst 9.3.2.  Edit these two entries in
the 'packages.yaml' file.

  1. Set the dyninst version to '9.3.2'.

  2. Add '+bgq' to the libmonitor variants.

Then, build hpctoolkit with '+bgq' turned on.  The '+bgq' variant adds
MPI and hpcprof-mpi and supersedes the '+mpi' variant.

     spack install hpctoolkit +bgq

The advanced method involves building a new compiler (gcc 4.8 or 4.9)
but allows using Dyninst 10.x.  This method requires TCL or Lmod modules
and an MPI C++ compiler built for a compatible version of g++.  Mira and
vesta at ANL have softenv modules for +mpiwrapper-gcc and
+bgqtoolchain-gcc484 but don't provide TCL or Lmod modules.

   First, use spack to build a new compiler.  Dyninst 10.x requires
gcc 4.8 or later, but libmonitor requires the true back-end
'powerpc64-bgq-linux-gcc' compiler which is version 4.4.  We settle on
gcc 4.8.5 as a compromise between these two constraints.  Edit
'config.yaml' to set 'module_roots' to a directory in which to install
the modules and then build gcc.

     spack install gcc @4.8.5

Assuming you have modules installed, then load the new gcc module and
rerun 'spack compiler find'.  This should add the new compiler to your
'compilers.yaml' file (twice, for front and back end).  For example, on
vulcan at LLNL,

     module use /path/to/modules/bgq-cnk-ppc64
     module load gcc-4.8.5-gcc-4.4.7-nzvjwva
     spack compiler find
     ==> Added 2 new compilers to /g/g21/krentel1/.spack/bgq/compilers.yaml
         gcc@4.8.5  gcc@4.8.5

Note: after running 'spack compiler find', you still need to add the
modules field to the compiler entry in 'compilers.yaml' as in the
section on Compilers.

   Note: in the above example, we used gcc 4.4.7 to build gcc 4.8.5 (so
the gcc-4.8.5 package is in the gcc-4.4.7 directory).  You could then
use gcc 4.8.5 to rebuild 4.8.5, if you like, but this is not necessary.

   Finally, use the new compiler to build hpctoolkit with the new
compiler.  If you have modified the dyninst entry in 'packages.yaml',
then reset the dyninst version to 10.1.0 (or later).

     spack install hpctoolkit +bgq %gcc@4.8.5

If you don't have modules installed, then use 'spack bootstrap' to build
the environment-modules package.  Then, source the bash or csh script in
the 'Modules/init' directory to add the 'module' function to your
environment.  For example,

     spack bootstrap
     cd /path/to/environment-modules-4.2.4-rwtvlanuss35fxd3xuyldbfopt2m4ecs/init
     (bash)  . ./bash
     (csh)   source ./csh

Alternatively, you could try adding the module environment variables
(PATH and LD_LIBRARY_PATH) manually, but then you may have to run spack
install with '--dirty'.  The '--dirty' option tells spack not to erase
LD_LIBRARY_PATH while building packages.

     spack install --dirty hpctoolkit +bgq %gcc@4.8.5

For developers, if you are building hpctoolkit directly (outside of
spack) but using spack prerequisites, then use a configure line similar
to the following.

     configure  \
         --prefix=/path/to/install/prefix  \
         --with-spack=/path/to/bgq-cnk-ppc64/gcc-4.8.5  \
         --enable-bgq  \
         --enable-all-static  \
         MPICXX=mpicxx

11.2 Cray
---------

Cray systems, like Blue Gene, have separate front and back-end
architecture types.  For example, on theta at ANL, the front-end arch is
'cray-sles12-haswell' (SuSE Linux version 12 for haswell) and the
back-end is 'cray-cnl6-mic_knl' (Compute Node Linux for KNL).

   Hpctoolkit needs to be built with the GNU Programming Environment and
the front-end x86_64 compilers, plus the 'CC' MPI C++ wrapper.  Switch
to the 'PrgEnv-gnu' module and unload the darshan module.  Darshan is a
profiling tool that monitors an application's use of I/O, but it
conflicts with hpctoolkit.

     module swap PrgEnv-intel PrgEnv-gnu
     module unload darshan

By default, spack does not correctly identify the gcc compiler modules
as front-end compilers.  Start by loading a recent gcc module (eg, 7.x
or 8.x) and try running 'spack compiler find' and 'spack compiler list'.
For example on theta at ANL,

     module load gcc/8.3.0
     spack compiler find /opt/gcc/8.3.0

     spack compiler list
     ==> Available compilers
     -- gcc cnl6-any -------------------------------------------------
     gcc@8.3.0  gcc@8.1.0  gcc@7.2.0  gcc@6.3.0  gcc@5.3.0
     gcc@8.2.0  gcc@7.3.0  gcc@7.1.0  gcc@6.1.0  gcc@4.9.3

     -- gcc sles12-x86_64 --------------------------------------------
     gcc@8.3.0  gcc@4.8

If spack identifies the compiler (as above), then great.  Note that we
need spack to accept this as a front-end compiler (sles12-x86_64), not
just for the back end (cnl6-any).  If spack doesn't find the compiler
automatically, then you need to add it manually.  Edit
'~/.spack/cray/compilers.yaml' and add an entry similer to the following
(your versions may differ).

     - compiler:
         environment: {}
         flags: {}
         modules:
         - PrgEnv-gnu/6.0.5
         - gcc/8.3.0
         - craype/2.6.1
         - cray-mpich/7.7.10
         operating_system: sles12
         paths:
           cc:  /opt/gcc/8.3.0/bin/gcc
           cxx: /opt/gcc/8.3.0/bin/g++
           f77: /opt/gcc/8.3.0/bin/gfortran
           fc:  /opt/gcc/8.3.0/bin/gfortran
         spec: gcc@8.3.0
         target: x86_64

Regardless of whether spack created the entry or you added it, you still
need to add the 'modules:' field manually (be sure to include all four
modules).  Although it takes several steps to create the correct entry,
the good news is that it should continue to work as long as the
underlying module exists.

   Next, review your 'packages.yaml' file.  On Cray systems with Xeon
Phi back-end nodes (KNL, KNH, etc), add '~tm' to intel-tbb to disable
transactional memory.

   Finally, build hpctoolkit with the front-end arch type (eg,
cray-sles12-x86_64) and option '+cray'.  Normally, you can use OS type
'fe' (front-end) in place of 'sles12'.  As with Blue Gene, the '+cray'
option adds MPI and hpcprof-mpi and supersedes the '+mpi' variant.

     spack install hpctoolkit +cray arch=cray-fe-x86_64

On some systems, spack fails to properly load the compiler modules
during the build and fails during hpctoolkit configure with an error
about, "MPICXX is not a valid compiler."  In this case, make sure that
you have the PrgEnv-gnu and gcc modules loaded and retry the install
with '--dirty'.

     spack install --dirty hpctoolkit +cray arch=cray-fe-x86_64

For developers, if you are building hpctoolkit directly (outside of
spack) but using spack prerequisites, then use a configure line similar
to the following.

     configure  \
         --prefix=/path/to/install/prefix  \
         --with-spack=/path/to/cray-sles12-x86_64/gcc-8.3.0  \
         --enable-all-static  \
         MPICXX=CC

12 HPCToolkit GUI Interface (Hpcviewer)
=======================================

Since 2021.04 release, HPCToolkit GUI interface provides both profile
and trace views in a single application, i.e.  hpcviewer.  Prior to
that, each view is a separate program: hpcviewer to analyze the profile
database, and hpctraceviewer to display the traces.

   We provide binary distributions for hpcviewer on Linux (x86_64,
ppc64/le and now aarch64), Windows and MacOS. HPCToolkit databases are
platform-independent and it is common to run hpcrun on one machine and
then view the results on another machine.

   Starting with 2021.01, the viewer now requires Java 11 or later (but
not 15), plus GTK+ 3.20 or later on Linux.  Older viewers, up through
2020.12, require Java 8 (not 9 or later), plus GTK+ 2.x for Linux.

12.1 Spack Install
------------------

The spack install is available on Linux x86_64, little-endian powerpc64
(power8 and 9) and now aarch64 ARM. This installs hpcviewer and includes
the Java prerequisite.

   For the current viewers, use openjdk with the most recent version of
Java 11 for all platforms.  Currently, this is the default, but if not,
then you can add an explicit openjdk dependency.  You can check this
with 'spack info' and 'spack spec'.

     spack info openjdk
     spack install hpcviewer
     spack install hpcviewer ^openjdk @11.0.8_10

For older viewers, up through 2020.12, the choice of Java depends on the
platform.  On x86_64, use openjdk, the most recent version of Java 8
(not 9).  For powerpc, use ibm-java, again the latest version 8.  Older
viewers are not supported on ARM. For example (the versions may change):

     (x86)  spack install hpcviewer ^openjdk @1.8.0_265-b01
     (ppc)  spack install hpcviewer ^ibm-java @8.0.6.20

12.2 Manual Install
-------------------

Binary distributions of the viewers for all supported platforms are
available at:

     <http://hpctoolkit.org/download.html>

On Linux, download the 'linux.gtk' version of hpcviewer (and also
hpctraceviewer for older versions), unpack the tar files and run the
install scripts (for both viewers) with the path to the desired install
prefix.

     ./install /path/to/install/directory

On Windows and MacOS, download the 'win32' or 'macosx.cocoa' versions
and unpack the zip or dmg files in the desired directory.  For older
versions prior to 2021.04 release, due to Apple's security precautions,
on MacOS, you will need to use curl or wget instead of a web browser.

13 Building a New Compiler
==========================

Some systems may have compilers that are too old for building HPCToolkit
or other packages.  For example, RedHat 7.x comes with gcc 4.8.x which
is very old.  If your system doesn't already have modules for later
compilers, then you may need to build a new compiler yourself.

   First, pick a directory in which to install the modules and make
subdirectories for the spack packages and module files.  In this
example, I'm using '/opt/spack/Modules' as my top-level directory and
subdirectories 'packages' and 'modules'.  Edit 'config.yaml' and
'modules.yaml' to add these paths.

     config:
       install_tree:
         root: /opt/spack/Modules/packages

     modules:
       default:
         roots:
           # just use one of these
           module_roots:
             tcl:  /opt/spack/Modules/modules
             lmod: /opt/spack/Modules/modules

Determine if your system uses TCL (environment) or Lmod modules.
Normally, the 'module' command is a shell function.  TCL modules use
'modulecmd' and Lmod modules eval 'LMOD_CMD'.  Edit 'modules.yaml' to
enable the module type for your system.  Again, you only need one of
these (and don't use dotkit unless you work at LLNL).

     modules:
       default:
         enable:
          - tcl
          - lmod

Then, choose a version of gcc and use the default compiler (normally
'/usr/bin/gcc') to build the newer version.  Currently, gcc 8.x is a
good choice that builds robustly, has enough modern features but is not
too new to cause problems for some packages.  For example,

     spack install gcc@8.4.0

Note: it is not necessary to rebuild the new compiler with itself.

13.1 Using the New Compiler
---------------------------

After building a new compiler, then you need to tell spack how to find
it.  First, use 'module use' and 'module load' to load the module.  For
TCL modules, the module files are in a subdirectory of 'module_roots'
named after the system architecture.  For example,

     module use /opt/spack/Modules/modules/linux-rhel7-x86_64
     module load gcc-8.4.0-gcc-4.8.5-qemsqrc

For Lmod modules, the module directory is one level below that and the
module names are a little different.

     module use /opt/spack/Modules/modules/linux-rhel7-x86_64/Core
     module load gcc/8.4.0-dan4vbm

For both TCL and Lmod modules, it's best to put the 'module use' command
in your shell's startup scripts so that 'module avail' and 'module load'
will know where to find them.  After loading the module, run 'spack
compiler find'.

     $ spack compiler find
     ==> Added 1 new compiler to /home/krentel/.spack/linux/compilers.yaml
         gcc@8.4.0

Finally, always check the new entry in 'compilers.yaml' and add the name
of the module to the 'modules:' field.

     - compiler:
         environment: {}
         extra_rpaths: []
         flags: {}
         modules:
         - gcc-8.4.0-gcc-4.8.5-qemsqrc
         operating_system: rhel7
         paths:
           cc:  /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gcc
           cxx: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/g++
           f77: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gfortran
           fc:  /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gfortran
         spec: gcc@8.4.0
         target: x86_64

Note: as long as the spack packages and modules directories remain
intact and you don't remove the 'compilers.yaml' entry, then this
compiler will always be available from within spack.  You can also use
this compiler outside of spack by using 'module load'.  If you want to
make this your default compiler for all spack builds, then you can
specify this in 'packages.yaml'.  For example,

     packages:
       all:
         compiler: [gcc@8.4.0]

Also, when using the compiler from within spack, it doesn't matter if
you have the module loaded or not.  Spack will erase your environment
and re-add the appropriate modules automatically.

     <https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration>

13.2 Bootstrapping Environment Modules
--------------------------------------

If your system does not support modules, then you will have to add it.
If you have root access, the easiest solution is to install a system
package for modules.  If not, then use spack to install the
environment-modules package.  Source the bash or csh script in the
'init' directory to add the 'module' function to your environment.  For
example,

     spack install environment-modules
     cd /path/to/environment-modules-4.3.1-ism7cdy4xverxywj27jvjstqwk5oxe2v/init
     (bash)  . ./bash
     (csh)   source ./csh

Again, add the setup command to your shell's startup scripts.

14 Spack Mirrors
================

A spack mirror allows you to download and save a source tar file in
advance.  This is useful if your system is behind a firewall, or if you
need to manually agree to a license, or if you just don't want to keep
downloading the same file over and over.

   A mirror has a simple directory structure and is easy to set up.
Create a top-level directory with subdirectories named after the spack
packages and copy the tar files into their package's subdirectory.  For
example,

     my-mirror/
        boost/
           boost-1.66.0.tar.bz2      (from boost_1_66_0.tar.bz2)
           boost-1.70.0.tar.bz2
        dyninst/
           dyninst-10.1.0.tar.gz     (from git checkout)
        ibm-java/
           ibm-java-8.0.5.30.None    (from ibm-java-sdk-8.0-5.30-ppc64le-archive.bin)
        intel-xed/
           intel-xed-2019.03.01.tar.gz    (from git checkout)
           mbuild-2019.03.01.tar.gz       (resource from git checkout)
        jdk/
           jdk-1.8.0_202.tar.gz      (from jdk-8u202-linux-x64.tar.gz)

Note: the names of the files in the spack mirror always follow the same,
specific format, regardless of the actual name of the tar file.  Version
is the spack name for the version (from 'spack info'), and extension is
the same extension as the tar file ('tar.gz', 'tar.bz2', etc) or else
'None' for other types of files.

     <package-name> - <version> . <extension>

For example, the boost 1.66.0 tar file is actually named
'boost_1_66_0.tar.bz2' but is stored in the mirror as
'boost-1.66.0.tar.bz2' and 'jdk-8u202-linux-x64.tar.gz' is renamed to
'jdk-1.8.0_202.tar.gz'.

   For packages that use a snapshot from a git repository (tag or commit
hash), clone the repository, checkout the desired version, make a tar
file and gzip the file.  (You should exclude the '.git' subdirectory.)
But note that spack refuses to use a cached file for the head of a
branch because it is a moving target.

   Finally, after creating the mirror directory, add it to spack with
'spack mirror add'.  For example,

     spack mirror add my-mirror file:///home/krentel/spack/my-mirror
     spack mirror list
     my-mirror    file:///home/krentel/spack/mirror

Note: by default, spack stores downloaded files inside the spack
repository at 'spack/var/spack/cache'.  This directory is a full spack
mirror, so instead of creating a separate directory tree, you could just
copy the files into the 'cache' directory.  This is useful when 'spack
fetch' has trouble downloading a file.  If you can download the file
manually, or copy it from another machine, then just rename the file as
above and copy it into the spack file cache.

   For more information on mirrors, see:

     <https://spack.readthedocs.io/en/latest/mirrors.html>

15 Common Problems
==================

15.1 Unable to fetch tar file
-----------------------------

Spack is somewhat fragile for how it downloads tar files and will often
fail for transitory network problems.  This is especially true for
packages with many dependencies.  For example:

     ==> Installing m4
     ==> Searching for binary cache of m4
     ==> No binary for m4 found: installing from source
     curl: (6) Could not resolve host: ftp.wayne.edu; Name or service not known
     ==> Fetching https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz
     ==> Fetching from https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz failed.
     ==> Error: FetchError: All fetchers failed for m4-1.4.18-vorbvkcjfac43b7vuswsvnm6xe7w7or5

There are two workarounds.  First, assuming the problem is temporary,
simply wait 10 minutes or an hour and try again.  Second, you could
manually download the file(s) by some other means and copy them to
spack's cache directory 'spack/var/spack/cache/<package>' or to a spack
mirror.

15.2 New releases break the build
---------------------------------

Normally, HPCToolkit should build and work correctly with the latest
version for all of its dependencies.  But sometimes a new release will
change something and break the build.  This has happened a couple times
where a new release of Boost has broken the build for Dyninst.  Or,
maybe the latest version of gcc/g++ disallows some usage and breaks the
build.

   The solution is to use 'packages.yaml' to specify an earlier version
until the rest of the code adapts to the change.

15.3 Failure to load modules
----------------------------

Spack is quite aggressive about compiling with a clean environment and
will unload modules unless they are specifically required by some config
file ('compilers.yaml' or 'packages.yaml').  This can result in a
situation where you think some compiler or build tool is available from
your environment but spack removes it during the build.

   In this example, I am using modules for 'GCC/8.3.0' and
'CMake/3.8.2'.  Spack finds the gcc 8.3.0 compiler and I added
'cmake@3.8.2' to 'packages.yaml'.  But I failed to add the 'modules:'
field for gcc 8.3.0 in 'compilers.yaml'.  As a result, the build fails
with:

     cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by cmake)
     cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by cmake)
     cmake: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by cmake)
     ==> Error: ProcessError: Command exited with status 1:

The problem is that cmake 3.8.2 was built with g++ 8.3.0, but spack is
running cmake without the GCC/8.3.0 libraries and so the build fails as
above.  One way to confirm this is to rerun 'spack install --dirty'
which then succeeds.  The '--dirty' option tells spack not to unload
your modules.  Whenever the build fails with a mismatched library as
above and especially when '--dirty' fixes the problem, this is a clear
sign that spack is missing a module during the build.

   Although '--dirty' may make the build succeed, there should be no
case where this is necessary.  The correct solution is to fill in the
'modules:' field in 'compilers.yaml' or some other config file.  See the
section on Compilers above.