Getting Started

Mayo 101

Assuming you've cloned mayo and put it under the directory whatever Notice you might need git lfs to download all the datasets (mnist and cifa10). Mayo now runs at tensorflow v1.8, which is the latest release.

Training

Make sure you are at whatever/mayo-dev, then run:

./my datasets/mnist.yaml trainers/lenet5.yaml models/lenet5.yaml train

The command above executes training of a lenet5 network on a mnist dataset. In Mayo, we define every hyperparameter in YAML files and as you can see from this command line, we fetch dataset yaml, trainer yaml and model yaml from different files. The final train is executed as an action.

You can also override some YAML definitions in the command line:

./my datasets/mnist.yaml \
     trainers/lenet5.yaml \
     models/lenet5.yaml \
     system.batch_size_per_gpu=1024 \ 
     system.num_gpus=2 train

In this case, I overrided the system definitions in system.yaml and changed the batch size and number of gpus in the command line. Notice system.yaml is read by default.

Datasets

Datasets are defined at whatever/mayo-dev/datasets, only two datasets come with the project (cifar10 and mnist).

YOU NEED GIT LFS FOR CLONING THE DATASETS!

However, datasets such as flowers and imagnet are declared in this directory, you can create a symbolic link for your imagnet dataset and point them to this directory, eg:

ln -s where_your_imagenet_is whatever/mayo-dev/datasets

Inference

./my datasets/mnist.yaml \
     trainers/lenet5.yaml \ 
     models/lenet5.yaml \ 
     system.checkpoint.load=pretrained eval

In this case, we specified a pretrained model that is under whatever/mayo-dev/checkpoints/lenet5/mnist/. We adopt this naming habbit checkpoints/$(model.name)/$(dataset.name)/, as declared in system.yaml.

Parameter Overriding

An important feature of Mayo is that it supports parameter overriding and thus opens opportunities for a number of optimizations.

To prune

./my datasets/mnist.yaml \
     models/override/lenet5.yaml \
     models/override/prune/dns.yaml \
     trainers/lenet5.yaml \
     system.checkpoint.load=pretrained \
     overriders-update train

This command asks mayo to train lenet5 from scratch using dynamic network surgery pruners.

! Variables missing in checkpoint:
    lenet5/conv0/weights/DynamicNetworkSurgeryPruner.mask
    lenet5/conv0/weights/DynamicNetworkSurgeryPruner.alpha
    lenet5/conv1/weights/DynamicNetworkSurgeryPruner.mask
    lenet5/conv1/weights/DynamicNetworkSurgeryPruner.alpha
    lenet5/fc1/weights/DynamicNetworkSurgeryPruner.mask
    lenet5/fc1/weights/DynamicNetworkSurgeryPruner.alpha
    lenet5/logits/weights/DynamicNetworkSurgeryPruner.mask
    lenet5/logits/weights/DynamicNetworkSurgeryPruner.alpha

Part of the command line output is shown above, this indicates the variables are created for the pruner and these variables would be stored later in the checkpoints.

To quantize

./my datasets/mnist.yaml \
     models/override/lenet5.yaml \
     models/override/quantize/fixed.yaml \
     trainers/lenet5.yaml \
     system.checkpoint.load=pretrained \
     overriders-update train

More overriders can be found at mayo/overriders

Instantiate your own layer

Instantiate layers is a common practice and if it is a common layer that Mayo has already supported, you can instantiate them in the YAML file:

conv0: &conv
    <<: *init
    type: convolution
    kernel_size: 5
    padding: valid
    num_outputs: 20

Mayo also supports instantiate customized modules in YAML files, for example, SqueezeNet has used a fire module, and we declare it as:

_fire: &fire
    type: module
    kwargs: {squeeze_depth: null, expand_depth: null}
    layers:
        squeeze:
            <<: *conv
            num_outputs: ^squeeze_depth
        expand1: &expand1
            <<: *conv
            num_outputs: ^expand_depth
        expand3: {<<: *expand1, kernel_size: 3}
        concat:
            type: concat
            axis: 3
    graph:
        - {from: input, with: squeeze, to: squeezed}
        - {from: squeezed, with: expand1, to: expanded1}
        - {from: squeezed, with: expand3, to: expanded3}
        - {from: [expanded1, expanded3], with: concat, to: output}

This graph definition instantiate a special module called fire and many complex vision models (Resnet, inception ...) use this way to instantiate some parts of the network.

Another common practice is to write your own layer! Here we show how to define a layer in Mayo and instantiate it in Yaml. First, all common layer definitions can be found in mayo/net/tf/layers.py. Layers with our own algorithmic changes are in their respective files in the same directory. For example, mayo/net/tf/hadamard.py defines a hadamard layer. This layer is later instantiated in models/hadamard/*.yaml model files.

Auto Run

Note:

1. This is currently on test phase and is only available on feature/autorun branch.
2. The scripts use subprocess, a simple ctrl c might not kill the processes, but you can always check whether all processes have been killed using htop and definitely kill them using sudo kill -9 PID.

Auto run helps to explore a wide range of configs, a sample file of running multiple fixed-point quantization with various precisions can be found at scripts/autorun/multiple_fixedpoint.py.

Now you have to copy this file to the parent directory:

cp scripts/autorun/multiple_fixedpoint.py whatever

You can execute this script: python3 multiple_fixedpoint.py This will generate a config folder which includes a bunch of auto-generated yamls. The checkpoints/fixedpoint stores the running results in checkpoints.