mOS for HPC v1.0 Utility Thread API

Utility Thread Application Programmer's Interface

The Utility Thread API (UTI API) has been developed by RIKEN Advanced Institute for Computational Science - Japan, Sandia National Laboratories, and Intel Corp. with feedback from Fujitsu and Argonne National Laboratory.

Why use this API?

The UTI API allows run-times and applications to control the placement of the threads which are not the primary computational threads within the application.

The API:

• Keeps these extra threads from interfering with computational threads.
• Allows grouping and placing of utility threads across the ranks within a node to maximize performance.
• Does not require the caller to have detailed knowledge of the system topology or the scheduler. Allows the kernel to provide intelligent placement and scheduling behavior.
• Does not require the caller to be aware of other potentially conflicting run-time or application thread placement actions. CPU selection is managed globally across the node by mOS.

Utility Thread API Compiling and Linking

• Header file /usr/include/uti.h contains the function and macro declarations.
  • #include <uti.h>
• Library /usr/lib/libmos.so contains the mOS implementation of the UTI API.  Link using the following:
  • "-lmos'

Behaviors and Locations

The programmer can provide behavior and location hints to the kernel. The kernel will then use its knowledge of the system topology and available scheduling facilities to intelligently place and run the utility thread.  The scheduler can optimize scheduling actions for the utility thread for the following behaviors: CPU intensive, e.g., constant polling, high or low scheduling priority, processes that block or yield infrequently, or processes that expect to run on a dedicated CPU.  The scheduler can also optimize placement considering:  L1/L2/L3/NUMA-domain, specific Linux CPU, lightweight kernel CPU, or CPUs that handle fabric interrupts.

Specifying a location

There are various ways of specifying a location:

• Explicit NUMA domain
  • Supply a bit mask containing NUMA domains.
• Location relative to the caller of the API.
  • Same L1, L2, L3, or NUMA domain
  • Different L1, L2, L3, or NUMA domain
• Location relative to other utility threads specifying a common key.
  • Allows grouping of utility threads used across ranks within the node.
  • Used in conjunction with a specification of "Same L1, L2, L3, or NUMA domain"
• Type of CPU
  • Can be used in conjunction with the above location specifications
  • FWK - Linux CPU running under the Linux scheduler
  • LWK - lightweight kernel controlled CPU
  • Fabric Interrupt handling CPU

Location Key Example

This example shows the required sequence of operations to place utility threads on Linux CPUs running under the same L2 cache.

• Run-time agrees on a unique key value to use across ranks within a node.
• Each rank creates a utility thread and specifies:
  • The same location key value.
  • Request Same L2
  • Request FWK CPU type
• When the first utility thread is created, mOS will pick an appropriate Linux CPU and L2 cache.
• All subsequent utility threads created with the same key will be placed on Linux CPUs and share the same L2 cache.
• The mOS kernel will assign the utility threads balanced across the available CPUs that satisfy the location requested. 

UTI Attribute Object

The UTI attribute object is an opaque object that contains the behavior and location information to be used by the kernel when a pthread is created. The definition of the fields within the object are OS specific and purposely hidden from the user interface. This object is treated similarly to the pthread_attr object within the pthread library. This object is passed to the uti_pthread_create() interface, along with the standard arguments passed to pthread_create(). The libmos.so library contains the functions used to prepare the attribute object for use. 

Initializing and Destroying the Attribute Object

The following function is provided for initializing the attribute object before use:

• int uti_attr_init(uti_attr_t *attr);

The following function is provided to destroy the attribute object:

• int uti_attr_destroy(uti_attr_t *attr);

Setting behaviors

This is the list of library functions used to set behaviors in the attribute object:

• int uti_attr_cpu_intensive(uti_attr_t *attr);
  • CPU intensive thread, e.g. constant polling
• int uti_attr_high_priority(uti_attr_t *attr);
  • Expects high scheduling priority
• int uti_attr_low_priority(uti_attr_t *attr);
  • Expects low scheduling priority
• int uti_attr_non_cooperative(uti_attr_t *attr);
  • Does not play nice with others. Infrequenct yields and/or blocks
• int uti_attr_exclusive_cpu(uti_attr_t *attr);
  • Expectes to run on a dedicated CPU

Setting locations

This is the list of library functions used to set location in the attribute object:

• int uti_attr_numa_set(uti_attr_t *attr, unsigned long *nodemask,   unsigned long maxnodes);
• int uti_attr_same_numa_domain(uti_attr_t *attr);
• int uti_attr_different_numa_domain(uti_attr_t *attr);
• int uti_attr_same_l1(uti_attr_t *attr);
• int uti_attr_different_l1(uti_attr_t *attr);
• int uti_attr_same_l2(uti_attr_t *attr);
• int uti_attr_different_l2(uti_attr_t *attr);
• int uti_attr_same_l3(uti_attr_t *attr);
• int uti_attr_different_l3(uti_attr_t *attr);
• int uti_attr_prefer_lwk(uti_attr_t *attr);
• int uti_attr_prefer_fwk(uti_attr_t *attr);
• int uti_attr_fabric_intr_affinity(uti_attr_t *attr);
• int uti_attr_location_key(uti_attr_t *attr, unsigned long key);

Determining results

The uti_pthread_create interface will return EINVAL if conflicting or invalid specifications are provided in the UTI attributes. For example, EINVAL will be returned if 'Same L2' and 'Different L2' are both requested. In these cases, no thread will be created. In other situations when there is no obvious conflict, the thread will be created, even if the requested location or behavior could not be satisfied. Location and behavior results can be determined using the interfaces listed below. The return values are 1=true, and 0=false. The setting of pthread attributes should be used with caution since, they will override the actions/results provided by the UTI attributes.

• int uti_result_different_numa_domain(uti_attr_t *attr);
• int uti_result_same_l1(uti_attr_t *attr);
• int uti_result_different_l1(uti_attr_t *attr);
• int uti_result_same_l2(uti_attr_t *attr);
• int uti_result_different_l2(uti_attr_t *attr);
• int uti_result_same_l3(uti_attr_t *attr);
• int uti_result_different_l3(uti_attr_t *attr);
• int uti_result_prefer_lwk(uti_attr_t *attr);
• int uti_result_prefer_fwk(uti_attr_t *attr);
• int uti_result_fabric_intr_affinity(uti_attr_t *attr);

• int uti_result_exclusive_cpu(uti_attr_t *attr);

• int uti_result_cpu_intensive(uti_attr_t *attr);

• int uti_result_high_priority(uti_attr_t *attr);

• int uti_result_low_priority(uti_attr_t *attr);

• int uti_result_non_cooperative(uti_attr_t *attr);

• int uti_result_location(uti_attr_t *attr);

• int uti_result_behavior(uti_attr_t *attr);

• int uti_result(uti_attr_t *attr);

Usage Example

Note: if your application could be running concurrently with another application using the UTI API, you may need to generate a location key that does not mistakenly match the key in the other application. This example simply uses a statically defined key value. 

#include <uti.h>

pthread_attr_t p_attr;

uti_attr_t uti_attr;

int ret;

..

/* Initialize the attribute objects */

if ((ret = pthread_attr_init(&p_attr)) ||

    (ret = uti_attr_init(&uti_attr)))

        goto uti_exit;

/* Request to put the thread on the same L2 as other utility threads.

 * Also indicate that the thread repeatedly monitors a device.

 */

if ((ret = uti_attr_same_l2(&uti_attr)) ||

    (ret = uti_attr_location_key(&uti_attr, 123456)) ||

    (ret = uti_attr_cpu_intensive(&uti_attr)))

        goto uti_exit;

/* Create the utility thread */ 

if ((ret = uti_pthread_create(idp, &p_attr, thread_start, thread_info, &uti_attr)))

        goto uti_exit;

/* Did the system accept our location and behavior request? */

if (!uti_result(&uti_attr))

        printf(“Warning: utility thread attributes not honored.\n”);

if ((ret = uti_attr_destroy(&uti_attr)))

        goto uti_exit;

..

uti_exit:

Interactions between pthread_attr and uti_attr

Avoid the use of pthread_attr_setaffinity_np when specifying a location with the uti_attr object. The pthread_attr_setaffinity_np directive is prioritized over the uti_attr location requests. If valid CPUs are specified, this action may alter the placement directives requested by the UTI attributes object. If invalid CPUs are provided, this will result in the uti_pthread_create interface returning EINVAL with no utility thread created.

Avoid the use of pthread_attr_set_schedparam and pthread_attr_setschedpolicy when specifying a behavior within the uti_attr_object. These attributes are prioritized over the uti_attr behavior requests. Usage may alter the actions that would have been taken based on the uti_attributes behavior hints. A policy or param that is invalid for an mOS process will result in the uti_pthread_create interface returning EINVAL with no utility thread created.