TIREx Tracker

Automatic resource and metadata tracking for information retrieval experiments.

CLI • C/C++ API • Python API • Java/Kotlin API • Citation

---

The TIREx tracker is a command line tool and API to automatically track resource usage, hardware specifications, and other metadata when running information retrieval experiments. It can be used either from the command line, from C/C++ code, in Python applications, or in Java/Kotlin applications.

Command Line Tool

Download a prebuilt binary for your hardware architecture and operating system from the latest release.

After downloading, run tirex-tracker --help to get a full description of all supported command line arguments.

In most cases, you would use the TIREx tracker CLI like this:

tirex-tracker "<command>"

This will measure all hardware metrics and metadata while executing the given shell command <command>.

C/C++ API

To use the TIREx tracker in your C/C++ projects, first include this repository to your CMake file like this:

include(FetchContent)
# Use GIT_TAG to request the tag (or branch) you would like
FetchContent_Declare(tirex_tracker GIT_REPOSITORY https://github.com/tira-io/tirex-tracker.git GIT_TAG 0.0.11)
FetchContent_MakeAvailable(tirex_tracker)
target_link_libraries(<yourtarget> tirex_tracker::tirex_tracker)

This will link the TIREx tracker C API to your binaries. Take a look at the examples to see how our C API can be used.

A minimal example would is shown below:

#include <tirex_tracker.h>

int main() {
    // Configure measures to track.
    tirexMeasureConf conf[] = { 
        {TIREX_TIME_ELAPSED_WALL_CLOCK_MS, TIREX_AGG_NO},
        // Further measures...
        tirexNullMeasureConf // sentinel value
    };
    size_t pollIntervalMs = 100;
    tirexTrackingHandle* handle;
    tirexStartTracking(conf, pollIntervalMs, &handle);

    // Do something...

    tirexResult* result;
    tirexStopTracking(handle, &result);
    // Analyze the results.
    tirexResultFree(result);
}

The TIREx tracker will automatically track the specified metrics and metadata for everything that is run between tirexStartTracking and tirexStopTracking.

You can customize the measures to track by adjusting the conf array in the example above.

Python API

First, install the TIREx tracker Python package from PyPI:

pip install tirex-tracker

Now, you can track the hardware metrics and metadata of your Python code by using the context manager:

from tirex_tracker import tracking

with tracking() as results:
    # Do something...

print(results)

Alternatively, you can track the hardware metrics and metadata of a Python function by using the function decorator:

from tirex_tracker import tracked

@tracked
def do_something():
    # Do something...

do_something()

print(do_something.results)

If you cannot use either the context manager or the function decorator from above, you can manually start and stop the tracking:

from tirex_tracker import start_tracking, stop_tracking

handle = start_tracking()
try:
    # Do something...
finally:
    results = stop_tracking(handle)

print(results)

Java/Kotlin/JVM API

The Java/Kotlin API can be installed via Gradle or Maven from the Maven Central Repository. After installing the package, you can use the TIREx tracker JVM API in your Java or Kotlin projects.

Alternative: GitHub Packages

Alternatively to the Maven Central Repository, the TIREx tracker JVM API is also published to GitHub Packages. To use GitHub Packages, you must first authenticate (Maven instructions, Gradle instructions).

Gradle Dependency

For Gradle, add the following line to the dependencies block of your build.gradle (or build.gradle.kts) file:

dependencies {
    implementation("io.tira:tirex-tracker:x.x.x")
}

Maven Dependency

For Maven projects, add these lines to your pom.xml file:

<dependency>
  <groupId>io.tira</groupId>
  <artifactId>tirex-tracker</artifactId>
  <version>x.x.x</version>
</dependency>

Replace the version placeholder (x.x.x) with the latest available version tag.

You can now use the TIREx tracker JVM API in your Java or Kotlin projects.

Java Usage

For pure Java projects, the easiest way is to use the track function and pass your code to be tracked as a lambda like this:

package io.tira.tirex.tracker.example;

import io.tira.tirex.tracker.*;

void main() {
    var result = Tracker.track(() -> {
        // Do something...
    });

    System.out.println(result);
}

Alternatively, use the try-with-resources syntax like this:

Tracked tracked = new Tracked();
try (tracked) {
    // Do something...
}
    
System.out.println(tracked.result);

Kotlin Usage

In Kotlin projects, the track function takes an inline block so that your code can be tracked like this:

package io.tira.tirex.tracker.example

import io.tira.tirex.tracker.*

fun main() {
  val result = track {
    // Do something...
  }

  println(result)
}

Tracked Measures

ir_metadata Extension

The resources, hardware specifications, and metadata tracked by the TIREx tracker can easily be exported to an ir_metadata file. Not all of our extensive metadata fits well into the current ir_metadata specification version 0.1. We therefore extend the specification and propose ir_metadata 0.2-beta:

Contributing

We happily accept pull requests, feature requests, and bug reports to the TIREx tracker! To get started for contributing to the development, first clone this repository:

git clone https://github.com/tira-io/tirex-tracker.git
cd tirex-tracker

The further steps will depend on which part of the TIREx tracker's API you work on: the C API, the Python API, or the JVM API.

C development

We use CMake to build the C library, command line tool, and examples. Different build targets are used so that you can select what to build. Doxygen is used to generate the documentation for the C API. If you haven't installed CMake and/or Doxygen, please install it as described here for CMake and here for Doxygen

First, setup CMake to use the correct version of GCC and enable all build targets by running:

cmake -S c/ -B c/build/ \
  -D CMAKE_BUILD_TYPE=Release \
  -D CMAKE_C_COMPILER=gcc-13 \
  -D CMAKE_CXX_COMPILER=g++-13 \
  -D BUILD_SHARED_LIBS=YES \
  -D TIREX_TRACKER_BUILD_DOCS=YES \
  -D TIREX_TRACKER_BUILD_DEB=YES \
  -D TIREX_TRACKER_BUILD_EXAMPLES=YES

(Hint: If you do not want to generate the documentation and have not installed Doxygen, you can disable it by setting TIREX_TRACKER_BUILD_DOCS=NO.)

You can then build the library (and examples) like this:

cmake --build c/build/ --config Release --target tirex_tracker_full

This will compile the C API into a statically linked library at c/build/extensions/libtirex_tracker_full.so.

The Debian package is built by running:

cmake --build c/build/ --config Release --target package

You will find the compiled Debian package file at c/build/tirex-tracker-*-Linux.deb (where * is the version).

Python development

When developing (parts of) the Python API, first create a virtual environment and then install all necessary developer tooling by running:

python3 -m venv venv/
source venv/bin/activate
pip install python/[tests]

Once the Python developer tools are set up like that, you can check the Python code, static typing, and security, and run all tests with:

ruff check python/  # Code format and LINT
mypy python/        # Static typing
bandit -c python/pyproject.toml -r python/  # Security
pytest python/      # Unit tests

JVM Development

The JVM wrapper API uses Gradle for build and test. Consistent builds are ensured by using a Gradle wrapper. To build the classes, run:

jvm/gradlew --project-dir jvm/ build

Tests and checks for Java and Kotlin usage can be run by:

jvm/gradlew --project-dir jvm/ check

Citation

The TIREx tracker's accompanying paper is under review. We will add citation instructions once it is published.

License

The TIREx tracker code is licensed under the MIT License. If you use the TIREx tracker in your experiments, we would appreciate you citing our paper.

Abstract

The reproducibility and transparency of retrieval experiments heavily depends on properly provided information on the experimental setup and conditions. But as manually curating such experiment metadata can be tedious, error-prone, and inconsistent, metadata should be systematically collected in an automatic way—similar to the collection of Python and git-specific settings in the ir_metadata reference implementation. To enable a platform-independent automatic metadata collection following the ir_metadata specification, we introduce the TIREx tracker: a tool realized via a lightweight C binary, pre-compiled with all dependencies for all major platforms to track hardware configurations, usage of power/CPUs/RAM/GPUs, and experiment/system versions. The TIREx tracker seamlessly integrates into Python, Java, or C/C++ workflows and can be easily incorporated in run submissions of shared tasks, which we showcase for the TIRA/TIREx platform. Code, binaries, and documentation are publicly available at https://github.com/tira-io/tirex-tracker.