examples/00_interfacing_DEHB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interfacing DEHB\n",
    "## How to read this notebook\n",
    "\n",
    "This notebook is designed to serve as a high-level, highly abstracted view of DEHB and how it can be used. The examples here are mere placeholders and *only* offer an interface to run DEHB on toy or actual problems.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import warnings\n",
    "import numpy as np\n",
    "import ConfigSpace\n",
    "from typing import Dict, Union, List\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting started with DEHB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DEHB was designed to be an algorithm for Hyper Parameter Optimization (HPO). DEHB uses Differential Evolution (DE) under-the-hood as an Evolutionary Algorithm to power the black-box optimization that HPO problems pose. DE is a black-box optimization algorithm that generates candidate configurations $x$, to the black-box function $f(x)$, that is being optimized. The $x$ is evaluated by the black-box and the corresponding response $y$ is made available to the DE algorithm, which can then use this observation ($x$, $y$), along with previous such observations, to suggest a new candidate $x$ for the next evaluation. \n",
    "\n",
    "DEHB also uses Hyperband along with DE, to allow for cheaper approximations of the actual evaluations of $x$. Let $f(x)$ be the validation error of training a multilayer perceptron (MLP) on the complete training set. Multi-fidelity algorithms such as Hyperband, allow for cheaper approximations along a possible *fidelity*. For the MLP, a subset of the dataset maybe a cheaper approximation to the full data set evaluation. Whereas the fidelity can be quantified as the fraction of the dataset used to evaluate the configuration $x$, instead of the full dataset. Such approximations can allow sneak-peek into the black-box, potentially revealing certain landscape features of *f(x)*, thus rendering it a *gray*-box and not completely opaque and black! \n",
    "\n",
    "The $z$ parameter is the fidelity parameter to the black-box function. If $z \\in [fidelity_{min}, fidelity_{max}]$, then $f(x, fidelity_{max})$ would be equivalent to the black-box case of $f(x)$.\n",
    "<!--- Here we need to use the weblink in order to correctly show the image in the docs--->\n",
    "![boxes](https://github.com/automl/DEHB/blob/master/examples/imgs/black-gray-box.png?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HPO algorithms optimize such black/gray box by wrapping around this *target* function an interface, by which the algorithms can suggest new $x$ and also consume the result of the corresponding evaluation to store a collection of such ($x$, $y$) pairs. Therefore, to run DEHB, the most essential component required as input is the target function to optimize. Since DEHB can leverage a Hyperband, the target function interface should account for possible input of fidelity too. \n",
    "\n",
    "## Sample interface for target function that DEHB optimizes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def target_function(\n",
    "    x: Union[ConfigSpace.Configuration, List, np.array], \n",
    "    fidelity: Union[int, float] = None,\n",
    "    **kwargs\n",
    ") -> Dict:\n",
    "    \"\"\" Target/objective function to optimize\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    x : configuration that DEHB wants to evaluate\n",
    "    fidelity : parameter determining cheaper evaluations\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    dict\n",
    "    \"\"\"\n",
    "    # ...\n",
    "    # write your code here\n",
    "    # ...\n",
    "    \n",
    "    # remove the code snippet below\n",
    "    start = time.time()\n",
    "    y = np.random.uniform()  # placeholder response of evaluation\n",
    "    time.sleep(fidelity)       # simulates runtime (mostly proportional to fidelity)\n",
    "    cost = time.time() - start\n",
    "    \n",
    "    # result dict passed to DE/DEHB as function evaluation output\n",
    "    result = {\n",
    "        \"fitness\": y,  # must-have key that DE/DEHB minimizes\n",
    "        \"cost\": cost,  # must-have key that associates cost/runtime \n",
    "        \"info\": dict() # optional key containing a dictionary of additional info\n",
    "    }\n",
    "    return result    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `target_function` is the problem that needs to be solved, or the function to be optimized. The other prerequisite for this function is therefore the domain for its input $x$. In other words, the definition and constraints of the *search space* for DEHB. \n",
    "\n",
    "The DE component inside DEHB, **assumes that the input domain is scaled to a unit hypercube**. This is essential for effective search. If the [ConfigSpace](https://pypi.org/project/ConfigSpace/) library is used to define the domain of $x$, or the parameters of the search space, DEHB can internally handle the scaling to and from the unit hypercube required for search. If ConfigSpace is not used, one needs to additionally handle the scaling of the parameters as an extra interface between DEHB and the target function (or encode it within the target function). \n",
    "\n",
    "For this template notebook, we will illustrate how a ConfigSpace parameter space can be created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining a sample search space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuration space object:\n",
      "  Hyperparameters:\n",
      "    x0, Type: UniformFloat, Range: [3.0, 10.0], Default: 6.5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import ConfigSpace\n",
    "\n",
    "\n",
    "def create_search_space():\n",
    "    # Creating a one-dimensional search space of real numbers in [3, 10]\n",
    "    cs = ConfigSpace.ConfigurationSpace()\n",
    "    cs.add_hyperparameter(ConfigSpace.UniformFloatHyperparameter(\"x0\", lower=3, upper=10, log=False))\n",
    "    return cs\n",
    "\n",
    "\n",
    "cs = create_search_space()\n",
    "print(cs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "# Finding dimensionality of search space\n",
    "dimensions = len(cs.get_hyperparameters())\n",
    "print(dimensions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Configuration(values={\n",
       "  'x0': 5.609491838639338,\n",
       "})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sampling a random configuration\n",
    "cs.sample_configuration()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [ConfigSpace documentation](https://automl.github.io/ConfigSpace/master/index.html) can be referred to for more complicated search space creation.\n",
    "\n",
    "In a similar vein, for a complete gray-box definition, the fidelity domain needs to be defined too. For the earlier example of dataset fractions, the fidelity upper limit cannot clearly exceed 1, and therefore $[0.3, 1]$ is a legitimate definition for such a fidelity. In this template example, we shall simply define the lower and upper range of the fidelity as two parameters that can be input to DEHB. Given that fidelity is being used to simulate cost of runtime in our sample `target_function`, we shall use a reasonable time range as a placeholder for the fidelity in this case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining fidelity range for the target function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_fidelity, max_fidelity = (0.1, 3) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above definitions are all the information that DEHB needs about a problem. We are now in a position to call upon DEHB and start running it to tune $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instantiating and running DEHB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-06-18 22:21:08.083\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36mdehb.optimizers.dehb\u001b[0m:\u001b[36m__init__\u001b[0m:\u001b[36m264\u001b[0m - \u001b[33m\u001b[1mA checkpoint already exists, results could potentially be overwritten.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from dehb import DEHB\n",
    "\n",
    "\n",
    "dehb = DEHB(\n",
    "    f=target_function,\n",
    "    dimensions=dimensions,\n",
    "    cs=cs,\n",
    "    min_fidelity=min_fidelity,\n",
    "    max_fidelity=max_fidelity,\n",
    "    output_path=\"./temp\",\n",
    "    n_workers=1        # set to >1 to utilize parallel workers\n",
    ")\n",
    "\n",
    "# NOTE: the other hyperparameters to DEHB have been set to certain defaults that were \n",
    "# empirically determined through related literature, ablation analysis and other experiments,\n",
    "# but can be tuned as desired"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0mdehb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mfevals\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mbrackets\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mtotal_cost\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msingle_node_with_gpus\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdebug\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m<\u001b[0m\u001b[0mbuilt\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;32min\u001b[0m \u001b[0mfunction\u001b[0m \u001b[0marray\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mbuilt\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;32min\u001b[0m \u001b[0mfunction\u001b[0m \u001b[0marray\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mbuilt\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;32min\u001b[0m \u001b[0mfunction\u001b[0m \u001b[0marray\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Main interface to run optimization by DEHB.\n",
       "\n",
       "This function waits on workers and if a worker is free, asks for a configuration and a\n",
       "fidelity to evaluate on and submits it to the worker. In each loop, it checks if a job\n",
       "is complete, fetches the results, carries the necessary processing of it asynchronously\n",
       "to the worker computations.\n",
       "\n",
       "The duration of the DEHB run can be controlled by specifying one of 3 parameters. If more\n",
       "than one are specified, DEHB selects only one in the priority order (high to low): <br>\n",
       "1) Number of function evaluations (fevals) <br>\n",
       "2) Number of Successive Halving brackets run under Hyperband (brackets) <br>\n",
       "3) Total computational cost (in seconds) aggregated by all function evaluations (total_cost)\n",
       "\n",
       "!!! note \"Using `tell` under the hood.\"\n",
       "\n",
       "    Please note, that `run` uses `tell` under the hood, therefore please have a\n",
       "    look at the documentation of `tell` for more information e.g. about the result format.\n",
       "\n",
       "!!! note \"Adjusting verbosity\"\n",
       "\n",
       "    The verbosity of DEHB logs can be adjusted via adding the `log_level` parameter to DEHBs\n",
       "    initialization. As we use loguru, the logging levels can be found on [their website](https://loguru.readthedocs.io/en/stable/api/logger.html#levels).\n",
       "\n",
       "Args:\n",
       "    fevals (int, optional): Number of functions evaluations to run. Defaults to None.\n",
       "    brackets (int, optional): Number of brackets to run. Defaults to None.\n",
       "    total_cost (int, optional): Wallclock budget in seconds. Defaults to None.\n",
       "    single_node_with_gpus (bool): Workers get assigned different GPUs. Default to False.\n",
       "    debug (bool): Activate debug output. Defaults to False.\n",
       "\n",
       "Returns:\n",
       "    Trajectory, runtime and optimization history.\n",
       "\u001b[0;31mFile:\u001b[0m      ~/DEHB/src/dehb/optimizers/dehb.py\n",
       "\u001b[0;31mType:\u001b[0m      method"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?dehb.run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DEHB allows the option of 3 different resources for its runtime budget:\n",
    "### 1) Running DEHB for a certain number of (successive halving) *brackets*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuration(values={\n",
      "  'x0': 3.5266014566419885,\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "_, _, _ = dehb.run(brackets=1)\n",
    "print(dehb.vector_to_configspace(dehb.inc_config))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Running DEHB for total number of *function evaluations*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(Configuration(values={\n",
      "  'x0': 7.072551207320847,\n",
      "}), 0.01580110736028184)\n"
     ]
    }
   ],
   "source": [
    "# allows optimization to restart from the beginning by forgetting all observations\n",
    "dehb.reset()  \n",
    "\n",
    "_, _, _ = dehb.run(fevals=20)\n",
    "print(dehb.get_incumbents())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Running DEHB for total amount of *wallclock time*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-06-18 22:21:41.523\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36mdehb.optimizers.dehb\u001b[0m:\u001b[36m_timeout_handler\u001b[0m:\u001b[36m351\u001b[0m - \u001b[33m\u001b[1mRuntime budget exhausted. Saving optimization checkpoint now.\u001b[0m\n",
      "(Configuration(values={\n",
      "  'x0': 7.209535942424234,\n",
      "}), 0.0074313010951959635)\n"
     ]
    }
   ],
   "source": [
    "# allows optimization to restart from the beginning by forgetting all observations\n",
    "dehb.reset()  \n",
    "\n",
    "_, _, _ = dehb.run(total_cost=10)  # run for 10s\n",
    "print(dehb.get_incumbents())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each `dehb` object initialized maintains a `log` file in the `output_path` specified, where the progress and other debugging information is updated. While every alternative DEHB evaluation (and after full optimization), an `incumbent.json` file is written to disk `output_path`, with the incumbent (best seen so far) configuration and its corresponding score. \n",
    "\n",
    "\n",
    "We now rerun DEHB in parallel with 2 workers, and show that the incumbents can be retrieved in any of the following manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-06-18 22:22:18.698\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36mdehb.optimizers.dehb\u001b[0m:\u001b[36m__init__\u001b[0m:\u001b[36m264\u001b[0m - \u001b[33m\u001b[1mA checkpoint already exists, results could potentially be overwritten.\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-06-18 22:22:18,708 - distributed.protocol.pickle - ERROR - Failed to serialize <ToPickle: HighLevelGraph with 1 layers.\n",
      "<dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\n",
      " 0. 140098536150272\n",
      ">.\n",
      "Traceback (most recent call last):\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/protocol/pickle.py\", line 63, in dumps\n",
      "    result = pickle.dumps(x, **dump_kwargs)\n",
      "TypeError: cannot pickle '_thread.lock' object\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/protocol/pickle.py\", line 68, in dumps\n",
      "    pickler.dump(x)\n",
      "TypeError: cannot pickle '_thread.lock' object\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/protocol/pickle.py\", line 81, in dumps\n",
      "    result = cloudpickle.dumps(x, **dump_kwargs)\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/cloudpickle/cloudpickle.py\", line 1479, in dumps\n",
      "    cp.dump(obj)\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/cloudpickle/cloudpickle.py\", line 1245, in dump\n",
      "    return super().dump(obj)\n",
      "TypeError: cannot pickle '_thread.lock' object\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-06-18 22:22:18.710\u001b[0m | \u001b[31m\u001b[1mERROR   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36m<module>\u001b[0m:\u001b[36m10\u001b[0m - \u001b[31m\u001b[1mAn error has been caught in function '<module>', process 'MainProcess' (12515), thread 'MainThread' (140100813578624):\u001b[0m\n",
      "\u001b[33m\u001b[1mTraceback (most recent call last):\u001b[0m\n",
      "\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/protocol/pickle.py\", line 63, in dumps\n",
      "    result = pickle.dumps(x, **dump_kwargs)\n",
      "             │      │     │    └ {'protocol': 5, 'buffer_callback': <built-in method append of list object at 0x7f6b9590c5c0>}\n",
      "             │      │     └ <ToPickle: HighLevelGraph with 1 layers.\n",
      "             │      │       <dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\n",
      "             │      │        0. 140098536150272\n",
      "             │      │       >\n",
      "             │      └ <built-in function dumps>\n",
      "             └ <module 'pickle' from '/home/fixja/miniconda3/envs/dehb/lib/python3.9/pickle.py'>\n",
      "\n",
      "\u001b[31m\u001b[1mTypeError\u001b[0m:\u001b[1m cannot pickle '_thread.lock' object\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1mDuring handling of the above exception, another exception occurred:\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[33m\u001b[1mTraceback (most recent call last):\u001b[0m\n",
      "\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/protocol/pickle.py\", line 68, in dumps\n",
      "    pickler.dump(x)\n",
      "    │       │    └ <ToPickle: HighLevelGraph with 1 layers.\n",
      "    │       │      <dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\n",
      "    │       │       0. 140098536150272\n",
      "    │       │      >\n",
      "    │       └ <method 'dump' of '_pickle.Pickler' objects>\n",
      "    └ <distributed.protocol.pickle._DaskPickler object at 0x7f6b40096d00>\n",
      "\n",
      "\u001b[31m\u001b[1mTypeError\u001b[0m:\u001b[1m cannot pickle '_thread.lock' object\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1mDuring handling of the above exception, another exception occurred:\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[33m\u001b[1mTraceback (most recent call last):\u001b[0m\n",
      "\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/protocol/serialize.py\", line 353, in serialize\n",
      "    header, frames = dumps(x, context=context) if wants_context else dumps(x)\n",
      "                     │     │          │           │                  │     └ <ToPickle: HighLevelGraph with 1 layers.\n",
      "                     │     │          │           │                  │       <dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\n",
      "                     │     │          │           │                  │        0. 140098536150272\n",
      "                     │     │          │           │                  │       >\n",
      "                     │     │          │           │                  └ <function pickle_dumps at 0x7f6b486c5a60>\n",
      "                     │     │          │           └ True\n",
      "                     │     │          └ None\n",
      "                     │     └ <ToPickle: HighLevelGraph with 1 layers.\n",
      "                     │       <dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\n",
      "                     │        0. 140098536150272\n",
      "                     │       >\n",
      "                     └ <function pickle_dumps at 0x7f6b486c5a60>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/protocol/serialize.py\", line 76, in pickle_dumps\n",
      "    frames[0] = pickle.dumps(\n",
      "    │           │      └ <function dumps at 0x7f6b4864fa60>\n",
      "    │           └ <module 'distributed.protocol.pickle' from '/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/protocol...\n",
      "    └ [None]\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/protocol/pickle.py\", line 81, in dumps\n",
      "    result = cloudpickle.dumps(x, **dump_kwargs)\n",
      "             │           │     │    └ {'protocol': 5, 'buffer_callback': <built-in method append of list object at 0x7f6b9590c5c0>}\n",
      "             │           │     └ <ToPickle: HighLevelGraph with 1 layers.\n",
      "             │           │       <dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\n",
      "             │           │        0. 140098536150272\n",
      "             │           │       >\n",
      "             │           └ <function dumps at 0x7f6b48940550>\n",
      "             └ <module 'cloudpickle' from '/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/cloudpickle/__init__.py'>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/cloudpickle/cloudpickle.py\", line 1479, in dumps\n",
      "    cp.dump(obj)\n",
      "    │  │    └ <ToPickle: HighLevelGraph with 1 layers.\n",
      "    │  │      <dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\n",
      "    │  │       0. 140098536150272\n",
      "    │  │      >\n",
      "    │  └ <function Pickler.dump at 0x7f6b489403a0>\n",
      "    └ <cloudpickle.cloudpickle.Pickler object at 0x7f6b40096880>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/cloudpickle/cloudpickle.py\", line 1245, in dump\n",
      "    return super().dump(obj)\n",
      "                        └ <ToPickle: HighLevelGraph with 1 layers.\n",
      "                          <dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\n",
      "                           0. 140098536150272\n",
      "                          >\n",
      "\n",
      "\u001b[31m\u001b[1mTypeError\u001b[0m:\u001b[1m cannot pickle '_thread.lock' object\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1mThe above exception was the direct cause of the following exception:\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[33m\u001b[1mTraceback (most recent call last):\u001b[0m\n",
      "\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/runpy.py\", line 197, in _run_module_as_main\n",
      "    return _run_code(code, main_globals, None,\n",
      "           │         │     └ {'__name__': '__main__', '__doc__': 'Entry point for launching an IPython kernel.\\n\\nThis is separate from the ipykernel pack...\n",
      "           │         └ <code object <module> at 0x7f6bc2574240, file \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel_launche...\n",
      "           └ <function _run_code at 0x7f6bc2571a60>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/runpy.py\", line 87, in _run_code\n",
      "    exec(code, run_globals)\n",
      "         │     └ {'__name__': '__main__', '__doc__': 'Entry point for launching an IPython kernel.\\n\\nThis is separate from the ipykernel pack...\n",
      "         └ <code object <module> at 0x7f6bc2574240, file \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel_launche...\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel_launcher.py\", line 17, in <module>\n",
      "    app.launch_new_instance()\n",
      "    │   └ <bound method Application.launch_instance of <class 'ipykernel.kernelapp.IPKernelApp'>>\n",
      "    └ <module 'ipykernel.kernelapp' from '/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelapp.py'>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/traitlets/config/application.py\", line 1075, in launch_instance\n",
      "    app.start()\n",
      "    │   └ <function IPKernelApp.start at 0x7f6bbea381f0>\n",
      "    └ <ipykernel.kernelapp.IPKernelApp object at 0x7f6bc2f996d0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelapp.py\", line 739, in start\n",
      "    self.io_loop.start()\n",
      "    │    │       └ <function BaseAsyncIOLoop.start at 0x7f6bbea38ee0>\n",
      "    │    └ <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f6bbd78c1c0>\n",
      "    └ <ipykernel.kernelapp.IPKernelApp object at 0x7f6bc2f996d0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/tornado/platform/asyncio.py\", line 205, in start\n",
      "    self.asyncio_loop.run_forever()\n",
      "    │    │            └ <function BaseEventLoop.run_forever at 0x7f6bc06a4e50>\n",
      "    │    └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n",
      "    └ <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f6bbd78c1c0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/asyncio/base_events.py\", line 601, in run_forever\n",
      "    self._run_once()\n",
      "    │    └ <function BaseEventLoop._run_once at 0x7f6bc06a79d0>\n",
      "    └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/asyncio/base_events.py\", line 1905, in _run_once\n",
      "    handle._run()\n",
      "    │      └ <function Handle._run at 0x7f6bc075be50>\n",
      "    └ <Handle <TaskWakeupMethWrapper object at 0x7f6b48bd1f10>(<Future finis...5c0>, ...],))>)>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/asyncio/events.py\", line 80, in _run\n",
      "    self._context.run(self._callback, *self._args)\n",
      "    │    │            │    │           │    └ <member '_args' of 'Handle' objects>\n",
      "    │    │            │    │           └ <Handle <TaskWakeupMethWrapper object at 0x7f6b48bd1f10>(<Future finis...5c0>, ...],))>)>\n",
      "    │    │            │    └ <member '_callback' of 'Handle' objects>\n",
      "    │    │            └ <Handle <TaskWakeupMethWrapper object at 0x7f6b48bd1f10>(<Future finis...5c0>, ...],))>)>\n",
      "    │    └ <member '_context' of 'Handle' objects>\n",
      "    └ <Handle <TaskWakeupMethWrapper object at 0x7f6b48bd1f10>(<Future finis...5c0>, ...],))>)>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 542, in dispatch_queue\n",
      "    await self.process_one()\n",
      "          │    └ <function Kernel.process_one at 0x7f6bbef27550>\n",
      "          └ <ipykernel.ipkernel.IPythonKernel object at 0x7f6bbd78c820>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 531, in process_one\n",
      "    await dispatch(*args)\n",
      "          │         └ ([<zmq.sugar.frame.Frame object at 0x7f6b4811cca0>, <zmq.sugar.frame.Frame object at 0x7f6bbc752eb0>, <zmq.sugar.frame.Frame ...\n",
      "          └ <bound method Kernel.dispatch_shell of <ipykernel.ipkernel.IPythonKernel object at 0x7f6bbd78c820>>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 437, in dispatch_shell\n",
      "    await result\n",
      "          └ <coroutine object IPythonKernel.execute_request at 0x7f6bbe9f50c0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/ipkernel.py\", line 359, in execute_request\n",
      "    await super().execute_request(stream, ident, parent)\n",
      "                                  │       │      └ {'header': {'date': datetime.datetime(2024, 6, 18, 20, 22, 18, 68000, tzinfo=tzutc()), 'msg_id': '4c0babc8-0868-46cf-aaf0-2fa...\n",
      "                                  │       └ [b'32f41178-6ac3-4e91-86b1-ccc69081afe1']\n",
      "                                  └ <zmq.eventloop.zmqstream.ZMQStream object at 0x7f6bbd77eee0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 775, in execute_request\n",
      "    reply_content = await reply_content\n",
      "                          └ <coroutine object IPythonKernel.do_execute at 0x7f6bbe9f5a40>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/ipkernel.py\", line 446, in do_execute\n",
      "    res = shell.run_cell(\n",
      "          │     └ <function ZMQInteractiveShell.run_cell at 0x7f6bbea24700>\n",
      "          └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f6bbd78cd90>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/zmqshell.py\", line 549, in run_cell\n",
      "    return super().run_cell(*args, **kwargs)\n",
      "                             │       └ {'store_history': True, 'silent': False, 'cell_id': '1b213cab-b993-425a-8843-42438a78557b'}\n",
      "                             └ ('dehb = DEHB(\\n    f=target_function,\\n    dimensions=dimensions,\\n    cs=cs,\\n    min_fidelity=min_fidelity,\\n    max_fidel...\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3048, in run_cell\n",
      "    result = self._run_cell(\n",
      "             │    └ <function InteractiveShell._run_cell at 0x7f6bbf976160>\n",
      "             └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f6bbd78cd90>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3103, in _run_cell\n",
      "    result = runner(coro)\n",
      "             │      └ <coroutine object InteractiveShell.run_cell_async at 0x7f6bbe9f5cc0>\n",
      "             └ <function _pseudo_sync_runner at 0x7f6bbf9e39d0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/async_helpers.py\", line 129, in _pseudo_sync_runner\n",
      "    coro.send(None)\n",
      "    │    └ <method 'send' of 'coroutine' objects>\n",
      "    └ <coroutine object InteractiveShell.run_cell_async at 0x7f6bbe9f5cc0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3308, in run_cell_async\n",
      "    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,\n",
      "                       │    │             │        │     └ '/tmp/ipykernel_12515/1264706599.py'\n",
      "                       │    │             │        └ [<ast.Assign object at 0x7f6b40b1b4c0>, <ast.Assign object at 0x7f6b400c7670>, <ast.Expr object at 0x7f6b400c7f10>]\n",
      "                       │    │             └ <ast.Module object at 0x7f6b40b1b4f0>\n",
      "                       │    └ <function InteractiveShell.run_ast_nodes at 0x7f6bbf976430>\n",
      "                       └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f6bbd78cd90>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3490, in run_ast_nodes\n",
      "    if await self.run_code(code, result, async_=asy):\n",
      "             │    │        │     │              └ False\n",
      "             │    │        │     └ <ExecutionResult object at 7f6b400c7640, execution_count=12 error_before_exec=None error_in_exec=None info=<ExecutionInfo obj...\n",
      "             │    │        └ <code object <module> at 0x7f6b959055b0, file \"/tmp/ipykernel_12515/1264706599.py\", line 10>\n",
      "             │    └ <function InteractiveShell.run_code at 0x7f6bbf9764c0>\n",
      "             └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f6bbd78cd90>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3550, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "         │         │    │               │    └ {'__name__': '__main__', '__doc__': 'Automatically created module for IPython interactive environment', '__package__': None, ...\n",
      "         │         │    │               └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f6bbd78cd90>\n",
      "         │         │    └ <property object at 0x7f6bbf9f2130>\n",
      "         │         └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f6bbd78cd90>\n",
      "         └ <code object <module> at 0x7f6b959055b0, file \"/tmp/ipykernel_12515/1264706599.py\", line 10>\n",
      "\n",
      "> File \"\u001b[32m/tmp/ipykernel_12515/\u001b[0m\u001b[32m\u001b[1m1264706599.py\u001b[0m\", line \u001b[33m10\u001b[0m, in \u001b[35m<module>\u001b[0m\n",
      "    \u001b[1mtrajectory\u001b[0m\u001b[1m,\u001b[0m \u001b[1mruntime\u001b[0m\u001b[1m,\u001b[0m \u001b[1mhistory\u001b[0m \u001b[35m\u001b[1m=\u001b[0m \u001b[1mdehb\u001b[0m\u001b[35m\u001b[1m.\u001b[0m\u001b[1mrun\u001b[0m\u001b[1m(\u001b[0m\n",
      "    \u001b[36m                               │    └ \u001b[0m\u001b[36m\u001b[1m<function DEHB.run at 0x7f6b40b01a60>\u001b[0m\n",
      "    \u001b[36m                               └ \u001b[0m\u001b[36m\u001b[1m<dehb.optimizers.dehb.DEHB object at 0x7f6b40b1b2e0>\u001b[0m\n",
      "\n",
      "  File \"\u001b[32m/home/fixja/DEHB/src/dehb/optimizers/\u001b[0m\u001b[32m\u001b[1mdehb.py\u001b[0m\", line \u001b[33m1156\u001b[0m, in \u001b[35mrun\u001b[0m\n",
      "    \u001b[1mself\u001b[0m\u001b[35m\u001b[1m.\u001b[0m\u001b[1m_submit_job\u001b[0m\u001b[1m(\u001b[0m\u001b[1mjob_info\u001b[0m\u001b[1m,\u001b[0m \u001b[35m\u001b[1m**\u001b[0m\u001b[1mkwargs\u001b[0m\u001b[1m)\u001b[0m\n",
      "    \u001b[36m│    │           │           └ \u001b[0m\u001b[36m\u001b[1m{}\u001b[0m\n",
      "    \u001b[36m│    │           └ \u001b[0m\u001b[36m\u001b[1m{'config': Configuration(values={\u001b[0m\n",
      "    \u001b[36m│    │             \u001b[0m\u001b[36m\u001b[1m  'x0': 4.823080883359318,\u001b[0m\n",
      "    \u001b[36m│    │             \u001b[0m\u001b[36m\u001b[1m}), 'config_id': 46, 'fidelity': 0.1111111111111111, 'parent_id'...\u001b[0m\n",
      "    \u001b[36m│    └ \u001b[0m\u001b[36m\u001b[1m<function DEHB._submit_job at 0x7f6b40b011f0>\u001b[0m\n",
      "    \u001b[36m└ \u001b[0m\u001b[36m\u001b[1m<dehb.optimizers.dehb.DEHB object at 0x7f6b40b1b2e0>\u001b[0m\n",
      "\n",
      "  File \"\u001b[32m/home/fixja/DEHB/src/dehb/optimizers/\u001b[0m\u001b[32m\u001b[1mdehb.py\u001b[0m\", line \u001b[33m732\u001b[0m, in \u001b[35m_submit_job\u001b[0m\n",
      "    \u001b[1mself\u001b[0m\u001b[35m\u001b[1m.\u001b[0m\u001b[1mclient\u001b[0m\u001b[35m\u001b[1m.\u001b[0m\u001b[1msubmit\u001b[0m\u001b[1m(\u001b[0m\u001b[1mself\u001b[0m\u001b[35m\u001b[1m.\u001b[0m\u001b[1m_f_objective\u001b[0m\u001b[1m,\u001b[0m \u001b[1mjob_info\u001b[0m\u001b[1m)\u001b[0m\n",
      "    \u001b[36m│    │      │      │    │             └ \u001b[0m\u001b[36m\u001b[1m{'config': Configuration(values={\u001b[0m\n",
      "    \u001b[36m│    │      │      │    │               \u001b[0m\u001b[36m\u001b[1m  'x0': 4.823080883359318,\u001b[0m\n",
      "    \u001b[36m│    │      │      │    │               \u001b[0m\u001b[36m\u001b[1m}), 'config_id': 46, 'fidelity': 0.1111111111111111, 'parent_id'...\u001b[0m\n",
      "    \u001b[36m│    │      │      │    └ \u001b[0m\u001b[36m\u001b[1m<function DEHB._f_objective at 0x7f6b40b00430>\u001b[0m\n",
      "    \u001b[36m│    │      │      └ \u001b[0m\u001b[36m\u001b[1m<dehb.optimizers.dehb.DEHB object at 0x7f6b40b1b2e0>\u001b[0m\n",
      "    \u001b[36m│    │      └ \u001b[0m\u001b[36m\u001b[1m<function Client.submit at 0x7f6b48317a60>\u001b[0m\n",
      "    \u001b[36m│    └ \u001b[0m\u001b[36m\u001b[1m<Client: 'tcp://127.0.0.1:33897' processes=2 threads=2, memory=12.44 GiB>\u001b[0m\n",
      "    \u001b[36m└ \u001b[0m\u001b[36m\u001b[1m<dehb.optimizers.dehb.DEHB object at 0x7f6b40b1b2e0>\u001b[0m\n",
      "\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/client.py\", line 1961, in submit\n",
      "    futures = self._graph_to_futures(\n",
      "              │    └ <function Client._graph_to_futures at 0x7f6b48319790>\n",
      "              └ <Client: 'tcp://127.0.0.1:33897' processes=2 threads=2, memory=12.44 GiB>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/client.py\", line 3158, in _graph_to_futures\n",
      "    header, frames = serialize(ToPickle(dsk), on_error=\"raise\")\n",
      "                     │         │        └ HighLevelGraph with 1 layers.\n",
      "                     │         │          <dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\n",
      "                     │         │           0. 140098536150272\n",
      "                     │         │          \n",
      "                     │         └ <class 'distributed.protocol.serialize.ToPickle'>\n",
      "                     └ <function serialize at 0x7f6b486cd040>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/distributed/protocol/serialize.py\", line 379, in serialize\n",
      "    raise TypeError(msg, str_x) from exc\n",
      "                    │    │           └ TypeError(\"cannot pickle '_thread.lock' object\")\n",
      "                    │    └ '<ToPickle: HighLevelGraph with 1 layers.\\n<dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\\n 0. 140098536150272...\n",
      "                    └ 'Could not serialize object of type HighLevelGraph'\n",
      "\n",
      "\u001b[31m\u001b[1mTypeError\u001b[0m:\u001b[1m ('Could not serialize object of type HighLevelGraph', '<ToPickle: HighLevelGraph with 1 layers.\\n<dask.highlevelgraph.HighLevelGraph object at 0x7f6b95909100>\\n 0. 140098536150272\\n>')\u001b[0m\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "cannot unpack non-iterable NoneType object",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[12], line 10\u001b[0m\n\u001b[1;32m      1\u001b[0m dehb \u001b[38;5;241m=\u001b[39m DEHB(\n\u001b[1;32m      2\u001b[0m     f\u001b[38;5;241m=\u001b[39mtarget_function,\n\u001b[1;32m      3\u001b[0m     dimensions\u001b[38;5;241m=\u001b[39mdimensions,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m      8\u001b[0m     n_workers\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m2\u001b[39m\n\u001b[1;32m      9\u001b[0m )\n\u001b[0;32m---> 10\u001b[0m trajectory, runtime, history \u001b[38;5;241m=\u001b[39m dehb\u001b[38;5;241m.\u001b[39mrun(\n\u001b[1;32m     11\u001b[0m     total_cost\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m20\u001b[39m,\n\u001b[1;32m     12\u001b[0m )\n\u001b[1;32m     14\u001b[0m \u001b[38;5;28mprint\u001b[39m(dehb\u001b[38;5;241m.\u001b[39mget_incumbents())\n",
      "\u001b[0;31mTypeError\u001b[0m: cannot unpack non-iterable NoneType object"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-06-18 22:22:38.699\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36mdehb.optimizers.dehb\u001b[0m:\u001b[36m_timeout_handler\u001b[0m:\u001b[36m351\u001b[0m - \u001b[33m\u001b[1mRuntime budget exhausted. Saving optimization checkpoint now.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "dehb = DEHB(\n",
    "    f=target_function,\n",
    "    dimensions=dimensions,\n",
    "    cs=cs,\n",
    "    min_fidelity=min_fidelity,\n",
    "    max_fidelity=max_fidelity,\n",
    "    output_path=\"./temp\",\n",
    "    n_workers=2\n",
    ")\n",
    "trajectory, runtime, history = dehb.run(\n",
    "    total_cost=20,\n",
    ")\n",
    "\n",
    "print(dehb.get_incumbents())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dehb.vector_to_configspace(dehb.inc_config))  # config as ConfigSpace configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-06-18 22:13:41.200\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36mdehb.optimizers.dehb\u001b[0m:\u001b[36m_timeout_handler\u001b[0m:\u001b[36m350\u001b[0m - \u001b[33m\u001b[1mRuntime budget exhausted. Saving optimization checkpoint now.\u001b[0m\n",
      "\u001b[32m2024-06-18 22:13:41.201\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mdehb.optimizers.dehb\u001b[0m:\u001b[36msave\u001b[0m:\u001b[36m946\u001b[0m - \u001b[1mSaving state to disk...\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "print(trajectory[-1], dehb.inc_score)\n",
    "print(dehb.vector_to_configspace(dehb.inc_config))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "As detailed above, the problem definition needs to be input to DEHB as the following information:\n",
    "* the *target_function* (`f`) that is the primary black-box function to optimize,\n",
    "* the fidelity range of `min_fidelity` and `max_fidelity` that allows the cheaper, faster gray-box optimization of `f` and\n",
    "* the search space or the input domain of the function `f`, that can be represented as a `ConfigSpace` object and passed to DEHB at initialization.\n",
    "\n",
    "\n",
    "Following which, DEHB can be run for any amount of practical real-world budget. It can be run for either:\n",
    "* a total amount of actual wallclock time, example one day (~86400 seconds), or\n",
    "* a total number of function evaluations, or the number of times we want the black-box to be accessed for evaluation, across all fidelities or\n",
    "* the total number of *brackets* we want to run the DEHB algorithm for.\n",
    "\n",
    "DEHB will terminate once its chosen runtime budget is exhausted, and report the incumbent found. DEHB, as an *anytime* algorithm, constantly writes to disk a lightweight `json` file with the best found configuration and its score seen till that point."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}