Configuration can be used with MP

Configuration can be now used with multiprocessing:

import multiprocessing as mp


def worker_config_checker(config):
    import numpy as np

    assert "a" in config
    assert config.a == 123

    assert "b" in config
    np.testing.assert_allclose(config.b, np.array([1, 2, 3]))

    assert "out" in config
    assert config.out == 12

if __name__ == "__main__":
    config = ConfigObject(config_filepath)
    ctx = mp.get_context("spawn")
    p = ctx.Process(target=worker_config_checker, args=(config,))
    p.start()
    p.join()

Deep Learning DDP example with Ignite :

• https://github.com/vfdev-5/py_config_runner/tree/master/examples#deep-learning-example-with-pytorch-ignite

Mutable configuration improvements

Configuration can be updated from the script before loading it and mutation can change types.

Limitations: mutation can not a local object or class instance.

Example

old_value = {"encoder": "E1", "decoder": "D1"} 
# old_value = "unet"
# old_value = [1, 2, 3]
# old_value = 5

new_value = "unet"
# new_value = [1, 2, 3]
# new_value = {"encoder": "E1", "decoder": "D1"}
# new_value = 5

# Create a configuration file:
filepath = os.path.join(dirname, "custom_module.py")
s = f"""
a = {old_value}
"""

with open(filepath, "w") as h:
   h.write(s)

# Load configuration:
config = ConfigObject(filepath, mutations={"a": new_value})
assert config.a == new_value

What's Changed

• Mutations can change types by @vfdev-5 in #16
• Updated docstring by @vfdev-5 in #17

Full Changelog: v0.3.0...v0.3.1

Mutable configuration

Configuration can be updated from the script before loading it.

Let's assume that configuration python file has learning_rate = 0.01 which is used to configure an optimizer:

# baseline.py configuration file

learning_rate = 0.01
optimizer = SGD(parameters, lr=learning_rate)

And we would like to override learning_rate from the script using above configuration file and has also optimizer updated accordingly:

# Script file using baseline.py configuration

config = ConfigObject("/path/to/baseline.py", mutations={"learning_rate": 0.05})
print(config)
print(config.optimizer)
assert config.lr == 0.05
assert config.optimizer.lr == 0.05

What's Changed

• Code updates by @vfdev-5 in #13
• Added mutations to config object by @vfdev-5 in #14
• Code cosmetics by @vfdev-5 in #15

Full Changelog: v0.2.1...v0.3.0

Minor internal update

• Fixed #10

ConfigObject as Python API without using CLI

• Expose ConfigObject as Python API without using CLI
• Another way to validate configuration (uses pydantic)

See example:

• https://github.com/vfdev-5/py_config_runner/tree/master/examples

Lazy config loading

• Lazy configuration loading
  • removed --manual-setup option

py_config_runner scripts/training.py configs/train/baseline.py

Bug fixes

Post release with bug fix of #1

Multiple executors + new options

1. Execute configuration as

• cmd : py_config_runner scripts/training.py configs/train/baseline.py
• module : python -u -m py_config_runner.__main__ scripts/training.py configs/train/baseline.py
• with specific launcher : python -m special_launcher py_config_runner_script scripts/training.py configs/train/baseline.py

2. Added option to load configuration manually in running script

py_config_runner scripts/training.py configs/train/baseline.py --manual_config_load

3. Documentation improvements

First tag

It is the first release of python configuration runner.
Added

• Documentation on ReadTheDocs