diff --git a/README.md b/README.md index e41afce..b995377 100644 --- a/README.md +++ b/README.md @@ -1,164 +1,223 @@ -multiprocess -============ -better multiprocessing and multithreading in Python +

+ + + + + + PyPI version + + Pypi ownload counter + Conda download counter + +

+

+ + StackOverflow help + + Donate + +

+ +## ⚡️ Introduction +[`multiprocess`](https://github.com/uqfoundation/multiprocess) is a Python library that provides multiprocessing and multithreading functionalities. +[`multiprocess`](https://github.com/uqfoundation/multiprocess) extends [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) by providing enhanced serialization through [`dill`](https://github.com/uqfoundation/dill). +[`multiprocess`](https://github.com/uqfoundation/multiprocess) leverages [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) to support the spawning of processes using the API of the Python standard library's [`threading`](https://docs.python.org/3/library/threading.html) module. +[`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) has been part of the standard library since Python 2.6. + +[`multiprocess`](https://github.com/uqfoundation/multiprocess) is part of [`pathos`](https://github.com/uqfoundation/pathos), a Python framework for heterogeneous computing. +[`multiprocess`](https://github.com/uqfoundation/multiprocess) is in active development, so any user feedback, bug reports, comments, +or suggestions are highly appreciated. +A list of known issues is located [here](https://github.com/uqfoundation/multiprocess/issues), with a legacy list maintained [here](https://uqfoundation.github.io/project/pathos/query). + +If you are looking for the dev version, click [here](#development-version) + +## ✨ Main Features +[`multiprocess`](https://github.com/uqfoundation/multiprocess) enables: -About Multiprocess ------------------- -``multiprocess`` is a fork of ``multiprocessing``. ``multiprocess`` extends ``multiprocessing`` to provide enhanced serialization, using `dill`. ``multiprocess`` leverages ``multiprocessing`` to support the spawning of processes using the API of the Python standard library's ``threading`` module. ``multiprocessing`` has been distributed as part of the standard library since Python 2.6. +* objects to be transferred between processes using pipes or multi-producer/multi-consumer queues +* objects to be shared between processes using a server process or (for simple data) shared memory -``multiprocess`` is part of ``pathos``, a Python framework for heterogeneous computing. -``multiprocess`` is in active development, so any user feedback, bug reports, comments, -or suggestions are highly appreciated. A list of issues is located at https://github.com/uqfoundation/multiprocess/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query. +[`multiprocess`](https://github.com/uqfoundation/multiprocess) provides: +* equivalents of all the synchronization primitives in [`threading`](https://docs.python.org/3/library/threading.html) +* a `Pool` class to facilitate submitting tasks to worker processes +* enhanced serialization, using [`dill`](https://github.com/uqfoundation/dill) -Major Features --------------- -``multiprocess`` enables: +## 🔌 Requirements +```bash +python>=3.7 +setuptools>=42 +dill>=0.3.6 +``` -* objects to be transferred between processes using pipes or multi-producer/multi-consumer queues -* objects to be shared between processes using a server process or (for simple data) shared memory +For Python 2, a C compiler is required to build the included extension module from source. +Python 3 and binary installs do not require a C compiler. -``multiprocess`` provides: +## 💾 Installation +[`multiprocess`](https://github.com/uqfoundation/multiprocess) can be installed with `pip` or `conda`: +```bash +pip install multiprocess +``` +```bash +conda install multiprocess +``` -* equivalents of all the synchronization primitives in ``threading`` -* a ``Pool`` class to facilitate submitting tasks to worker processes -* enhanced serialization, using ``dill`` +## 💡 Basic Usage +The [``multiprocess.Process``](https://multiprocess.readthedocs.io/en/latest/multiprocess.html#module-multiprocess.process) class follows the API of [``threading.Thread``](https://docs.python.org/3/library/threading.html#thread-objects): -Current Release -[![Downloads](https://static.pepy.tech/personalized-badge/multiprocess?period=total&units=international_system&left_color=grey&right_color=blue&left_text=pypi%20downloads)](https://pepy.tech/project/multiprocess) -[![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/multiprocess?color=blue&label=conda%20downloads)](https://anaconda.org/conda-forge/multiprocess) -[![Stack Overflow](https://img.shields.io/badge/stackoverflow-get%20help-black.svg)](https://stackoverflow.com/questions/tagged/multiprocess) ---------------- -The latest released version of ``multiprocess`` is available from: - https://pypi.org/project/multiprocess +```python +from multiprocess import Process, Queue -``multiprocess`` is distributed under a 3-clause BSD license, and is a fork of ``multiprocessing``. +def f(q): + q.put('hello world') +if __name__ == '__main__': + q = Queue() + p = Process(target=f, args=[q]) + p.start() + print(q.get()) + p.join() +``` -Development Version -[![Support](https://img.shields.io/badge/support-the%20UQ%20Foundation-purple.svg?style=flat&colorA=grey&colorB=purple)](http://www.uqfoundation.org/pages/donate.html) -[![Documentation Status](https://readthedocs.org/projects/multiprocess/badge/?version=latest)](https://multiprocess.readthedocs.io/en/latest/?badge=latest) -[![Build Status](https://travis-ci.com/uqfoundation/multiprocess.svg?label=build&logo=travis&branch=master)](https://travis-ci.com/github/uqfoundation/multiprocess) -[![codecov](https://codecov.io/gh/uqfoundation/multiprocess/branch/master/graph/badge.svg)](https://codecov.io/gh/uqfoundation/multiprocess) -------------------- -You can get the latest development version with all the shiny new features at: - https://github.com/uqfoundation +Synchronization primitives like [locks](https://en.wikipedia.org/wiki/Lock_(computer_science)), [semaphores](https://en.wikipedia.org/wiki/Semaphore_(programming)) and [conditions](https://en.wikipedia.org/wiki/Conditional_(computer_programming)) are available: +```python +from multiprocess import Condition -If you have a new contribution, please submit a pull request. +c = Condition() + +print(c) +>>> ), 0> + +c.acquire() +>>> True + +print(c) +>>> ), 0> +``` + +One can also use a manager to create shared objects either in shared memory or in a server process: + +```python +from multiprocess import Manager + +manager = Manager() +l = manager.list(range(10)) +l.reverse() + +print(l) +>>> [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] + +print(repr(l)) +>>> +``` + +Tasks can be offloaded to a [pool](https://en.wikipedia.org/wiki/Pool_(computer_science)) of worker processes in various ways: + +```python +from multiprocess import Pool + +def f(x): + return x*x + +p = Pool(4) +result = p.map_async(f, range(10)) +result.get(timeout=1) +``` +Output: +```python +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81] +``` + +When [`dill`](https://github.com/uqfoundation/dill) is installed, serialization is extended to most objects: + +```python +from multiprocess import Pool + +p = Pool(4) +p.map(lambda x: (lambda y:y**2)(x) + x, range(10)) +``` +Output: +```python +[0, 2, 6, 12, 20, 30, 42, 56, 72, 90] +``` + +## 📚 Additional Information + +You can find [`multiprocess`](https://github.com/uqfoundation/multiprocess) documentation [here](http://multiprocess.rtfd.io). + +``multiprocess.tests`` contains scripts that demonstrate how [`multiprocess`](https://github.com/uqfoundation/multiprocess) can be used to leverage multiple processes to execute Python in parallel. +You can run the test suite with ``python -m multiprocess.tests``. + +As [`multiprocess`](https://github.com/uqfoundation/multiprocess) conforms to the [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) interface, the examples and documentation found [here](http://docs.python.org/library/multiprocessing.html) also apply to [`multiprocess`](https://github.com/uqfoundation/multiprocess) if one will ``import multiprocess as multiprocessing``. + +Examples that demonstrate some basic use cases and benchmarking for Python code in parallel can be found [here](https://github.com/uqfoundation/multiprocess/tree/master/py3.11/examples). + +Please feel free to submit a [ticket on GitHub](https://github.com/uqfoundation/multiprocess/issues/new), or ask a question on [StackOverflow](https://stackoverflow.com/users/2379433/mike-mckerns) ([**@Mike McKerns**](https://stackoverflow.com/users/2379433/mike-mckerns)). If you would like to share how you use [`multiprocess`](https://github.com/uqfoundation/multiprocess) in your work, please send an email (to **mmckerns at uqfoundation dot org**). + + +## 🎓 Citation +If you use [`multiprocess`](https://github.com/uqfoundation/multiprocess) to do research that leads to publication, we ask that you +acknowledge the use of [`multiprocess`](https://github.com/uqfoundation/multiprocess) by citing the following in your publication: + +``` +M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis, +"Building a framework for predictive science", Proceedings of +the 10th Python in Science Conference, 2011; +http://arxiv.org/pdf/1202.1056 +``` +
+BibTeX + +```bibtex + @inproceedings{McKerns_1, + author = {Michael M. McKerns and + Leif Strand and + Tim Sullivan and + Alta Fang and + Michael A. G. Aivazis}, + title = {Building a Framework for Predictive Science}, + booktitle = {10th Python in Science Conference}, + year = {2011}, + url = {http://arxiv.org/pdf/1202.1056} + } +``` +
+ +``` +Michael McKerns and Michael Aivazis, +"pathos: a framework for heterogeneous computing", 2010- ; +https://uqfoundation.github.io/project/pathos +``` + +
+BibTeX + +```bibtex + @misc{McKerns_2, + author = {Michael M. McKerns and + Michael A. G. Aivazis}, + title = {pathos: a framework for heterogeneous computing}, + year = {2010}, + url = {https://uqfoundation.github.io/project/pathos} + } +``` +
-Installation ------------- -``multiprocess`` can be installed with ``pip``:: - - $ pip install multiprocess - -For Python 2, a C compiler is required to build the included extension module from source. Python 3 and binary installs do not require a C compiler. - - -Requirements ------------- -``multiprocess`` requires: - -* ``python`` (or ``pypy``), **>=3.7** -* ``setuptools``, **>=42** -* ``dill``, **>=0.3.6** - - -Basic Usage ------------ -The ``multiprocess.Process`` class follows the API of ``threading.Thread``. -For example :: - - from multiprocess import Process, Queue - - def f(q): - q.put('hello world') - - if __name__ == '__main__': - q = Queue() - p = Process(target=f, args=[q]) - p.start() - print (q.get()) - p.join() - -Synchronization primitives like locks, semaphores and conditions are -available, for example :: - - >>> from multiprocess import Condition - >>> c = Condition() - >>> print (c) - ), 0> - >>> c.acquire() - True - >>> print (c) - ), 0> - -One can also use a manager to create shared objects either in shared -memory or in a server process, for example :: - - >>> from multiprocess import Manager - >>> manager = Manager() - >>> l = manager.list(range(10)) - >>> l.reverse() - >>> print (l) - [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] - >>> print (repr(l)) - - -Tasks can be offloaded to a pool of worker processes in various ways, -for example :: - - >>> from multiprocess import Pool - >>> def f(x): return x*x - ... - >>> p = Pool(4) - >>> result = p.map_async(f, range(10)) - >>> print (result.get(timeout=1)) - [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] - -When ``dill`` is installed, serialization is extended to most objects, -for example :: - - >>> from multiprocess import Pool - >>> p = Pool(4) - >>> print (p.map(lambda x: (lambda y:y**2)(x) + x, xrange(10))) - [0, 2, 6, 12, 20, 30, 42, 56, 72, 90] - - -More Information ----------------- -Probably the best way to get started is to look at the documentation at -http://multiprocess.rtfd.io. Also see ``multiprocess.tests`` for scripts that -demonstrate how ``multiprocess`` can be used to leverge multiple processes -to execute Python in parallel. You can run the test suite with -``python -m multiprocess.tests``. As ``multiprocess`` conforms to the -``multiprocessing`` interface, the examples and documentation found at -http://docs.python.org/library/multiprocessing.html also apply to -``multiprocess`` if one will ``import multiprocessing as multiprocess``. -See https://github.com/uqfoundation/multiprocess/tree/master/py3.11/examples -for a set of examples that demonstrate some basic use cases and benchmarking -for running Python code in parallel. Please feel free to submit a ticket on -github, or ask a question on stackoverflow (**@Mike McKerns**). If you would -like to share how you use ``multiprocess`` in your work, please send an email -(to **mmckerns at uqfoundation dot org**). - - -Citation --------- -If you use ``multiprocess`` to do research that leads to publication, we ask that you -acknowledge use of ``multiprocess`` by citing the following in your publication:: - - M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis, - "Building a framework for predictive science", Proceedings of - the 10th Python in Science Conference, 2011; - http://arxiv.org/pdf/1202.1056 - - Michael McKerns and Michael Aivazis, - "pathos: a framework for heterogeneous computing", 2010- ; - https://uqfoundation.github.io/project/pathos Please see https://uqfoundation.github.io/project/pathos or http://arxiv.org/pdf/1202.1056 for further information. +## 📄 License +[`multiprocess`](https://github.com/uqfoundation/multiprocess) is distributed under a 3-clause BSD license, and is a fork of [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html). + +## Development Version + +[![Support](https://img.shields.io/badge/support-the%20UQ%20Foundation-purple.svg?style=flat&colorA=grey&colorB=purple)](http://www.uqfoundation.org/pages/donate.html) +[![Documentation Status](https://readthedocs.org/projects/multiprocess/badge/?version=latest)](https://multiprocess.readthedocs.io/en/latest/?badge=latest) +[![Build Status](https://travis-ci.com/uqfoundation/multiprocess.svg?label=build&logo=travis&branch=master)](https://travis-ci.com/github/uqfoundation/multiprocess) +[![codecov](https://codecov.io/gh/uqfoundation/multiprocess/branch/master/graph/badge.svg)](https://codecov.io/gh/uqfoundation/multiprocess) + +You can get the latest development version with all the shiny new features [here](https://github.com/uqfoundation). +If you have a new contribution, please submit a pull request.