Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions docs/source/getting_started/environment_setup.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
.. _Set_up_Python:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As GitHub warning says, this needs to be included in the top level toctree.


Set up Python
=============

Please follow the instructions below to set up an appropriate environment to use PyHyperScattering. All of these setup steps should be run in a terminal, not a Jupyter notebook.

These instructions have been tested for the use of PyHyperScattering in a local JupyterLab notebook using the Anaconda distribution (https://www.anaconda.com/download) on a Windows computer. The instructions below *might* work for other platforms (e.g., NSLS II Jupyterhub, Google Colab), but there are no guarantees; recently, the NSLS II Jupyterhub has been especially incompatible PyHyperScattering.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A reference to a specific facility's platform is entirely inappropriate to have here. Also, simply setting this incompatibility in stone is unproductive; it would be prudent for NSLS2 staff to actually engage and make this work on their systems, especially with recent fixes to dependency management, things might work a lot better.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it is worth mentioning that Colab and Jupyterhub are container-based platforms but you do not really have shell access (or you only have them inside the container) so I wouldn't offhand expect the command line parts to work.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, also we shouldn't be endorsing a commercial product like Anaconda, miniforge is probably what I would recommend, or refer out to somebody else who can endorse a product


Download Git
------------

To aid this workflow, download Git (https://git-scm.com/download/win). Then in the command prompt (not Anaconda Prompt), run ``winget install --id Git.Git -e --source winget``. After this, if you are able to run ``git --version`` and have a version number outputted, the installation was successful. If Anaconda Prompt was open, it may need to be restarted.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said in the comment chain, if the user is going to use git - they don't need it unless they plan to develop - they should really use GitHub desktop which is a lot easier.


Create and activate an environment
----------------------------------

- Open the Anaconda Prompt. Do not use the terminal feature after opening JupyterLab.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idk what the second sentence means? Do not use the terminal within jupyterlab? I don't offhand know why.


- Identify the current conda environment, which appears in parentheses at the beginning of the command prompt. If a conda environment that is not base and is not the desired environment is active, deactivate the conda environment. Do not deactivate the base environment.

.. code-block:: bash

conda deactivate

- Create a new environment if a suitable one does not exist. Replace ``YOUR_ENVIRONMENT_NAME`` with a an environment name of choice that does not contain any spaces. If needed, the ``...`` can be replaced with other conda packages to be installed in this environment. The current notebook was run with the ``...`` omitted. After loading some packages, you will be asked if you want to proceed. Enter y (yes).

.. code-block:: bash

conda create -n YOUR_ENVIRONMENT_NAME ipykernel ...

If there already is an appropriate environment, skip this step. The purpose of the conda environment is to contain the necessary package versions that will enable data reduction and not conflict with other packages.

- Activate the desired environment. After running this command, the selected environment name should appear in parentheses in the command prompt.

.. code-block:: bash

conda activate YOUR_ENVIRONMENT_NAME

If you do not remember your environment name, you can run ``conda env list`` to display a list of environments that currently exist. If there is an environment you want to delete, first ensure it is not active, and then run ``conda remove -n YOUR_ENVIRONMENT_NAME --all``. The flat ``--all`` removes the entire environment.

- Run the following to add the environment to your Jupyter notebook selection. The display name and environment name do not have to be the same.

.. code-block:: bash

python -m ipykernel install --user --name YOUR_ENVIRONMENT_NAME --display-name YOUR_ENVIRONMENT_NAME

Install packages and dependencies
---------------------------------

Install Python
~~~~~~~~~~~~~~

Check the Python version. Use version 3.11 or lower for PyHyperScattering to work.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comment in thread -- is this true? it would be better to have a softer recommendation like "use a recent Python version but perhaps not the newest one"

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As of 0.2.9 we are now verified to be compatible with all Python up to current 3.13, so this is incorrect.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still running into issues. Below is a screenshot for the error I encountered with Python 3.13.2, and I ran into a similar error while when I downgraded to Python 3.13. I meant to attach the full log, but when I enter the file path into my Windows browser, I get a "Windows can't find" error. I don't encounter this error when I use Python 3.12. Noting this for reference until I can troubleshoot further, and then I'll update the instructions.

image

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those error messages look like you don't have a C compiler to build a 1.x numpy against Python 3.13; idk why it wouldn't be able to find it assuming you do have one. The CI build chain does fail on 3.13/windows but due to the container running out of resources while compiling, so I figured it would work. It does test successfully against 3.13 on Mac and linux.


.. code-block:: bash

python --version

If needed, install the correct Python version.

.. code-block:: bash

conda install python==3.11
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is really bad, it will default match to python==3.11.0 which doesn't have a bunch of security and bug fixes. instead, use something like "python<3.12" so you match to 3.11.11 or whatever the newest is.


If a CondaSSL error is encountered during this step, the following solution can be run, and then Python installation can be retried: https://github.com/conda/conda/issues/8273

Install PyHyperScattering
~~~~~~~~~~~~~~~~~~~~~~~~~

``pip install pyhyperscattering[bluesky,ui]`` installs PyHyperScattering. The ``bluesky`` portion installs Bluesky-related dependencies needed to access the NSLS II Tiled database. The ``ui`` portion installs the necessary dependencies to draw a mask. In some cases, it may be necessary to clone and check out a later PyHyperScattering commit or branch instead of the default version. Below are some examples.

- ``pip install "git+https://github.com/usnistgov/PyHyperScattering.git#egg=PyHyperScattering[bluesky, ui]"`` installs the latest commit on the main branch.

- ``pip install "git+https://github.com/usnistgov/PyHyperScattering.git@Issue170_UpdateDatabrokerImports#egg=PyHyperScattering[bluesky, ui]"`` installs the latest commit on the branch named ``Issue170_UpdateDatabrokerImports``.

- ``pip install "git+https://github.com/usnistgov/PyHyperScattering.git@6657973#egg=PyHyperScattering[bluesky, ui]"`` installs commit ``6657973``.


Install JupyterLab
~~~~~~~~~~~~~~~~~~

``pip install jupyterlab`` is required if using the Anaconda distribution but might not be necessary in other cases (e.g., NSLS II JupyterHub, Google Colab)

Other installs
~~~~~~~~~~~~~~

If there are errors during installation or later on, it might be necessary to install additional packages and then retry the pip installs. Below is a list of what might be needed.

- Microsoft C++ Build Tools (https://visualstudio.microsoft.com/visual-cpp-build-tools/). This is installed outside the Anaconda prompt. Computer should be restarted after this installation.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should explain that some dependencies of PHS use compiled C code and if there is no binary for your system, you need a C compiler such as MS Visual C++, as indicated in the error message you get.


- ``pip install --upgrade holoviews`` This may be necessary if mask drawing is not working. The ``--upgrade`` is necessary to ensure that the package will get upgraded even if some version of it is currently installed.

- ``pip install natsort`` allows use of the natsort package, but is not necessary for the main functioning of PyHyperScattering.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...do we use natsort anywhere? it should be added to requirements if so. if not, it isn't really in scope here?



Open JupyterLab
---------------

- Start up JupyterLab from the Anaconda command prompt. Do not open JupyterLab using Anaconda's GUI menu.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused as to why.


.. code-block:: bash

jupyter-lab

- When prompted to select a kernel, choose the desired environment. If not prompted, ensure that the kernel on the top right-hand corner of the page is set to the correct environment name.

- Proceed to using a Jupyter notebook of choice to reduce and analyze data.


Additional resources
--------------------
- Full list of PyHyperScattering dependencies: https://github.com/usnistgov/PyHyperScattering/blob/main/pyproject.toml
- Further guidance on creating and managing environments: https://jupyter.nsls2.bnl.gov/hub/guide
- Conda documentation: https://docs.conda.io/projects/conda/en/stable/
- Xarray documentation: https://docs.xarray.dev/en/stable/
- Numpy documentation: https://numpy.org/doc/2.1/
- MatPlotLib documentation: https://matplotlib.org/stable/index.html

Loading