diff --git a/.gitignore b/.gitignore index a72fa3d0..5bf8e751 100644 --- a/.gitignore +++ b/.gitignore @@ -133,3 +133,5 @@ docs/ cls/stylesheets/ssheet.txt bcm/backend.py + +**/scan_time_data/ diff --git a/README.md b/README.md index c9de9996..290ac9c8 100644 --- a/README.md +++ b/README.md @@ -25,9 +25,9 @@ pyStxm can be configured to connect to two different types of data acquisition s - **BlueSky with Epics** - This software is completely dependant on Epics for providing the connection to all positioners and counters as well as - the engine for doing scans. When the software is started a connection is made to all configured devices and therefore - the Epics side of the software must be running before attempting to start pyStxm. + This software is completely dependant on Epics for providing the connection to all positioners and counters as well + as the engine for doing scans. When the software is started a connection is made to all configured devices and + therefore the Epics side of the software must be running before attempting to start pyStxm. Next there is a second process called **nx_server** that must be running in order to save NeXus **nxstxm** files, typically the nx_server process is running on a Windows or Linux computer that has a drive mapping to the base data @@ -40,41 +40,66 @@ pyStxm can be configured to connect to two different types of data acquisition s - **Pixelator with Epics or Tango** - If running pyStxm with Pixelator then the Pixelator controller must be running and publishing/replying on ports 56561 - and 56562 respectively. The Pixelator controller is a separate piece of software that is used to control the Pixelator - server which is a device that controls the beamline devices and the data acquisition process. + If running pyStxm with Pixelator then the Pixelator controller must be running and publishing/replying on default + ports 56561 and 56562 respectively and the data will be published on port 56563. The Pixelator controller is a + separate piece of software that is used to control the Pixelator server which is a device that controls the beamline + devices and the data acquisition process. - To use pyStxm with Pixelator follow the instructions below for **Configure pyStxm with Pixelator** + To use pyStxm with Pixelator follow the instructions below for **Configure pyStxm with Pixelator** -------------------------------------- -### Installing +### Installation On the machine where you are going to run pyStxm: 1. Install miniforge: https://conda-forge.org/download/ 2. ```conda create -n pyStxm python=3.11.5``` -3. ```git clone https://github.com/Canadian-Light-Source/pyStxm``` -4. ```pip install -r requirements.txt``` -5. Edit the file ```reqd_env_vars.sh``` and set: +3. ```conda activate pyStxm``` +4. ```git clone https://github.com/Canadian-Light-Source/pyStxm``` +5. ```cd pyStxm``` +6. ```pip install -r requirements.txt``` +7. Edit the file ```reqd_env_vars.sh``` and set: ```shell - export DCS_HOST="localhost" + # required for EPICS only, if using Pixelator only you can leave these as is + export OPHYD_CONTROL_LAYER="caproto" + + # this is the host name of the computer that is running the DCS (Pixelator Controller) + export DCS_HOST="IP ADDR HERE" export DCS_HOST_PROC_NAME="Pixelator" - export DCS_SUB_PORT="55561" - export DCS_REQ_PORT="55562" - export DATA_SUB_PORT="55563" + export DCS_SUB_PORT="56561" + export DCS_REQ_PORT="56562" + + #this is the IP addr of the computer that is running the NXserver or Pixelator depending on your setup + export DATA_SERVER_HOST="IP ADDR HERE" + # this is the port that the data will be published on for pyStxm to display + export DATA_SUB_PORT="56563" - export PYSTXM_DATA_DIR =”/tmp” - export LINUX_DATA_DIR =”/tmp” + # this is the base path to the data + # NOTE: for Pixelator this path must match the path configured in the Pixelator software settings.json file + # for the entries `NeXusBaseDirectory` and `NeXusDiscardSubDirectory` + export DATA_DIR="PATH HERE" + + # This is the IP address of the computer that has the binary cnxvalidate on it, + # note this is only relevant for BlueSky configurations + export CNXVALIDATE_HOST_IPADDR="IP ADDR HERE" + export PATH_TO_CNXVALIDATE="DIR PATH HERE" + # this is the path to the directory that contains the nexus definitions cloned repo, Only needed if planning + # to validate nexus files using cnxvalidate via pyStxm + export PATH_TO_NX_DEFINITIONS="PATH HERE" + ``` -The rest you can leave as is. If your configuration is for BlueSky with Epics it will not actively use the information +If your configuration is for BlueSky with Epics it will not actively use the information in the DCS variables but they do need to exist. -The reason for the two data directories is because pyStxm can run on Windows OS and if it is configured for BlueSky -then pyStxm needs to send the process that saves the files (nx_server) the paths to the data files. A Pixelator -configuration does not use nx_server so just set both ```PYSTXM_DATA_DIR``` and ```LINUX_DATA_DIR``` to the same -location of where Pixelator will be saving the data. +8. Edit the `cls/applications/pyStxm/app.ini` file and set the **bl_config** to the configuration name that has been +created for your beamline. + +Example: +```shell +bl_config = sls_x07da_polLux +``` -6. The display needs to support a resolution of 3840x2160, find out the name of the display: +9. The display needs to support a resolution of 3840x2160, find out the name of the display: Run xrandr and check its output: ```shell @@ -87,17 +112,14 @@ Run xrandr and check its output: Here the display name is ```rdp0``` If connecting over noMachine, edit the ```.bashrc``` file of the user account and add these at the bottom, - NOTE you need the path to the ```reqd_env_vars.sh``` file you edited earlier: - ```shell export QT_SCREEN_SCALE_FACTORS=1.0 export QT_AUTO_SCREEN_SCALE_FACTOR=0 export QT_SCALE_FACTOR=1.0 xrandr --output rdp0 --mode 3840x2160 - source /home/control/github/pyStxm/reqd_env_vars.sh ``` -7. to run pyStxm execute the file runpyStxm.sh: +10. To run pyStxm execute the file runpyStxm.sh: ```shell ./runpyStxm.sh @@ -105,8 +127,9 @@ Run xrandr and check its output: You should see the pyStxm splash screen come up as well as as see output on the console about devices that are being connected to. -If the process appears to stop for a lengthy amount of time it is likely that pyStxm cannot see the Pixelator -Controller, make sure its running and that it is **PUB**lishing on port ```56561``` and **REP**lying on port ```56562```. +If the process appears to stop for a lengthy amount of time it is likely that pyStxm cannot see **PixelatorController** +(Pixelator configurations) or **nx_server** (EPICS), make sure that the appropriate server is running and check that the +the ports used to publish match those for subscribing. -------------------------------------- ## nx_server ```(BlueSky with Epics only)``` @@ -126,7 +149,8 @@ the service is used to: (so that the instance of pyStxm can determine the appropriate data_dir format) The nx_server service must be installed on a computer that -has a mount point to the data computer such that the ```/etc/fstab``` file contains the entry in its list of mount points. +has a mount point to the data computer such that the ```/etc/fstab``` file contains the entry in its list of mount +points. **Service Commands**: @@ -149,18 +173,20 @@ has a mount point to the data computer such that the ```/etc/fstab``` file conta -------------------------------------- ### Configure pyStxm with Pixelator -1. First complete steps 1 through 6 above under **Installing** +1. First complete steps 1 through 7 above under **Installation** -2. Edit the file ```reqd_env_vars.sh``` and modify the following variables to your Pixelator settings: - - ```shell - export DCS_HOST="localhost" - export DCS_HOST_PROC_NAME="Pixelator" - export DCS_SUB_PORT="56561" - export DCS_REQ_PORT="56562" - ``` +2. Ensure that the pixelator settings file located in teh Pixelator config directory, specifies the same port numbers as +above for the PUB, REP and data publishing sockets. + +Open the settings.json file and check/add the following entries: + +```json + "publisherPort": 56561, + "requestPort": 56562, + "dataPublisherPort": 56563, +``` +The _dataPublisherPort_ is used to send requested data to pyStxm from a threaded task in PixelatorController. - 3. If Pixelator is running on the same machine as ```pyStxm``` **then you can skip this step**. @@ -175,59 +201,19 @@ If Pixelator is going to run on a different machine than ```pyStxm```, setup por ssh @ -L 56561:localhost:56561 -L 56562:localhost:56562 ``` - - 4. run pyStxm execute the file runpyStxm.sh: ```shell >./runpyStxm.sh ``` -pyStxm should start up, If the process appears to stop for a lengthy amount of time it is likely that pyStxm cannot see the Pixelator -Controller, make sure its running and that it is **PUB**lishing and **REP**lying on the port numbers that you entered -in ```reqd_env_vars.sh```. - --------------------------------------- -### Prerequisites - -The pyStxm software is dependent on the following python modules (note that their individual dependencies are not listed): - - - guidata (3.1.1) - - plotpy (2.0.0) - - h5py (3.10.0) - - NeXpy (1.0.5) - - nexusformat (1.0.2) - - numpy (1.26.1) - - pyepics (3.5.2) - - qwt (0.11.1) - - QtPy (2.4.1) - - bluesky (1.12.0) - - ophyd (1.9.0) - - databroker (1.2.5) - - caproto (1.1.0) - - suitcase (0.11.0) - - scipy (1.11.3) - - simplejson (3.19.2) - - pymongo (4.6.0) - - +pyStxm should start up, If the process appears to stop for a lengthy amount of time it is likely that pyStxm cannot see +the PixelatorController, make sure its running and that it is **PUB**lishing and **REP**lying on the port numbers that +you entered in ```reqd_env_vars.sh```. ## Documentation The documentation for the software has been started and will soon be available on github pages: -## Built With - -* [Python](https://www.python.org/) - The open source programming used -* [Qt](https://www.qt.io/) - The open source application framework used -* [BlueSky](https://nsls-ii.github.io/bluesky/) - Bluesky Data Collection Framework -* [Epics](http://www.aps.anl.gov/epics/) - The open source device and data acquisition control -* [Synapps](https://www1.aps.anl.gov/bcda/synapps/) The device and data collection software the - - -## Contributing - -Please read [CONTRIBUTING.md](https://gist.github.com/PurpleBooth/b24679402957c63ec426) for details on our code of conduct, and the process for submitting pull requests to us. - ## Author * **Russ Berg** - [pyStxm](https://github.com/Canadian-Light-Source/pyStxm) diff --git a/cls/appWidgets/main_object.py b/cls/appWidgets/main_object.py index e148aebc..7a3a08dc 100644 --- a/cls/appWidgets/main_object.py +++ b/cls/appWidgets/main_object.py @@ -15,7 +15,7 @@ from cls.utils.arrays import nulls_to_nans from cls.utils.dict_utils import dct_put, dct_get, dct_merge, find_key_in_dict from cls.utils.version import get_version -from cls.utils.process_utils import check_windows_procs_running + from cls.data_io.zmq_utils import send_to_server from cls.utils.roi_dict_defs import * from cls.utils.log import get_module_logger @@ -44,8 +44,7 @@ devq = Query() -NX_SERVER_DATA_SUB_PORT = os.getenv('NX_SERVER_DATA_SUB_PORT', 56566) -PIX_DATA_SUB_PORT = os.getenv('PIX_DATA_SUB_PORT', 55563) +DATA_SUB_PORT = os.getenv('DATA_SUB_PORT', None) DATA_SERVER_HOST = os.getenv('DATA_SERVER_HOST', None) _logger = get_module_logger(__name__) @@ -133,8 +132,7 @@ def __init__(self, name, endstation, beamline_cfg_dct=None, splash=None, main_cf self.device_backend = BACKEND #default self.zmq_dev_mgr = None self.dcs_settings = None - self.win_data_dir = self.data_dir = beamline_cfg_dct["BL_CFG_MAIN"]['data_dir'] - self.linux_data_dir = beamline_cfg_dct["BL_CFG_MAIN"]['linux_data_dir'] + self.data_dir = beamline_cfg_dct["BL_CFG_MAIN"]['data_dir'] self.default_detector = beamline_cfg_dct["BL_CFG_MAIN"].get('default_detector', None) self.data_sub_context = zmq.Context() @@ -147,29 +145,27 @@ def __init__(self, name, endstation, beamline_cfg_dct=None, splash=None, main_cf self.mongo_db_nm = None self.nx_server_host = None self.nx_server_port = None - self.nx_server_is_windows = None # SUB socket: Subscribing to the publisher - print(f"Connecting to data server at tcp://{DATA_SERVER_HOST}:{PIX_DATA_SUB_PORT}") + print(f"Connecting to data server at tcp://{DATA_SERVER_HOST}:{DATA_SUB_PORT}") self.data_sub_socket = self.data_sub_context.socket(zmq.SUB) - self.data_sub_socket.connect(f"tcp://{DATA_SERVER_HOST}:{PIX_DATA_SUB_PORT}") # Connect to the PUB socket + self.data_sub_socket.connect(f"tcp://{DATA_SERVER_HOST}:{DATA_SUB_PORT}") # Connect to the PUB socket self.data_sub_socket.setsockopt_string(zmq.SUBSCRIBE, "") # Subscribe to all messages else: # SUB socket: Subscribing to the publisher self.data_sub_socket = self.data_sub_context.socket(zmq.SUB) - print(f"Connecting to data server at tcp://{DATA_SERVER_HOST}:{NX_SERVER_DATA_SUB_PORT}") - self.data_sub_socket.connect(f"tcp://{DATA_SERVER_HOST}:{NX_SERVER_DATA_SUB_PORT}") # Connect to the PUB socket + print(f"Connecting to data server at tcp://{DATA_SERVER_HOST}:{DATA_SUB_PORT}") + self.data_sub_socket.connect(f"tcp://{DATA_SERVER_HOST}:{DATA_SUB_PORT}") # Connect to the PUB socket self.data_sub_socket.setsockopt_string(zmq.SUBSCRIBE, "") # Subscribe to all messages if main_cfg: self.mongo_db_nm = main_cfg.get_value("MAIN", "mongo_db_nm") - self.nx_server_host = main_cfg.get_value("MAIN", "nx_server_host") + self.nx_server_host = DATA_SERVER_HOST self.nx_server_port = main_cfg.get_value("MAIN", "nx_server_port") else: self.mongo_db_nm = 'mongo_databroker' self.nx_server_host = 'localhost' self.nx_server_port = 5555 - self.nx_server_is_windows = self.check_is_nx_server_windows() ver_dct = get_version() @@ -341,11 +337,11 @@ def check_dcs_settings(self, settings: dict=None): #update the dcs server # self.engine_widget.engine.set_data_directory(self.data_dir) - if int(settings['dataPublisherPort']) != int(PIX_DATA_SUB_PORT): - _logger.error(f"Data publisher port in DCS server [{settings['dataPublisherPort']}] does not match the one in the GUI [{PIX_DATA_SUB_PORT}]") - print(f"ERROR: Data publisher port in DCS server [{settings['dataPublisherPort']}] does not match the one in the GUI [{PIX_DATA_SUB_PORT}]") + if int(settings['dataPublisherPort']) != int(DATA_SUB_PORT): + _logger.error(f"Data publisher port in DCS server [{settings['dataPublisherPort']}] does not match the one in the GUI [{DATA_SUB_PORT}]") + print(f"ERROR: Data publisher port in DCS server [{settings['dataPublisherPort']}] does not match the one in the GUI [{DATA_SUB_PORT}]") #update the dcs server - #self.engine_widget.engine.set_data_publisher_port(PIX_DATA_SUB_PORT) + #self.engine_widget.engine.set_data_publisher_port(DATA_SUB_PORT) if int(settings['publisherPort']) != DCS_SUB_PORT: _logger.error(f"Command publisher port in DCS server [{settings['publisherPort']}] does not match the one in the GUI [{DCS_SUB_PORT}]") @@ -368,9 +364,6 @@ def reload_data_directory(self, data_dir: str=None): if data_dir is None: return - if not self.nx_server_is_windows: - data_dir = self.make_linux_data_dir(data_dir) - if data_dir is None: data_dir = self.data_dir current_date = datetime.now().strftime('%Y-%m-%d') @@ -665,14 +658,6 @@ def dev_exists(self, app_devname): return False - def make_linux_data_dir(self, data_dir: str) -> str: - """ - translate the data_dir from windows to linux - """ - data_dir = data_dir.replace(self.win_data_dir, self.linux_data_dir) - data_dir = data_dir.replace("\\", "/") - return data_dir - def send_to_nx_server(self, cmnd, run_uids=[], fprefix='', data_dir='', nx_app_def=None, fpaths=[], host='localhost', port=5555, verbose=False, cmd_args={}): """ @@ -714,9 +699,6 @@ def save_nx_files(self, run_uids: list, fprefix: str, data_dir: str, nx_app_def: """ makes calls to save a scan file(s) """ - if not self.nx_server_is_windows: - data_dir = self.make_linux_data_dir(data_dir) - res = self.send_to_nx_server(NX_SERVER_CMNDS.SAVE_FILES, run_uids, fprefix, data_dir, nx_app_def=nx_app_def, host=self.nx_server_host, port=self.nx_server_port, verbose=verbose) @@ -732,9 +714,6 @@ def remove_ptycho_tif_files(self, data_dir: str, fpaths: list, host: str = 'loca makes calls to save a scan file(s) """ final_paths = fpaths - if not self.nx_server_is_windows: - data_dir = self.make_linux_data_dir(data_dir) - res = self.send_to_nx_server(NX_SERVER_CMNDS.REMOVE_FILES, data_dir=data_dir, fpaths=final_paths, host=self.nx_server_host, port=self.nx_server_port, verbose=verbose) return res['status'] @@ -766,14 +745,7 @@ def check_nx_server_running(self): that nx_server is running, the MAIN_OBJ has already determined if the process is running on windows or linux """ - if self.nx_server_is_windows: - return check_windows_procs_running(procs_to_check={ - "DataRecorder Process": ("python.exe", "nx_server.py"), - "MongoDB": ("mongod.exe", None), - }) - else: - # linux - return self.check_linux_nx_server_running() + return self.check_linux_nx_server_running() def check_linux_nx_server_running(self): ''' diff --git a/cls/applications/pyStxm/app.ini b/cls/applications/pyStxm/app.ini index ad864996..663b0076 100644 --- a/cls/applications/pyStxm/app.ini +++ b/cls/applications/pyStxm/app.ini @@ -4,11 +4,10 @@ autoSaveData = true # which beamline configuration to use for the devices, this is the name of the .ini file in the bl_configs directory # of the same name __.ini -# bl_config = sls_x07da_polLux -bl_config = amb_bl10ID1 +bl_config = sls_x07da_polLux +# bl_config = amb_bl10ID1 # bl_config = uhv_bl10ID1 mongo_db_nm = pystxm_amb_bl10ID1 -nx_server_host = ${NX_SERVER_HOST} nx_server_port = 5555 #use_laser, 0 = no, 1 = yes diff --git a/cls/applications/pyStxm/bl_configs/amb_bl10ID1/amb_bl10ID1.ini b/cls/applications/pyStxm/bl_configs/amb_bl10ID1/amb_bl10ID1.ini index 3773746c..04600829 100644 --- a/cls/applications/pyStxm/bl_configs/amb_bl10ID1/amb_bl10ID1.ini +++ b/cls/applications/pyStxm/bl_configs/amb_bl10ID1/amb_bl10ID1.ini @@ -5,11 +5,7 @@ endstation_name=Ambient STXM endstation_prefix=amb_bl10ID1 dcs_backend=epics datafile_prefix=A - -# NOTE: data dirs need to end with the same last directory name -# WINDOWS data_dir = T:\operations\STXM-data\ASTXM_upgrade_tmp\2024 -data_dir = ${PYSTXM_DATA_DIR} -linux_data_dir = ${LINUX_DATA_DIR} +data_dir = ${DATA_DIR} # data_sub_dir can specifiy an explicit string or a keyword such as _cur_date_ which will internally then generate # the data_sub_dir to be the string "2025-02-19" for example diff --git a/cls/applications/pyStxm/bl_configs/sls_x07da_polLux/sls_x07da_polLux.ini b/cls/applications/pyStxm/bl_configs/sls_x07da_polLux/sls_x07da_polLux.ini index 0f7ecc41..1b75c8c8 100644 --- a/cls/applications/pyStxm/bl_configs/sls_x07da_polLux/sls_x07da_polLux.ini +++ b/cls/applications/pyStxm/bl_configs/sls_x07da_polLux/sls_x07da_polLux.ini @@ -5,10 +5,8 @@ endstation_name=PolLux endstation_prefix=sls_x07da_polLux dcs_backend=zmq datafile_prefix=S +data_dir = ${DATA_DIR} -# NOTE: data dirs need to end with the same last directory name -data_dir = ${PYSTXM_DATA_DIR} -linux_data_dir = ${LINUX_DATA_DIR} # data_sub_dir can specifiy an explicit string or a keyword such as _cur_date_ which will internally then generate # the data_sub_dir to be the string "2025-02-19" for example # the keyword _default_ will create sub dirs of /guest/<2 digit day 2 digit month> diff --git a/cls/scanning/nexus/nxstxm.py b/cls/scanning/nexus/nxstxm.py index 7d853695..3aeccb41 100644 --- a/cls/scanning/nexus/nxstxm.py +++ b/cls/scanning/nexus/nxstxm.py @@ -19,7 +19,7 @@ # import nxs import numpy as np import h5py -import pkg_resources +import importlib.metadata import nexpy from cls.utils.json_threadsave import dict_to_json_string diff --git a/reqd_env_vars.sh b/reqd_env_vars.sh index 3aeef657..0d5897c2 100644 --- a/reqd_env_vars.sh +++ b/reqd_env_vars.sh @@ -1,27 +1,27 @@ #!/bin/bash +# required for EPICS only, if using Pixelator only you can leave these as is +export OPHYD_CONTROL_LAYER="caproto" -# this is the host name of th ecomputer that is running the DCS (Pixelator Controller) -export DCS_HOST="localhost" +# this is the host name of the computer that is running the DCS (Pixelator Controller) +export DCS_HOST="IP ADDR HERE" export DCS_HOST_PROC_NAME="Pixelator" export DCS_SUB_PORT="56561" export DCS_REQ_PORT="56562" -# This is the IP address of the computer that has the binary cnxvalidate on it, note this is only relevant for BlueSky configurations -# Pixelator configs save their own files -export CNXVALIDATE_HOST_IPADDR="IP ADDR HERE" -export PATH_TO_CNXVALIDATE="DIR PATH HERE" -#this is the IP addr of the computer that is running the NXserver, again only relevant for BlueSky configs -export NX_SERVER_HOST="IP ADDR HERE" - -# this is the path to the directory that contains the nexus definitions cloned repo -export PATH_TO_NX_DEFINITIONS="PATH HERE" - - -# if pyStxm is run on windows then the path to the main data directory will needs to be specified here as a windows path -export PYSTXM_DATA_DIR="/tmp" -# this is the path used by the areaDetector plugin to save the data for tomography scans, NOTE: -export LINUX_DATA_DIR="/tmp" - - +#this is the IP addr of the computer that is running the NXserver or Pixelator depending on your setup +export DATA_SERVER_HOST="IP ADDR HERE" +# this is the port that the data will be published on for pyStxm to display +export DATA_SUB_PORT="56563" +# this is the base path to the data +# NOTE: for Pixelator this path must match the path configured in the Pixelator software settings.json file +# for the entries `NeXusBaseDirectory` and `NeXusDiscardSubDirectory` +export DATA_DIR="/tmp" +# This is the IP address of the computer that has the binary cnxvalidate on it, +# note this is only relevant for BlueSky configurations +export CNXVALIDATE_HOST_IPADDR="IP ADDR HERE" +export PATH_TO_CNXVALIDATE="DIR PATH HERE" +# this is the path to the directory that contains the nexus definitions cloned repo, Only needed if planning +# to validate nexus files using cnxvalidate via pyStxm +export PATH_TO_NX_DEFINITIONS="PATH HERE" \ No newline at end of file