41 error data server host environment variable is not set cannot continue

.gitignore

@@ -133,3 +133,5 @@ docs/
 cls/stylesheets/ssheet.txt
 
 bcm/backend.py
+
+**/scan_time_data/

README.md

@@ -25,9 +25,9 @@ pyStxm can be configured to connect to two different types of data acquisition s
 
 - **BlueSky with Epics**
 
-	This software is completely dependant on Epics for providing the connection to all positioners and counters as well as 
-	the engine for doing scans. When the software is started a connection is made to all configured devices and therefore 
-	the Epics side of the software must be running before attempting to start pyStxm.
+	This software is completely dependant on Epics for providing the connection to all positioners and counters as well 
+    as the engine for doing scans. When the software is started a connection is made to all configured devices and 
+    therefore the Epics side of the software must be running before attempting to start pyStxm.
 
 	Next there is a second process called **nx_server** that must be running in order to save NeXus **nxstxm** files, 
 	typically the nx_server process is running on a Windows or Linux computer that has a drive mapping to the base data 
@@ -40,41 +40,66 @@ pyStxm can be configured to connect to two different types of data acquisition s
 
 - **Pixelator with Epics or Tango**
 
-	If running pyStxm with Pixelator then the Pixelator controller must be running and publishing/replying on ports 56561 
-	and 56562 respectively. The Pixelator controller is a separate piece of software that is used to control the Pixelator
-	server which is a device that controls the beamline devices and the data acquisition process.
+	If running pyStxm with Pixelator then the Pixelator controller must be running and publishing/replying on default 
+    ports 56561 and 56562 respectively and the data will be published on port 56563. The Pixelator controller is a 
+    separate piece of software that is used to control the Pixelator server which is a device that controls the beamline 
+    devices and the data acquisition process.
 
-	To use pyStxm with Pixelator follow the instructions below for **Configure pyStxm with Pixelator**
+    To use pyStxm with Pixelator follow the instructions below for **Configure pyStxm with Pixelator**
 
 --------------------------------------
-### Installing
+### Installation
 
 On the machine where you are going to run pyStxm:
 1.	Install miniforge: https://conda-forge.org/download/
 2.	```conda create -n pyStxm python=3.11.5```
-3.	```git clone https://github.com/Canadian-Light-Source/pyStxm```
-4.	```pip install -r requirements.txt```
-5.	Edit the file ```reqd_env_vars.sh``` and set:
+3.  ```conda activate pyStxm```
+4.	```git clone https://github.com/Canadian-Light-Source/pyStxm```
+5.  ```cd pyStxm```
+6.	```pip install -r requirements.txt```
+7.	Edit the file ```reqd_env_vars.sh``` and set:
 
       ```shell
-      export DCS_HOST="localhost"
+      # required for EPICS only, if using Pixelator only you can leave these as is
+      export OPHYD_CONTROL_LAYER="caproto"
+
+      # this is the host name of the computer that is running the DCS (Pixelator Controller)
+      export DCS_HOST="IP ADDR HERE"
       export DCS_HOST_PROC_NAME="Pixelator"
-      export DCS_SUB_PORT="55561"
-      export DCS_REQ_PORT="55562"
-      export DATA_SUB_PORT="55563"
+      export DCS_SUB_PORT="56561"
+      export DCS_REQ_PORT="56562"
+
+      #this is the IP addr of the computer that is running the NXserver or Pixelator depending on your setup
+      export DATA_SERVER_HOST="IP ADDR HERE"
+      # this is the port that the data will be published on for pyStxm to display
+      export DATA_SUB_PORT="56563"
 
-      export PYSTXM_DATA_DIR =”/tmp”
-      export LINUX_DATA_DIR =”/tmp”
+      # this is the base path to the data
+      # NOTE: for Pixelator this path must match the path configured in the Pixelator software settings.json file
+      # for the entries `NeXusBaseDirectory` and `NeXusDiscardSubDirectory`
+      export DATA_DIR="PATH HERE"
+
+      # This is the IP address of the computer that has the binary cnxvalidate on it, 
+      # note this is only relevant for BlueSky configurations
+      export CNXVALIDATE_HOST_IPADDR="IP ADDR HERE"
+      export PATH_TO_CNXVALIDATE="DIR PATH HERE"
+      # this is the path to the directory that contains the nexus definitions cloned repo, Only needed if planning
+      # to validate nexus files using cnxvalidate via pyStxm
+      export PATH_TO_NX_DEFINITIONS="PATH HERE"
+
       ```
-The rest you can leave as is. If your configuration is for BlueSky with Epics it will not actively use the information
+If your configuration is for BlueSky with Epics it will not actively use the information
 in the DCS variables but they do need to exist.
 
-The reason for the two data directories is because pyStxm can run on Windows OS and if it is configured for BlueSky 
-then pyStxm needs to send the process that saves the files (nx_server) the paths to the data files. A Pixelator 
-configuration does not use nx_server so just set both ```PYSTXM_DATA_DIR``` and ```LINUX_DATA_DIR``` to the same 
-location of where Pixelator will be saving the data.
+8. Edit the `cls/applications/pyStxm/app.ini` file and set the **bl_config** to the configuration name that has been 
+created for your beamline.
+
+Example:
+```shell
+bl_config = sls_x07da_polLux
+```
 
-6.	The display needs to support a resolution of 3840x2160, find out the name of the display:
+9. The display needs to support a resolution of 3840x2160, find out the name of the display:
 Run xrandr and check its output:
 
       ```shell
@@ -87,26 +112,24 @@ Run xrandr and check its output:
       Here the display name is ```rdp0```
       If connecting over noMachine, edit the ```.bashrc``` file of the user account and add these at the bottom, 
 
-      NOTE you need the path to the ```reqd_env_vars.sh``` file you edited earlier:
-
       ```shell
       export QT_SCREEN_SCALE_FACTORS=1.0
       export QT_AUTO_SCREEN_SCALE_FACTOR=0
       export QT_SCALE_FACTOR=1.0
       xrandr --output rdp0 --mode 3840x2160
-      source /home/control/github/pyStxm/reqd_env_vars.sh
       ```
 
-7.	to run pyStxm execute the file runpyStxm.sh:
+10.	To run pyStxm execute the file runpyStxm.sh:
 
       ```shell
         ./runpyStxm.sh
       ```
 
 You should see the pyStxm splash screen come up as well as as see output on the console about devices that are being 
 connected to. 
-If the process appears to stop for a lengthy amount of time it is likely that pyStxm cannot see the Pixelator 
-Controller, make sure its running and that it is **PUB**lishing on port ```56561``` and **REP**lying on port ```56562```.
+If the process appears to stop for a lengthy amount of time it is likely that pyStxm cannot see **PixelatorController** 
+(Pixelator configurations) or **nx_server** (EPICS), make sure that the appropriate server is running and check that the
+the ports used to publish match those for subscribing.
 
 --------------------------------------
 ## nx_server ```(BlueSky with Epics only)```
@@ -126,7 +149,8 @@ the service is used to:
   (so that the instance of pyStxm can determine the appropriate data_dir format)
 
 The nx_server service must be installed on a computer that 
-has a mount point to the data computer such that the ```/etc/fstab``` file contains the entry in its list of mount points.
+has a mount point to the data computer such that the ```/etc/fstab``` file contains the entry in its list of mount 
+points.
 
 **Service Commands**:
 
@@ -149,18 +173,20 @@ has a mount point to the data computer such that the ```/etc/fstab``` file conta
 --------------------------------------
 ### Configure pyStxm with Pixelator
 
-1. First complete steps 1 through 6 above under **Installing**
+1. First complete steps 1 through 7 above under **Installation**
 
-2. Edit the file ```reqd_env_vars.sh``` and modify the following variables to your Pixelator settings:
-
-      ```shell
-      export DCS_HOST="localhost"
-      export DCS_HOST_PROC_NAME="Pixelator"
-      export DCS_SUB_PORT="56561"
-      export DCS_REQ_PORT="56562"
-      ```
+2. Ensure that the pixelator settings file located in teh Pixelator config directory, specifies the same port numbers as 
+above for the PUB, REP and data publishing sockets.
+
+Open the settings.json file and check/add the following entries:
+
+```json
+  "publisherPort": 56561,
+  "requestPort": 56562,
+  "dataPublisherPort": 56563,
+```
+The _dataPublisherPort_ is used to send requested data to pyStxm from a threaded task in PixelatorController.
 
-
 3. If Pixelator is running on the same machine as ```pyStxm``` **then you can skip this step**.
 
 
@@ -175,59 +201,19 @@ If Pixelator is going to run on a different machine than ```pyStxm```, setup por
   ssh <user>@<Pixelator server> -L 56561:localhost:56561 -L 56562:localhost:56562
   ```
 
-
-
 4. run pyStxm execute the file runpyStxm.sh:
     ```shell
       >./runpyStxm.sh
     ```
 
-pyStxm should start up, If the process appears to stop for a lengthy amount of time it is likely that pyStxm cannot see the Pixelator 
-Controller, make sure its running and that it is **PUB**lishing and **REP**lying on the port numbers that you entered
-in ```reqd_env_vars.sh```.
-
---------------------------------------
-### Prerequisites
-
-The pyStxm software is dependent on the following python modules (note that their individual dependencies are not listed):
-
- - guidata (3.1.1)
- - plotpy (2.0.0) 
- - h5py (3.10.0)
- - NeXpy (1.0.5)
- - nexusformat (1.0.2)
- - numpy (1.26.1)
- - pyepics (3.5.2)
- - qwt (0.11.1)
- - QtPy (2.4.1)
- - bluesky (1.12.0)
- - ophyd (1.9.0)
- - databroker (1.2.5)
- - caproto (1.1.0)
- - suitcase (0.11.0)
- - scipy (1.11.3)
- - simplejson (3.19.2)
- - pymongo (4.6.0)
-
-
+pyStxm should start up, If the process appears to stop for a lengthy amount of time it is likely that pyStxm cannot see 
+the PixelatorController, make sure its running and that it is **PUB**lishing and **REP**lying on the port numbers that 
+you entered in ```reqd_env_vars.sh```.
 
 ## Documentation
 
 The documentation for the software has been started and will soon be available on github pages:
 
-## Built With
-
-* [Python](https://www.python.org/) - The open source programming used
-* [Qt](https://www.qt.io/) - The open source application framework used
-* [BlueSky](https://nsls-ii.github.io/bluesky/) - Bluesky Data Collection Framework
-* [Epics](http://www.aps.anl.gov/epics/) - The open source device and data acquisition control
-* [Synapps](https://www1.aps.anl.gov/bcda/synapps/) The device and data collection software the 
-
-
-## Contributing
-
-Please read [CONTRIBUTING.md](https://gist.github.com/PurpleBooth/b24679402957c63ec426) for details on our code of conduct, and the process for submitting pull requests to us.
-
 ## Author
 
 * **Russ Berg** -  [pyStxm](https://github.com/Canadian-Light-Source/pyStxm)

cls/appWidgets/main_object.py

@@ -15,7 +15,7 @@
 from cls.utils.arrays import nulls_to_nans
 from cls.utils.dict_utils import dct_put, dct_get, dct_merge, find_key_in_dict
 from cls.utils.version import get_version
-from cls.utils.process_utils import check_windows_procs_running
+
 from cls.data_io.zmq_utils import send_to_server
 from cls.utils.roi_dict_defs import *
 from cls.utils.log import get_module_logger
@@ -44,8 +44,7 @@
 
 devq = Query()
 
-NX_SERVER_DATA_SUB_PORT = os.getenv('NX_SERVER_DATA_SUB_PORT', 56566)
-PIX_DATA_SUB_PORT = os.getenv('PIX_DATA_SUB_PORT', 55563)
+DATA_SUB_PORT = os.getenv('DATA_SUB_PORT', None)
 DATA_SERVER_HOST = os.getenv('DATA_SERVER_HOST', None)
 
 _logger = get_module_logger(__name__)
@@ -133,8 +132,7 @@ def __init__(self, name, endstation, beamline_cfg_dct=None, splash=None, main_cf
         self.device_backend = BACKEND #default
         self.zmq_dev_mgr = None
         self.dcs_settings = None
-        self.win_data_dir = self.data_dir = beamline_cfg_dct["BL_CFG_MAIN"]['data_dir']
-        self.linux_data_dir = beamline_cfg_dct["BL_CFG_MAIN"]['linux_data_dir']
+        self.data_dir = beamline_cfg_dct["BL_CFG_MAIN"]['data_dir']
         self.default_detector = beamline_cfg_dct["BL_CFG_MAIN"].get('default_detector', None)
         self.data_sub_context = zmq.Context()
 
@@ -147,29 +145,27 @@ def __init__(self, name, endstation, beamline_cfg_dct=None, splash=None, main_cf
             self.mongo_db_nm = None
             self.nx_server_host = None
             self.nx_server_port = None
-            self.nx_server_is_windows = None
             # SUB socket: Subscribing to the publisher
-            print(f"Connecting to data server at tcp://{DATA_SERVER_HOST}:{PIX_DATA_SUB_PORT}")
+            print(f"Connecting to data server at tcp://{DATA_SERVER_HOST}:{DATA_SUB_PORT}")
             self.data_sub_socket = self.data_sub_context.socket(zmq.SUB)
-            self.data_sub_socket.connect(f"tcp://{DATA_SERVER_HOST}:{PIX_DATA_SUB_PORT}")  # Connect to the PUB socket
+            self.data_sub_socket.connect(f"tcp://{DATA_SERVER_HOST}:{DATA_SUB_PORT}")  # Connect to the PUB socket
             self.data_sub_socket.setsockopt_string(zmq.SUBSCRIBE, "")  # Subscribe to all messages
 
         else:
             # SUB socket: Subscribing to the publisher
             self.data_sub_socket = self.data_sub_context.socket(zmq.SUB)
-            print(f"Connecting to data server at tcp://{DATA_SERVER_HOST}:{NX_SERVER_DATA_SUB_PORT}")
-            self.data_sub_socket.connect(f"tcp://{DATA_SERVER_HOST}:{NX_SERVER_DATA_SUB_PORT}")  # Connect to the PUB socket
+            print(f"Connecting to data server at tcp://{DATA_SERVER_HOST}:{DATA_SUB_PORT}")
+            self.data_sub_socket.connect(f"tcp://{DATA_SERVER_HOST}:{DATA_SUB_PORT}")  # Connect to the PUB socket
             self.data_sub_socket.setsockopt_string(zmq.SUBSCRIBE, "")  # Subscribe to all messages
 
             if main_cfg:
                 self.mongo_db_nm = main_cfg.get_value("MAIN", "mongo_db_nm")
-                self.nx_server_host = main_cfg.get_value("MAIN", "nx_server_host")
+                self.nx_server_host = DATA_SERVER_HOST
                 self.nx_server_port = main_cfg.get_value("MAIN", "nx_server_port")
             else:
                 self.mongo_db_nm = 'mongo_databroker'
                 self.nx_server_host = 'localhost'
                 self.nx_server_port = 5555
-            self.nx_server_is_windows = self.check_is_nx_server_windows()
 
         ver_dct = get_version()
 
@@ -341,11 +337,11 @@ def check_dcs_settings(self, settings: dict=None):
                 #update the dcs server
                 # self.engine_widget.engine.set_data_directory(self.data_dir)
 
-            if int(settings['dataPublisherPort']) != int(PIX_DATA_SUB_PORT):
-                _logger.error(f"Data publisher port in DCS server [{settings['dataPublisherPort']}] does not match the one in the GUI [{PIX_DATA_SUB_PORT}]")
-                print(f"ERROR: Data publisher port in DCS server [{settings['dataPublisherPort']}] does not match the one in the GUI [{PIX_DATA_SUB_PORT}]")
+            if int(settings['dataPublisherPort']) != int(DATA_SUB_PORT):
+                _logger.error(f"Data publisher port in DCS server [{settings['dataPublisherPort']}] does not match the one in the GUI [{DATA_SUB_PORT}]")
+                print(f"ERROR: Data publisher port in DCS server [{settings['dataPublisherPort']}] does not match the one in the GUI [{DATA_SUB_PORT}]")
                 #update the dcs server
-                #self.engine_widget.engine.set_data_publisher_port(PIX_DATA_SUB_PORT)
+                #self.engine_widget.engine.set_data_publisher_port(DATA_SUB_PORT)
 
             if int(settings['publisherPort']) != DCS_SUB_PORT:
                 _logger.error(f"Command publisher port in DCS server [{settings['publisherPort']}] does not match the one in the GUI [{DCS_SUB_PORT}]")
@@ -368,9 +364,6 @@ def reload_data_directory(self, data_dir: str=None):
         if data_dir is None:
             return
 
-        if not self.nx_server_is_windows:
-            data_dir = self.make_linux_data_dir(data_dir)
-
         if data_dir is None:
             data_dir = self.data_dir
             current_date = datetime.now().strftime('%Y-%m-%d')
@@ -665,14 +658,6 @@ def dev_exists(self, app_devname):
             return False
 
 
-    def make_linux_data_dir(self, data_dir: str) -> str:
-        """
-        translate the data_dir from windows to linux
-        """
-        data_dir = data_dir.replace(self.win_data_dir, self.linux_data_dir)
-        data_dir = data_dir.replace("\\", "/")
-        return data_dir
-
     def send_to_nx_server(self, cmnd, run_uids=[], fprefix='', data_dir='', nx_app_def=None, fpaths=[],
                           host='localhost', port=5555, verbose=False, cmd_args={}):
         """
@@ -714,9 +699,6 @@ def save_nx_files(self, run_uids: list, fprefix: str, data_dir: str, nx_app_def:
         """
         makes calls to save a scan file(s)
         """
-        if not self.nx_server_is_windows:
-            data_dir = self.make_linux_data_dir(data_dir)
-
         res = self.send_to_nx_server(NX_SERVER_CMNDS.SAVE_FILES, run_uids, fprefix, data_dir, nx_app_def=nx_app_def,
                                      host=self.nx_server_host, port=self.nx_server_port,
                                      verbose=verbose)
@@ -732,9 +714,6 @@ def remove_ptycho_tif_files(self, data_dir: str, fpaths: list, host: str = 'loca
         makes calls to save a scan file(s)
         """
         final_paths = fpaths
-        if not self.nx_server_is_windows:
-            data_dir = self.make_linux_data_dir(data_dir)
-
         res = self.send_to_nx_server(NX_SERVER_CMNDS.REMOVE_FILES, data_dir=data_dir, fpaths=final_paths,
                                      host=self.nx_server_host, port=self.nx_server_port, verbose=verbose)
         return res['status']
@@ -766,14 +745,7 @@ def check_nx_server_running(self):
         that nx_server is running, the MAIN_OBJ has already determined if the process is running on windows
         or linux
         """
-        if self.nx_server_is_windows:
-            return check_windows_procs_running(procs_to_check={
-                "DataRecorder Process": ("python.exe", "nx_server.py"),
-                "MongoDB": ("mongod.exe", None),
-            })
-        else:
-            # linux
-            return self.check_linux_nx_server_running()
+        return self.check_linux_nx_server_running()
 
     def check_linux_nx_server_running(self):
         '''

cls/applications/pyStxm/app.ini

@@ -4,11 +4,10 @@ autoSaveData = true
 
 # which beamline configuration to use for the devices, this is the name of the .ini file in the bl_configs directory
 # of the same name <lab>_<beamline>_<endstation>.ini
-# bl_config = sls_x07da_polLux
-bl_config = amb_bl10ID1
+bl_config = sls_x07da_polLux
+# bl_config = amb_bl10ID1
 # bl_config = uhv_bl10ID1
 mongo_db_nm = pystxm_amb_bl10ID1
-nx_server_host = ${NX_SERVER_HOST}
 nx_server_port = 5555
 
 #use_laser, 0 = no, 1 = yes