Update CI by AbbyGi · Pull Request #24 · NSLS2/pdf-profile-collection

AbbyGi · 2021-03-02T16:23:11Z

Updating to test latest environment

mrakitin · 2021-03-03T15:39:06Z

Power-cycled the PR to pick up the new version of xpdconf with the fix for the missing examples/*.yaml files (nsls-ii-forge/xpdconf-feedstock#5).

AbbyGi · 2021-03-09T16:45:08Z

@tacaswell, we're trying to update the CI at PDF and we came across this error:

[E 2021-03-09 16:18:44,840.840 nslsii.ipython  logutils:56] could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:numpy.core.multiarray.scalar'
  in "/home/vsts/.config/acq/glbl.yml", line 15, column 13
Traceback (most recent call last):
  File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3418, in run_code
    return self.construct_document(node)
  File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 60, in construct_document
    for dummy in generator:
  File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
  File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 143, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 100, in construct_object
    data = constructor(self, node)
  File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 429, in construct_undefined
    node.start_mark)
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:numpy.core.multiarray.scalar'
  in "/home/vsts/.config/acq/glbl.yml", line 15, column 13

We were able to reproduce the error locally by copying the pdf.yml and glbl.yml files from your comment (#23 (comment)) and then importing

from xpdacq.xpdacq_conf import (glbl_dict, configure_device,
                                _reload_glbl, _set_glbl,
                                _load_beamline_config)

from https://github.com/NSLS-II-PDF/profile_collection/blob/mnt_prep_for_demo/startup/94-load.py#L19-L21

ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:numpy.core.multiarray.scalar' 
  in "/Users/abigailgiles/.config/acq/glbl.yml", line 15, column 13

cc @st3107

st3107 · 2021-03-09T17:02:29Z

I will look into this.

…

On Tue, Mar 9, 2021 at 11:50 AM AbbyGi ***@***.***> wrote: @tacaswell <https://github.com/tacaswell>, we're trying to update the CI at PDF and we came across this error: [E 2021-03-09 16:18:44,840.840 nslsii.ipython logutils:56] could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:numpy.core.multiarray.scalar' in "/home/vsts/.config/acq/glbl.yml", line 15, column 13Traceback (most recent call last): File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3418, in run_code return self.construct_document(node) File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 60, in construct_document for dummy in generator: File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 413, in construct_yaml_map value = self.construct_mapping(node) File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 218, in construct_mapping return super().construct_mapping(node, deep=deep) File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 143, in construct_mapping value = self.construct_object(value_node, deep=deep) File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 100, in construct_object data = constructor(self, node) File "/home/vsts/miniconda/envs/collection-2021-1.0/lib/python3.7/site-packages/yaml/constructor.py", line 429, in construct_undefined node.start_mark)yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:numpy.core.multiarray.scalar' in "/home/vsts/.config/acq/glbl.yml", line 15, column 13 We were able to reproduce the error locally by copying the pdf.yml and glbl.yml files from your comment (#23 (comment) <#23 (comment)>) and then importing from xpdacq.xpdacq_conf import (glbl_dict, configure_device, _reload_glbl, _set_glbl, _load_beamline_config) from https://github.com/NSLS-II-PDF/profile_collection/blob/mnt_prep_for_demo/startup/94-load.py#L19-L21 ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:numpy.core.multiarray.scalar' in "/Users/abigailgiles/.config/acq/glbl.yml", line 15, column 13 cc @st3107 <https://github.com/st3107> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI27NDZUUCS4ITSA6LIGGFTTCY7UVANCNFSM4YPJHUFA> .

-- [image: photo] *Songsheng Tao* Ph. D. student, Columbia University 347-749-7947 Create your own email signature <https://www.wisestamp.com/create-own-signature/?utm_source=promotion&utm_medium=signature&utm_campaign=create_your_own&srcid=>

mrakitin · 2021-03-09T17:06:01Z

I will look into this.

Thanks, @st3107. I think we are using glbl.yml in the wrong location. @AbbyGi and I are trying to move it to a different place.

Are there any instructions on how to deal with the xpdAcq configuration? It looks like a very complicated system, so good guidance may be needed.

AbbyGi · 2021-03-09T17:15:53Z

TODO next:
Address this error:

[E 2021-03-09 17:12:57,318.318 nslsii.ipython  logutils:56] [Errno 2] No such file or directory: '/mnt/data/bnl/xpdacq_special/data/xpdConfig/xpd_beamline_config.yml'

st3107 · 2021-03-12T16:49:47Z

I just figured out the xpdAcq configuration system. In total, the configuration requires two files: (a) a basic configuration yaml file for the software (b) a xpd_beamline_config.yaml for the metadata of the beamline.

How the configuration are loaded:

First, the software will search the yaml file in the following places:

~/.config/acq
python/etc/acq
/etc/acq

If it finds one, load it to dictionary, else use default simulation configuration dictionary. Then, it will process the data in the dictionary and generate more key value pairs (detail in xpdConf package). This dicionary is the glbl used in the ipython session. It contains info like where to save the data and how long time the dark window is.

When the beamtime is started, another yaml file xpd_beamline_config.yaml will be loaded. This file should be allocated at glbl[BASE_DIR]/glbl[BLCONFIG_DIR_NAME]/glbl[BLCONFIG_NAME]. Here, glbl is the directory from step 1. This file should contain a message like this:

Verification time: '2021-01-09 17:45:59'
Verified by: AUTO VERIFIED IN TEST
hadtoedit: false
is_pytest: false

st3107 · 2021-03-12T16:51:27Z

Thus, the solution to the error is that we can add a command in CI to create a yaml file with the content shown below at the location glbl[ARCHIVE_ROOT_DIR]/glbl[BLCONFIG_DIR_NAME]/glbl[BLCONFIG_NAME]. Or we can add the command in ipython startup to create this file.

Verification time: '2021-01-09 17:45:59'
Verified by: AUTO VERIFIED IN TEST
hadtoedit: false
is_pytest: false

mrakitin · 2021-03-12T18:37:14Z

Hi @st3107, thanks for the extra details! We will try it with @AbbyGi next week.

Why are there repeating lines in the following block?

~/.config/acq
/etc/acq
/etc/acq

Is the last /etc/acq record supposed to be something else?

st3107 · 2021-03-12T18:50:46Z

Sorry for the typos. It should be like this:

~/.config/acq
python/etc/acq
/etc/acq

Also, the path to xpd_beamline_config.yaml should be glbl[BASE_DIR]/glbl[BLCONFIG_DIR_NAME]/glbl[BLCONFIG_NAME]. I corrected them in my last comment.

This was a syntax error that never got exercised.

mrakitin

We worked on it together with @AbbyGi, so it looks great!

mrakitin · 2021-03-19T14:27:48Z

Thanks, @tacaswell.

AbbyGi added 3 commits March 2, 2021 11:22

Update ci to test latest environment

daebac3

Update beamline specific steps

9bd6c19

Add timeout fix for CI

4e42daa

AbbyGi mentioned this pull request Mar 2, 2021

Add MANIFEST.in file to fix source distribution xpdAcq/xpdConf#16

Merged

mrakitin closed this Mar 3, 2021

mrakitin reopened this Mar 3, 2021

mrakitin assigned tacaswell, MikeHart85, mrakitin, AbbyGi and dmgav Mar 3, 2021

mrakitin mentioned this pull request Mar 3, 2021

5LS Demo NSLS-II/project-management#39

Closed

17 tasks

AbbyGi added 2 commits March 9, 2021 10:39

Added config file for CI

62a2e37

Added glbl.yml file

07bcfb4

AbbyGi changed the base branch from master to mnt_prep_for_demo March 9, 2021 16:12

Changed where glbl.yml is located

2b50cbd

Added xpd_beamline_config.yml file

182884b

AbbyGi changed the base branch from mnt_prep_for_demo to master March 17, 2021 19:32

tacaswell added 3 commits March 17, 2021 15:35

HACK: make the profile not ask for config confirmation

f02293a

MNT: make forgiving of missing slack token

d644a5d

MNT: add missing commas

6d5305a

This was a syntax error that never got exercised.

AbbyGi marked this pull request as ready for review March 17, 2021 20:43

AbbyGi requested review from DanOlds, dmgav and tacaswell March 17, 2021 20:44

mrakitin approved these changes Mar 17, 2021

View reviewed changes

tacaswell merged commit 97ea48b into master Mar 19, 2021

tacaswell deleted the update-ci branch March 19, 2021 14:05

tacaswell mentioned this pull request Mar 19, 2021

Mnt prep for demo #23

Draft

mrakitin mentioned this pull request Mar 24, 2021

Fix CI NSLS-II-XPDD/profile_collection#5

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update CI#24

Update CI#24
tacaswell merged 10 commits intomasterfrom
update-ci

AbbyGi commented Mar 2, 2021

Uh oh!

mrakitin commented Mar 3, 2021

Uh oh!

AbbyGi commented Mar 9, 2021 •

edited

Loading

Uh oh!

st3107 commented Mar 9, 2021 via email

Uh oh!

mrakitin commented Mar 9, 2021

Uh oh!

AbbyGi commented Mar 9, 2021

Uh oh!

st3107 commented Mar 12, 2021 •

edited

Loading

Uh oh!

st3107 commented Mar 12, 2021

Uh oh!

mrakitin commented Mar 12, 2021

Uh oh!

st3107 commented Mar 12, 2021 •

edited

Loading

Uh oh!

mrakitin left a comment

Uh oh!

mrakitin commented Mar 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

AbbyGi commented Mar 2, 2021

Uh oh!

mrakitin commented Mar 3, 2021

Uh oh!

AbbyGi commented Mar 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

st3107 commented Mar 9, 2021 via email

Uh oh!

mrakitin commented Mar 9, 2021

Uh oh!

AbbyGi commented Mar 9, 2021

Uh oh!

st3107 commented Mar 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

st3107 commented Mar 12, 2021

Uh oh!

mrakitin commented Mar 12, 2021

Uh oh!

st3107 commented Mar 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrakitin left a comment

Choose a reason for hiding this comment

Uh oh!

mrakitin commented Mar 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AbbyGi commented Mar 9, 2021 •

edited

Loading

st3107 commented Mar 12, 2021 •

edited

Loading

st3107 commented Mar 12, 2021 •

edited

Loading