diff --git a/how-to/author-guide.md b/how-to/author-guide.md index f5820967..72f40491 100644 --- a/how-to/author-guide.md +++ b/how-to/author-guide.md @@ -9,6 +9,7 @@ Reviewer Guide Editor Guide Editor-in-Chief Guide Triage Team Guide +Peer Review Lead ::: ```{toctree} @@ -271,7 +272,7 @@ are usually not accepted by JOSS. Be sure to review JOSS's before writing up a paper about your package. ``` -## Post review - welcome to the pyOpenSci community! +## Post review - welcome to the pyOpenSci community Congratulations! Once your package has been accepted into the pyOpenSci ecosystem, you'll be invited to join our [community Slack](https://join.slack.com/t/pyopensci/shared_invite/zt-39qitgkqb-gZTIo79xCJhS5kSxW1yNfg) where you can connect with other package maintainers, get help with maintenance questions, and stay updated on community developments. diff --git a/how-to/finding-reviewers.md b/how-to/finding-reviewers.md index a081ad2a..41f1bcfb 100644 --- a/how-to/finding-reviewers.md +++ b/how-to/finding-reviewers.md @@ -1,6 +1,7 @@ +(finding-reviewers)= # Finding Reviewers -Sometimes the most challenging part of our pyOpenSci open peer review process +Sometimes, the most challenging part of our pyOpenSci software peer review process is finding reviewers. This page provides tips and tricks to help you find the right people to review a scientific Python package. @@ -35,13 +36,13 @@ up using our [reviewer signup form](https://docs.google.com/forms/d/e/1FAIpQLSeV ## Criteria for Choosing Reviewers -Here are criteria to keep in mind when choosing a reviewer. You might need to +Here are the key criteria to consider when selecting a reviewer. You might need to piece this information together by searching `PyPI`, `Conda` / `Conda-forge` and the potential reviewer’s GitHub page and general online presence (personal website, social media profiles). * Has not reviewed a package for us within the last 6 months. -* Some package development / contribution experience. +* Some package development/contribution experience. * Some domain experience in the field of the package or data source. * No [conflicts of interest](coi). @@ -55,18 +56,18 @@ complexity of the package. * **Openness** - reviewers should also have demonstrated interest in open source or Python community activities, although blind emailing is fine. -Each submission should be reviewed by _two_ package reviewers. Although it is +Two package reviewers should review each submission. Although it is fine for one of them to have less package development experience and more domain knowledge, the review should not be split into two parts. Both reviewers need to review the package comprehensively, from their particular -perspectives. In general, at least one reviewer should have prior reviewing -experience, and of course inviting one new reviewer expands our pool of +perspectives. If possible, at least one reviewer should have prior reviewing +experience, and of course, inviting one new reviewer expands our pool of reviewers. -Reviewers should ideally have some subject matter expertise associated with +At least one reviewer of the two should have some subject matter expertise associated with the package functionality. It is ok and even welcome if one reviewer has more -technical expertise and the other focuses on usability and is less technical. -Read through the Guidelines for Reviewers Section to learn more about finding +technical knowledge and the other focuses on usability and is less technical. +Read through the [Guidelines for Reviewers Section](reviewer-guide-sphinx) to learn more about finding and selecting reviewers. (review-mentorship)= @@ -74,7 +75,7 @@ and selecting reviewers. pyOpenSci encourages those who are newer to review to become involved in our open peer review process. As such, we offer a reviewer mentorship program -where we pair a new reviewer with someone in the community that has previous +where we pair a new reviewer with someone in the community who has previous review experience. It is useful for reviewers to not only review the technical content of a diff --git a/how-to/peer-review-lead.md b/how-to/peer-review-lead.md new file mode 100644 index 00000000..b1dee038 --- /dev/null +++ b/how-to/peer-review-lead.md @@ -0,0 +1,69 @@ +# Peer Review Lead Guide + +:::{tip} +There are several resources that will help you with this role. +Make sure that you have access to the: + +* shared Google folder that contains a list of editors and reviewers who have recently signed up to participate in our review process +* Check out our [peer review status dashboard](https://www.pyopensci.org/metrics/peer-review/peer-review-status-dashboard.html) for the state of peer review +* [Check out our editorial dashboard](https://www.pyopensci.org/metrics/peer-review/editorial-dashboard.html) to get a sense of how many of our editors are busy, and who might be available to lead a review. +* Check out the current review status for the [current state of peer review](https://www.pyopensci.org/metrics/peer-review/current-review-status.html) +::: + + +The Peer Review Lead is responsible for overseeing the entire software review process. They are responsible for: + +* Keeping the review process moving forward by checking in on stalled reviews and supporting the editorial team. +* Ensuring a diverse and active editorial board. +* Onboarding and offboarding new editors. +* Making updates to the [pyOpenSci Software Peer Review Guide](https://www.pyopensci.org/software-peer-review/) as needed. +* Updating [software peer review policies](https://www.pyopensci.org/software-peer-review/our-process/policies.html) as needed. +* Helping editors find reviewers as necessary. + +This role will also help manage conflicts that may arise in the software peer review process. + +:::{note} +The pyOpenSci Executive Director has historically held the lead role in peer review. However, we are transitioning this role to a stipend position, which a community volunteer with excellent organizational and communication skills will hold. + +Ideally, this person has experience with our peer review process as an editor or has been part of our community for some time and is familiar with the process. +::: + +## How to keep peer review moving forward + +A volunteer editorial team runs software peer review; it also relies on volunteer community reviewers. As with any volunteer-led effort, it is common for the review process to stall or slow down. + +As Peer Review Lead, you should monitor the status of reviews and help editors move stalled reviews forward. Our [review status dashboard](https://www.pyopensci.org/metrics/peer-review/current-review-status.html) will help you stay informed about the overall review process. + +Ideally, you should check in on peer reviews weekly or every other week, depending on the volume of reviews submitted. We anticipate that your time allocation in this role will be 4-8 hours a month. + +Keep an eye on a few things when you check in: + +### 1. Dashboard: check the date the comment date for every review + +Before you begin, ensure the peer review dashboard is up to date. At the top of the dashboard, you will see the **Last Updated** date. The metrics repository has a cron job that runs weekly to update review status metrics. If the date at the top of the page is older than 1-2 weeks, likely, a pull request, [similar to this one](https://github.com/pyOpenSci/metrics/pull/147) needs to be merged. Go ahead and merge that PR if one exists. If the dashboard date is old and there is no open PR to merge, our cron job has likely failed. In that case, please leave a note in our Slack `#pyos-infrastructure` channel. Our infrastructure lead can investigate and resolve any issues that arise with our build. + +:::{image} /images/peer-review-metrics-dashboard.png +:alt: Dashboard last updated date +::: + +### 2. Check in on prereview checks + +Next, check in on the prereview checks by sorting the [all-open-reviews table](https://www.pyopensci.org/metrics/peer-review/current-review-status.html#all-open-reviews) table on the **Active_Status** column. Sometimes our Editor in Chief becomes overwhelmed with too many submissions at once. It's easy to forget that they are also able to [delegate the prereview checks to another editor on the team](pre-review-checks) if they fall behind. You can remind them and even help them find other editors to step in if needed. + +### 3. Identify and check in on stalled reviews + +The [All Open Reviews](https://www.pyopensci.org/metrics/peer-review/current-review-status.html#all-open-reviews) section of this page will also help you keep tabs on the current activity status of our reviews. + +Check out the **The last date a comment was left on a review.** You can sort the table by this column to see the date that someone last commented on the issue. + +If a review has been quiet for over a month, it's a good idea to check in on things. You can either leave a note on the issue to see if anyone responds (this is ideal, as you can then track when someone responds to the issue or if you were the last person to respond!). If momentum is not gained by leaving a note for the editor and reviewers on the issue, then follow up with the editor in Slack after 2 weeks to see if they need some support. If Slack doesn't work normally, email works as a last attempt to connect with the editor. + +Also, check out the days open column. Keep an eye out for reviews that have been open for longer than 6 months. In some cases, you may want to check in with the editor to see how things are going and whether there is a way to move the review forward. + +In some cases a review hasn't moved forward because the editor is struggling to find reviewers. [This page](finding-reviewers) will help you with some tips on helping an editor find reviewers. Sometimes this is as easy as posting in our pyOpenSci #software-review channel. Other times we might need to run a call for reviewers. + +### 4. Check the reviews seeking editors section + +The [**Editors Needed** section of the dashboard](https://www.pyopensci.org/metrics/peer-review/current-review-status.html#editors-needed) is useful for you to check in and see how the Editor in Chief (EiC) is doing. In some cases, the Editor in Chief needs support in onboarding a new editor. If packages are seeking editors for more than a month, it's time to check in. Sometimes, all that is needed is for the package to have an editor, but the YAML at the top of the issue hasn't been filled out properly. This is an easy fix - just add the editor to the `editor:` field at the top of the issue. In other cases, we may need to run a call for more editors. In that case, you can consult + * Post in the software-review channel to see if anyone in our Slack community is interested in stepping into an editorial role + * Please reach out to the Executive Director about posting on social media to find a new editor to join our team. [The editorial signup form can be found here](https://forms.gle/VEUxEzN6YmeWSb6t8), and the responses (names, emails, and domains only) to that form can be provided to you by the Executive Director upon request. It's not publicly shared to maintain the privacy of those who sign up to volunteer with us. diff --git a/how-to/reviewer-guide.md b/how-to/reviewer-guide.md index b286f3e6..9194811c 100644 --- a/how-to/reviewer-guide.md +++ b/how-to/reviewer-guide.md @@ -1,3 +1,4 @@ +(reviewer-guide-sphinx)= # Guide for Reviewers ```{epigraph} diff --git a/images/dashboard-last-comment-date.png b/images/dashboard-last-comment-date.png new file mode 100644 index 00000000..84df122f Binary files /dev/null and b/images/dashboard-last-comment-date.png differ diff --git a/images/dashboard-last-updated.png b/images/dashboard-last-updated.png new file mode 100644 index 00000000..b9dae3db Binary files /dev/null and b/images/dashboard-last-updated.png differ diff --git a/images/peer-review-metrics-dashboard.png b/images/peer-review-metrics-dashboard.png new file mode 100644 index 00000000..4376941d Binary files /dev/null and b/images/peer-review-metrics-dashboard.png differ