Questions or comments can be directed to the IRIS DMC Quality Assurance Group at dmc_qa@iris.washington.edu.
+
+
Questions or comments can be directed to the EarthScope Quality Assurance Group at qa-qc@earthscope.org.
QuARG is a Python client that allows network operators to generate quality assurance (QA) reports from start to finish. These reports utilize IRIS’s database of MUSTANG data quality metrics to find and highlight potential issues in the data, reducing the amount of time that analysts need to spend scanning the data for problems.
+QuARG is a Python client that allows network operators to generate quality assurance (QA) reports from start to finish. These reports utilize EarthScope's database of MUSTANG data quality metrics to find and highlight potential issues in the data, reducing the amount of time that analysts need to spend scanning the data for problems.
Users have the ability to customize QuARG to adapt to their particular network. Some features that can be personalized:
- Add, edit, or remove Thresholds based on what best fits the network instrumentation.
- Group instrumentation by Network, Station, Locations, or Channels, defining thresholds individually for each group. -
- Use metric values sourced from either IRIS or a local ISPAQ database. Similarly, locally-sourced or IRIS-provided metadata. +
- Use metric values sourced from either EarthScope or a local ISPAQ database. Similarly, locally-sourced or EarthScope-provided metadata.
- Create preference files to minimize the number of fields that users need to input, easily track what thresholds were used to find issues, and potentially create a series of files to be utilized for different use-cases within a network.
- Use the built-in ticketing system or an external one, whichever works better for your workflow. @@ -434,7 +434,7 @@
Threshold
-Thresholds take a metric from a simple description of the data and moves it into a way to determine if the data is “good”. We use the term Threshold to mean aMetricplut aCutoff Value… or, in many cases, a combination of metrics and cutoff values. For QuARG, we actually focus on the data that is “bad”, since Network Operators need to know where things are going wrong so that they can fix it.
One thing to keep in mind is that different instrument types - broadband, short period, strong motion, etc - may have different cutoff values. For example, strong motion instruments have a very different noise profile than broadband instruments, and even a healthy strong motion instrument will have a significant portion of the noise profile that is above the New High Noise Model (Peterson, J, 1993, Observations and Modeling of Seismic Background Noise, U.S.G.S. OFR-93-322) . So thresholds using pct_above_nhnm probably ought to have different cutoff values if applied to strong motion and broadband.
+Thresholds take a metric from a simple description of the data and moves it into a way to determine if the data is “good”. We use the term Threshold to mean aMetricplut aCutoff Value… or, in many cases, a combination of metrics and cutoff values. For QuARG, we actually focus on the data that is “bad”, since Network Operators need to know where things are going wrong so that they can fix it.
One thing to keep in mind is that different instrument types - broadband, short period, strong motion, etc - may have different cutoff values. For example, strong motion instruments have a very different noise profile than broadband instruments, and even a healthy strong motion instrument will have a significant portion of the noise profile that is above the New High Noise Model (Peterson, J, 1993, Observations and Modeling of Seismic Background Noise, U.S.G.S. OFR-93-322) . So thresholds using pct_above_nhnm probably ought to have different cutoff values if applied to strong motion and broadband.
It should be noted that in QuARG, some Thresholds are based on metadata as well. This can help you find cases where the metadata may be incorrect or incomplete.
Here are some examples of thresholds:
pct_above_nhnm > 90 && dead_channel_lin > 2
@@ -467,11 +467,11 @@Definitions
You will notice that there are a variety of types of thresholds in those examples: there are simple cases in the form of METRIC [operator] VALUE. There are also METRIC1 / METRIC2 [operator] VALUE. There are also thresholds that that an AVERAGE of a METRIC over the reporting period and compare that to a VALUE. There are some thresholds that apply only to certain subsets, such as only to the Horizontal or only to the Vertical channels. There are many possible ways to define a Threshold, and theThreshold Definitions Formallows you to do all of these types of comparisons, plus a couple more. This may make the thresholds a little more complicated, but we think that it is worth it to have greater flexibility for you, the user.
IMPORTANT NOTE: The Thresholds, and particularly the cutoff values, that come with QuARG are ones that we have found empirically to balance between false positives and false negatives. They are not set in stone, and will very likely benefit from refinement based on your own network. We have made it so that you can edit, add, or remove thresholds based on your own needs.MUSTANG
-MUSTANG is the Quality Assurance system that we have built at IRIS. It is essentially an entire workflow that ingests data from our archives and outputs a range of about 45 metrics. When data comes into the IRIS DMC, whether in realtime or latent, it triggers a series of steps that lead to metric calculation on that data. The UTC day after data is archived, MUSTANG will begin calculating metrics on the data. Note that archiving can be up to about a day after realtime data streams in, due to the way that the data is pooled prior to archiving.
-
We store the metrics we have calculated in a series of databases that are accessible to users through our web services. Most of the metrics are accessed through the measurements service, though there are also a handful of other services that are primarily related to PSDs and PDFs.
+MUSTANG is the Quality Assurance system that we have built at EarthScope. It is essentially an entire workflow that ingests data from our archives and outputs a range of about 45 metrics. When data comes into EarthScope, whether in realtime or latent, it triggers a series of steps that lead to metric calculation on that data. The UTC day after data is archived, MUSTANG will begin calculating metrics on the data. Note that archiving can be up to about a day after realtime data streams in, due to the way that the data is pooled prior to archiving.
+
We store the metrics we have calculated in a series of databases that are accessible to users through our web services. Most of the metrics are accessed through the measurements service, though there are also a handful of other services that are primarily related to PSDs and PDFs.
If you are unfamiliar with our web services, in simple terms it is a way to input a specific URL into your webbrowser and have the requested metric values returned to you. Or, you can use your favorite language (python in the case of QuARG) to do the work for you.ISPAQ
-Because MUSTANG is inherently built into the IRIS DMC, and we know that not all data streams into our archive, we have created a portable version of MUSTANG that users can install on their own computer to run metrics on their local data. This utility, ISPAQ, is a command line python tool that can write metrics to a file system or to a sqlite database (in ISPAQ 3.0, to be released soon). Those ISPAQ metrics that are written to a sqlite database can be accessed by QuARG by specifying theMetric Sourcefrom within thePreference File Form. This allows greater flexibility - networks can still use QuARG to find issues in their network even if the data does not get archived at the IRIS DMC and we do not have MUSTANG metrics for that data.
+Because MUSTANG is inherently built to work with the EarthScope data archive, and we know that not all data streams into our archive, we have created a portable version of MUSTANG that users can install on their own computer to run metrics on their local data. This utility, ISPAQ, is a command line python tool that can write metrics to a file system or to a sqlite database (in ISPAQ 3.0, to be released soon). Those ISPAQ metrics that are written to a sqlite database can be accessed by QuARG by specifying the - If using an external ticketing system, then it must be able to export the tickets as a ‘|’-delimited file with the following fields:
# Threshold|Target|Start|End|Ndays|Status|Value|Notes. If that’s the case, then the file generated using the external ticketing system must match this filename.
- Metric Source: QuARG can retrieve metric values from either the IRIS MUSTANG web services or from a local sqlite database. ISPAQ is a portable version of MUSTANG that can be downloaded from GitHub and used to calculated metrics on your local machine from local data. A new feature in ISPAQ is that it can now write to a sqlite database, which can then easily be read by QuARG. If a local database will be used, use the
Metric Sourcedropdown menu to select “Local ISPAQ SQLite Database` and then either browse to or type in the name of the database file to be used. Otherwise,”IRIS" should be selected from the dropdown menu.
-- Metadata Source: Similarly, QuARG can use metadata from the IRIS station service or from a local metadata file. If using a local metadata file, there are two options: 1) station xml at the channel level, or 2) ‘|’-delimited text file at the channel level. For the second option, the file must end with ‘.txt’, else QuARG will assume that the file is xml. Either option should match the fields and structure that the IRIS station service uses. If a local metadata file is to be used, then use the
Metadata Sourcedropdown menu to select “Local Metadata File” and either use theBrowsebutton or type in the name of the file containing the metadata. Otherwise, “IRIS” should be selected in the dropdown menu.
+ Metric Source: QuARG can retrieve metric values from either the EarthScope MUSTANG web services or from a local sqlite database. ISPAQ is a portable version of MUSTANG that can be downloaded from GitHub and used to calculated metrics on your local machine from local data. A new feature in ISPAQ is that it can now write to a sqlite database, which can then easily be read by QuARG. If a local database will be used, use the
Metric Sourcedropdown menu to select “Local ISPAQ SQLite Database` and then either browse to or type in the name of the database file to be used. Otherwise,”EarthScope" should be selected from the dropdown menu.
+- Metadata Source: Similarly, QuARG can use metadata from the EarthScope station service or from a local metadata file. If using a local metadata file, there are two options: 1) station xml at the channel level, or 2) ‘|’-delimited text file at the channel level. For the second option, the file must end with ‘.txt’, else QuARG will assume that the file is xml. Either option should match the fields and structure that the EarthScope station service uses. If a local metadata file is to be used, then use the
Metadata Sourcedropdown menu to select “Local Metadata File” and either use theBrowsebutton or type in the name of the file containing the metadata. Otherwise, “EarthScope” should be selected in the dropdown menu.
An example text file:
@@ -793,7 +793,7 @@ - Author: The name of the person that generated the report
- Project Name: This is where you can provide some information about what the report covers. For example, if it was for the IU network, the project name could be something like “IU: Global Seismograph Network - IRIS/USGS”. Or if the report covers just the strong motion instruments, it could be something like “IU: Strong Motion”. There is a lot of flexibility here. -
- Email: The contact email for the person or group that generated the report. This field was created because initailly the IRIS DMC QA group was generating reports for a select few networks and we wanted to provide our contact information. But it may still be useful for you to include. If not, just put a space in there or something similar.
+ - Email: The contact email for the person or group that generated the report. This field was created because initailly the EarthScope QA group was generating reports for a select few networks and we wanted to provide our contact information. But it may still be useful for you to include. If not, just put a space in there or something similar.
- Directory: The directory for Weekly reports is set up like YYYYMMDD. Again, if run on August 13, 2020, the new subdirecotry would be 20200803/.
- Daily: Daily will run for just a single day. This option resolves to ‘Two days ago’ to give time for MUSTANG metrics to have calculated after IRIS recieves the data. +
- Daily: Daily will run for just a single day. This option resolves to ‘Two days ago’ to give time for MUSTANG metrics to have calculated after EarthScope recieves the data.
- Dates: If run on August 13, 2020, the dates will be:
Start Date: 2020-08-11andEnd Date: 2020-08-12 - Directory: The subdirectory for daily reports is also formatted as YYYYMMDD. If the report is started on August 13, 2020, the new subdirecotry would be 20200811/.
Report Frequency
@@ -1014,7 +1014,7 @@Thresholds File Editor
-In many ways, Thresholds are the entire basis of of QuARG and these Quality Assurance (QA) Reports. They are a way to take pre-computed MUSTANG or ISPAQ metric values and use those metrics as a way to find potential issues in the data. The Thresholds File is what QuARG uses to keep track of Instrument Groups (see
Preference File Form) and Threshold Groups, as well as actually defining the thresholds. To edit this file, you use theThreshold Definitions Form. This file isthresholds.txtand is necessary for QuARG toFind Issues, which creates the file that is used toExamine Issues.
+In many ways, Thresholds are the entire basis of of QuARG and these Quality Assurance (QA) Reports. They are a way to take pre-computed MUSTANG or ISPAQ metric values and use those metrics as a way to find potential issues in the data. The Thresholds File is what QuARG uses to keep track of Instrument Groups (see
Preference File Form) and Threshold Groups, as well as actually defining the thresholds. To edit this file, you use theThreshold Definitions Form. This file isthresholds.txtand is necessary for QuARG toFind Issues, which creates the file that is used toExamine Issues.
In case you need a refresher, some defintions are listed here
Threshold Definitions Form
In More Detail:
Metrics
+At the top is a selectable list of all of the MUSTANG metrics. This list comes from the EarthScope MUSTANG webservices and is refreshed whenever QuARG is connected to the internet so it should stay up to date as we add new metrics. When a metric is selected, it will fill in the text box labeled
-At the top is a selectable list of all of the MUSTANG metrics. This list comes from the IRIS MUSTANG webservices and is refreshed whenever QuARG is connected to the internet so it should stay up to date as we add new metrics. When a metric is selected, it will fill in the text box labeledFieldbelow. While you can simply type the metric you are interested inFieldbox directly, the list makes it easy to know what metrics are availble to use.Fieldbelow. While you can simply type the metric you are interested inFieldbox directly, the list makes it easy to know what metrics are availble to use.Channel Options
The channel options allow you to specify whether a threshold, or part of a threshold, should apply to only the horizontal or vertical channels. In most cases, these will not be used since you will want to find issues associated with any and all of the channels. But there are some cases where you would want to limit things. For example, when looking for issues in the metadata you may want to find all cases where the horizontal channels have a Dip != 0. If you applied this threshold to all channels, then every vertical channel should get triggered since they ought to have a non-0 Dip. Another example would be rmsRatio, which compares the sample_rms of the vertical channel to and average of the horizontals.There are 4 buttons for
@@ -1041,7 +1041,7 @@Channel Options:Threshold Definitions Form
- Dates: If run on August 13, 2020, the dates will be:
- Save
- Diagnostic Information: At the bottom of the report are a number of links that may be useful for QA or for better understanding the metrics. There are also definitions of the thresholds used so that there is a record
- MUSTANG Metrics - a list of MUSTANG metrics, as well as a link to the service page -
- Links - links to various MUSTANG web services, as well as other IRIS sites that may be helpful in diagnosing issues +
- Links - links to various MUSTANG web services, as well as other EarthScope sites that may be helpful in diagnosing issues
- MUSTANG noise-psd service: the service interface for our PSD service. This page lists information about how to use the service, and the URL builder (near top) can be used to create queries related to your network.
- MUSTANG noise-pdf service: the service interface for our PDF service. This page lists information about how to use the service, and the URL builder (near top) can be used to create queries related to your network.
- MUSTANG noise-mode-timeseries service: the service interface for our noise mode timeseries service. This service plots the values of various frequencies (the mode value for each frequency for the day) over time so that you can see trends in the noise levels. This page lists information about how to use the service, and the URL builder (near top) can be used to create queries related to your network.
- GOAT/data_available: GOAT is a visual way to view when and where there are data gaps. This link takes you to a page that describes how to construct a URL to view the gaps for the data and timerange you are interested in.
- Metadata Aggregator: MDA displays all of the metadata that we have in our holdings, as well as the metadata from other federated data centers (the source of the metadata is listed under the “Data Center” field). -
- BUD stats: this is currently the only public-facing interface where you can get information about whether you have data in Purgatory, meaning that the data is streaming into IRIS but cannot be archived because we do not have metadata for it. The page contains a lot more information than just that, though. This page describes what we have in the BUD (Buffer of Uniform Data, the data streaming into IRIS in realtime). For each day of data that we have in the BUD, there is a link. That link will have information about all of the networks that have data in the BUD for that day. If your network has a blue M next to it, then there is metadata for all of the data for that day. But if it has a grey M, then there are some channels coming into IRIS that do not have metadata. Click on the network to dive deeper, where you now get a list of stations. Again, a blue M means that the station has complete metadata, a grey one means there is some missing. Click on the station and it will break down by channel. A blue M means that we have metadata, but channels without the icon next to it are missing metadata. For those stations, the “Purgatory” column should be marked as “True” - meaning that it cannot be archived. These channels should either have metadata provided, the data streams should be shut off, or there could be an error in the channel name in the data or the metadata so that they do not match. By correcting the situation, we can ensure that all data that should be getting archived at the IRIS DMC will be archived.
+ - BUD stats: this is currently the only public-facing interface where you can get information about whether you have data in Purgatory, meaning that the data is streaming into EarthScope but cannot be archived because we do not have metadata for it. The page contains a lot more information than just that, though. This page describes what we have in the BUD (Buffer of Uniform Data, the data streaming into EarthScope in realtime). For each day of data that we have in the BUD, there is a link. That link will have information about all of the networks that have data in the BUD for that day. If your network has a blue M next to it, then there is metadata for all of the data for that day. But if it has a grey M, then there are some channels coming into EarthScope that do not have metadata. Click on the network to dive deeper, where you now get a list of stations. Again, a blue M means that the station has complete metadata, a grey one means there is some missing. Click on the station and it will break down by channel. A blue M means that we have metadata, but channels without the icon next to it are missing metadata. For those stations, the “Purgatory” column should be marked as “True” - meaning that it cannot be archived. These channels should either have metadata provided, the data streams should be shut off, or there could be an error in the channel name in the data or the metadata so that they do not match. By correcting the situation, we can ensure that all data that should be getting archived at EarthScope will be archived.
- - SeismiQuery: this is a way to investigate the data holdings at IRIS, both the metadata and the actual data. There are many ways to find or view information from this page. +
- SeismiQuery: this is a way to investigate the data holdings at EarthScope, both the metadata and the actual data. There are many ways to find or view information from this page.
- Thresholds - a list of the definitions of all thresholds. Will print the broadband thresholds if
broadbandis included in the instruments list in the preference file; will print the short period thresholds ifshort periodis included in the instruments list.
diff --git a/findIssues.py b/findIssues.py
index a993666..eed3c66 100755
--- a/findIssues.py
+++ b/findIssues.py
@@ -27,15 +27,20 @@
import pandas as pd
-
# TODO: If ts_ metrics are used, must propagate through into the thresholds file
# ============================#
# LOAD INPUT ARGUMENTS
-network = ''; station = ''; location = ''; channels = ''; start = '';end = ''; outfile = ''
+network = ""
+station = ""
+location = ""
+channels = ""
+start = ""
+end = ""
+outfile = ""
args = reportUtils.getArgs()
-start= args.start
+start = args.start
end = args.end
# month = args.month
@@ -43,26 +48,57 @@
if not preferenceFile:
# If no preference file included, run everything
- thresholdGroups = ['Completeness','Amplitudes','Timing','State of Health','Metadata']
- groupsDict = {'Completeness':['avgGaps','gapsRatioGt12','noData'],
- 'Amplitudes' : ['flat','lowRms','hiAmp','lowAmp','badResp',
- 'avgSpikes','pegged','dead','noise1','noise2',
- 'medianUnique','rmsRatio','xTalk',
- 'gainRatio','nonCoher','polarity',
- 'dcOffsets','nSpikes','rmsRatio'],
- 'Timing' : ['poorTQual','suspectTime','noTime'],
- 'State of Health' : ['ampSat','filtChg','clip',
- 'spikes','glitch','padding','tSync'],
- 'Metadata' : ['zDip','horDip','zeroZ','lowScale','nonMSUnits']}
-
+ thresholdGroups = [
+ "Completeness",
+ "Amplitudes",
+ "Timing",
+ "State of Health",
+ "Metadata",
+ ]
+ groupsDict = {
+ "Completeness": ["avgGaps", "gapsRatioGt12", "noData"],
+ "Amplitudes": [
+ "flat",
+ "lowRms",
+ "hiAmp",
+ "lowAmp",
+ "badResp",
+ "avgSpikes",
+ "pegged",
+ "dead",
+ "noise1",
+ "noise2",
+ "medianUnique",
+ "rmsRatio",
+ "xTalk",
+ "gainRatio",
+ "nonCoher",
+ "polarity",
+ "dcOffsets",
+ "nSpikes",
+ "rmsRatio",
+ ],
+ "Timing": ["poorTQual", "suspectTime", "noTime"],
+ "State of Health": [
+ "ampSat",
+ "filtChg",
+ "clip",
+ "spikes",
+ "glitch",
+ "padding",
+ "tSync",
+ ],
+ "Metadata": ["zDip", "horDip", "zeroZ", "lowScale", "nonMSUnits"],
+ }
+
else:
try:
with open(preferenceFile) as f:
exec(compile(f.read(), preferenceFile, "exec"))
except OSError:
- print('Cannot open', preferenceFile)
+ print("Cannot open", preferenceFile)
quit()
-
+
# Commandline arguments override preference file values, if provided
if args.network:
network = args.network
@@ -71,7 +107,7 @@
if args.locations:
location = args.locations
if args.channels:
- channels= args.channels
+ channels = args.channels
if args.outfile:
outfile = args.outfile
if args.metricsource:
@@ -102,53 +138,64 @@
if os.path.isfile(outfile):
resp1 = input("This file already exists - overwrite?[y/n]: ")
- if (resp1.upper() == 'Y') or (resp1.upper() == 'YES'):
- print('Removing existing file')
+ if (resp1.upper() == "Y") or (resp1.upper() == "YES"):
+ print("Removing existing file")
os.remove(outfile)
-
- elif (resp1.upper() == 'N') or (resp1.upper()== 'NO'):
- resp2= input('Should I append to the existing file?[y/n]: ')
- if (not resp2.upper() == 'Y') and (not resp2.upper() == 'YES'):
+
+ elif (resp1.upper() == "N") or (resp1.upper() == "NO"):
+ resp2 = input("Should I append to the existing file?[y/n]: ")
+ if (not resp2.upper() == "Y") and (not resp2.upper() == "YES"):
quit("Exiting")
else:
- print('Input not recognized, cancelling')
+ print("Input not recognized, cancelling")
quit()
-
+
# Load up list of metrics and metadata, for reference later on
if os.path.isfile(metrics_file):
- with open(metrics_file,'r') as f:
+ with open(metrics_file, "r") as f:
metricsList = f.read().splitlines()
else:
# This should not happen unless running outside of QuARG since QuARG.py has a check before running findIssues.py
- print("WARNING: Could not find list of MUSTANG metrics in file %s - does it exist?" % metrics_file)
- print(" You can create this list by entering the Thresholds Editor - it will automatically generate there")
+ print(
+ "WARNING: Could not find list of MUSTANG metrics in file %s - does it exist?"
+ % metrics_file
+ )
+ print(
+ " You can create this list by entering the Thresholds Editor - it will automatically generate there"
+ )
quit()
-
+
if os.path.isfile(metadata_file):
- with open(metadata_file,'r') as f:
+ with open(metadata_file, "r") as f:
metadataList = f.read().splitlines()
else:
# This should not happen unless running outside of QuARG since QuARG.py has a check before running findIssues.py
- print("WARNING: Could not find list of IRIS metadata fields in file %s - does it exist?" % metadata_file)
- print(" You can create this list by entering the Thresholds Editor - it will automatically generate there")
+ print(
+ "WARNING: Could not find list of EarthScope metadata fields in file %s - does it exist?"
+ % metadata_file
+ )
+ print(
+ " You can create this list by entering the Thresholds Editor - it will automatically generate there"
+ )
quit()
-
+
# ============================#
# GO THROUGH THRESHOLDS
# Add the header to the file
-with open(outfile, 'w') as f:
+with open(outfile, "w") as f:
f.write("# Threshold|Target|Start|End|Ndays|Status|Value|Notes\n")
-f.close()
+f.close()
# Get metadata dataframe at the beginning to use wherever necessary, since it is always the same
-metadataDF = reportUtils.getMetadata(network, station, location, channels, start, end, metadataSource)
+metadataDF = reportUtils.getMetadata(
+ network, station, location, channels, start, end, metadataSource
+)
failedMetricsAll = list()
failedThresholdsAll = list()
-# thresholdFile = './groupsTEST.txt'
for thresholdGroup in thresholdGroups:
print()
print("Running %s Thresholds" % thresholdGroup)
@@ -157,46 +204,65 @@
except:
print(" Could not find any thresholds for %s" % thresholdGroup)
continue
-
+
thresholdsList.sort()
-
- allMetrics, failedThresholds = thresholds.get_threshold_metrics(thresholdsList, thresholdFile)
+
+ allMetrics, failedThresholds = thresholds.get_threshold_metrics(
+ thresholdsList, thresholdFile
+ )
metadatas = [e for e in metadataList if e in allMetrics]
metrics = [e for e in metricsList if e in allMetrics]
-# hasMetadata = False;
hasMetrics = False
-# if len(metadatas) > 0:
-# print("This thresholds Group contains some metadata fields")
-# hasMetadata = True
if len(metrics) > 0:
hasMetrics = True
-
if hasMetrics:
- metricDF, failedMetrics = reportUtils.mergeMetricDF(network, station, location, channels, start, end, metrics, metricSource)
+ metricDF, failedMetrics = reportUtils.mergeMetricDF(
+ network, station, location, channels, start, end, metrics, metricSource
+ )
else:
- metricDF = pd.DataFrame(columns=['value','target','start','end','network','station','location','channel'])
+ metricDF = pd.DataFrame(
+ columns=[
+ "value",
+ "target",
+ "start",
+ "end",
+ "network",
+ "station",
+ "location",
+ "channel",
+ ]
+ )
failedMetrics = list()
-
+
for failedThreshold in failedThresholds:
if not failedThreshold in failedThresholdsAll:
failedThresholdsAll.append(failedThreshold)
-
+
for failedMetric in failedMetrics:
if not failedMetric in failedMetricsAll:
failedMetricsAll.append(failedMetric)
-
-# if hasMetrics == True and not metricDF.empty:
+
for threshold in thresholdsList:
if not threshold in failedThresholds:
- thresholds.do_threshold(threshold, thresholdFile, metricDF, metadataDF, outfile, instruments, start, end, hasMetrics, chanTypes)
+ thresholds.do_threshold(
+ threshold,
+ thresholdFile,
+ metricDF,
+ metadataDF,
+ outfile,
+ instruments,
+ start,
+ end,
+ hasMetrics,
+ chanTypes,
+ )
-with open('failedMetrics.txt','w') as f:
+with open("failedMetrics.txt", "w") as f:
for failedThreshold in failedThresholdsAll:
- f.write('threshold: %s\n' % failedThreshold)
+ f.write("threshold: %s\n" % failedThreshold)
for failedMetric in failedMetricsAll:
- f.write('metric: %s\n' % failedMetric)
+ f.write("metric: %s\n" % failedMetric)
print("INFO: Completed generating issue file")
-
diff --git a/generateHTML.py b/generateHTML.py
index 22ed6fa..d067668 100644
--- a/generateHTML.py
+++ b/generateHTML.py
@@ -31,26 +31,23 @@
import reportUtils
args = reportUtils.getArgs()
-start= args.start
+start = args.start
end = args.end
-# month = args.month
zipDir = args.htmldir
report_fullPath = args.html_file_path
iShort = 0
iBroad = 0
iStrong = 0
-# global iFlag
-# iFlag = 0
metricsFile = args.metrics_file
thresholdFile = args.thresholds_file
preferenceFile = args.preference_file
-if not preferenceFile:
+if not preferenceFile:
quit("WARNING: Preference File required")
-
+
else:
try:
with open(preferenceFile) as f:
@@ -58,36 +55,42 @@
except:
print("Cannot open ", preferenceFile)
quit()
- if ("short period" in map(str.lower, instruments)) or ("shortperiod" in map(str.lower, instruments)):
+ if ("short period" in map(str.lower, instruments)) or (
+ "shortperiod" in map(str.lower, instruments)
+ ):
iShort = 1
- if ("broad band" in map(str.lower, instruments)) or ("broadband" in map(str.lower, instruments)):
+ if ("broad band" in map(str.lower, instruments)) or (
+ "broadband" in map(str.lower, instruments)
+ ):
iBroad = 1
- if ("strong motion" in map(str.lower, instruments)) or ("strongmotion" in map(str.lower, instruments)):
+ if ("strong motion" in map(str.lower, instruments)) or (
+ "strongmotion" in map(str.lower, instruments)
+ ):
iStrong = 1
-
-if start == '' or end == '':
+
+if start == "" or end == "":
pref_start, pref_end, subdir = reportUtils.calculate_dates(reportFrequency)
- if start == '':
+ if start == "":
start = pref_start
- if end == '':
+ if end == "":
end = pref_end
try:
- startDate = datetime.datetime.strptime(start, '%Y-%m-%d').strftime('%B %d, %Y')
- endDate = datetime.datetime.strptime(end, '%Y-%m-%d').strftime('%B %d, %Y')
+ startDate = datetime.datetime.strptime(start, "%Y-%m-%d").strftime("%B %d, %Y")
+ endDate = datetime.datetime.strptime(end, "%Y-%m-%d").strftime("%B %d, %Y")
dates = [startDate, endDate]
except:
print("ERROR: Are the dates properly formatted? YYYY-mm-dd")
-
+
quit("")
-
+
try:
with open(thresholdFile) as f:
exec(compile(f.read(), thresholdFile, "exec"))
-
+
except Exception as e:
print("WARNING: Cannot open thresholds File - %s" % e)
-
+
if args.network:
network = args.network
if args.ticketsfile:
@@ -98,263 +101,338 @@
print("Creating new directory: %s" % zipDir)
os.mkdir(zipDir)
-
infile = csvfile
print(infile)
-#infile = directory + 'issues.csv'
if not os.path.isfile(infile):
quit("Input csv file does not exist")
-summaryFile = report_fullPath + '.summary'
-detailFile = report_fullPath + '.detail'
-
-
-
-
-# date = datetime.datetime.strptime(month, '%Y%m').strftime('%B %Y')
-#author = "Laura Keyson"
-
-
-#os.chdir(directory)
+summaryFile = report_fullPath + ".summary"
+detailFile = report_fullPath + ".detail"
#########################
# Define useful utilities
#########################
-def printPreamble(net,dates,authors,email,outfile):
+def printPreamble(net, dates, authors, email, outfile):
# This prints the header of the html
- with open(outfile,'a+') as f:
- #print("Writing Header")
+ with open(outfile, "a+") as f:
+ # print("Writing Header")
f.write("\n\n")
f.write(" \n")
- f.write("\t\n");
- f.write("\t - %s" % thresholdName) -# f.write("%s \t" % thresholdName); for instrumentGroup in thresholdsDict[thresholdName].keys(): - + if instrumentGroup in instruments: - defStr = ' && '.join(thresholdsDict[thresholdName][instrumentGroup]) + defStr = " && ".join(thresholdsDict[thresholdName][instrumentGroup]) - f.write("
- %s - %s
" % (instrumentGroup,defStr)); + f.write("- %s - %s
" % (instrumentGroup, defStr)) f.write("\n\n"); - f.write("\t\n\n"); - f.write("\n"); - + f.write("\t - %s - %s
Table of Contents
Background
-IRIS (Incorporated Research Institutions for Seismology) DMC (Data Management Center) has been performing quality assurance checks on data since the Transportable Array began in 2004. Since that time, we have expanded and improved our quality assurance efforts, including developing a comprehensive quality assurance system called MUSTANG with over 40 metrics available through our webservices.
+EarthScope (formerly IRIS) Data Services has been performing quality assurance checks on data since the Transportable Array began in 2004. Since that time, we have expanded and improved our quality assurance efforts, including developing a comprehensive quality assurance system called MUSTANG with over 40 metrics available through our webservices.
In addition to the weekly QA performed on the TA network, we developed monthly quality assurance reports for the _GSN virtual network. Since then, we added a few more networks to our monthly and quarterly network reports as we refined our methods and improved the QuARG utility. We wrapped up our final network report in Summer 2019 with the goal of providing the QuARG utility to individual networks for the purpose of performing their own QA. While this tool was born at the DMC, intended for generating reports on very specific networks and leveraging tools that we have available in Seattle, the utility has since been expanded to be useful to network operators working on their own networks.
Over the years, we have refined the process of generating a report into four primary steps:
-
@@ -453,9 +453,9 @@
Definitions
“Metric” is a term that is used to describe some quality or aspect of the data. This can range from quite simple - say, the number of gaps in a day or the maximum value recorded in a day. It can also be more complex, such as taking the cross correlation of two channels at a station and recording the maximum correlation value. Metrics do not inherently tell us whether the data is “good” or “bad” - they simply tell us something about the data. It is up to us, as Network Operators or Researchers, to use those metrics to decide what data is good or and what data bad. And remember, what is “bad” for one purpose may be very “good” for another, and vice versa.For both MUSTANG and ISPAQ, metrics are calculated for each N.S.L.C. target, and generally are calculated on a per-day basis. There are a few exceptions, such as the metrics that are based on the data from a window surrounding the start time of an earthquake.
The usage of the term “Metric” can get complicated and confusing. Generally, the term refers to a concept or algorithm, such as the metric “num_gaps”. A value is calculated for each metric-target-day. We tend to call these values “value” or “measurement”, hence accessing them via our measurements webservice. So IU.ANMO.01.BHZ has a sample_median value of 6241 counts for 2020-01-01 - here, the “metric” is sample_median and the “value” (or measurement) is 6241.
-
It is suggested that you familiarize yourself with the metrics that MUSTANG uses. A full list with a brief description can be found here - click on the red
Current list of all metrics button to pull down the full list. Clicking on the Detailed Documentation button for any given metric will take you to a page that describes it more fully.
+It is suggested that you familiarize yourself with the metrics that MUSTANG uses. A full list with a brief description can be found here - click on the red
Current list of all metrics button to pull down the full list. Clicking on the Detailed Documentation button for any given metric will take you to a page that describes it more fully.
Metric Source from within the Preference File Form. This allows greater flexibility - networks can still use QuARG to find issues in their network even if the data does not get archived at EarthScope and we do not have MUSTANG metrics for that data.
@@ -480,7 +480,7 @@ Definitions
Installation
-QuARG is distributed through GitHub, via IRIS’s public repository (iris-edu). You will use a git client command to get a copy of the latest stable release. In addition, you will use the miniconda python package manager to create a customized Python environment designed to run QuARG properly.
QuARG is distributed through GitHub, via EarthScope’s public repository (EarthScope). You will use a git client command to get a copy of the latest stable release. In addition, you will use the miniconda python package manager to create a customized Python environment designed to run QuARG properly.
If running macOS, Xcode command line tools should be installed. Check for existence and install if missing:
xcode-select --installFollow the steps below to begin running QuARG.
@@ -489,7 +489,7 @@Download the Source Code
You must first have git installed your system. This is a commonly used source code management system and serves well as a mode of software distribution as it is easy to capture updates. See the Git Home Page to begin installation of git before proceeding further.
After you have git installed, you will download the QuARG distribution into a directory of your choosing from GitHub by opening a text terminal and typing:
-git clone https://github.com/iris-edu/quarg.gitgit clone https://github.com/EarthScope/quarg.gitThis will produce a copy of this code distribution in the directory you have chosen. When new quarg versions become available, you can update QuARG by typing:
cd quarg
git pull origin mainDirectories and Filenames
Directories and Filenames
Targets
-This section defines which targets (network, station, channel, location) will be used when retrieving quality assurance (likely from MUSTANG, but could also be ISPAQ) metrics and metadata, and therefore which channels will be included in the issue list. It also defines the category of instrumentation used in the report.
+This section defines which targets (network, station, channel, location) will be used when retrieving quality assurance (likely from MUSTANG, but could also be ISPAQ) metrics and metadata, and therefore which channels will be included in the issue list. It also defines the category of instrumentation used in the report.
Report Header Information
Report Frequency
In More Detail:
-Metadata List This is a scrollable, selectable list of all metadata fields that can be used in QuARG. These are based on the IRIS station service headers at the channel level in the text format. When a field is selected, it will turn blue and will automatically fill in the Field in column 3. The metadata list is disabled by default, and only becomes available when the Metadata toggle button is selected (see below).
Metadata List This is a scrollable, selectable list of all metadata fields that can be used in QuARG. These are based on the EarthScope station service headers at the channel level in the text format. When a field is selected, it will turn blue and will automatically fill in the Field in column 3. The metadata list is disabled by default, and only becomes available when the Metadata toggle button is selected (see below).
Threshold Options
There are five options available:
Databrowser is a tool that allows users to plot MUSTANG metrics. These include Metric Timeseries (plotting metric values over time), Gap Duration plots, Network and Station boxplots, as well as some other options. It can be useful in looking at a network’s overall health, or to quickly view patterns in metric values over long periods of time. The Databrowser button does not require any of the Input fields to be filled. |
|||
| Waveforms | -This button will retreive and display waveform data from the IRIS timeseriesplot service. This requires all target fields to be specified, though it can accomodate a comma-separated list. Users must be careful with the requested Start and End times, as the service limits the length of time that can be plotted. Note: this returns a static image and is not recommended to be the primary way of viewing waveforms - we expect the analyst to use another more dynamic tool to view waveforms, this is simply for use as a quick view of the data. |
+Waveforms | +This button will retreive and display waveform data from the EarthScope timeseriesplot service. This requires all target fields to be specified, though it can accomodate a comma-separated list. Users must be careful with the requested Start and End times, as the service limits the length of time that can be plotted. Note: this returns a static image and is not recommended to be the primary way of viewing waveforms - we expect the analyst to use another more dynamic tool to view waveforms, this is simply for use as a quick view of the data. |
| Metrics | +Metrics | The Metrics button opens a web browser page that displays metric values from the MUSTANG Measurements web service. It uses input from all of the input fields except for Threshold. Start and End are used to limit the time range for the metrics retrieved; Metrics can be a comma-separated list of any desired metrics; Network, Station, Location, and Channel can all be wildcarded, lists, or left blank. Be careful of leaving fields blank, particularly Network, as that can create a very large query. |
|
The Metric Plot button uses the same inputs as the Metrics button, but rather than opening a web page with tabular data, it generates a simple timeseries plot of the requested values. |
|||
| PDFs | +PDFs | Opens a webpage with monthly PDFs for the requested targets, beginning with the month of Start. |
|
| Spectrograms | +Spectrograms | Opens a webpage with the spectorams for the requested targets, for the time span of Start to End. If no dates are provided, will do for the entire span of the targets (from the beginning of the earliest target until the end of the latest target). |
|
| Noise Modes | +Noise Modes | Opens a webpage to the Noise Mode Timeseries plot. All Network, Station, Location, and Channel fields must be filled, with only one target allowed (ie, no wildcarding or lists). Will use the Start and End dates. |
|
Opens a webpage of the USGS event service based on the Start and End dates specified. It will list all earthquakes M5.5 and larger, as MUSTANG event-based metrics do not calculate on smaller events. |
|||
| Station | -Opens a channel-level web page of the IRIS Station service, using provided target information. Any blank field will be wildcarded, and lists and wildcards are allowed; start and end times are ignored for this diagnosis tool. | +Station | +Opens a channel-level web page of the EarthScope Station service, using provided target information. Any blank field will be wildcarded, and lists and wildcards are allowed; start and end times are ignored for this diagnosis tool. |
Ticket File
id,tracker,target,start_date,category,subject,thresholds,images,caption,links,status,end_date,descriptionThe lines that come after that follow that pattern, with quotation marks (‘"’) around any fields that may have a comma in them. For example:
id,tracker,target,start_date,category,subject,thresholds,images,caption,links,status,end_date,description
-4,Support,UU BEI 01 EHZ,2019-12-01,Other,Example Ticket,"gapsRatioGt12, glitch",/Users/laura/QA_reports/testImage.jpg,"This is a figure caption, with a comma so it has quotation marks",http://service.iris.edu/mustang/measurements/1/query?metric=percent_availability&net=YO&cha=?XH&format=text&nodata=404&orderby=start_asc,In Progress,2019-12-03,"This one has a start and end date, and a link!"The most important thing is that the ticketing system used either has these fields, or has an equivalent, and that the tickets can be exported into a csv file of this format. Any missing fields can be left blank if necessary. For example, using a Redmine ticketing system, we are able to use the ‘Export to CSV’ function and choose what columns are exported. It may take an intermediate step to convert the CSV into the correct format, in which case it is probably worth setting up a workflow to do the conversion for you. Depending on the complexity, it might be worth delving into the code to change the required format - just be wary of doing that: it may create unintended consequences.
Final Report
Data Quality Report for " + str(', '.join(net.split(','))) + "
"); - f.write("\t" + str(' - '. join(dates)) + "
\n\n"); + f.write( + '\t\n' + ) + f.write( + "\tData Quality Report for " + str(", ".join(net.split(","))) + "
" + ) + f.write("\t" + str(" - ".join(dates)) + "
\n\n") + + f.write("\t " + str(authors) + "\n") + f.write("\t " + str(email) + "
\n") + + today = datetime.datetime.today().strftime("%B %d, %Y") + f.write("\t Issued " + str(today) + "\n\n") - f.write("\t " + str(authors) + "
\n"); - f.write("\t " + str(email) + "
\n"); - - today = datetime.datetime.today().strftime('%B %d, %Y'); - f.write("\t Issued " + str(today) + "\n\n"); + f.close() - f.close(); def printFirstProject(project, summaryFile, detailFile): # Start the summary and detail files, which will be combined into one later - with open(summaryFile,'a+') as f: - f.write("\t
Summary
\n\n"); - - f.write("\tClicking on each issue Summary link takes you to a more detailed description of \n"); - f.write("\t that issue, including the metrics used to identify the problem.\n"); - f.write("\t Sorted by category, then station.\n"); - f.write("\t
\n"); - f.write("\t\n"); - f.write("\t "+ str(project) +"\n\n"); - f.write("\t
| Category | \n"); - f.write("\tChannel(s) | \n"); - f.write("\tStatus | \n"); - f.write("\tStart Date | \n"); - f.write("\tSummary | \n"); - f.write("\t
| Category | \n") + f.write("\tChannel(s) | \n") + f.write("\tStatus | \n") + f.write("\tStart Date | \n") + f.write("\tSummary | \n") + f.write("\t
\n"); - f.write("\t "+ str(project) +"\n\n"); - f.write("\t
| Category | \n"); - f.write("\tChannel(s) | \n"); - f.write("\tStatus | \n"); - f.write("\tStart Date | \n"); - f.write("\tSummary | \n"); - f.write("\t
\n") + f.write("\t " + str(project) + "\n\n") + f.write("\t
| Category | \n") + f.write("\tChannel(s) | \n") + f.write("\tStatus | \n") + f.write("\tStart Date | \n") + f.write("\tSummary | \n") + f.write("\t
| " + str(category) + " | \n"); - f.write("\t" + str(sncl).replace(" ",".").replace('--',"") + " | \n"); - f.write("\t" + str(status) + " | \n"); - f.write("\t" + str(start) + " | \n"); - f.write("\t" + str(summary) + " | \n"); - f.write("\t
| " + str(category) + " | \n") + f.write( + "\t" + + str(sncl).replace(" ", ".").replace("--", "") + + " | \n" + ) + f.write("\t" + str(status) + " | \n") + f.write("\t" + str(start) + " | \n") + f.write( + '\t' + + str(summary) + + " | \n" + ) + f.write("\t
"+ str(snclq).replace(" ",".").replace('--',"") +" "+ str(subject) + " -- " + str(start) +"
\n");
+ start = "(Start not identified)"
+ if status == "New":
+ status = "Open"
+ f.write(
+ '\t
'
+ + str(snclq).replace(" ", ".").replace("--", "")
+ + " "
+ + str(subject)
+ + " -- "
+ + str(start)
+ + "
\n"
+ )
else:
- f.write("\t
"+ str(snclq) +" "+ str(subject) + " -- " +str(start) +" to " + str(end) +"
\n");
- f.write("\t STATUS: "+ str(status) +"
\n");
- #f.write("\t Diagnostics: \n");
- #f.write("\t "+ str(diagnostics) +"\n");
- #f.write("\t (what is this?)
\n");
- f.write("\t Thresholds: \n");
- f.write("\t "+ str(thresholds) +"\n");
- f.write("\t (what is this?)
\n");
- f.write("\t "+ str(str(description).replace('\n','
')) +"\n");
- f.write("\t
'
+ + str(snclq)
+ + " "
+ + str(subject)
+ + " -- "
+ + str(start)
+ + " to "
+ + str(end)
+ + "
\n"
+ )
+ f.write('\t STATUS: ' + str(status) + "
\n")
+ f.write('\t Thresholds: \n')
+ f.write('\t ' + str(thresholds) + "\n")
+ f.write('\t (what is this?)
\n')
+ f.write("\t " + str(str(description).replace("\n", "
")) + "\n")
+ f.write("\t
") for thisLink in links: - f.write("\t " + thisLink +"" ) - f.write("
"); - + f.write( + '\t ' + + thisLink + + "" + ) + f.write("
") + if not imageurl == "": - images = imageurl.split(';;;;') - captions = imagecaption.split(';;;;') + images = imageurl.split(";;;;") + captions = imagecaption.split(";;;;") nImages = len(images) - + for image_number in range(nImages): thisImage = images[image_number] thisCaption = captions[image_number] - printTicketDetails.iFlag = 1; -# imgfile = str(inum) + ".png"; - imgfile = "%s_%s.png" % (inum,image_number) - + printTicketDetails.iFlag = 1 + imgfile = "%s_%s.png" % (inum, image_number) + try: - + try: - shutil.copyfile(thisImage, zipDir + '/' + imgfile) + shutil.copyfile(thisImage, zipDir + "/" + imgfile) except: - urllib.request.urlretrieve(thisImage, zipDir + '/' + imgfile) - - gotPhoto=1 + urllib.request.urlretrieve(thisImage, zipDir + "/" + imgfile) + + gotPhoto = 1 except: print("WARNING: Unable to retrieve image: %s" % thisImage) - gotPhoto=0 - - f.write("\t \n"); - - + gotPhoto = 0 + + f.write("\t \n") + if gotPhoto == 1: if not thisCaption == "": - f.write("\t "+ str(thisCaption) +":
\n"); - f.write("\t \n"); - f.write("\t
\n") + f.write('\t \n') + f.write( + '\t
Diagnostics
\n"); - f.write("\tThe links below take you to the metrics and other data quality tools used to identify the data issues in this report.\n"); - f.write("\t
\n\n"); - - f.write("\tMUSTANG measurement service metrics:\n"); - f.write("\t
| %s | \n" % metric); - if (ii % nCol == 0): - f.write("\t
MUSTANG noise-mode-timeseries service
\n"); - f.write("\t \n"); - for net in network.split(','): + f.write("\tMUSTANG noise-mode-timeseries service
\n' + ) + f.write( + '\t \n' + ) + for net in network.split(","): net = net.strip() - f.write("\t \n" % (net, net)); - f.write("\t \n"); - f.write("\t \n"); - + f.write( + '\t \n' + % (net, net) + ) + f.write( + '\t \n' + ) + f.write( + '\t \n' + ) # Loop over the thresholds dictionary to print the definitions for instrument groups that are being used. - f.write("\tThresholds
\n"); - f.write("\tThresholds used to identify potential data issues for this report were:\n"); - f.write("\t
\n\n"); - -# f.write("\t- \n");
-
+ f.write('\t
Thresholds
\n') + f.write( + "\tThresholds used to identify potential data issues for this report were:\n" + ) + f.write("\t
\n\n") + for thresholdName in sorted(thresholdsDict.keys()): f.write("- ")
f.write("