Add results and collection scripts for Helix evaluation (presented in paper)

Author: florinzaicu

README.md

@@ -9,6 +9,10 @@ deploying TE on WANs. The emulation framework allows evaluating the control
 plane failure resilience, data-plane failure resilience and TE optimisation
 performance of SDN systems.
 
+The raw results collected to evaluate Helix (presented in the paper) can be
+found in the `SOSR2021_RESULTS/` folder of this repo. Please refer to the
+folder readme file for info on how results were collected.
+
 
 
 ## Repo Overview ##

SOSR2021_RESULTS/README.md

@@ -0,0 +1,108 @@
+# SOSR2021 Raw Results & Process Scripts #
+
+This folder contains all raw results and process scripts used to evaluate
+Helix. This directory is divided into three sub-directories that evaluate
+different aspects.
+
+_Note: The paper only presented part of the control-plane evaluation results
+(scenario 1) and part of the TE algorithm evaluation results (AT&T-MPLS
+Topology)._
+
+
+
+## Data Plane Failure Evaluation #
+
+**Folder:** `SOSR21_ReactVsProact/`
+
+Data plane failure recovery performance evaluation of Helix and comparison
+between reactive versus proactive (Helix) recovery collected using two control
+channel latencies:
+* 4ms
+* 20ms - our calculated average WAN latency (see paper for details).
+
+
+To generate graph using GNU Plot (of processed results data found in
+`reactive_vs_proactive.dat` file) use `gnuplot reactive_vs_proactive.p`.
+Running this script will generate an SVG image `reactive_vs_proactive.svg`.
+
+Raw results were collected using the collection script
+`SOSR2021_collect_react_vs_proact.sh` from the root folder of the repository.
+
+
+### RAW RESULTS ###
+
+`RAW_RESULTS/` - Contains collected raw and processed
+results. Folder also contains process scripts and collection script.
+
+`RAW_RESULTS/ResLinkFail_20ms` - Contains data plane recovery results collected
+using a control channel latency of 20ms. Reactive controller results extended
+Helix to use a restoration based recovery approach while the proactive and
+proactive_alt controllers use Helix's standard protection based recovery method.
+The proactive_alt controller slightly modifies how Helix computes path splices.
+
+`RAW_RESULTS/ResLinkFail_4ms` - Contains data plane recovery results collected
+using a control channel latency of 4ms.  This folder only contains results for
+the reactive controller (proactive controller results were consistent with the
+20ms latency results - omitted to save space).
+
+To process the raw results use the `RAW_RESULTS/ResLinkFail_20ms/proc_all.sh`
+and `RAW_RESULTS/ResLinkFail_4ms/proc_all.sh` to process the 20ms and 4ms
+raw results and output the average recovery time and CI intervals.
+
+
+
+## TE Algorithm Evaluation #
+
+**Folder:** `SOSR21_TE/`
+
+Helix TE algorithm evaluation results collected using YATES and presented in
+the paper. The paper contained the results for the AT&T MPLS topology (using
+the three traffic multipliers). This folder also contains results collected
+when evaluating Helix against the other algorithms using the Abilene topology
+across three traffic multipliers.
+
+`ConLoss` - Folder that contains congestion loss results. Files used by the
+GNU plot scripts which generate graphs for the results (presented in the
+paper).
+
+`PathChurn` - Same as `ConLoss` folder but contains path change results.
+
+To generate the graphs presented in the paper for the (AT&T topology) use the
+commands:
+* `gnuplot attmpls_500.p` - Generate graph for AT\&T MPLS topology (500x
+    multiplier)
+* `gnuplot attmpls_550.p` - Generate graph for AT\&T MPLS topology (550x
+    multiplier)
+* `gnuplot attmpls_560.p` - Generate graph for AT\&T MPLS topology (600x
+    multiplier)
+* `gnuplot abi_2_2.p` - Generate graph for Abilene topology (2.2x multiplier)
+* `gnuplot abi_2_8.p` - Generate graph for Abilene topology (2.8x multiplier)
+* `gnuplot abi_3_0.p` - Generate graph for Abilene topology (3.0x multiplier)
+
+
+
+## Control-Plane Resilience Evaluation ##
+
+**Folder:** `SOSR21_CtrlPlaneFail/`
+
+Helix control plane failure resilience evaluation results collect using the
+emulation framework with the first scenario (presented in paper) and second
+scenario. The folder contains a script to check if the output data contains
+a validation error (`process_find_validation_error.py`) and several scripts
+to calculate the metrics based on the emulation framework event output. The
+process scripts use the file name format: `process_scen<scen>_<stage>.py` where
+`<scen>` represents the scenario number (i.e. 1 and 2) and `<stage>` the
+stage number of the experiment we are extracting relevant metrics for.
+
+The `scen1_out.nostat100.txt` and `scen2_out.nostat100.txt` files contain the
+raw emulation framework output for our experiments (100 iterations) using the
+first and second scenario. `scen1_processed.txt` and `scen2_processed.txt`
+files contain the processed metric results from the raw results file (output
+of the process scripts collected in a single file).
+
+The raw results were collected using the collection script
+`SOSR2021_SOSR2021_collect_ctrlfail.sh` from the root folder of the repository.
+The evaluation experiments were collected using two Helix switch-to-controller
+mapping files. `mdc_v2.sw_ctrl_map.json` was used to collect results for the
+first scenario while `mdc_v2.sw_ctrl_map.v2.json` for the second (contains
+extra controller instances).

SOSR2021_RESULTS/SOSR21_CtrlPlaneFail/process_find_validation_error.py

@@ -0,0 +1,68 @@
+#!/usr/bin/python3
+
+
+# -----------------------------------------------------------------------------
+# Process scenario 1 stage 1, compute CI info for all submetrics
+# Iterate through a control-plane failure result experiment file to find any
+# validation errors in the output. The script will search for the keywords
+# "VALIDATION ERROR". If found, the script specifies the iteration number and
+# stage the validation was encountered.
+#
+# NOTE: The run and stage numbers start at 0, i.e. the first stage of the first
+# run is stage 0 of run 0 and the second stage of the second run is stage 1 of
+# run 1.
+# -----------------------------------------------------------------------------
+
+
+import sys
+
+
+if __name__ == "__main__":
+    # Current run and scen being processed
+    run = -1
+    scen = 99999999
+
+    # Did we find and pass the first list of actions of the framework output?
+    found_first = False
+
+    # Did we find a validation error that needs to be processed. If yes
+    # wait for the actuall data before consuming the error (write the stage and
+    # iteration to the console).
+    found_error = False
+
+    with open(sys.argv[1]) as fin:
+        for line in fin:
+            line = line.strip()
+
+            # Check for the validation error line
+            if "validation error" in line.lower():
+                found_error = True
+                continue
+
+            tok = line.split(",")
+            if len(tok) == 1 and tok[0] == "-----":
+                if found_first == False:
+                    found_first = True
+                continue
+
+            # Get the scenario and compare
+            try:
+                tmp_scen = int(tok[0])
+            except ValueError as e:
+                # Skip lines with invalid scenarios (i.e. first token is not a
+                # number)
+                continue
+
+            if tmp_scen != scen:
+                # Scenario wrapped, this is data for the next run
+                if tmp_scen < scen:
+                    run += 1
+                scen = tmp_scen
+                found_first = False
+
+            if found_first and scen == 0:
+                if found_error:
+                    # Consume the validation error
+                    print("Found validation error during run %s stage %s" %
+                                                                (run, scen))
+                    found_error = False

SOSR2021_RESULTS/SOSR21_CtrlPlaneFail/process_get_run.v2.py

@@ -0,0 +1,92 @@
+#!/usr/bin/python3
+
+
+
+import sys
+from argparse import ArgumentParser
+
+
+def in_range(ranges, value):
+    """ Check if `value` is betweny a range defined by `ranges`.
+
+    Returns:
+        bool: True if `value` in a range of `ranges`, false otherwise.
+    """
+    for r in ranges:
+        if value >= r["start"] and value <= r["end"]:
+            return True
+    return False
+
+
+if __name__ == "__main__":
+    parser = ArgumentParser("Emulate Ctrl Fail Result Processor")
+    parser.add_argument("--file", required=True, type=str,
+        help="File to filter through")
+    parser.add_argument("--run", required=False, type=int,
+        help="Print results for iteration r")
+    parser.add_argument("--multi", metavar="r1:r2", type=str, nargs="+",
+        help="Prin results for iteratuin range r1 to r2 (inclusive)")
+    parser.add_argument("--stage", required=False, type=int,
+        help="Only print results for stage i (applies to both filter types)")
+    args = parser.parse_args()
+
+    # Make sure run number is specified (otherwise print error and exit)
+    if args.run is None and args.multi is None:
+        print("Please provide either a run or stage number")
+        exit(0)
+
+    # Process the single or multi run argument into a list of ranges
+    run_range = []
+    if args.run is not None:
+        run_range.append({"start": args.run, "end": args.run})
+
+    if args.multi is not None:
+        for tmp in args.multi:
+            if ":" not in tmp:
+                print("Invalid multi line argument format: %s" % tmp)
+                continue
+            tok = tmp.split(":")
+            start = int(tok[0])
+            end = int(tok[1])
+            run_range.append({"start": start, "end": end})
+
+    # Current run and scen being processed
+    run = -1
+    scen = 99999999
+    skip_valid_err = 0
+
+    with open(args.file) as fin:
+        for line in fin:
+            line = line.strip()
+
+            # Ignore empty lines
+            if line == "":
+                continue
+
+            # Ingore the validation error output
+            if "!!" in line:
+                skip_valid_err = 3
+                continue
+            if skip_valid_err > 0:
+                skip_valid_err -= 1
+                continue
+
+            # Tokenize the line
+            tok = line.split(",")
+
+            # Get the scenario and compare
+            try:
+                tmp_scen = int(tok[0])
+            except ValueError as e:
+                # If line has no scenario process as the current scenario
+                tmp_scen = scen
+
+            if tmp_scen != scen:
+                # Scenario wrapped, this is data for the next run
+                if tmp_scen < scen:
+                    run += 1
+                scen = tmp_scen
+
+            if in_range(run_range, run):
+                if args.stage is None or args.stage == scen:
+                    print(line)