Rebrand as INNDiE (#166)

* Rename util module * Rename ui-javafx module * Rename training-test-util module * Rename training module * Rename tf-layer-loader module * Rename tf-data-code module * Rename tf-data module * Rename test-util module * Rename test-runner module * Rename plugin module * Rename pattern-match module * Rename logging module * Rename example-models module * Rename dsl-test-util module * Rename db-test-util module * Rename db module * Rename aws module * Rename gradle projcet * Axon -> INNDiE * axon -> inndie * Fix S3PreferencesManager letting an exception get loose * Update Azure build ID * Poke azure Co-authored-by: Austin Shalit <[email protected]>
WPIRoboticsProjects · May 12, 2020 · bf16091 · bf16091
1 parent 77e9bc8
commit bf16091
Show file tree

Hide file tree

Showing 348 changed files with 1,895 additions and 1,555 deletions.
diff --git a/README.md b/README.md
@@ -1,10 +1,10 @@
-[![Build Status](https://dev.azure.com/wpilib/DesktopTools/_apis/build/status/wpilibsuite.Axon?branchName=master)](https://dev.azure.com/wpilib/DesktopTools/_build/latest?definitionId=34&branchName=master)
-[![Github Releases](https://img.shields.io/github/downloads/wpilibsuite/Axon/total.svg)](https://github.com/wpilibsuite/Axon/releases/latest)
-[![codecov](https://codecov.io/gh/wpilibsuite/Axon/branch/master/graph/badge.svg)](https://codecov.io/gh/wpilibsuite/Axon)
+[![Build Status](https://dev.azure.com/wpilib/DesktopTools/_apis/build/status/wpilibsuite.INNDiE?branchName=master)](https://dev.azure.com/wpilib/DesktopTools/_build/latest?definitionId=44&branchName=master)
+[![Github Releases](https://img.shields.io/github/downloads/wpilibsuite/INNDiE/total.svg)](https://github.com/wpilibsuite/INNDiE/releases/latest)
+[![codecov](https://codecov.io/gh/wpilibsuite/INNDiE/branch/master/graph/badge.svg)](https://codecov.io/gh/wpilibsuite/INNDiE)
 
-# Axon
+# INNDiE
 
-Axon - A Graphical Neural Network Editor
+INNDiE: An Integrated Neural Network Development Environment
 
 A Computer Science and Robotic Engineering Major Qualifying Project submitted to
 the Faculty of Worcester Polytechnic Institute in partial fulfillment of the
@@ -66,7 +66,7 @@ loads each one.
 
 ### Plugins
 
-Axon uses a simple plugin system to generalize over many different datasets and models.
+INNDiE uses a simple plugin system to generalize over many different datasets and models.
 
 #### Dataset Plugins
 
@@ -105,7 +105,7 @@ for how to use `steps`.
 After the inference step of a test run has completed, the input and output to/from the model is
 given to a test output processing plugin. This plugin is responsible for interpreting the output of
 the model and writing any test results to a folder in the current directory called `output`. Any
-files put into this directory will be presented to the user in Axon's test view UI. These plugins
+files put into this directory will be presented to the user in INNDiE's test view UI. These plugins
 must implement this function:
 ```python
 def process_model_output(model_input, model_output):
@@ -120,34 +120,34 @@ the output directly from the model.
 
 ## AWS Integration
 
-### S3 Directories Managed by Axon
+### S3 Directories Managed by INNDiE
 
-Inside Axon's autogenerated S3 bucket (named with a prefix `axon-autogenerated-` followed by some
-random alphanumeric characters for uniqueness), Axon manages these directories:
+Inside INNDiE's autogenerated S3 bucket (named with a prefix `inndie-autogenerated-` followed by some
+random alphanumeric characters for uniqueness), INNDiE manages these directories:
 
-- axon-untrained-models
+- inndie-untrained-models
     - Contains “untrained” models that the user can use to create a new Job with
     - These models cannot be used for testing because they are assumed to not contain any weights (or at least not any meaningful weights)
-- axon-training-results
+- inndie-training-results
     - Contains all results from running a training script
-- axon-test-data
+- inndie-test-data
     - Contains test data files that can be used with the test view
-- axon-datasets
+- inndie-datasets
     - Contains the user's custom datasets
-- axon-training-scripts
+- inndie-training-scripts
     - Contains generated training scripts
-- axon-training-progress
-    - Contains training progress files that Axon polls to get training progress updates
-- axon-plugins
+- inndie-training-progress
+    - Contains training progress files that INNDiE polls to get training progress updates
+- inndie-plugins
     - Unofficial plugins are stored here
 
 ### AWS Configuration
 
-- Security Group for ECS named `axon-autogenerated-ecs-sg`
-- Security Group for EC2 named `axon-autogenerated-ec2-sg`
-- Security Group for RDS named `axon-autogenerated-rds-sg`
-- Task role for ECS named `axon-autogenerated-ecs-task-role`
-- IAM role for EC2 named `axon-autogenerated-ec2-role`
-- Instance profile for EC2 named `axon-autogenerated-ec2-instance-profile`
-- ECS Cluster named `axon-autogenerated-cluster`
-- ECS Task Definition named `axon-autogenerated-task-family`
+- Security Group for ECS named `inndie-autogenerated-ecs-sg`
+- Security Group for EC2 named `inndie-autogenerated-ec2-sg`
+- Security Group for RDS named `inndie-autogenerated-rds-sg`
+- Task role for ECS named `inndie-autogenerated-ecs-task-role`
+- IAM role for EC2 named `inndie-autogenerated-ec2-role`
+- Instance profile for EC2 named `inndie-autogenerated-ec2-instance-profile`
+- ECS Cluster named `inndie-autogenerated-cluster`
+- ECS Task Definition named `inndie-autogenerated-task-family`
diff --git a/...ain/kotlin/edu/wpi/axon/aws/EC2Manager.kt → ...n/kotlin/edu/wpi/inndie/aws/EC2Manager.kt b/...ain/kotlin/edu/wpi/axon/aws/EC2Manager.kt → ...n/kotlin/edu/wpi/inndie/aws/EC2Manager.kt
@@ -1,4 +1,4 @@
-package edu.wpi.axon.aws
+package edu.wpi.inndie.aws
 
 import java.util.Base64
 import mu.KotlinLogging
@@ -21,7 +21,7 @@ class EC2Manager {
      *
      * @param scriptData The data for the EC2 instance to run when it boots. This should not
      * contain the entire training script, as that would be too much data. Instead, this script
-     * should use Axon's CLI to download the training script from S3 at runtime.
+     * should use INNDiE's CLI to download the training script from S3 at runtime.
      * @param instanceType The type of the instance to start.
      * @return The ID of the instance that was started.
      */
@@ -32,9 +32,9 @@ class EC2Manager {
                 .maxCount(1)
                 .minCount(1)
                 .userData(scriptData.toBase64())
-                .securityGroups("axon-autogenerated-ec2-sg")
+                .securityGroups("inndie-autogenerated-ec2-sg")
                 .instanceInitiatedShutdownBehavior(ShutdownBehavior.TERMINATE)
-                .iamInstanceProfile { it.name("axon-autogenerated-ec2-instance-profile") }
+                .iamInstanceProfile { it.name("inndie-autogenerated-ec2-instance-profile") }
         }
 
         return runInstancesResponse.instances().first().instanceId()

diff --git a/...wpi/axon/aws/EC2TrainingResultSupplier.kt → ...i/inndie/aws/EC2TrainingResultSupplier.kt b/...wpi/axon/aws/EC2TrainingResultSupplier.kt → ...i/inndie/aws/EC2TrainingResultSupplier.kt
@@ -1,8 +1,9 @@
-package edu.wpi.axon.aws
+package edu.wpi.inndie.aws
 
 import java.io.File
 
-class EC2TrainingResultSupplier(private val s3Manager: S3Manager) : TrainingResultSupplier {
+class EC2TrainingResultSupplier(private val s3Manager: S3Manager) :
+    TrainingResultSupplier {
 
     override fun listResults(id: Int): List<String> = s3Manager.listTrainingResults(id)
 

diff --git a/...pi/axon/aws/EC2TrainingScriptCanceller.kt → .../inndie/aws/EC2TrainingScriptCanceller.kt b/...pi/axon/aws/EC2TrainingScriptCanceller.kt → .../inndie/aws/EC2TrainingScriptCanceller.kt
@@ -1,4 +1,4 @@
-package edu.wpi.axon.aws
+package edu.wpi.inndie.aws
 
 /**
  * A [TrainingScriptCanceller] that is designed for an [EC2TrainingScriptRunner].

diff --git a/.../aws/EC2TrainingScriptProgressReporter.kt → .../aws/EC2TrainingScriptProgressReporter.kt b/.../aws/EC2TrainingScriptProgressReporter.kt → .../aws/EC2TrainingScriptProgressReporter.kt
@@ -1,7 +1,7 @@
-package edu.wpi.axon.aws
+package edu.wpi.inndie.aws
 
-import edu.wpi.axon.db.data.TrainingScriptProgress
-import edu.wpi.axon.util.getLatestEpochFromProgressCsv
+import edu.wpi.inndie.db.data.TrainingScriptProgress
+import edu.wpi.inndie.util.getLatestEpochFromProgressCsv
 import java.lang.NumberFormatException
 import mu.KotlinLogging
 import software.amazon.awssdk.services.ec2.model.InstanceStateName
@@ -78,12 +78,13 @@ class EC2TrainingScriptProgressReporter(
                 """.trimMargin()
             }
 
-            val progressAssumingEverythingIsFine = computeProgressAssumingEverythingIsFine(
-                heartbeat,
-                progress,
-                status,
-                epochs
-            )
+            val progressAssumingEverythingIsFine =
+                computeProgressAssumingEverythingIsFine(
+                    heartbeat,
+                    progress,
+                    status,
+                    epochs
+                )
 
             if ((status == InstanceStateName.SHUTTING_DOWN ||
                     status == InstanceStateName.TERMINATED ||

diff --git a/...u/wpi/axon/aws/EC2TrainingScriptRunner.kt → ...wpi/inndie/aws/EC2TrainingScriptRunner.kt b/...u/wpi/axon/aws/EC2TrainingScriptRunner.kt → ...wpi/inndie/aws/EC2TrainingScriptRunner.kt
@@ -1,8 +1,8 @@
-package edu.wpi.axon.aws
+package edu.wpi.inndie.aws
 
-import edu.wpi.axon.db.data.TrainingScriptProgress
-import edu.wpi.axon.tfdata.Dataset
-import edu.wpi.axon.util.FilePath
+import edu.wpi.inndie.db.data.TrainingScriptProgress
+import edu.wpi.inndie.tfdata.Dataset
+import edu.wpi.inndie.util.FilePath
 import mu.KotlinLogging
 import org.apache.commons.lang3.RandomStringUtils
 import org.koin.core.KoinComponent
@@ -26,9 +26,15 @@ class EC2TrainingScriptRunner(
 
     private val instanceIds = mutableMapOf<Int, String>()
     private val scriptDataMap = mutableMapOf<Int, RunTrainingScriptConfiguration>()
-    private val progressReporter = EC2TrainingScriptProgressReporter(ec2Manager, s3Manager)
-    private val canceller = EC2TrainingScriptCanceller(ec2Manager)
-    private val resultSupplier = EC2TrainingResultSupplier(s3Manager)
+    private val progressReporter =
+        EC2TrainingScriptProgressReporter(
+            ec2Manager,
+            s3Manager
+        )
+    private val canceller =
+        EC2TrainingScriptCanceller(ec2Manager)
+    private val resultSupplier =
+        EC2TrainingResultSupplier(s3Manager)
 
     override fun startScript(
         config: RunTrainingScriptConfiguration
@@ -65,7 +71,7 @@ class EC2TrainingScriptRunner(
         val downloadDatasetString = when (config.dataset) {
             is Dataset.ExampleDataset -> ""
             is Dataset.Custom ->
-                """axon download-dataset "${config.dataset.path.path}""""
+                """inndie download-dataset "${config.dataset.path.path}""""
         }
 
         val scriptForEC2 = """
@@ -81,17 +87,17 @@ class EC2TrainingScriptRunner(
             |apt-cache policy docker-ce
             |apt install -y docker-ce
             |systemctl status docker
-            |pip3 install https://github.com/wpilibsuite/axon-cli/releases/download/v0.1.17/axon-0.1.17-py2.py3-none-any.whl
-            |axon create-heartbeat ${config.id}
-            |axon update-training-progress ${config.id} "initializing"
-            |axon download-model "${config.oldModelName.path}"
+            |pip3 install https://github.com/wpilibsuite/INNDiE-cli/releases/download/v0.1.18/inndie-0.1.18-py2.py3-none-any.whl
+            |inndie create-heartbeat ${config.id}
+            |inndie update-training-progress ${config.id} "initializing"
+            |inndie download-model "${config.oldModelName.path}"
             |$downloadDatasetString
-            |axon download-training-script "$scriptFileName"
-            |docker run -v ${'$'}(eval "pwd"):/home wpilib/axon-ci:latest "/usr/bin/python3.6" "/home/$scriptFileName"
-            |axon upload-training-results ${config.id} "${config.workingDir}"
-            |axon update-training-progress ${config.id} "completed"
-            |axon remove-heartbeat ${config.id}
-            |axon set-training-log-file ${config.id} "/var/log/syslog"
+            |inndie download-training-script "$scriptFileName"
+            |docker run -v ${'$'}(eval "pwd"):/home wpilib/inndie-ci:latest "/usr/bin/python3.6" "/home/$scriptFileName"
+            |inndie upload-training-results ${config.id} "${config.workingDir}"
+            |inndie update-training-progress ${config.id} "completed"
+            |inndie remove-heartbeat ${config.id}
+            |inndie set-training-log-file ${config.id} "/var/log/syslog"
             |shutdown -h now
             """.trimMargin()
 

diff --git a/...otlin/edu/wpi/axon/aws/LoadPreferences.kt → ...lin/edu/wpi/inndie/aws/LoadPreferences.kt b/...otlin/edu/wpi/axon/aws/LoadPreferences.kt → ...lin/edu/wpi/inndie/aws/LoadPreferences.kt
@@ -1,4 +1,4 @@
-package edu.wpi.axon.aws
+package edu.wpi.inndie.aws
 
 import arrow.core.None
 import arrow.core.Option
@@ -8,19 +8,19 @@ import software.amazon.awssdk.core.exception.SdkClientException
 import software.amazon.awssdk.services.s3.S3Client
 
 /**
- * Finds the S3 bucket Axon will work out of. Returns [None] if there is no matching bucket, which
- * causes Axon to run locally and not interface with AWS. The AWS region MUST be auto-detectable
+ * Finds the S3 bucket INNDiE will work out of. Returns [None] if there is no matching bucket, which
+ * causes INNDiE to run locally and not interface with AWS. The AWS region MUST be auto-detectable
  * from the environment (like when running on ECS). To use AWS when running locally, set
  * `AWS_REGION` to your preferred region. To not use AWS when running locally, do not set
  * `AWS_REGION`.
  *
  * @return The name of the bucket or [None] if the bucket could not be found.
  */
-fun findAxonS3Bucket(): Option<String> = try {
+fun findINNDiES3Bucket(): Option<String> = try {
     val s3Client = S3Client.builder().build()
 
     val bucket = s3Client.listBuckets().buckets().first {
-        it.name().startsWith("axon-autogenerated-")
+        it.name().startsWith("inndie-autogenerated-")
     }
 
     LOGGER.info { "Starting with S3 bucket: $bucket" }

diff --git a/...i/axon/aws/LocalTrainingResultSupplier.kt → ...inndie/aws/LocalTrainingResultSupplier.kt b/...i/axon/aws/LocalTrainingResultSupplier.kt → ...inndie/aws/LocalTrainingResultSupplier.kt
@@ -1,4 +1,4 @@
-package edu.wpi.axon.aws
+package edu.wpi.inndie.aws
 
 import java.io.File
 import java.nio.file.Path

diff --git a/.../axon/aws/LocalTrainingScriptCanceller.kt → ...nndie/aws/LocalTrainingScriptCanceller.kt b/.../axon/aws/LocalTrainingScriptCanceller.kt → ...nndie/aws/LocalTrainingScriptCanceller.kt
@@ -1,6 +1,6 @@
-package edu.wpi.axon.aws
+package edu.wpi.inndie.aws
 
-import edu.wpi.axon.db.data.TrainingScriptProgress
+import edu.wpi.inndie.db.data.TrainingScriptProgress
 
 /**
  * A [TrainingScriptCanceller] that is designed for an [LocalTrainingScriptRunner].
@@ -23,7 +23,7 @@ class LocalTrainingScriptCanceller : TrainingScriptCanceller {
     }
 
     /**
-     * Adds a Job that was pulled from the DB after Axon was restarted (so there is no thread data).
+     * Adds a Job that was pulled from the DB after INNDiE was restarted (so there is no thread data).
      *
      * @param jobId The Job ID.
      * @param updateProgress A callback used to update the progress of the Job.

diff --git a/...ws/LocalTrainingScriptProgressReporter.kt → ...ws/LocalTrainingScriptProgressReporter.kt b/...ws/LocalTrainingScriptProgressReporter.kt → ...ws/LocalTrainingScriptProgressReporter.kt
@@ -1,8 +1,8 @@
-package edu.wpi.axon.aws
+package edu.wpi.inndie.aws
 
-import edu.wpi.axon.db.data.TrainingScriptProgress
-import edu.wpi.axon.util.createLocalProgressFilepath
-import edu.wpi.axon.util.getLatestEpochFromProgressCsv
+import edu.wpi.inndie.db.data.TrainingScriptProgress
+import edu.wpi.inndie.util.createLocalProgressFilepath
+import edu.wpi.inndie.util.getLatestEpochFromProgressCsv
 import java.lang.NumberFormatException
 import mu.KotlinLogging
 
@@ -36,7 +36,7 @@ class LocalTrainingScriptProgressReporter : TrainingScriptProgressReporter {
     }
 
     /**
-     * Adds a Job that was pulled from the DB after Axon was restarted (so there is no progressMap
+     * Adds a Job that was pulled from the DB after INNDiE was restarted (so there is no progressMap
      * or thread data).
      *
      * @param config The config the Job was started with.
@@ -62,7 +62,8 @@ class LocalTrainingScriptProgressReporter : TrainingScriptProgressReporter {
 
         // Create the progress file up here to share code but don't read from it unless we have to,
         // to avoid reading from it if it's not there but didn't have to be
-        val progressFile = createLocalProgressFilepath(config.workingDir).toFile()
+        val progressFile = createLocalProgressFilepath(config.workingDir)
+            .toFile()
 
         LOGGER.debug {
             "Getting training progress for Job $jobId from path ${progressFile.path}"
@@ -83,7 +84,10 @@ class LocalTrainingScriptProgressReporter : TrainingScriptProgressReporter {
                 TrainingScriptProgress.Initializing
             } else {
                 try {
-                    val latestEpoch = getLatestEpochFromProgressCsv(progressText)
+                    val latestEpoch =
+                        getLatestEpochFromProgressCsv(
+                            progressText
+                        )
                     if (latestEpoch == epochs) {
                         // We need to make sure the progress gets to Completed because the thread is no
                         // longer alive to do it for us.
@@ -141,7 +145,10 @@ class LocalTrainingScriptProgressReporter : TrainingScriptProgressReporter {
                             else -> {
                                 // Otherwise it must be InProgress
                                 try {
-                                    val latestEpoch = getLatestEpochFromProgressCsv(progressText)
+                                    val latestEpoch =
+                                        getLatestEpochFromProgressCsv(
+                                            progressText
+                                        )
                                     LOGGER.debug { "Latest epoch: $latestEpoch" }
                                     TrainingScriptProgress.InProgress(
                                         latestEpoch / epochs.toDouble(),

diff --git a/...wpi/axon/aws/LocalTrainingScriptRunner.kt → ...i/inndie/aws/LocalTrainingScriptRunner.kt b/...wpi/axon/aws/LocalTrainingScriptRunner.kt → ...i/inndie/aws/LocalTrainingScriptRunner.kt
@@ -1,11 +1,11 @@
-package edu.wpi.axon.aws
-
-import edu.wpi.axon.db.data.TrainingScriptProgress
-import edu.wpi.axon.tfdata.Dataset
-import edu.wpi.axon.util.FilePath
-import edu.wpi.axon.util.createLocalProgressFilepath
-import edu.wpi.axon.util.getOutputModelName
-import edu.wpi.axon.util.runCommand
+package edu.wpi.inndie.aws
+
+import edu.wpi.inndie.db.data.TrainingScriptProgress
+import edu.wpi.inndie.tfdata.Dataset
+import edu.wpi.inndie.util.FilePath
+import edu.wpi.inndie.util.createLocalProgressFilepath
+import edu.wpi.inndie.util.getOutputModelName
+import edu.wpi.inndie.util.runCommand
 import java.nio.file.Paths
 import kotlin.concurrent.thread
 import mu.KotlinLogging
@@ -103,7 +103,7 @@ class LocalTrainingScriptRunner : TrainingScriptRunner {
                     )
                     is Dataset.ExampleDataset -> emptyList()
                 } + listOf(
-                    "wpilib/axon-ci:latest",
+                    "wpilib/inndie-ci:latest",
                     "/usr/bin/python3.6",
                     "/home/$scriptFilename"
                 ),
@@ -129,7 +129,11 @@ class LocalTrainingScriptRunner : TrainingScriptRunner {
                     }
 
                     val newModelFile = config.workingDir
-                        .resolve(getOutputModelName(oldModelName.filename))
+                        .resolve(
+                            getOutputModelName(
+                                oldModelName.filename
+                            )
+                        )
                         .toFile()
                     if (newModelFile.exists()) {
                         scriptProgressMap[config.id] = TrainingScriptProgress.Completed

diff --git a/...xon/aws/RunTrainingScriptConfiguration.kt → ...die/aws/RunTrainingScriptConfiguration.kt b/...xon/aws/RunTrainingScriptConfiguration.kt → ...die/aws/RunTrainingScriptConfiguration.kt
@@ -1,7 +1,7 @@
-package edu.wpi.axon.aws
+package edu.wpi.inndie.aws
 
-import edu.wpi.axon.tfdata.Dataset
-import edu.wpi.axon.util.FilePath
+import edu.wpi.inndie.tfdata.Dataset
+import edu.wpi.inndie.util.FilePath
 import java.nio.file.Path
 
 /**