sealingtech · dlohin · Jun 18, 2018 · Jun 18, 2018 · Jun 18, 2018 · Jun 19, 2018
diff --git a/.DS_Store b/.DS_Store
diff --git a/Jenkinsfile b/Jenkinsfile
@@ -0,0 +1,32 @@
+#!/usr/bin/groovy
+
+// load pipeline functions
+// Requires pipeline-github-lib plugin to load library from github
+
+node {
+  def app
+
+
+
+  def pwd = pwd()
+  def user_id = ''
+  wrap([$class: 'BuildUser']) {
+      echo "userId=${BUILD_USER_ID},fullName=${BUILD_USER},email=${BUILD_USER_EMAIL}"
+      user_id = "${BUILD_USER_ID}"
+  }
+
+  sh "env"
+
+  stage('Clone repository') {
+      /* Let's make sure we have the repository cloned to our workspace */
+      checkout scm
+  }
+
+  stage('Build image') {
+    sh "make iso" 
+  }
+
+  stage('Move ISO to /tmp of the Jenkins CONTAINER') {
+    sh "mv EDCOP-dev.iso /tmp/" 
+  }
+}
diff --git a/README.md b/README.md
@@ -21,116 +21,13 @@ The EDCOP is a scalable cluster to deploy virtual network defense tools on. It i
 
 ![EDCOP Architecture](https://github.com/sealingtech/EDCOP/raw/master/docs/images/stacked_platform_concept.png)
 
-EDCOP is designed to be a platform to deploy any CND tools.  Once deployed you will have Bro, Suricata, ELK stack and other tools made available to you.  There is a seperate Github Repository available here: [EDCOP-TOOLS](https://github.com/sealingtech/EDCOP-TOOLS)
+EDCOP is designed to be a platform to deploy any CND tools.  Once deployed you will have Bro, Suricata, ELK stack and other tools made available to you.  Each tool has a seperate Github repository viewable here:
+https://github.com/sealingtech/
 
-## Quickstart
----
-For the more adventurous, you can [download the latest release installation ISO](https://github.com/sealingtech/EDCOP/releases/download/0.9.1/EDCOP-0.9.1.iso), and give it a try (we'd love the feedback).
-
-To build the ISO, simply clone the repo and run `make iso` (requires docker on the host system and ~10GB free space). Validated on Mac, CentOS, and Ubuntu.
-
-```shell
-git clone https://github.com/sealingtech/EDCOP.git
-make iso
-```
-This will create the docker build container and build the installer ISO from online.
-
-If successful, you should have a file called "EDCOP-dev.iso" in your folder.
-
-The system is installed with the following default UN/PW:
-
-**Default Username:** root
-
-**Default Password:** open.local.box
-
-## Installation
-Deploying EDCOP first involves booting the ISO and running the install setup on the master node.  Once this is complete, minions can be automatically built over the network through PXE services.
-
-### Hardware Pre-Requisites
-The EDCOP installer has been tested on both physical and VMs, however it expects a minimum amount of resources on both the Master server and Minions. At this time, the **Master and Minion must have the same hardware specs**.
-
-| Resource                 | Minimum Spec  |
-| ------------------------ |:-------------:|
-| CPU                      | 4 cores       |
-| Memory                   | 8 GB          |
-| Harddrive 1 (OS)         | 80 GB         |
-| Harddrive 2 (Data)       | 300 GB        |
-| Network Interfaces       | 2 NICs        |
-
-
-
-After booting from the install disk, you'll be asked a series of questions to set the Network and Storage:
-
-![Install Prompt](https://github.com/sealingtech/EDCOP/raw/master/docs/images/installation_prompt.png)
-
-### Network Settings:
-
-The installer will make a feeble attempt to guess what your network settings should be. This is normally incorrect and should probably be answered _**N**_ and entered manually. 
-
-+ **Enter hostname (entire FQDN):**
-
-   EDCOP requires a FQDN and corresponding DNS entry (e.g. "edcop.example.com" or "dev.edcop.io"). After installation, you must access the Admin panel with the FQDN (not IP address).
-
-+ **TEAM the network interfaces on Master? (Y/N):**
-
-   For large clusters, it's recommended to team multiple interfaces (if testing in VMs, recommend answering **_N_**). Answering **_Y_** will use LACP to team the provided interfaces, which must be configured on the switch as well. The new interface will be called "team0".
-
-+ **Enter CLUSTER Interface:**
+EDCOP is designed to work in a number of deployment scenarios including a single physical system as well as large cluster with a traffic load balancer.
 
-   If you answer **_N_** to the teaming, you must enter an interface to communicate with the rest of the cluster.
+Installation takes place by building a "master" node which is then used to build more "minion" nodes.  Once this process is completed then it is possible to deploy tools in a scalable fashion.  
 
-+ **Would you like to set the CLUSTER interface for DHCP? (Y/N)**
 
-   You can set the CLUSTER interface for DHCP, however remember that this required a corresponding DNS entry. If answering _**N**_ you'll be prompted for IP Address, Netmask, Gateway, and DNS. 
-
-+ **Enter PXE-Server Interface:**
-
-   EDCOP installs a PXEboot server on the Master server that allows for auto-installing the minions. The PXE-interface should be on a separate network/vlan. This network should have no DHCP servers on it (the master will start a DHCP server for PXE).
-
-+ **Enter PXE-Server IP Address:**
-
-   Since this is on closed network, any IP address should work (e.g. 10.50.50.10)
-
-+ **Enter PXE-Server Netmask:**
-
-   Ensure a large enough network to cover all minions/nodes to be installed.
-
-+ **Enter last octet of DHCP starting IP:**
-
-   Enter only the last octet for the DHCP server, for example _**100**_
-
-+ **Enter last octet of DHCP ending IP:**
-
-   Enter only the last octet for the DHCP server, for example entering _**150**_ will give you 51 IP addresses for the PXEboot server
-
-### Storage Settings:
-
-At this time, EDCOP allows for an OS disk and a DATA disk. The installation will show the device-id (e.g. sda or sdb) and the corresponding size. Follow the instructions to select which disk is for the OS and which is for the DATA (such as ElasticSearch event storage)
-
-## Using EDCOP
-
-The system is installed with the following default UN/PW:
-
-**Default Username:** root
-
-**Default Password:** open.local.box
-
-After installation, EDCOP runs a service called "EDCOP-firstboot" to finish installing kubernetes, calico, multus, and the other internal cluster components. For normal operations, this requires internet access (a completely offline installer is in development). The service will attempt to ping 8.8.8.8 to verify internet connectivity. If no connectivity is found, the service will fail.
-
-You can validate the service is running with: `systemctl status EDCOP-firstboot`
-
-Once the service has finished installing everything, the follwing URLs can be accessed:
-
-| URL                         | Role                         |
-| --------------------------- |:----------------------------:|
-| https://\<fqdn\>/admin        | Cockpit Admin Panel        |
-| https://\<fqdn\>/kubernetes-ui|Kubernetes Dashboard        |
-| https://\<fqdn\>/loadbalancer |Traefik Ingress Loadbalancer|
-| https://apps.\<fqdn\>         |Kubeapps DCO deployment UI  |
-
-EDCOP uses [Cockpit ](https://github.com/cockpit-project/cockpit) for server/cluster administration. Login with the UN/PW shown above. 
-
-## Building all required packages
----
-The Makefile and Dockerfile pull the necessary RPM packages from both CentOS and EDCOP repos. If you want to build/update the RPMs yourself, you can use the steps outlined in build-packages.md.
+Install: https://github.com/sealingtech/EDCOP/blob/master/docs/installation_guide.rst
 
diff --git a/build/online-configure.sh b/build/online-configure.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 BUILDDIR=~/build
 ISOLINUXDIR=$BUILDDIR/isolinux
-BASEURL=http://mirror.umd.edu/centos/7.4.1708/os/x86_64
+BASEURL=http://mirror.umd.edu/centos/7/os/x86_64/
 
 mkdir -p $ISOLINUXDIR
 curl -s --list-only $BASEURL/repodata/ | sed -n 's/.*href="\([^"]*comps.xml\)[^.].*/\1/p' | xargs -I% curl  --create-dirs $BASEURL/repodata/% -o $BUILDDIR/comps.xml

diff --git a/docs/ingress_design.rst b/docs/ingress_design.rst
@@ -0,0 +1,41 @@
+
+####################
+EDCOP Ingress Design
+####################
+
+By default it is not possible to communicate with services internal to the cluster from outside of the cluster..  To communicate with services inside the cluster two methods are used.  ![Traefik](https://traefik.io) provides access to all web services over https and nodeports provide access to all non-web services.  It is important to understand some of these concepts for deploying applications as well as designing scalable solutions.
+
+
+#######
+Traefik
+#######
+
+Traefik is deployed on all nodes including the masterin EDCOP.  This means that if a web request comes in to any node the traffic will be forwarded to the proper location inside of the cluster regardless of which host the service is running on.  Traefik terminates SSL and uses a certificate called edcop-wild that is created when you install.  When new capabilities are deployed an ingress is created which is how Traefik know where to send traffic.  Treafik uses ![SNI]() https://en.wikipedia.org/wiki/Server_Name_Indication) which means that users must browse to websites using the domain name and on the IP address.  The purpose of the wild card DNS entry is to make sure that all requests to a specific sub-domain is forwarded to traefik so that it can figure out where to send data.
+
+In smaller deployments you can point these DNS records to the master.  In larger deployments this can become an issue as all traffic is being processed by the master node.  A more scalable option is to use a network load balancer which forwards network traffic to all nodes (master and minions).  This serves to spread the load of this traffic as well as provides redundancy in case a node fails.  In this situation, the wild card DNS entry must be pointed at the load balancer IP.  
+
+When configuring any charts with an ingress (such as Kibana), it is important to look for the value host and change this value to any subdomain under the FQDN value you selected when you built the cluster.  If this option is not selected then the default value will be deployed incorrectly.  Once deployed and the service is up, it is possible to browse to this host name.
+
+To view Traefik ingresses, browse to loadbalancer.<fqdn> where you can view all current ingresses as well as view their status.  From the console it is possible to run the command from the command line.
+
+.. code-block:: bash
+
+  kubectl get ingress --all-namespaces
+
+
+
+########
+NodePort
+########
+
+NodePort services traffic for all non web traffic.  NodePorts instruct kube-proxy which lives on every node to forward traffic to the proper location inside of the cluster.  Nodeports are a port number between 30,000 and 32,767.  When deploying capabilities (such as ingress) containing node ports set the port number to a unique number.  To view current node ports from the command line use the following command:
+
+.. code-block:: bash
+
+  kubectl get service --all-namespaces | grep NodePort
+
+Traffic can be load balanced here if desired using an external load balancer or it is possible to point clients to specific nodes and spread out the traffic in the way.
+
+
+
+
diff --git a/docs/installation_guide.rst b/docs/installation_guide.rst
@@ -30,11 +30,16 @@ EDCOP requires a specific network configuration to work properly.  A switch will
 .. image:: images/network_configuration.png
 
 DNS Requirements
-EDCOP requires three DNS entries currently (Eventually only two will be required).  It is recommended to make a seperate sub-domain for the entire cluster.  All servers in the cluster must be able to resolve this as well as the workstations which are accessing services inside of the cluster.
+================
+EDCOP requires two DNS entries. When you setup the master you will be asked for the fully qualified domain name (FQDN).  The same domain name must be entered into the DNS server pointing to the master system.  In addition, there must be a wild card subdomain under that same FQDN.  The wild card DNS entry can be pointed either at the master as well or it can be load balanced by an external load balancer to the master and all minions on 80 and 443.  Load balancing is optional
+
+Example:
+
+You name the master, master.edcop.io and give it the IP of 192.168.1.10.  You create a wild card subdomain of *.master.edcop.io of 192.168.1.10.
+
+It is necessarry to be able to resolve these entries from both inside the cluster as well as any clients accessing the cluster.  You must access the cluster by using the domain name and not by IP.
+
 
-- <subdomain>:       This will point to the network address of the master server
-- apps.<subdomain>:  This will also point to the master and be used for the EDCOP marketplace.
-- minion addresses:  This can be setup after the  minions are built.  It is reccomended to load balance traffic to  the minions if possible.  In this case, a VIP would be created on a load balancer and then forward requests down to the minions in a round robin fashion.
 
 
 Building ISO image
@@ -123,20 +128,21 @@ Accessing Cockpit
 
 If you have configured the DNS entry correctly, then Cockpit should be available at this point.  Open a web browser and go to:
 
-https://<hostname of master>/admin/
+https://admin.<fqdn>/
 
-(Note that the trailing slash is important)
 
 Logon with root as the user and the password you set earlier
 
 Building the Minions
 ====================
 
+Once the master is successfully running, minions can be PXE booted off of the main system.  This is not needed on single node deployments.
+
 Boot off of the PXE Interface in startup (see system manual for this process)
 
 If the PXE is configured correctly, an Install the Expandable DCO Platform (EDCOP) option will be displayed, select Enter
 
-After the installation process is completed and the system reboots.  Access cockpit and select Cluster -> Nodes and your new node should appear here after a bit.
+After the installation process is completed and the system reboots.  Access cockpit and select Cluster -> Nodes and your new node should appear here after a bit and the status should be set to ready.
 
 From the command line, it is also possible to do this from the command line on the master using:
 
@@ -150,39 +156,65 @@ Labeling nodes
 
 NOTE: This section will need to change when more granular roles are configured
 
-Nodes must be given roles in order to take certain tasks.  In the Helm charts there are often options to select NodeSelectors.  Log on to the master node and run the command:
+Nodes must be given roles in order to take certain tasks.  Each of these labels must be applied somwhere throughout the cluster.  For small deployments, simply label the master as all of them.  For larger deployments it is possible to selectively apply the labels to specific nodes throughout the cluster.
+
 
 .. code-block:: bash
 
-  kubectl label node <name of minion node> nodetype=worker
+  kubectl label node <name of node> nodetype=worker
+  kubectl label node <name of node> sensor=true
+  kubectl label node <name of node> data=true
+  kubectl label node <name of node> infrastructure=true
+  kubectl label node <name of node> ingest=true
+
+
+Please see the node labelling guide  https://github.com/sealingtech/EDCOP/blob/master/docs/node_labels.rst
 
-This process will be repeated for each node.
 
-Datastorage workaround
+Verifying installation
 ======================
-This is temporary fix in the prototype.
 
-On the master and all minions run:
+After a few minutes all the pods should be either in a "running" or "completed" state.  To verify these have come up, run the command.  
 
 .. code-block:: bash
+
+  kubectl get pods --all-namespaces
 
-  mkdir /EDCOP/bulk/esdata
-  chmod 777 /EDCOP/bulk/esdata
 
 
-Configuring Nodes
-=================
-An application called host-setup will need to be run in order to configure all the interfaces and neworks.  
+Accessing other Services
+========================
+
+EDCOP has deployed a number of internal web inferfaces automatically for you.  To view these:
+
+- https://admin.<fqdn>/
+- https://kubernetes.<fqdn>/
+- https://loadbalancer.<fqdn>/
+- https://apps.<fqdn>/
+- https://ceph.<fqdn>/
+
+Please view the ingress guide https://github.com/sealingtech/EDCOP/blob/master/docs/ingress_design.rst for more details.
+
+
+SSL Certificate Management
+==========================
+
+By default EDCOP will create a wild card certificate that is used by all domains.  This certificate has been signed by an auto-generated Certificate Authority (CA) that is used for internal CA operations.  This CA is generally not trusted by your browser.  To make SSL error messages go away a user can trust the internal kubernetes certificate authority.  
+
+The certificate is stored in /root/ca.cer and can be added to user's internal Root CA store.
+
+For windows follow this guide:
+https://blogs.technet.microsoft.com/sbs/2008/05/08/installing-a-self-signed-certificate-as-a-trusted-root-ca-in-windows-vista/
+
+
+Deploying Capabilities
+======================
+
+To deploy additional tools users can go to apps.<fqdn> and select the applications to they want to deploy.  Selecting "Available Capabilities" will bring up a number of charts that can then be deployed.  Each chart will have built in instructions.  Many of these charts values are set to defaults that will work with smaller deployments but more planning is required for larger deployments to get more performance out of the tools.
+
 
-Go to apps.<subdomain>
 
-#. Select Deploy one
-#. Select Host-setup
-#. Select Deploy using Helm
 
-View the Optimization Guide for how to configure interfaces.  If this is EDCOP supported hardware this process will have been done for you.
 
-#. Enter in a name such as hostsetup
-#. Select Submit