diff --git a/.gitignore b/.gitignore index 6aca310..83c270e 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,5 @@ Cargo.lock *~ .*.sw? .cargo +*.pyc +*.bk diff --git a/README.md b/README.md index 70e402d..d89ed71 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,10 @@ recommended to manage the build process. ``` git clone XXX cargo build -sudo RR_DEVICE=eth0 cargo run +sudo ethtool -K eth0 tx off rx off gro off tso off gso off +sudo insmod netmap +# sudo RUST_BACKTRACE=1 RR_DEVICE=eth0 RR_TARGET_IPS=10.1.0.1 target/debug/rusty_rail +sudo RUST_BACKTRACE=1 RR_DEVICE=eth0 RR_TARGET_IPS=server1ip target/debug/rusty_rail ``` # Configuration @@ -39,6 +42,118 @@ Configuration is via environment variables. * ``RR_DEVICE`` should be the name of the interface to receive and transmit GRE wrapped packets on. +# Testing + +## Test bed overview + +The initial deployment architecture we want to emulate is: + +Clients -> I/N -> router [GRE encap] -> LoadBalancers -> Servers [GRE decap, DSR] + +Key characteristics: + - servers need GRE 1/2 sided, and the route back to the client src IP must not + be within the tunnel network + - clients need to be able to send packets to the loadbalancers that are GRE + +So - minimal setup: + - V is the network of your hypervisor virtual network + - pick T not in V as the subnet for traffic within GRE (used for e.g. ICMP + originating from the GRE receiver) Suggest a /22 (net, peer1, peer2, broad) + - pick C not in V,T as the client src IPs + - pick S not in V,T or C as the service src IP [decoupled from server count as + this is the virtual IP for the service] + - disable anti-spoofing in your hypervisor network + - add one (or more) addresses from C to the client workload VMs + - add one (or more if you're running more than one test service) address from + S to the server workload VMs + - have a tunnel from C to the load balancer VM(s) over GRE for traffic destined for S + - disable return path spoof protection on the clients (as traffic from S will + arrive unencaped) + - have a tunnel from LB to S on the server VMs for traffic destined to S + - possibly disable the outbound route matching? + +iteration 1: + one client, one server, one LB + V: 192.168.137.0/24 + T(client): 172.16.0.1/22 + T(server): 172.16.0.2/22 + C: 10.0.0.0/24 + C1: 10.0.0.1 + S: 10.1.0.0/24 + S1: 10.1.0.1 +iteration 2: + one client, two servers, one LB +iteration 3: + one client, two servers, two LB +iteration 4: + two clients, two servers, two LB + +Test automation: salt. Just because. + +## Install a salt master + +Anywhere you like. Have fun. Knock yourself out. The states for this project +are in salt/, so add this path to your file_roots in /etc/salt/master (or +wherever your master conf is). + +## Test node prep + +Test nodes - http://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/11.0-RELEASE/amd64/Latest/FreeBSD-11.0-RELEASE-amd64.vhd.xz + +See https://blogs.msdn.microsoft.com/kylie/2014/12/25/running-freebsd-on-hyper-v/ ; gen 1, dynamic memory off. Give 256M to each VM. + +username root; no password +change password e.g. foo + +install ssh for mgmt: +vi /etc/rc.conf + +```sshd_enable=YES``` + +vi /etc/ssh/sshd_config +enable root logins + +then: +```sh /etc/rc.d/sshd start``` + +ssh-copy-key as desired +disable challenge-response logins to disable passwords + +Enable GRE: +echo if_gre_load="YES" > /boot/loader.conf + +reboot and check everything came up ok. +kldstat + +Install salt: +``` +pkg install py27-salt +cat << EOF > /usr/local/etc/salt/minion +master: $masterip +id: $uniqueid +minion_id_caching: False +grains: + roles: + - unassigned +EOF +sysrc salt_minion_enable="YES" +rm -fr /usr/local/etc/salt/pki/minion +rm -fr /usr/local/etc/salt/minion_id +``` + +Clone the VM at this point to permit rapid creation of additional machines. + +## Per node +1. Boot the node. +2. Change the hostname +# Avoid salt crashing on start - or hand this out via DHCP etc. +sysrc hostname="$ROLE-$N.local" +hostname $ROLE-$N.local +2. restart salt: + ```service salt_minion start``` + + + # License Apache-2.0. diff --git a/salt/_modules/freebsd_common.py b/salt/_modules/freebsd_common.py new file mode 100644 index 0000000..534d9ad --- /dev/null +++ b/salt/_modules/freebsd_common.py @@ -0,0 +1,10 @@ +def sysrc(value): + """Call sysrc. + CLI Example: + + .. code-block:: bash + + salt '*' freebsd_common.sysrc sshd_enable=YES + salt '*' freebsd_common.sysrc static_routes + """ + return __salt__['cmd.run_all']("sysrc %s" % value) diff --git a/salt/_modules/freebsd_ip.py b/salt/_modules/freebsd_ip.py new file mode 100644 index 0000000..e996a91 --- /dev/null +++ b/salt/_modules/freebsd_ip.py @@ -0,0 +1,275 @@ +# Derived from the Salt codebase (due to implementing the ip interface); that +# is Apache-2 so license compatible. + +import logging + +import salt.utils.validate.net + +# Define the module's virtual name +__virtualname__ = 'ip' + +# From rh_ip.py +_CONFIG_TRUE = ['yes', 'on', 'true', '1', True] +_CONFIG_FALSE = ['no', 'off', 'false', '0', False] +_IFACE_TYPES = [ + 'gre', 'alias' +] + + +log = logging.getLogger(__name__) +# TODO: error handling throughout. + + +def __virtual__(): + '''Confine this module to FreeBSD based distros''' + if __grains__['os'] == 'FreeBSD': + return __virtualname__ + return False + + +def _raise_error(message): + log.error(message) + raise AttributeError(message) + + +def build_interface(iface, iface_type, enabled, **settings): + '''Build an interface script for a network interface. + + CLI Example: + + .. code-block:: bash + + salt '*' ip.build_interface eth0 eth + salt '*' ip.build_interface eth0.0 alias + ''' + iface = iface.lower() + iface_type = iface_type.lower() + + if iface_type not in _IFACE_TYPES: + _raise_error( + "Bad interface type %s not known in %s" % (iface_type, _IFACE_TYPES)) + + opts = _parse_int_settings(settings, iface_type, enabled, iface) + # Could read in current value here for retain-settings + # XXX: should be cross cutting scatter-gather across all interfaces? + cloned_interfaces = set() + if iface_type in ['eth']: + ifcfg = 'ifconfig_%(major)s="inet %(ipaddr)s netmask %(netmask)s"' % opts + elif iface_type in ["alias"]: + ifcfg = 'ifconfig_%(major)s_alias%(minor)s="inet %(ipaddr)s netmask %(netmask)s"' % opts + elif iface_type in ["gre"]: + ifcfg = 'ifconfig_%(major)s="inet %(ipaddr)s %(peer_inner_addr)s netmask %(netmask)s tunnel %(tunnel_addr)s %(tunnel_peer)s"' % opts + cloned_interfaces.add(opts['major']) + else: + _raise_error("Unexpected interface type %s" % iface_type) + + if 'test' in settings and settings['test']: + return [ifcfg] + + __salt__['freebsd_common.sysrc'](ifcfg) + if cloned_interfaces: + cloned_cfg = 'cloned_interfaces="%s"' % ' '.join(cloned_interfaces) + __salt__['freebsd_common.sysrc'](cloned_cfg) + return [ifcfg] + + +def build_network_settings(**settings): + '''Build the global network script.''' + # Not implemented yet + return '' + + +def get_interface(iface): + '''Return the contents of an interface script + + Mocked out to get minimal virtual IP management working. + ''' + # Not implemented yet + return '' + + +def get_network_settings(): + '''Return the contents of the global network script. + + Mocked to get virtual IP management working. + ''' + return '' + + +def up(iface, iface_type): + '''Start up a network interface + + CLI Example: + + .. code-block:: bash + + salt '*' ip.up eth0 + ''' + if '.' not in iface: + major = iface + else: + major = iface[:iface.find('.')] + return __salt__['cmd.run']('/etc/rc.d/netif restart %s' % major) + + +def down(iface, iface_type): + '''Take an interface down.''' + if '.' not in iface: + major = iface + else: + major = iface[:iface.find('.')] + return __salt__['cmd.run']('ifconfig %s down' % major) + + +def apply_network_settings(**settings): + """Apply global network configuration.""" + if 'require_reboot' not in settings: + settings['require_reboot'] = False + if 'apply_hostname' not in settings: + settings['apply_hostname'] = False + + hostname_res = True + if settings['apply_hostname'] in _CONFIG_TRUE: + if 'hostname' in settings: + hostname_res = __salt__['network.mod_hostname'](settings['hostname']) + else: + log.warning( + 'The network state sls is trying to apply hostname ' + 'changes but no hostname is defined.' + ) + hostname_res = False + + res = True + if settings['require_reboot'] in _CONFIG_TRUE: + log.warning( + 'The network state sls is requiring a reboot of the system to ' + 'properly apply network configuration.' + ) + res = True + else: + res = __salt__['cmd.run']('/etc/netstart restart') + + return hostname_res and res + + +def get_routes(iface): + """Find the routes for an interface.""" + # 1: get the static_routes entry. + res = __salt__['freebsd_common.sysrc']("static_routes") + output = res["stdout"] + if res["retcode"] or not output.startswith("static_routes:"): + _raise_error( + "Invalid sysrc output %r" % (res,)) + routekeys = output[len("static_routes:"):].strip().split() + if not routekeys or routekeys == ['NO']: + return [] + result = ['static_routes+="'] + for key in routekeys: + route_iface, line = _get_static_route(key) + if route_iface != iface: + continue + result.append(line) + result[0] += " " + key + result[0] += '"' + return result + + +def build_routes(iface, **settings): + """Build/apply routes for iface.""" + iface = iface.lower() + settings = dict((k.lower(), v) for (k, v) in settings.items()) + if 'routes' not in settings: + _raise_error("No routes supplied for " + iface) + routes = [] + for route in settings['routes']: + # Required options: name, ipaddr, netmask, gateway + routes.append({ + 'name': route['name'], + 'ipaddr': route['ipaddr'], + 'netmask': route['netmask'], + 'gateway': route['gateway'] + }) + routekeys = set() + static_line = 'static_routes+="' + routelines = [] + for route in routes: + if route['name'] in routekeys: + _raise_error("Duplicate route " + route['name']) + static_line += " %s:%s" % (route['name'], iface) + routekeys.add(route['name']) + routelines.append('route_%s="%s %s %s"' % ( + route['name'], route['ipaddr'], route['gateway'], route['netmask'] + )) + static_line += '"' + if settings['test']: + return [static_line] + routelines + + # Serialise route entries. + for routeline in routelines: + __salt__['freebsd_common.sysrc'](routeline) + # And the root keys + __salt__['freebsd_common.sysrc'](static_line) + return [static_line] + routelines + + +def _get_static_route(key): + if ':' in key: + name, iface = key.split(':') + else: + name, iface = key, None + syskey = "route_" + name + res = __salt__['freebsd_common.sysrc'](syskey) + if res["retcode"] == 1: + return "" + if res["retcode"]: + _raise_error("Failed to get route %r" % syskey) + return iface, res["stdout"] + + +def _parse_int_settings(opts, iface_type, enabled, iface): + ''' + Filters given options and outputs valid settings for a + network interface. + ''' + result = {'name': iface} + if '.' in iface: + result['major'] = iface[:iface.find('.')] + result['minor'] = iface[iface.find('.')+1:] + else: + result['major'] = iface + result['minor'] = None + + for opt in ['ipaddr', 'master', 'netmask', 'srcaddr', 'delay', 'domain', 'gateway', 'zone']: + if opt in opts: + result[opt] = opts[opt] + + result.update(_parse_type[iface_type](iface, opts)) + return result + + +def _parse_gre(iface, opts): + """Parse GRE specific options: + + - peer_inner_addr: the far end address to route packets to. + - tunnel_addr: the address to send/receive GRE packets at. + - tunnel_peer: the address to send GRE packets to. + """ + result = {} + result.update(_require_opt('peer_inner_addr', iface, opts)) + result.update(_require_opt('tunnel_addr', iface, opts)) + result.update(_require_opt('tunnel_peer', iface, opts)) + return result + + +def _parse_alias(iface, opts): + return {} + + +_parse_type = {'gre': _parse_gre, 'alias': _parse_alias} + + +def _require_opt(optname, iface, opts): + if optname not in opts: + _raise_error( + "option %r not supplied for interface %s" % (optname, iface)) + return {optname: opts[optname]} diff --git a/salt/_modules/rr.py b/salt/_modules/rr.py new file mode 100644 index 0000000..1286064 --- /dev/null +++ b/salt/_modules/rr.py @@ -0,0 +1,2 @@ +def hostnumber(host): + return ''.join(d for d in host if d.isdigit()) diff --git a/salt/bsd.sls b/salt/bsd.sls new file mode 100644 index 0000000..45b7dc5 --- /dev/null +++ b/salt/bsd.sls @@ -0,0 +1,12 @@ +py27-salt: + pkg.latest: + - name: py27-salt + service.running: + - names: + - salt_minion + - require: + - pkg: py27-salt + - watch: + - file: /usr/local/etc/salt/minion + +/usr/local/etc/salt/minion: file.exists diff --git a/salt/client.sls b/salt/client.sls new file mode 100644 index 0000000..d94e43d --- /dev/null +++ b/salt/client.sls @@ -0,0 +1,34 @@ +system: + network.system: + - retain_settings: True + +hn0.0: + network.managed: + - type: alias + - ipaddr: 10.0.0.{{ salt['rr.hostnumber'](grains['id']) }} + - netmask: 255.255.255.0 + +gre0: + network.managed: + - type: gre + - ipaddr: 172.16.0.1 + - netmask: 255.255.255.252 + - peer_inner_addr: 172.16.0.2 + - tunnel_addr: 10.0.0.{{ salt['rr.hostnumber'](grains['id']) }} +# NB: sharing this address over all clients as we're not going to generate +# traffic from it, and the server endpoint is hardcoded. + - tunnel_peer: 192.168.137.188 +# Sending traffic to the load balancer node + +routes: + network.routes: + - name: hn0 + - routes: + - name: via_lb_1 + ipaddr: 10.1.0.1 + netmask: 255.255.255.255 + gateway: 172.16.0.2 + +netperf: + pkg.installed: + - name: netperf diff --git a/salt/server.sls b/salt/server.sls new file mode 100644 index 0000000..39c4d0e --- /dev/null +++ b/salt/server.sls @@ -0,0 +1,48 @@ +# TODO: the mshome.net glue is because my node ids aren't correct - fix that to +# make this easy for others to use. + +system: + network.system: + - retain_settings: True + +hn0.0: + network.managed: + - type: alias + - ipaddr: 10.1.0.{{ salt['rr.hostnumber'](grains['id']) }} + - netmask: 255.255.255.0 + +gre0: + network.managed: + - type: gre + - ipaddr: 172.16.0.2 + - peer_inner_addr: 172.16.0.1 + - netmask: 255.255.255.252 + - tunnel_addr: server{{salt['rr.hostnumber'](grains['id'])}}.mshome.net +# nodes own hostname because the LB can't route to the non-on-net address yet +# (and perhaps we simply don't need to do that?) +# - tunnel_addr: 10.1.0.{{ salt['rr.hostnumber'](grains['id']) }} +# NB: tunnel_peer won't be used as we won't ever send traffic to 172.16.0.1... +# and we're not going to route the client through the tunnel since we want DSR +# - tunnel peer is the LB because the src address is filtered by the gre stack. + - tunnel_peer: 192.168.137.188 + +# netsteps: +# - add a route to 10.0.0.X for each slave, via their client address +# ... put those in a reported back value of some sort? .. dns works clientN.mshome.net +routes: + network.routes: + - name: hn0 + - routes: + - name: dsr_client_1 + ipaddr: 10.0.0.1 + netmask: 255.255.255.255 + gateway: client1.mshome.net + - name: dsr_client_2 + ipaddr: 10.0.0.2 + netmask: 255.255.255.255 + gateway: client2.mshome.net + + +netperf: + pkg.installed: + - name: netperf diff --git a/salt/top.sls b/salt/top.sls new file mode 100644 index 0000000..aa48007 --- /dev/null +++ b/salt/top.sls @@ -0,0 +1,8 @@ +base: + 'client*': + - client + 'server*': + - server + 'os:FreeBSD': + - match: grain + - bsd