Skip to content

Commit

Permalink
Add fault injection in Books application:
Browse files Browse the repository at this point in the history 
Signed-off-by: Kush Trivedi <[email protected]>
  • Loading branch information
@kushthedude
kushthedude committed Oct 10, 2020
1 parent ab691bc commit ee27f2d
Show file tree
Hide file tree
Showing 3 changed files with 125 additions and 134 deletions.
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ A service mesh workshop on the fundamentals of Linkerd. All training resources a
### Traffic Control

- [Lab 7 - Traffic Splitting using SMI](lab-7/README.md)
- [Lab 8 - Fault Injection and Circuit Breaking](lab-8/README.md)
- [Lab 8 - Fault Injection using SMI](lab-8/README.md)

## Segment 6

Expand Down
17 changes: 14 additions & 3 deletions lab-7/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ linkerd inject ./sample/bookinfo.yaml | kubectl apply -f -

This will give you the if the Linkerd injection was successful or not.
```sh
linkerd stat deployments -n bookinfo
linkerd stat deploy
```

You will see the following services running in your cluster
Expand Down Expand Up @@ -77,7 +77,7 @@ Now, let's apply traffic-split CRD from SMI :
apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
name: reviews-rollout
name: reviews-split
spec:
service: reviews
backends:
Expand All @@ -89,4 +89,15 @@ spec:
This tells Linkerd’s control plane that whenever there are requests to the reviews service, to split them across the `reviews-v1` and `reviews-v2` based on the weights provided.

If we now go back to our product page, we can only see the reviews with orange or no stars appear on each refresh.
If we now go back to our product page, we can only see the reviews with orange or no stars appear on each refresh.

## Cleanup
```sh
kubectl delete trafficsplit/reviews-split
kubectl delete -f ./sample/service.yaml
```

- Remove the bookinfo application from the `Meshery Dashboard` by clicking on the `trash icon` in the `sample application` card on the linkerd adapters' page.


## [Continue to Lab 8 - Fault Injection using SMI](lab-8/README.md)
240 changes: 110 additions & 130 deletions lab-8/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,170 +1,150 @@
# Lab 8 - Mutual TLS & Identity Verification
# Lab 8 - Fault Injection using SMI in Linkerd

Istio provides transparent mutual TLS to services inside the service mesh where both the client and the server authenticate each others' certificates as part of the TLS handshake. As part of this workshop, we have deployed Istio with mTLS.
Application failure injection is a form of chaos engineering where we artificially increase the error rate of certain services in a microservice application to see what impact that has on the system as a whole. Traditionally, you would need to add some kind of failure injection library into your service code in order to do application failure injection. Thankfully, the service mesh gives us a way to inject application failures without needing to modify or rebuild our services at all.

By default istio sets mTLS in `PERMISSIVE` mode which allows plain text traffic to be sent and accepted by a mesh. We first disallow plain text traffic using `PeerAuthentication` and setting mTLS mode to STRICT.
## Using SMI Traffic Split API to inject errors

## Confirm mTLS is being enforced
This can be easily done by executing a simple command:-
```sh
kubectl get peerauthentication --all-namespaces
```
We can easily inject application failures by using the Traffic Split API of the Service Mesh Interface. This allows us to do failure injection in a way that is implementation agnostic and works across service meshes.

## 8.1 Verify mTLS
Citadel is Istio’s key management service. As a first step, confirm that Citadel is up and running:
```sh
kubectl get deploy -l istio=citadel -n istio-system
```
Output will be similar to:
```
NAME READY UP-TO-DATE AVAILABLE AGE
istio-citadel 1/1 1 1 3m23s
```
We will do this first by deploying a new service which only return errored responses. We will be using a simple NGINX service which has configured to only return HTTP 500 responses.

To experiment with mTLS, let's do so by logging into the sidecar proxy of the `productpage` pod by executing this command:
```sh
kubectl exec -it $(kubectl get pod | grep productpage | awk '{ print $1 }') -c istio-proxy -- /bin/bash
```
We will hten create a traffic split which would redirect the service mesh to send a sample percentage of traffic to the error service instead, let's say 20% of service's traffic to error, then we would have injected an artificial 20% error rate in service.

We are now in the proxy of the `productpage` pod. Check that all the ceritificates are loaded in this proxy:
```sh
ls /etc/certs/
```

You should see 3 entries:
```sh
cert-chain.pem key.pem root-cert.pem
```
### Deploy Linkerd Books Application

Now, try to make a curl call to the `details` service over HTTP:
```sh
curl http://details:9080/details/0
```
We will be deploying [Linkerd Books application](https://github.com/BuoyantIO/booksapp) for this part of the demo

Since, we have TLS between the sidecar's, an HTTP call will not work. The request will timeout. You will see an error like the one below:
```sh
curl: (7) Failed to connect to details port 9080: Connection timed out
```
Use meshery to deploy the bookinfo application :
- In Meshery, navigate to the Linkerd adapter's management page from the left nav menu.
- On the Linkerd adapter's management page, please enter `default` in the `Namespace` field.
- Then, click the (+) icon on the `Sample Application` card and select `Books Application` from the list.

Let us try to make a curl call to the details service over HTTPS but **WITHOUT** certs:
Inject linkerd into sample application using
```sh
curl https://details:9080/details/0 -k
linkerd inject https://run.linkerd.io/booksapp.yml | kubectl apply -f -
```

The request will be denied and you will see an error like the one below:
In the following, one of the service has already beeen configured with the error let's remove the error rate from the same :
```sh
curl: (16) SSL_write() returned SYSCALL, errno = 104
kubectl edit deploy/authors
```

Now, let us use curl over HTTPS with certificates to the details service:
Remove the lines
```sh
curl https://details:9080/details/0 -v --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k
- name: FAILURE_RATE
value: "0.5
```
Output will be similar to this:
Now if you will see `linkerd stat`, the success rate would be 100%
```sh
* Trying 10.107.35.26...
* Connected to details (10.107.35.26) port 9080 (#0)
* found 1 certificates in /etc/certs/root-cert.pem
* found 0 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification SKIPPED
* server certificate status verification SKIPPED
* error fetching CN from cert:The requested data were not available.
* common name: (does not match 'details')
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: O=#1300
* start date: Thu, 26 Oct 2018 14:36:56 GMT
* expire date: Wed, 05 Jan 2019 14:36:56 GMT
* issuer: O=k8s.cluster.local
* compression: NULL
* ALPN, server accepted to use http/1.1
> GET /details/0 HTTP/1.1
> Host: details:9080
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< server: envoy
< date: Thu, 07 Jun 2018 15:19:46 GMT
< content-length: 178
< x-envoy-upstream-service-time: 1
< x-envoy-decorator-operation: default-route
<
* Connection #0 to host details left intact
{"id":0,"author":"William Shakespeare","year":1595,"type":"paperback","pages":200,"publisher":"PublisherA","language":"English","ISBN-10":"1234567890","ISBN-13":"123-1234567890"}
linkerd stat deploy
```
This proves the existence of mTLS between the services on the Istio mesh.
Now lets come out of the container before we go to the next section:
### Create the errored service
```sh
exit
```
## 8.2 [Secure Production Identity Framework for Everyone (SPIFFE)](https://spiffe.io/)
Istio uses [SPIFFE](https://spiffe.io/) to assert the identify of workloads on the cluster. SPIFFE consists of a notion of identity and a method of proving it. A SPIFFE identity consists of an authority part and a path. The meaning of the path in spiffe land is implementation defined. In k8s it takes the form `/ns/$namespace/sa/$service-account` with the expected meaning. A SPIFFE identify is embedded in a document. This document in principle can take many forms but currently the only defined format is x509.
Now we will create our error service, we have NGINX pre-configured to only respond with HTTP 500 status code
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: error-injector
labels:
app: error-injector
spec:
selector:
matchLabels:
app: error-injector
replicas: 1
template:
metadata:
labels:
app: error-injector
spec:
containers:
- name: nginx
image: nginx:alpine
ports:
- containerPort: 80
name: nginx
protocol: TCP
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
volumes:
- name: nginx-config
configMap:
name: error-injector-config
---
apiVersion: v1
kind: Service
metadata:
labels:
app: error-injector
name: error-injector
spec:
clusterIP: None
ports:
- name: service
port: 7002
protocol: TCP
targetPort: nginx
selector:
app: error-injector
type: ClusterIP
---
apiVersion: v1
data:
nginx.conf: |2
events {
worker_connections 1024;
}
To start our investigation, let us check if the certs are in place in the productpage sidecar:
```sh
kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- ls /etc/certs
```
Output will be similar to:
```sh
cert-chain.pem
key.pem
root-cert.pem
http {
server {
location / {
return 500;
}
}
}
kind: ConfigMap
metadata:
name: error-injector-config
```
Mac users, MacOS should have openssl available. If your machine does not have openssl install, install it using your preferred method.
Here is one way to install it on RHEL or CentOS or its derivatives:
```sh
sudo yum install -y openssl-devel
```
After deploying the above errored service, we will create a traffic split resource which will be responsible to direct 20% of the book service to the error.
Here is one way to install it on Ubuntu or Debian or its derivatives:
```sh
sudo apt install -y libssl-dev
```yaml
apiVersion: split.smi-spec.io/v1alpha3
kind: TrafficSplit
metadata:
name: fault-inject
spec:
service: books
backends:
- service: books
weight: 800m
- service: error-injector
weight: 200m
```
Now that we have found the certs, let us verify the certificate of productpage sidecar by running this command:
You can now see an 20% error rate for calls from webapp to books
```sh
kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- cat /etc/certs/cert-chain.pem | openssl x509 -text -noout | grep Validity -A 2
linkerd routes deploy/webapp --to service/books
```
Output will be similar to:
You can also see the error on the web browser
```sh
Not Before: Sep 23 17:32:28 2019 GMT
Not After : Dec 22 17:32:28 2019 GMT
kubectl port-forward deploy/webapp 7000 && open http://localhost:7000
```
Lets also verify the URI SAN:
```sh
kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- cat /etc/certs/cert-chain.pem | openssl x509 -text -noout | grep 'Subject Alternative Name' -A 1
```
If you refresh page few times, you will see `Internal Server Error`.
Output will be similar to:
## Cleanup
```sh
X509v3 Subject Alternative Name: critical
URI:spiffe://cluster.local/ns/default/sa/bookinfo-productpage
kubectl delete trafficsplit/error-split
```
You can see that the subject isn't what you'd normally expect, URI SAN extension has a `spiffe` URI.
This wraps up this lab and training. Thank you for attending!
---
# Additional Resources
For future updates and additional resources, check out [layer5.io](https://layer5.io).
Join the Layer5 service mesh community on [Slack](http://slack.layer5.io) or point questions to [@Layer5](https://twitter.com/layer5) on Twitter.
- Remove the book info application from the `Meshery Dashboard` by clicking on the `trash icon` in the `sample application` card on the linkerd adapters' page.

0 comments on commit ee27f2d

Please sign in to comment.