Skip to content

Latest commit

 

History

History
347 lines (273 loc) · 12.7 KB

ec2-test-client-setup.md

File metadata and controls

347 lines (273 loc) · 12.7 KB

EC2 test client setup

In this lab exercise, we will create an EC2 Virtual Machine Instance as an external test client. EC2 Virtual Machine is a suitable alternative where you may not want to test your local machine due to security, latency, or other reasons. Creating a dedicated Virtual Machine Instance in the same VPC allows us to easily size our test client adequately for performance benchmarking

external client machine using ec2 instance to load test kubernetes services

Create Security group rule and key-pair for Public Key authentication for SSH

To access our client externally need to have an appropriate security groupset and a key-pair to connect over SSH: you can make a security group in the AWS management console website or follow the instructions to set up a security group, key-pair, and EC2 instance using the AWS CLI

  1. Create a Security Group to open only the SSH port in inbound and opens all ports in outbound. You can make a security group on the AWS website or create a security group, for instance, for remote SSH port 22 Access:
MY_SECURITY_GROUP=security-group-uswest2-armand

aws ec2 create-security-group --group-name $MY_SECURITY_GROUP --description "security group with open port 22"

aws ec2 authorize-security-group-ingress --group-name $MY_SECURITY_GROUP \
   --protocol tcp \
   --port 22 \
   --cidr 0.0.0.0/0
                                             
aws ec2 describe-security-groups --group-names $MY_SECURITY_GROUP

To get the security group ID for later steps, you can use the following query filter

MY_SECURITY_GROUP=security-group-uswest2-armand

MY_SECURITY_GROUP_ID=$(aws ec2 describe-security-groups --group-names $MY_SECURITY_GROUP --query 'SecurityGroups[*].[GroupId]' --output text)

echo $MY_SECURITY_GROUP_ID
  1. Create a key-pair for Public Key authentication for SSH. You can use your key pair. However, the public key should be in AWS, and the private key should be kept on your computer.

Important: Because AWS doesn't keep a copy of your private key, there is no way to recover a private key if you lose it. However, there can still be a way to connect to instances that use a lost key pair

Create a key-pair

MY_KEY_PAIR=keypair-armand

aws ec2 create-key-pair --key-name $MY_KEY_PAIR --output text > $MY_KEY_PAIR.pem

Some extra steps

# Describe your key-pair at any time
MY_KEY_PAIR=keypair-armand
aws ec2 describe-key-pairs --key-name $MY_KEY_PAIR 

# IF you need to delete your key pair
MY_KEY_PAIR=keypair-armand
aws ec2 delete-key-pair --key-name $MY_KEY_PAIR

  1. Clean up Private key for ssh use using a text editor (e.g., vi) and make sure the file starts and ends with the standard, -----BEGIN RSA PRIVATE KEY----- at the beginning of the file and -----END RSA PRIVATE KEY----- at the end of the file

    -----BEGIN RSA PRIVATE KEY-----
[your private key here]
-----END RSA PRIVATE KEY-----

For example, the raw private key generated by AWS needs can be cleaned up like so

clean up private key for ssh client

  1. Set secure permission on the file:
MY_KEY_PAIR=keypair-armand
chmod 400 $MY_KEY_PAIR.pem

Create EC2 Instance

  1. Launch a t2.micro instance in the specified subnet of a VPC. Replace the [my_image_id] parameter values with your own. To find the image-id you need, check out Finding a Linux AMI or other resources such as Ubuntu Cloud Image Finder. Note that AMI IDs are unique to each AWS Region, therefore ensuring you have found an AMI available in your VPC region.
  • --image-id: Find an image in your region using Ubuntu Cloud Image Finder, e.g., ami-082e4f383a98efbe9: Ubuntu 20.04 LTS (HVM), SSD Volume Type, in us-west-2
  • --count: 1
  • --key-name: reference your key-pair, e.g.,$MY_KEY_PAIR, the key pair that we made above
  • --security-group-ids: Security group ID
MY_KEY_PAIR=keypair-armand
MY_NAME=armand
MY_SECURITY_GROUP=security-group-uswest2-armand
MY_SECURITY_GROUP_ID=$(aws ec2 describe-security-groups --group-names $MY_SECURITY_GROUP --query 'SecurityGroups[*].[GroupId]' --output text)
MY_EC2_NAME=ec2-uswest2-armand
MY_IMAGE_ID=ami-082e4f383a98efbe9

aws ec2 run-instances \
   --image-id $MY_IMAGE_ID \
   --count 1 \
   --instance-type t2.micro \
   --key-name $MY_KEY_PAIR \
   --security-group-ids $MY_SECURITY_GROUP_ID \
   --tag-specifications "ResourceType=instance,Tags=[{Key=user,Value=$MY_NAME},{Key=Name,Value=$MY_EC2_NAME}]"
  1. Check the status of the EC2 instance. When the instance is ready, the output should have "running"
# Get instance state details [InstanceId, ImageId, State.Code, State.Name] 
MY_EC2_NAME=ec2-apsoutheast2-armand

aws ec2 describe-instances \
   --filters "Name=tag:Name,Values=$MY_EC2_NAME" \
   --query "Reservations[*].Instances[*].[InstanceId, ImageId, State.code, State.Name]" --o text \
   --output text
  1. Retrieve the EC2 instance Public IP Address. This information will be useful later
MY_EC2_NAME=ec2-uswest2-armand

aws ec2 describe-instances \
   --filters "Name=tag:Name,Values=$MY_EC2_NAME" \
  --query "Reservations[*].Instances[*].PublicIpAddress" \
  --output=text
  1. Check out General prerequisites for connecting to your instance, to find out the default username for the AMI you used to launch your Virtual Machine instance. Generally, it is:
* For Amazon Linux 2 or the Amazon Linux AMI, the user name is ec2-user.
* For a CentOS AMI, the user name is centos.
* For a Debian AMI, the user name is admin.
* For a Fedora AMI, the user name is ec2-user or fedora.
* For a RHEL AMI, the user name is ec2-user or root.
* For a SUSE AMI, the user name is ec2-user or root.
* For an Ubuntu AMI, the user name is ubuntu.
* Otherwise, if ec2-user and root don't work, check with the AMI provider..

Connect to EC2 instance

  1. Set key file permission and Access to the Virtual Machine instance using ssh
MY_KEY_PAIR=keypair-armand
MY_EC2_NAME=ec2-uswest2-armand
MY_EC2_PUBLIC_IP=$(aws ec2 describe-instances \
   --filters "Name=tag:Name,Values=$MY_EC2_NAME" \
  --query "Reservations[*].Instances[*].PublicIpAddress" \
  --output=text)
MY_USERNAME=ubuntu #e.g. Ubuntu AMI

ssh -i $MY_KEY_PAIR.pem $MY_USERNAME@$MY_EC2_PUBLIC_IP

Troubleshooting: If you are getting a Invalid Formaterror, make sure your private key started with -----BEGIN RSA PRIVATE KEY-----and ends with -----END RSA PRIVATE KEY-----. See Clean up Private key for ssh

Run HTTP tools on the client machine using a utility Docker container

We can deploy a utility docker container, network-tools, to run various HTTP tools to make requests to Externally accessible applications in our Kubernetes cluster. See the readme for the complete list of tools

For example We can run the following tools from the client machine using a alias to the docker container

# HTTPIE - https://httpie.io/ - HTTP client for the API era
alias http='docker run --rm -it armsultan/network-tools http'
# WRK - https://github.com/wg/wr) - modern HTTP benchmarking tool 
# capable of generating significant load when run on a single multi-core CPU
alias wrk='docker run --rm -it armsultan/network-tools wrk'
# WRK2 - https://github.com/giltene/wrk2 - wrk2 is wrk modifed to produce 
# a constant throughput load, and accurate latency
alias wrk2='docker run --rm -it armsultan/network-tools wrk2'
# VEGETA - https://github.com/tsenart/vegeta - Versatile HTTP load testing 
# tool built out of a need to drill HTTP services with a constant request rate
alias vegeta='docker run --rm -it armsultan/network-tools vegeta'
# FORTIO - https://github.com/fortio/fortio - started as, and is, Istio's 
# load testing tool and now graduated to be its own project
alias fortio='docker run --rm -it armsultan/network-tools fortio'
# testssl.sh - https://testssl.sh/ - tool which checks a server's service 
# on any port for the support of TLS/SSL ciphers, protocols as well as recent
# cryptographic flaws and more.
alias testssl.sh='docker run --rm -ti armsultan/network-tools testssl.sh'

Alternatively, you can run any command in the container using the docker run command. For example:

# docker run --rm -i armsultan/network-tools sh -c [command here]
# e.g.
docker run --rm -i armsultan/network-tools sh -c \
   "echo 'GET https://www.example.com' | vegeta attack -rate=10 -duration=30s -insecure | tee results.bin | vegeta report"
  1. Install Docker on your Client machine (i.e., your EC2 Virtual Machine instance, where you will l running your HTTP test). Follow the correct instructions to install on your OS

Note: To run docker as non-root, According to the official Docker docs, You need to do the following:

# To create the docker group and add your user:
# Create the docker group.
sudo groupadd docker
# Add your user to the docker group.
sudo usermod -aG docker ${USER}
# Now LOG OUT and log back in so that your group membership is 
# re-evaluated or type the following command:
su -s ${USER}
  1. Pull the docker image so that it is readily available
docker pull armsultan/network-tools

  1. From your AWS CLI Client: Get the External IP address of your ingress so that we can run a vegeta load test against your ingress controller and applications from your HTTP Client Machine (EC2) in the next step

    Just like in the Lab excerise: Expose a sample application with ingress, we want to get the external IP address. This time we will use vegeta to make high volume requests for load testing

From your AWS CLI Client: For example, if we have exposed Ingress with a loadBalancer, grab the external IP address of that Load Balancer and copy it to be used for out load testing tool on our EC2 HTTP client Machine

# Get External loadBalancer address
LB_FQDN=$(kubectl get services/nginx-ingress -n nginx-ingress -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')
# Get loadBalancer IP address
dig +short $LB_FQDN A |  awk 'NR==1'
  1. From your HTTP Client Machine (EC2): Run a few test curl requests with the External IP Address ($EXT_IP) into the and if successful, run a vegeta load test

Run some curl commands

EXT_IP=13.236.129.173
# Get coffee
curl https://cafe.example.com/coffee --insecure \
    --resolve cafe.example.com:80:$EXT_IP \
    --resolve cafe.example.com:443:$EXT_IP

# Get tea:
curl https://cafe.example.com/tea --insecure \
    --resolve cafe.example.com:80:$EXT_IP \
    --resolve cafe.example.com:443:$EXT_IP

Now run a vegeta load test

EXT_IP=13.236.129.173
docker run --rm -i armsultan/network-tools sh -c \
   "echo 'GET https://$EXT_IP/coffee' | vegeta attack -rate=10 -duration=30s -header 'Host: cafe.example.com' -insecure | tee results.bin | vegeta report"

Here is an example result. All successful HTTP 200 with no errors:

Requests      [total, rate, throughput]         300, 10.03, 10.03
Duration      [total, attack, wait]             29.905s, 29.9s, 5.135ms
Latencies     [min, mean, 50, 90, 95, 99, max]  2.578ms, 4.568ms, 4.82ms, 5.367ms, 5.551ms, 8.29ms, 18.658ms
Bytes In      [total, mean]                     15343345, 51144.48
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:300
Error Set:

Clean up

  1. Delete EC2 instance
MY_EC2_NAME=ec2-uswest2-armand
MY_EC2_INSTANCE_ID=$(aws ec2 describe-instances \
   --filters "Name=tag:Name,Values=$MY_EC2_NAME" \
   --query 'Reservations[].Instances[].InstanceId' \
   --output text)
   
aws ec2 terminate-instances --instance-ids $MY_EC2_INSTANCE_ID
  1. Delete security group
MY_SECURITY_GROUP=security-group-uswest2-armand

aws ec2 create-security-group --group-name $MY_SECURITY_GROUP
  1. Delete your key pair
MY_KEY_PAIR=keypair-armand
aws ec2 delete-key-pair --key-name $MY_KEY_PAIR