Sam Stoelinga

Deploying Stable Diffusion Web UI on GKE Autopilot

2023-05-30T19:36:00+02:00

Deploying Stable Diffusion WebUI on K8s can be challenging, but no more. Learn how to do it by:

Creating a GKE autopilot cluster
Deploying AUTOMATIC1111 Stable Diffusion Web UI
Accessing the UI through port forwarding

Not up for a challenge? Maybe deploying Stable Diffusion on a GCE VM is easier for you.

Create a GKE Autopilot cluster:

gcloud container clusters create-auto sd \
    --region us-central1

Build and push the docker image to Google Artifact Registry:

export PROJECT_ID=$(gcloud config get project)
export IMAGE=gcr.io/$PROJECT_ID/sd-webui:latest
git clone https://github.com/samos123/stable-diffusion-webui-docker
cd stable-diffusion-webui-docker
gcloud builds submit

Your Stable Diffusion Web UI image will now be available in Google Container Registry.

Create a file named stable-diffusion.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: stable-diffusion-webui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: stable-diffusion-webui
  template:
    metadata:
      labels:
        app: stable-diffusion-webui
    spec:
      nodeSelector:
        cloud.google.com/gke-accelerator: "nvidia-tesla-t4"
        # remove this node selector if you don't want to use spot
        cloud.google.com/gke-spot: "true"
      containers:
      - name: sd
        image: $IMAGE
        env:
        - name: LD_LIBRARY_PATH
          value: /usr/local/nvidia/lib64
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            cpu: "3500m"
            memory: "14Gi"
            ephemeral-storage: 10Gi

Create the deployment:

envsubst < stable-diffusion.yaml | kubectl apply -f -

Creating the pod will take a while, because GKE Autopilot will create nodepool that meets your pod resource requirements. You can check the latest status by running:

kubectl describe pod -l app=stable-diffusion-webui

After a few minutes you should see an event that the pod triggered scale up and the pod should become running. This might take up to 15 minutes due to scale up being slow and pulling a 17GB docker image isn't fast.

Verify that Stable Diffusion is running:

kubectl logs deployment/stable-diffusion-webui

Access the Stable Diffusion WebUI by forwarding ports:

kubectl port-forward deployment/stable-diffusion-webui 7860:7860

In your browser go to the following URL: http://localhost:7860

You should now see Stable Diffusion Web UI like this:

Exposing Stable Diffusion Web UI to the internet

This isn't recommended and there is a risk that your service will get abused by others unless you add some form of authentication.

You can expose the deployment using the following:

kubectl expose deployment stable-diffusion-webui \
  --type=LoadBalancer --name=stable-diffusion-webui \
  --port=80 --target-port=7860

After a while you should see an external IP address assigned to the service. Verify by running:

kubectl get services stable-diffusion-webui

Now you should be able to go to http://external_ip to access Stable Diffusion Web UI

Relevant resources:

Source for the docker image: https://github.com/samos123/stable-diffusion-webui-docker.
Instructions on how to build the stable diffusion webui docker image in GCE.

Deploying Stable Diffusion Web UI using Docker on GCE T4 VM

2023-05-23T19:36:00+02:00

A short guide on how to deploy Stable Diffusion Web UI on a Google Cloud Compute Engine VM with 1 T4 GPU.

Creating the GCE VM with T4 using the Google Cloud Deep Learning Image:

export IMAGE_FAMILY="common-cu110-ubuntu-2004"
export ZONE="us-west1-b"
export INSTANCE_NAME="stable-diffusion"
export INSTANCE_TYPE="n1-standard-4"
gcloud compute instances create $INSTANCE_NAME \
        --zone=$ZONE \
        --image-family=$IMAGE_FAMILY \
        --image-project=deeplearning-platform-release \
        --maintenance-policy=TERMINATE \
        --accelerator="type=nvidia-tesla-t4,count=1" \
        --machine-type=$INSTANCE_TYPE \
        --boot-disk-size=120GB \
        --boot-disk-type=pd-ssd \
        --provisioning-model=SPOT \
        --instance-termination-action=STOP \
        --metadata="install-nvidia-driver=True"

Note: I'm using a spot VM to reduce cost. You might want to use a regular VM instead by removing --provisioning-model=SPOT from the parameters.

The VM will come pre-installed with the nvidia drivers.

SSH into the VM:

gcloud compute ssh $INSTANCE_NAME

Verify that docker can use GPUs:

sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

Add yourself to the docker group so you don't have to sudo:

sudo usermod -aG docker $USER
newgrp docker

Build the docker image inside your VM:

git clone https://github.com/samos123/stable-diffusion-webui-docker
cd stable-diffusion-webui-docker
docker build -t sd-webui-docker .

Run the docker image:

DOCKER_BUILDKIT=1 docker run --name sd -p 7860:7860 -d --runtime=nvidia --gpus all sd-webui-docker

Verify that the Stable Diffusion web UI has launched succesfully:

docker logs sd

You should see something like this after ~30 seconds in the logs:

Running on local URL:  http://0.0.0.0:7860

Exit out of your SSH session by running:

exit

Setup port-forwarding to access the Stable Diffusion Web UI securely:

gcloud compute ssh $INSTANCE_NAME -- -NL 7860:localhost:7860

This is helpful for when your virtual machine doesn't have a public IP address or has firewall rules setup.

You should now be able to acces the Stable Diffusion Web UI by going to the following URL in your browser: http://localhost:7860

Want to be even cooler? Learn how to deploy Stable Diffusion on GKE autopilot

GKE Nodepool Add Labels Without overwriting existing labels

2023-04-07T21:55:00+02:00

GKE has a feature to add node labels to all nodes in the nodepool. GKE will add the label to both the nodes already running in the cluster and also to newly added nodes.

You can use the feature like this:

gcloud container node-pools update my-node-pool \
  --cluster my-cluster --labels sam=test

That would add the label sam: test all your nodepools. The docs however do mention the following: "The label update overwrites any existing labels on the node pool. If the node pool has existing labels that you want to keep, you must include those labels along with any new labels that you want to add."

Well that's lame. But worry not. It's relatively easy to script this with a few lines of bash.

Create a file named gke-nodepool-add-label.sh with the following content:

#!/usr/bin/env bash

set -x

CLUSTER_NAME="$1"
POOL_NAME="$2"
NEW_LABEL="$3"
LABELS=$(gcloud container node-pools describe $POOL_NAME \
         --cluster $CLUSTER_NAME --format "get(config.labels)" | tr ";" ",")
echo "Current labels: $LABELS"
gcloud container node-pools update $POOL_NAME \
  --cluster $CLUSTER_NAME --node-labels="$LABELS,$NEW_LABEL"

Make the file executable:

chmod +x gke-nodepool-add-label.sh

Now you can run the script like this to add a label without overwriting existing labels:

./gke-nodepool-add-label.sh my-cluster my-nodepool sam=test

GKE list tainted nodepools with a specific taint

2023-03-09T19:47:00+01:00

A use case for upgrades involved being able to list all the node pools that have scaled down back to 0 and have a specific taint. This blog post shows the commands you can use to get this information.

List the GKE nodepools that have been tainted with key=upgrade using GKE nodepool:

gcloud container node-pools list --cluster test-np-upgrades --flatten \
  --filter "config.taints.key=upgrade" --format 'value(name)'

For example in my case that returns:

default-pool

Now if you want to get the number of nodes in that nodepool you could run:

kubectl get nodes -l cloud.google.com/gke-nodepool=default-pool \
  --output name --no-headers | wc -l

3 tips for GKE ML/batch workloads

2023-03-05T19:36:00+01:00

There has been an influx of large batch and ML training workloads on GKE. I've personally had the please of working with one of those workloads. The things that batch and ML workload often require from GKE are the following:

Minimize pod disruptions since pods often can't simply be restarted on another node
A single pod might need to stay running for many days
Only spin up VM resources when a large job is launched and scale to 0 when job is done -> Cluster Autoscaler

Top 3 tips for Batch ML workloads

Move your GKE system services to a separate nodepool

You should create a separate nodepool for system services and run your batch/ML job in it's own nodepool. The largest benefit is this will allow you to scale down your most expensive GPU/large VM nodepool back to 0. Otherwise, there might be a teeny-tiny kube-dns pod on your very expensive n2-highmem-128 VM preventing it to be removed by the autoscaler.

Read my blog post on how to move your GKE system services to a separate nodepool
Utilize release channels but prevent upgrades to the nodepools and minor release upgrades to control plane

You can continiously set up a new 180 day maintenance exclusion using the no_minor_or_node_upgrades scope. This will prevent the control plane from going through a minor version upgrade and also prevent nodepools from being upgraded. (Unverified, planning to verify soon and write a blog post on this)
Create new nodepools instead of upgrading

Do not rely on GKE upgrades instead create a new nodepool and safely decommission the old nodepool.

GKE upgrades will forcefully drain your nodes causing pod disruptions. GKE only respects your pod disruption budget for up to 1 hour, however Batch/ML pods often require it be respected multiple days.

So instead it's better to create a new nodepool with the new version and taint the old nodepool so no new pods get scheduled to the old nodepool. Eventually the old nodepool will get to 0 and then it's safe to delete it.

View this blog post to learn how to safely decommission a nodepool without pod disruptions

GKE Safely Drain a Nodepool without pod disruptions

2023-03-04T14:06:00+01:00

GKE/K8s wasn't originally designed for workloads that spin up single pods and want those pods to stay up and running on the same node for very time. That doesn't mean those kind of workloads aren't running on GKE. In fact, there are large GKE ML/batch platform workloads running in production that have these characteristics.

This post will show how to destroy / decommission a nodepool that's running pods that should not be disrupted. The requirements for our decommission of nodepool is as follows:

Any node that's running a batch job pod should stay up for as long as the batch job is running
The max lifetime of batch job pod is 16 days
The nodepool should automatically scale back to 0 after all batch job pods have finished running
New pods should not trigger the nodepool that's being decommissioned to scale up again

How to safely drain the node without pod disruptions?

The approach we will be taking is using the new GKE nodepool taint feature. This feature allows us to taint all existing nodes in the nodepool and also apply the taint on any newly added nodes.

Let's try this feature on the nodepool named default-pool in the cluster test-drain:

export POOL=default-pool
export CLUSTER=test-np-upgrades
gcloud container nodepools update $POOL --cluster $CLUSTER \
  --node-taints upgrade=true:NoSchedule

Note: You might have to set a zone or region using gcloud config set compute/region us-central1.

You would get the following error if cluster autoscaler is enabled on the nodepool:

ERROR: (gcloud.container.node-pools.update) ResponseError: code=400, message=Updates for 'taints' are not supported in node pools with autoscaling enabled (as a workaround, consider temporarily disabling autoscaling or recreating the node pool with the updated values.).

In our case the nodepools all have cluster autoscaler enabled since this is a ML/batch platform. So disabling autoscaler isn't an option.

You might be about to give up but do not! Let's carefully read the error message. It says as a workaround consider temporarily disabling autoscaling.

Let's give that a try, disable autoscaler, apply taints, enable autoscaler again. Run the following commands:

gcloud container node-pools update $POOL \
  --cluster $CLUSTER --no-enable-autoscaling 
gcloud container node-pools update $POOL \
  --cluster $CLUSTER --node-taints upgrade=true:NoSchedule
gcloud container node-pools update $POOL \
  --cluster $CLUSTER --enable-autoscaling --max-nodes 1000 --min-nodes 0

The --max-nodes value doesn't matter because the nodepool got tainted so it will never scale up again.

So looks like this workaround is working well to taint all nodes in a nodepool

Why not use kubectl cordon or taint?

The issue with that approach is that it will cause the nodepool to scale back up again from 0 to 1 because of cluster autoscaler. The cluster autoscaler does not try to scale up again if the taint is set using the GKE API. This is likely an implementation detail of how the cluster autoscaler behaves on GKE.

Verifying this meets our requirements

The test scenario is as follows. Before applying the taints, 4 pods were scheduled and running that caused 4 nodes to be scaled up. So right now I still expect those 4 nodes to be there.

Verify the pods and nodes are still running:

kubectl get pods -o wide

Returns the following:

NAME   READY   STATUS   NODE
pod1   1/1     Running  gke-test-np-upgrades-default-pool-f9fd8128-cd6h
pod2   1/1     Running  gke-test-np-upgrades-default-pool-f9fd8128-tdsf
pod3   1/1     Running  gke-test-np-upgrades-default-pool-a6056569-c577
pod4   1/1     Running  gke-test-np-upgrades-default-pool-a6056569-dqff

So that looks good.

Let's try spinning up another pod that's targeting this tainted nodepool.

Create a file named pod5.yaml with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: pod5
spec:
  nodeSelector:
    cloud.google.com/gke-nodepool: default-pool
  containers:
  - name: sleepy
    image: gcr.io/google_containers/pause-amd64:3.2
    resources:
      requests:
        memory: "2000Mi"

Create the pod:

kubectl apply -f pod5.yaml

Check to ensure cluster autoscaler won't scale up the nodepool. Run the following:

kubectl describe pod pod5

You should see this included in the output:

Status: Pending
..
Warning  FailedScheduling   default-scheduler   0/4 nodes are available: 4 node(s) had taint {upgrade: true}, that the pod didn't tolerate.
Normal   NotTriggerScaleUp  cluster-autoscaler  pod didn't trigger scale-up: 2 max node group size reached

This shows that the pod is pending and not triggering a scale up event. This is great and exactly what was desired.

Now, let's verify the nodepool scales down automatically all the way to 0 when all pods are deleted.

Delete all the batch jobs:

kubectl delete pod pod1 pod2 pod3 pod4

Note that in a normal situation you would wait for the pods to finish execution instead of having to delete them. Once finished, the pod automatically is deleted and autoscaler does the same scale down logic.

Wait for 10 to 15 minutes and then run:

kubectl get nodes

In my case, there seems to be 1 node still running. You might wonder why that is? It turns out that kube-dns and other GKE system services are still running on one of the nodes and are preventing the cluster autoscaling from scaling down. You can solve this by following my blog post on moving GKE system services to a dedicated nodepool.

If you already moved your GKE system services then you would have seen 0 nodes left.

So this concludes that the GKE nodepool tainting feature is an effective way to safely decommission nodepools without disrupting existing pods.

This was tested on GKE 1.21 and behavior is not guaranteed to stay the same in future releases

Removing sidebar from storefront theme

2023-02-12T21:32:00+01:00

By default, storefront comes with a large sidebar that isn't really useful for ecommerce websites. This post shows you how to remove the sidebar by creating a functions.php file in your storefront child theme.

See my previous post on creating a storefront child theme if you don't have a child theme yet.

Inside your child theme directory, create a file functions.php that has the following content:

<?php
function remove_storefront_sidebar() {
    remove_action('storefront_sidebar', 'storefront_get_sidebar', 10);
}

add_action('get_header', 'remove_storefront_sidebar');
?>

Note if you already have a functions.php file then simply append the content above.

Afterwards, upload your new version of your theme with the updated functions.php and you should now see the sidebar removed.

You might also need to update your style.css such that the content is able to utilize the space where the sidebar was before.

Creating a woocommerce storefront child theme

2023-02-11T09:34:00+01:00

The free storefront theme has a lot of features but it doesn't look great by default. Luckily, it's quite easy to create a child theme to make it more attractive. For example, using a child theme you can remove the sidebar and remove the footer credits. This blog post will guide you to creating your child them in 2 simple steps:

Creating a style.css file with metadata about the theme
Packaging and uploading the theme

Why you should use Wordpress child themes instead of directly editing theme files?

Because when you directly edit the theme files, any updates to the parent theme will overwrite your changes. This means all your customizations will be lost and you'll have to start from scratch. But when you use a child theme, your customizations are kept separate from the parent theme, so updates won't affect them.

Now that you know why, let's create the child theme

1. Creating a new `style.css` file with metadata

Create a new directory/folder called mychildtheme and change your working directory to it:

mkdir mychildtheme
cd mychildtheme

Create a file style.css inside the mychildtheme directory with the following content:

/* 
Theme Name: sambal child theme of storefront
Version: 1.0 
Author: Sam Stoelinga
Author URI: https://samos-it.com
Template: storefront
*/

/* optional just an example of how you can change the style */
.product_title h1 {
    color: pink;
}

That's all you need for your child theme to get started. You might wonder, how does it specify the parent theme? That's done through this line:

Template: storefront

It tells Wordpress that the parent theme of our theme is storefront. You can view the theme names by listing directories under the wp-content/themes. In our case there is a directory called storefront and that's how it finds the parent theme.

For fun, and this is optional, I made all product titles pink. So you can see how easy it is to add additional CSS rules to storefront this way.

2. Upload and activate the child theme

There are a few options to upload the child theme to your WooCommerce site:

Use FTP to place the mychildtheme directly in your wp-content/themes directory.
Create a zip file of the mychildtheme and uploading the zip file using the Wordpress admin UI under /wp-admin/themes.php

I think the zip file is the most straight forward so that's what we'll use here.

Create a zip file named mychildtheme.zip by running:

zip -r mychildtheme.zip mychildtheme

Steps to upload theme as zip:

Go to the following URL: https://yourwpblog.com/wp-admin/themes.php
Click on "Add Theme" button
Click on "Upload Theme" button
Click on "Choose File" button
Select the zip file mychildtheme.zip that you created before
Click on "Install Now" button

You should now see your new child theme and be able to preview and activate it. Notice that any product titles are now pink due to our rule in style.css.

Interested in all the cool stuff you can do with a child theme? Then check out these blog posts:

Removing the sidebar in the WooCommerce storefront theme using a child theme

Find files with ack and replace string with sed

2023-02-10T12:10:00+01:00

ack is a nice to tool to search in files in your current directory and subdirectores, but ack won't do search and replace. Luckily, we can easily combine ack and sed using xargs.

ack -l "app.kubernetes.io/name: myapp" | xargs sed -i 's|app.kubernetes.io/name: myapp|app.kubernetes.io/name: "{{ .Release.name }}"|g'

This example searches for the string app.kubernetes.io/name: myapp through all files and subdirs from current directory and tell ack to only output file names with the -l option. Then we use xargs to append the filename to the end of the sed command.

Python Create UUID from random string of words

2023-02-08T10:18:00+01:00

Context: I'm loading data from one database (A) to another database (Weaviate). Weaviate only supports UUID, however database A is using strings as the primary identifier for some tables.

Problem: In order to not duplicate data I need to ensure that the string identifier gets converted to an UUID in a deterministic way.

Solution: Use a hashing mechanism that produces a 128 bit integer, because UUID also uses a 128 bit integer. The MD5 hashing algo is used as the solution. The string is converted into an md5 hash and then the resulting 128 bit integer is passed to Python uuid.UUID method.

Code:

import uuid
import hashlib

def create_uuid_from_string(val: str):
    hex_string = hashlib.md5(val.encode("UTF-8")).hexdigest()
    return uuid.UUID(hex=hex_string)

my_string = "a string used as unique identifier"
print(create_uuid_from_string(my_string))

That should show the following output:

b97c65e2-23a3-e77a-2225-b577e0bbbcd3

Note that result will be the same no matter how many times you run it for the same string.

Deploying a Weaviate cluster on GKE

2023-02-07T12:14:00+01:00

Weaviate has great docs on how to deploy on K8s using Helm, however this guide is specifically focused on an end-to-end deployment of Weaviate on GKE with replication turned on. The following topics will be covered:

Creating and configuring your GKE cluster
Deploying Weaviate with Helm
Tweaking the Weaviate helm values.yml

The Weaviate cluster deployed won't have authentication enabled and will only allow you to run smaller datasets. A follow up blog post for production ready Weaviate cluster that can host larger datasets will be posted later.

1. Creating and configuring the GKE cluster

For this sandbox environment, smaller nodes will be used to optimize for costs. The e2-highmem-2 machine type is a great shape to get started with and comes with 2 CPUs and 16 GB of memory but not recommended for production.

For production with large datasets, you likely will need more memory, because Weaviate stores the vectors in memory. For sandbox however the e2-highmem-2 should suffice.

Create a 3 node e2-highmem-2 GKE cluster by running:

gcloud container clusters create weaviate \
  --disk-size 500GB --disk-type pd-ssd \
  --enable-shielded-nodes --shielded-integrity-monitoring \
  --shielded-secure-boot --image-type COS_CONTAINERD \
  --machine-type e2-highmem-2 --num-nodes 3 \
  --region us-central1 --node-locations=us-central1-a \
  --release-channel stable

2. Deploying Weaviate with Helm

Verify you have access to the GKE cluster and that helm is installed:

kubectl get nodes
helm ls

Get the helm chart

helm repo add weaviate https://weaviate.github.io/weaviate-helm
helm show values weaviate/weaviate > values.yml

Modify values.yaml to utilize PD-SSD instead of default balanced disk by running:

sed -i 's/storageClassName: ""/storageClassName: "premium-rwo"/g' values.yml

Modify values.yaml to set Weaviate replication factor from 1 to 3 by running:

sed -i 's/^replicas:.*/replicas: 3/g' values.yml

Install using the helm chart:

helm install my-weaviate weaviate/weaviate --values values.yml

Verify that the statefulset has been created:

kubectl get statefulset

After a while, you should see that the statefulset will have 3 pods running.

You should now be able to access your website cluster on the LoadBalancer external IP address. Run the following command to see the external IP:

kubectl get svc weaviate

You can access weaviate on the following URL: http://$LB_EXTERNAL_IP

Follow the official Weaviate docs for a quickstart tutorial on how to use your deployed cluster.

Note that anyone on the internet by default can access that IP address

(Optional) Restricting access to K8s cluster only

You have 2 choices if you do not want anyone on the internet to be able to access your cluster:

Change the K8s Service Type from LoadBalancer to Cluster IP, which we cover in this section
Enable Authentication for Weaviate which requires configuring an OIDC provider, which will be covered in another blog post

Changing the K8s Service Type of the Weaviate service to ClusterIP requires changing the values.yml that was used during helm install.

Modify the values.yml by running the following command:

sed -i 's/type: LoadBalancer/type: ClusterIP/g' values.yml

Apply the change by executing helm upgrade with our updated values.yml file:

helm upgrade -f values.yml my-weaviate weaviate/weaviate

Verify the external-ip for the weaviate service is now gone by running:

kubectl get svc weaviate

Leave a comment if you have any other topics you want me to cover or if you enjoyed this post.

Python Create UUID from integer

2023-02-06T11:16:00+01:00

Code:

import uuid
my_uuid = uuid.UUID(int=1)
print(my_uuid)

That should show the following string:

00000000-0000-0000-0000-000000000001

You can also convert back to integer by doing:

my_uuid.int()

For my use case this was needed because my source database has integer based IDs, however the destination database was using UUID based IDs. So converting from integer to UUID was helpful to ensure data can easily be deduplicated.

Review the Python UUID docs for more detailed explanation: Python official docs UUID

Preventing Privileged pods using Pod Security Admission / Standards

2023-01-14T18:38:00+01:00

In a Kubernetes cluster, a privileged pod is a pod that has been given extended permissions beyond the default set of permissions. These extended permissions can include the ability to access the host's network, devices, and other sensitive resources. While privileged pods can be useful in certain situations, they also present a significant security risk.

In this blog post, you will learn how to prevent privileged pods/containers using Pod Security Admission and applying Pod Security Standards. Note that using Pod Security Policy has been deprecated in 1.23 and removed in 1.25.

Introduction to Pod Security Standards and Pod Security Admission Controller

K8s comes with three predefined Pod Security Standards (PSS):

Privileged: No restrictions at all, which is the same as having no PSS applied at all.
Baseline: Minimally restrictive and prevents known high security risk configurations such as privileged pods
Restricted: Most restrictive following security hardening best practices

K8s offers a built-in Pod Security Admission (PSA) controller that to enforce Pod Security Standards across namespaces. The built-in Pod Security Admission controller is included by default since K8s 1.23.

Preventing privileged pods with PSS

The baseline and restricted Pod Security Standard would both prevent privileged pods. However, the restricted PSS would likely be too restrictive for your pod and would require you to update your Pod Spec and potentially your application. So if all you need is preventing privileged pods then Baseline would likely be an easier option.

Enforcing the baseline Pod Security Standard

Enforcing a pod security standard to a namespace has the risk of preventing new pods from being deployed to the namespace. So lets do a dry-run first instead of directly enforcing baseline.

Run a dry-run and check if any warnings are thrown:

kubectl label --dry-run=server --overwrite ns default \
    pod-security.kubernetes.io/enforce=baseline

If you saw namespace/default labeled without any warnings then that means all the currently running pods inside the namespace default would have been admitted if baseline was enforced.

Assuming you had no warnings. Let's start by enforcing the baseline standard on the default namespace:

kubectl label --overwrite ns default \
    pod-security.kubernetes.io/enforce=baseline

Notice that this time the --dry-run=server parameter is not added.

Let's verify that privileged pods indeed are getting blocked.

Create a file named nginx-priv.yaml with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-priv
spec:
  containers:
  - name: nginx-priv
    image: nginx:1.14.2
    ports:
    - containerPort: 80
    securityContext:
      privileged: true

Try to create the privileged pod:

kubectl apply -f nginx-priv.yaml

You should see the following output:

Error from server (Forbidden): error when creating "nginx-priv.yaml": pods "nginx-priv" is forbidden: violates PodSecurity "baseline:latest": privileged (container "nginx-priv" must not set securityContext.privileged=true)

ChatGPT Python: using an unofficial library

2023-01-12T22:44:00+01:00

ChatGPT at the time of writing this blog does not have an official API nor libraries published. Luckily, many folks have inspected the API calls made when using the ChatGPT web UI and created unofficial Python clients.

This blog post, will show you how to use the unofficial chatGPT python library by acheong08.

The blog post contains the following sections:

Installing ChatGPT python library
Configuring authentication
Creating a small Python application that uses the library

1. Installing the ChatGPT Python Library

Create a new directory named chatgpt-python and create a virtualenv in .venv directory:

mkdir chatgpt-python
cd chatgpt-python
python3 -m venv .venv
source .venv/bin/activate

Install revChatGPT using pip:

pip3 install --upgrade revChatGPT

2. Configuring authentication

The unofficial library requires a session token to authenticate to ChatGPT.

Here are the steps to get the session token: 1. Go to https://chat.openai.com and login with your OpenAI account 2. Open the Chrome Developer Tools e.g. by pressing Ctrl + Shift + I 3. Click on the application tab 4. Copy the value for the __Secure-next-auth.session-token cookie

You should now have the session token in your clipboard.

Create a file named chatgpt.json with the following content (don't forget to paste your session token):

{
  "session_token": "<COPY_PASTE_YOUR_SESSION_TOKEN_HERE>"
}

3. Using the ChatGPT python library

Create a chatbot instance:

import json
from revChatGPT.ChatGPT import Chatbot

conf = json.load(open("chatgpt.json"))

chatbot = Chatbot(conf, conversation_id=None, parent_id=None)

Let's ask the ChatGPT something using Python:

response = chatbot.ask("How to use ChatGPT using python")
print(response)

You should see something like this:

{
  'message': 'You can use the OpenAI Python SDK to interact with the ChatGPT model. Here is an example of how you might use the SDK to generate text using the model:\n\n1. First, you will need to install the SDK by running `pip install openai`.\n\n2. Next, you will need to import the `openai` module and set up an API key. You can get an API key from the OpenAI website.\n```\nimport openai\nopenai.api_key = "YOUR_API_KEY"\n```\n3. To generate text using the model, you can use the `openai.Completion.create()` function. \nThis function takes several parameters, including the `prompt` (the text you want the model to complete), the `model` (the name of the model you want to use), and the `max_tokens` (the maximum number of tokens the model should generate).\n```\nresponse = openai.Completion.create(\n    prompt=\'What is the meaning of\',\n    model=\'text-davinci-002\',\n    max_tokens=2048,\n)\nprint(response.choices[0].text)\n```\nThis will output the generated text from the model.\n\nYou can also use GPT-3 fine-tuning and retrain the model with your own dataset using Hugging face\'s transformers library.\n\nYou can also find more information on the OpenAI website, which has documentation and examples for using the SDK.\n',
  'conversation_id': '<SNIP>',
  'parent_id': '<SNIP>'
}

Trying ValidationAdmissionPolicy aka CEL Admission in K8s 1.26

2022-12-14T22:44:00+01:00

CEL for admission control is a new 1.26 feature. With the feature, define ValidationAdmissionPolicy to express your desired policy and ValidationAdmissionPolicyBinding to assign the policy to e.g. a namespace.

This post has the following sections:

Creating a 1.26 cluster with ValidationAdmissionPolicy / CEL Admission enabled
Configure the policies
Verifying the policy is working as expected

Creating a 1.26 cluster

We will be using kind to create a cluster with ValidatingAdmissionPolicy feature gate enabled and runtimeConfig set to admissionregistration.k8s.io/v1alpha1.

Create a kind configuration file named cluster.yaml with the following content:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  "ValidatingAdmissionPolicy": true
  runtimeConfig:
    "admissionregistration.k8s.io/v1alpha1": true
    nodes:
    - role: control-plane
      image: kindest/node:v1.26.0@sha256:691e24bd2417609db7e589e1a479b902d2e209892a10ce375fab60a8407c7352

Create the kind cluster by running:

kind create cluster --config cluster.yaml

Verify the cluster is up and running:

kubectl get nodes

Configure the policies

For this example, we want a simple policy that prevents creating a deployment that has the name samos.

Create a file named cel-policy.yaml with the following content:

apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicy
metadata:
  name: "example-policy"
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["apps"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["deployments"]
  validations:
    - expression: "object.metadata.name != 'samos'"

Create the policy by running:

kubectl apply -f cel-policy.yaml

Create a file named cel-policy-binding.yaml with the following content:

apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: "example-policy-test"
spec:
  policyName: "example-policy"
  matchResources:
    namespaceSelector:
      matchLabels:
        test: sam

The binding tells K8s to apply the ValidationAdmissionPolicy named example-policy to any namespace that has the label test=sam.

Let's add the label test=sam to the default namespace by running:

kubectl label namespaces default test=sam

The policy should now be effective on the default namespace.

Verify everything works

Create a file named deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: samos
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Try and create the deployment by running:

kubectl apply -f deployment.yaml

You should see an error like this:

The deployments "samos" is invalid: : ValidatingAdmissionPolicy 'example-policy' with binding 'example-policy-test' denied request: failed expression: object.metadata.name != 'samos'

Deploying K8s on your laptop with minikube

2022-12-09T08:44:00+01:00

K8s on your laptop is helpful for initial development and testing environment. Minikube makes it easy to get K8s deployed on your laptop. Let's get K8s installed by doing the following:

Installing required tools: docker, minikube and kubectl
Deploying the minikube cluster with minikube start
Verifying you can deploy an application to your minikube cluster

Installing required tools

Install docker by following the steps outlined here:

# These steps are for ubuntu, you can follow steps for another distro if needed
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

Verify docker was installed succesfully by running:

docker run hello-world

Note if you get permission denied then you might need to add yourself to the docker group. You can do this by running:

sudo usermod -a -G docker $USER
# make the change effective in current shell
newgrp docker

Install kubectl by following the steps outlined here:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Install minikube by following the steps outlined here:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

All required tools have now been installed.

Deploying K8s on your laptop

Follow steps of the "Start your cluster" section outlined here:

minikube start

Verify that you can access your cluster by running:

kubectl get nodes

Deploying an application to your K8s cluster

Let's deploy a simple web application by running:

kubectl create deployment hello-minikube \
  --image=kicbase/echo-server:1.0

You can expose the application on port 8080 by running:

kubectl expose deployment hello-minikube --type=NodePort --port=8080
minikube service hello-minikube

The minikube service command should have setup port forwarding for you and opened a browser so you can access the application.

Full Text & Vector Search for Firestore with Weaviate

2022-11-03T11:36:00+01:00

Many applications require searching through large text fields in your firestore database. For example, you might need to search through articles containing a specific word or searching for a concept.

Cloud Firestore does not have support for indexing of text fields. So 3rd party solutions such as Weaviate or ElasticSearch are required for search.

This blog post describes how to integrate Weaviate with Firestore to enable Full Text and Semantic search for an example Note taking application. At a high-level the following will be done:

Deploy Weaviate and define the Schema in Weaviate
Create a Firebase Function to index Notes automatically when a note is created
Create a Firebase Function to Search the indexed Notes using Weaviate

Deploy Weaviate and define Schema

You can use Weaviate Cloud Service or you can deploy Weaviate yourself by choosing one of the self-deploy options here.

This blog post uses the Weaviate Cloud Service because that provides us with an easy to use public endpoint for Weaviate. Ensure Enable OIDC Authenication is set to "Disabled". Note that this endpoint is accessible by anyone on the internet and doesn't require authentication. So this isn't suitable for production and for demo purposes only

Install Weaviate Javascript client:

mkdir firestore-weaviate-test
cd firestore-weaviate-test
npm install weaviate-client

Let's define the Weaviate Schema for our Note taking application. Currently this is stored in Firestore in the following way:

// /notes/${ID}
{
  owner: "{UID}", // Firebase Authentication's User ID of note owner
  text: "This is my first note!"
}

Now let's define our Schema to index the Firestore Note. Create a file named schema-creation.js with the following content:

const weaviate = require("weaviate-client");

const client = weaviate.client({
    scheme: "https",
    host: "your-wcs-instance.semi.network",
});

var classObj = {
    class: "Note",
    description: "An index for Full-Search of notes in Firestore",
    vectorizer: "text2vec-openai",
    moduleConfig: {
        "text2vec-openai": {
            model: "babbage",
            type: "text",
        },
    },
    properties: [
        {
            dataType: ["string"],
            description: "The ID of the note",
            name: "nid",
        },
        {
            dataType: ["string"],
            description:
                "The Firebase Authentication UID of the creator of the note",
            name: "uid",
        },
        {
            dataType: ["text"],
            description: "The content of the note",
            name: "text",
            moduleConfig: {
                "text2vec-openai": {
                    skip: false,
                    vectorizePropertyName: false,
                },
            },
        },
    ],
};


client.schema
    .classCreator()
    .withClass(classObj)
    .do()
    .then((res) => {
        console.log(res);
    })
    .catch((err) => {
        console.error(err);
    });

Review the file and make sure you changed your-wcs-instance to your actual instance name. Alternatively, change the connection section to connect to your self deployed Weaviate instance.

Execute the Schema creation script by running:

node schema-creation.js

Weaviate is now ready to start indexing notes. You can import existing notes by following the documention on importing data into Weaviate.

Indexing Firestore Notes automatically on creation

A Firebase Function can be used to automatically insert a Note into the Weaviate index when it is created. This assumes you've already created a Firebase project, if not, please create a firebase project first.

Initialize Firebase in your current working directly:

firebase init firestore
firebase init functions # say no to eslint

The above command will create a directory named functions inside your current working directory.

Change to the functions directory and install weaviate-client:

cd functions
npm install weaviate-client

Open the file index.js and replace it with the following content:

const functions = require("firebase-functions");
const weaviate = require("weaviate-client");
const WEAVIATE_HOST = functions.config().weaviate.host;
const OPENAI_APIKEY = functions.config().weaviate.openaiapikey;

const client = weaviate.client({
    scheme: "https", // http or https
    host: WEAVIATE_HOST,
    headers: { "X-OpenAI-Api-Key": OPENAI_APIKEY },
});

// Update the search index every time a note is created.
exports.onNoteCreated = functions.firestore
    .document("notes/{noteId}")
    .onCreate(async (snap, context) => {
        // Get the note document
        const note = snap.data();

        // Use the 'nodeId' path segment as the identifier for Elastic
        const id = context.params.noteId;
        client.data
            .creator()
            .withClassName("Note")
            .withProperties({
                nid: id,
                uid: note.uid,
                text: note.text,
            })
            .do()
            .then((res) => {
                console.log(res);
            })
            .catch((err) => {
                console.error(err);
            });
    });

Configure the Firebase Environment based on your own values. Run the following command but replace the values to match your environment:

firebase functions:config:set weaviate.host="your-wcs-instance.semi.network" weaviate.openaiapikey="yourverysecretAPIkey"

Deploy the function by running:

firebase deploy --only functions

Let's try creating many notes (7k+) in Firestore to test the function. Create a file named create-notes.js with the following content:

const {
    initializeApp,
    applicationDefault,
    cert,
} = require("firebase-admin/app");
const {
    getFirestore,
    Timestamp,
    FieldValue,
} = require("firebase-admin/firestore");
const fs = require("fs");
const { parse } = require("csv-parse");

initializeApp({
    credential: applicationDefault(),
});

const db = getFirestore();

var parser = parse({ columns: true }, function (err, records) {
    records.forEach(async (record) => {
        const docRef = await db.collection("notes").add({
            uid: "same",
            text: record.description,
        });
    });
});

fs.createReadStream(__dirname + "/wine_reviews.csv").pipe(parser);

Download a demo dataset that contains wine reviews. We'll use the content for the notes.

curl -O https://raw.githubusercontent.com/semi-technologies/weaviate-examples/main/semanticsearch-transformers-wines/data/wine_reviews.csv

The code for creating notes requires application default credentials. Let's configure your Google Application Default credentials by running the following:

gcloud config set project your-firebase-project-name
gcloud auth application-default login

Install the node-csv dependency by running:

npm install csv-parse

Create the notes in Firestore by running:

node create-notes.js

Verify the notes have also been created in Weaviate by running this GraphQL query in Weaviate:

{
  Get {
    Note {
      text
    }
  }
}

Creating a Firebase Function to search using Weaviate

Let's add another firebase function by adding the following code at the bottom of index.js:

exports.searchNotes = functions.https.onCall(async (data, context) => {
    const concepts = data.concepts;

    const notes = await client.graphql
        .get()
        .withClassName("Note")
        .withFields("nid uid text")
        .withNearText({
            concepts: concepts,
            distance: 0.6,
        })
        .do();
    console.log(notes)

    return {notes: notes.data.Get.Note};
});

Deploy the updated functions to Firebase by running:

firebase deploy --only functions

Afterwards, you can use the firebase callable function in your Firebase app like this:

var searchNotes = firebase.functions().httpsCallable('searchNotes');
searchNotes({concepts: ["light fruity wine"]}).then( (result) => {
    const notes = result.data.notes;
    console.log(notes[0]); // print the 1st result
});

Conclusion

This blog post demonstrated how to implement Full Text Semantic Search for Firestore by using Weaviate. The index in Weaviate is automatically updated whenever a new document is added to Firestore. OpenAI is used to automatically generate a vector for the text field. The firebase callable function makes it easy to search through the database using Weaviate without having to call Weaviate directly.

Security note

The setup in this blog post uses an unauthenticated Weaviate setup. Depending on your requirements, you might need to configure some form of authentication between the Cloud Function and your Weaviate instance.

GKE GPU timesharing and resource quotas experiment

2022-08-26T22:02:00+02:00

You only got a few GPUs and want to pretend to your end-users that you got many? Then GKE GPU timesharing might just be the feature for you to save costs on GPUs that are underutilized. In this blog post you will learn:

Creating a GKE nodepool with timesharing enabled
Configure GPU based ResourseQuotas on multiple namespaces
How it's possible to request 4 GPUs in different namespaces even though there is only a single physical GPU

Creating the cluster and nodepool

Create a cluster that meets the requirements of timesharing. At the time of writing the blog post this requires using the rapid release channel. This creates a cluster with a default CPU only nodepool using e2-medium. System services will run on the default CPU only nodepool.

gcloud container clusters create gpu-gpu-timesharing \
  --region us-central1 --node-locations=us-central1-a \
  --machine-type e2-medium --max-nodes=3 --min-nodes=1 \
  --enable-autoscaling --release-channel rapid \
  --shielded-integrity-monitoring --shielded-secure-boot

Create the T4 GPU nodepool with timesharing enabled

gcloud container node-pools create gpu-time-sharing \
    --cluster=gke-gpu-timesharing \
    --machine-type=n1-standard-4 \
    --num-nodes=1 \
    --region=us-central1 \
    --node-locations=us-central1-a \
    --accelerator=type=nvidia-tesla-t4,count=1,gpu-sharing-strategy=time-sharing,max-shared-clients-per-gpu=8 \
    --shielded-secure-boot --shielded-integrity-monitoring

Install the nvidia GPU device drivers:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded-latest.yaml

You should now have a working GKE cluster with 2 nodepools. A CPU only nodepool and a GPU nodepool with 1 GPU node that has 1 T4. This T4 GPU can be used by up to 8 clients at the same time. So it could be used by 8 pods each requesting 1 GPU or it could be used by 2 pods each requesting 4 GPUs.

Creating the namespaces and resource quotas

The example will use 2 fictional tenants: tenant-a and tenant-b.

Create the namespaces for both tenants:

kubectl create ns tenant-a
kubectl create ns tenant-b

Create a quota for tenant-a

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ResourceQuota
metadata:
  name: test-gpu-quota-1
  namespace: tenant-a
spec:
  hard:
    requests.nvidia.com/gpu: 1
    limits.nvidia.com/gpu: 1
    requests.cpu: 1
    limits.cpu: 1
EOF

Create a quota for tenant-b

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ResourceQuota
metadata:
  name: test-gpu-quota-1
  namespace: tenant-b
spec:
  hard:
    requests.nvidia.com/gpu: 10
    limits.nvidia.com/gpu: 10
    requests.cpu: 2
    limits.cpu: 2
EOF

Creating a GPU pod in tenant-a and tenant-b

Let's create 4 pods in tenant-a and 4 pods in tenant-b where each pod is requesting a single GPU.

The job that we will use in both tenants.

cat > gpu-deployment.yml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cuda-simple
spec:
  replicas: 4
  selector:
    matchLabels:
      app: cuda-simple
  template:
    metadata:
      labels:
        app: cuda-simple
    spec:
      containers:
      - name: cuda-simple
        image: nvidia/cuda:11.0.3-base-ubi7
        command:
        - bash
        - -c
        - |
          /usr/local/nvidia/bin/nvidia-smi -L; sleep 300
        resources:
          limits:
            nvidia.com/gpu: 1
EOF

launch it in tenant-a and tenant-b

kubectl apply -f gpu-deployment.yml -n tenant-a
kubectl apply -f gpu-deployment.yml -n tenant-b

You should see 1 pod running in tenant-a:

kubectl get pods -n tenant-a

this demonstrates that resourcequotas are effective and able to limit GPUs that are timeshared.

You should see 4 pods running in tenant-a:

kubectl get pods -n tenant-b

Verify that pods in both tenant-a see the whole GPU being usable

kubectl exec -n tenant-a deploy/cuda-simple -ti -- sh
nvidia-smi
exit

Verify that pods in tenant-b see the whole GPU being usable

kubectl exec -n tenant-b deploy/cuda-simple -ti -- sh
nvidia-smi
exit

You can also take a look at the GPU node to see that it will report having 8 GPU devices even though it only has a single GPU attached to the VM. This is due to the fact of timesharing being enabled with the setting max-shared-clients-per-gpu=8.

Run the following to get node details of the GPU nodes with tesla t4s

kubectl get node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 -o yaml

You should see something like this in the output:

    allocatable:
      attachable-volumes-gce-pd: "127"
      cpu: 3920m
      ephemeral-storage: "47093746742"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 12663300Ki
      nvidia.com/gpu: "1"
      pods: "110"

Summary

Demonstrated how to use timesharing GPUs in GKE and verified that each tenant will think they're getting the whole GPU even when they are being shared. We are able to limit how much GPUs a single tenant can request by applying resource quotas. GPU timesharing is a great way to reduce costs of GPUs when the GPU utilization of a single tenant/user is low.

GKE move system services (kube-dns, calico) to dedicated nodepool

2021-10-11T14:32:00+02:00

GKE by default deploys kube-dns and other system services to any of your nodepools. This is probably fine for most cases, but certain use cases might require preventing system services from running on the same nodes as your where your applications are running. This blog post provides instructions on how to force kube-dns and other GKE system services onto a specific nodepool.

Some of the use cases this is helpful for:

Cluster autoscaler unable to scale down a node that's running kube-dns
Ensuring all capacity of a nodepool is usable instead of some of it being taken up by system services
Preventing an application from possibly messing around with your system services

The easiest way to achieve this is to use nodeSelector and specify the label of the system services nodepool. Let's take a look at how to modify the deployments in the kube-system namespace and add a nodeSelector.

First, create the patch to add a nodeSelector by creating a file named nodeSelector-patch.yaml with the following content:

spec:
  template:
    spec:
      nodeSelector:
        cloud.google.com/gke-nodepool: system

Note: this assumes you have a nodepool named system where all your system services will be deployed. You will need to adjust the name of the nodepool to match your own environment. In my case I have multiple nodepools where 1 of the nodepools is named system.

Now let's patch all the deployments in the kube-system namespace. Before we do that let's take a quick look what's in the kube-system namespace:

kubectl get deployment -n kube-system
NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
calico-node-vertical-autoscaler      1/1     1            1           342d
calico-typha                         1/1     1            1           342d
calico-typha-horizontal-autoscaler   1/1     1            1           342d
calico-typha-vertical-autoscaler     1/1     1            1           342d
config-management-operator           1/1     1            1           236d
event-exporter-gke                   1/1     1            1           342d
gke-oidc-envoy                       1/1     1            1           137d
kube-dns                             2/2     2            2           342d
kube-dns-autoscaler                  1/1     1            1           342d
l7-default-backend                   1/1     1            1           342d
metrics-server-v0.3.6                1/1     1            1           342d

That looks fine in my case, but your output may differ so make sure to review before you add a nodeSelector to all these deployments.

Let's add the nodeSelector by patching all the deployments in kube-system:

kubectl get deployment -o NAME -n kube-system | xargs -n 1 -I {} kubectl patch {} -n kube-system --patch "$(cat nodeSelector-patch.yaml)"

In my case this was the output:

deployment.apps/calico-node-vertical-autoscaler patched
deployment.apps/calico-typha patched (no change)
deployment.apps/calico-typha-horizontal-autoscaler patched
deployment.apps/calico-typha-vertical-autoscaler patched
deployment.apps/config-management-operator patched
deployment.apps/event-exporter-gke patched
deployment.apps/gke-oidc-envoy patched
deployment.apps/kube-dns patched (no change)
deployment.apps/kube-dns-autoscaler patched
deployment.apps/l7-default-backend patched
deployment.apps/metrics-server-v0.3.6 patched

Now the pods belonging to those deployment should be terminated in the old nodepool and be running in the nodepool named system. You can verify this by checking for example on which node the kube-dns pods are running:

kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP         NODE                                 NOMINATED NODE   READINESS GATES
kube-dns-848bb8c5d-fzh7h   4/4     Running   0          28m   10.4.3.3   gke-cluster-1-system-2db0a54b-jx6s   <none>           <none>
kube-dns-848bb8c5d-l72fh   4/4     Running   0          28m   10.4.4.3   gke-cluster-1-system-2db0a54b-ll3p   <none>           <none>

GKE docker registry with HTTP proxy

2021-05-21T10:52:00+02:00

You are at one of those places that requires you to use a proxy to access your company wide Docker registry. Sometimes HTTP proxies are used to supposedly improve security or to workaround IP based rate limits. Well good luck, you're in for a ride on how to do this with GKE and I've made this easier for you by documenting the steps needed to get this to work on GKE.

The high-level solution is to configure HTTP proxy environment variables for the docker daemon running on the GKE nodes. This is done by following these steps:

Create a script that places a file with proxy config under under /etc/systemd/system/docker.service.d/ on the GKE nodes.
Create a K8s DaemonSet that runs the script on each node

Creating the docker HTTP proxy script as ConfigMap

There are 2 things the script needs to do:

Place the docker HTTP proxy configuration under /etc/systemd/system/docker.service.d/
Reload systemd configuration and restart docker daemon

Now create the file called proxy-inject-cm.yml with the following content:

apiVersion: v1
kind: ConfigMap
metadata:
  name: proxy-injector
  namespace: kube-system
data:
  proxy-injector.sh: |
    #!/bin/bash

    cat > /mnt/etc/systemd/system/docker.service.d/http-proxy.conf << EOF
    [Service]
    Environment="HTTP_PROXY=http://sams-proxy:3128"
    EOF
    nsenter --target 1 --mount systemctl daemon-reload
    nsenter --target 1 --mount systemctl restart docker

Note that you will need to edit this line in proxy-inject-cm.yml:

    Environment="HTTP_PROXY=http://sams-proxy:3128"

You need to change HTTP_PROXY=http://sams-proxy:3128 to your actual HTTP proxy.

Create the K8s configmap:

kubectl apply -f proxy-inject-cm.yml

Creating the proxy injector DaemonSet

The DaemonSet simply runs the script on each node during node startup. Create a file called proxy-injector-dm.yml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: proxy-injector
  namespace: kube-system
  labels:
    app: proxy-injector
spec:
  selector:
    matchLabels:
      app: proxy-injector
  template:
    metadata:
      labels:
        app: proxy-injector
    spec:
      hostPID: true
      initContainers:
      - name: proxy-injector
        image: ubuntu
        command:
          - /usr/local/bin/proxy-injector.sh
        volumeMounts:
          - name: etc
            mountPath: "/mnt/etc"
          - name: bins
            mountPath: "/usr/local/bin"
        securityContext:
          privileged: true
      volumes:
      - name: etc
        hostPath:
          path: /etc
      - name: bins
        configMap:
          name: proxy-injector
      containers:
      - name: pause
        image: gcr.io/google_containers/pause
      priorityClassName: system-node-critical
      tolerations:
        - effect: NoSchedule
          operator: Exists

Before you execute this next command, please note that this will cause disruption to existing pods because docker is force restarted. So be extra careful before running it. Create the K8s DaemonSet:

kubectl apply -f proxy-injector-dm.yml

Summary

We have rolled out a daemonset that modifies the docker HTTP proxy settings. Your GKE nodes will now use the HTTP proxy when pulling down images. Note that you would need to create a similar DaemonSet or simple bash script that SSHs into the GKE nodes to remove the proxy settings because simply delete the DaemonSet will leave the existing proxy settings.

GKE custom OSS K8s cluster autoscaler

2021-03-12T22:02:00+01:00

Update 2023-03-27: Added instructions for clusters using Workload Identity

This blog post described how to deploy your own K8s cluster autoscaler instead of the cluster autoscaler that's bundled with GKE. This can be helpful in the rare case that the bundled GKE cluster autoscaler doesn't work for you.

Note that the GKE bundled cluster autoscaler is vastly different from the OSS cluster autoscaler. GKE has done optimizations to make it perform better and scale based on cost. So in general you are strongly recommended to use the bundled GKE cluster autoscaler.

Why?

Now you might ask why would I do this instead of just using the bundled GKE cluster autoscaler? The short answer is you shouldn't unless you:

Require to tweak parameters of the K8s cluster autoscaler that are not exposed to you. Ask yourself are tweaking these parameters really that important?
Hit scaling limitations with the bundled GKE cluster autoscaler when running more than 5,000 nodes per cluster. Note that K8s OSS cluster autoscaler would likely also hit scaling limitations. Instead I would recommend working closely with the Google team when you're at that scale to see what options are available.
Bug in the GKE bundled autoscaler that isn't fixed yet but is already fixed in the OSS K8s cluster autoscaler

How?

You're stubborn and persistent and still believe you have to do this then let's look at how this can be done.

1. Create a GKE cluster without cluster autoscaler enabled

PROJECT_ID=$(gcloud config get-value project)
GKE_VERSION="1.25.6-gke.1000"
gcloud container clusters create no-autoscaler \
  --no-enable-autoscaling \
  --region us-central1 --node-locations us-central1-a \
  --release-channel None \
  --cluster-version $GKE_VERSION \
  --workload-pool $PROJECT_ID.svc.id.goog

2. Configure Workload Identity

Create a serviceAccount to be used for cluster autoscaler:

gcloud iam service-accounts create k8s-cluster-autoscaler

Grant permission to the service account:

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member "serviceAccount:k8s-cluster-autoscaler@$PROJECT_ID.iam.gserviceaccount.com" \
    --role "roles/compute.instanceAdmin.v1"

Allow K8s Service Account to use the Google Service Account:

gcloud iam service-accounts add-iam-policy-binding k8s-cluster-autoscaler@$PROJECT_ID.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$PROJECT_ID.svc.id.goog[kube-system/cluster-autoscaler-gce-cluster-autoscaler]"

3. Deploy the K8s cluster autoscaler using helm

Create a file named values.yaml that contains the config for autoscaler:

autoscalingGroupsnamePrefix:
  # The name prefix of of the GCE managed instance groups
  # this will be different in your case and in general follows this pattern
  # gke-$CLUSTER_NAME (in my case cluster name is no-autoscaler)
  - name: gke-no-autoscaler
    minSize: 1
    maxSize: 10

autoDiscovery:
  # the cluster name of the GKE cluster, change to your cluster name
  clusterName: no-autoscaler

extraArgs:
  # I had to do this to prevent this error:
  # lock is held by gke-2ff9d20265d64621837d-14fa-96a3-vm and has not yet expired
  # failed to acquire lease kube-system/cluster-autoscaler
  leader-elect: false

cloudProvider: gce
image:
  # Change the image tag version to match the GKE version. So if running
  # GKE 1.25.6 make sure this version also is 1.25.0. there is no 1.25.6 image
  tag: v1.25.0

Deploy K8s cluster autoscaler with the just created values.yaml:

helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm install cluster-autoscaler autoscaler/cluster-autoscaler -f values.yaml --namespace kube-system

You should now have a K8s cluster autoscaler working. Try spinning up many pods with high resource requests and the cluster should automatically add more nodes.

Custom DNS entry with KubeDNS stubdomain

2021-03-11T22:02:00+01:00

An example use case that I've seen is where you have a K8s service exposed on the ClusterIP and you want to make that service accessible over a domain name that you don't control.

You can do to the following steps to set this up:

Deploy CoreDNS with custom DNS entries
Configure kube-dns to use stubDomain that points to CoreDNS

1. Deploying CoreDNS with custom DNS entries

Create the namespace for coredns k8s resources:

kubectl create ns coredns

Create a file called core-dns-cm.yaml with the following content:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: coredns
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        cache 30
        log
        reload
        loadbalance
        file /etc/coredns/example.db example.org
    }
  example.db: |
    ; example.org test file
    example.org.            IN      SOA     sns.dns.icann.org. noc.dns.icann.org. 2015082541 7200 3600 1209600 3600
    example.org.            IN      NS      b.iana-servers.net.
    example.org.            IN      NS      a.iana-servers.net.
    example.org.            IN      A       10.200.0.1

In the configmap you definte the custom dns entries. In the example, the A record for example.org has been overriden to point to 10.200.0.1. You would change example.org and the specific DNS entries depending on your needs.

Create the configmap:

kubectl apply -f core-dns-cm.yaml

Create a file named core-dns-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: coredns
  labels:
    k8s-app: coredns
    kubernetes.io/name: "CoreDNS"
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: coredns
  template:
    metadata:
      labels:
        k8s-app: coredns
    spec:
      containers:
      - name: coredns
        image: coredns/coredns:1.8.3
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
            - key: example.db
              path: example.db

Create the deployment that uses the configmap:

kubectl apply -f core-dns-deployment.yaml

Create a file named core-dns-svc.yaml:

apiVersion: v1
kind: Service
metadata:
  name: coredns
  namespace: coredns
  labels:
    k8s-app: coredns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  clusterIP: 10.0.6.177
  selector:
    k8s-app: coredns
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP

Notice that we set a static cluster IP so this blog post is easier to follow. You will probably want to remove clusterIP: 10.0.6.177 and let K8s asign a random clusterIP, afterwards note note down the cluster IP. You will need the CoreDNS cluster IP in the next step during configuring kube-dns.

Create the coredns k8s service:

kubectl apply -f core-dns-svc.yaml

2. Configure kube-dns to use stubDomain that points to CoreDNS

In this step, you will be configuring kube-dns to go to our coredns server that holds the custom DNS entries. This is done by configuring a stubDomain that for e.g. example.org that points to the coreDNS clusterIP.

Create a file named kube-dns-cm.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
        {"example.org" : ["10.0.6.177"]}

Apply the config map:

kubectl apply -f kube-dns-cm.yaml

Summary

You have deployed a custom DNS server that holds custom DNS entries by using coreDNS deployed on K8s. Afterwards you configured kube-dns to point to that DNS server by configuring a stubdomain. Now you can test it out by deploying a pod and running dig example.org from a pod. That should return 10.200.0.1.

Mounting FUSE without privileges on K8s/GKE

2021-03-02T10:02:00+01:00

This might become a full blog post but for now read the steps here: https://github.com/samos123/gke-gcs-fuse-unprivileged

Securing Redis with Istio TLS origination

2020-12-28T10:42:00+01:00

Istio is daunting and not all use cases are well documented. The public docs focus mostly on using the egress gateway for TLS orignation. The use case of using the sidecar for TLS origination with a database isn't documented well. This blog post hopes to solve that.

So you've actually done security well and are using an external Redis provider that only allows TLS to talk to it. You could simply configure each of your applications to use TLS from the application pod or you can use Istio to handle the TLS part. This blog is focused on how to use Istio to do TLS origination for Redis (TCP) using the sidecar instead of the egress gateway.

TLS origination occurs when an Istio proxy (sidecar or egress gateway) is configured to accept unencrypted internal TCP connections, encrypt the requests, and then forward them to servers that are secured using simple or mutual TLS. This is the opposite of TLS termination where an ingress proxy accepts incoming TLS connections, decrypts the TLS, and passes unencrypted requests on to internal mesh services.

The blog post contains the following sections:

Creating the Redis instance with Aiven or Redislabs
Creating the Istio resources
Validating that TLS origination worked

1. Creating the Redis instance with Aiven or Redislabs

Aiven is a managed database provider and provides a managed Redis service. The Redis service allows you to require TLS which is what will be used in this blog post. You can use any TLS Redis service instead of Aiven.

Go to aiven.io/redis and follow the steps to create a Redis service using the 30 day free trial. Aiven was really convenient for me. Redislabs the creators of Redis provide similar service with a free 30MB Redis instance. You might want to support the creators of Redis by using Redislabs.

In this blog post the variables will be used:

export REDIS_HOST=redis-1425a1d9-google-bc39.aivencloud.com
export REDIS_PORT=16222

You will need to set these variables to your environment specifics.

2. Creating the Istio resources

In this section, you will create the following Istio resources: - DestinationRule: To configure how outgoing connections to Redis should be handled. For example, this is how we configure the TLS settings. - ServiceEntry: Such that Istio knows about the external Redis service

Create the Redis namespace which is used for testing:

kubectl create ns redis
# If you use kubectx
kubens redis

Create the following ServiceEntry:

cat <<EOF | kubectl apply --namespace=redis -f -
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-aiven-redis
spec:
  hosts:
  - $REDIS_HOST
  location: MESH_EXTERNAL
  resolution: DNS
  ports:
  - number: $REDIS_PORT
    name: tcp-redis
    protocol: TCP
EOF

Create the DestinationRule:

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: external-aiven-redis
  namespace: redis
spec:
  host: $REDIS_HOST
  trafficPolicy:
    tls:
      mode: SIMPLE
      caCertificates: /etc/ssl/certs/ca-certificates.crt
EOF

Note that Istio has a security issue that if you do not specify the caCertificates that it will not validate the cert at all. So in the destination rule we simply point it to the system certs. For more info on this issue see #25652.

The destination rule tells Istio that TLS should be used and mode simple means that this is using standard TLS instead of mutual TLS.

That was all that's needed from the Istio side.

3. Validating that TLS origination worked

Create a pod that has redis-client installed:

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-client
  namespace: redis
  labels:
    app: redis-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-client
  template:
    metadata:
      labels:
        app: redis-client
      annotations:
        sidecar.istio.io/logLevel: debug
        sidecar.istio.io/inject: "true"
    spec:
      containers:
        - image: redis
          name: redis-client
          command: [ "/bin/bash", "-c", "--" ]
          args: [ "while true; do sleep 30; done;" ]
EOF

Wait for the pods to come up....

Get a shell to the redis-client container:

kubectl exec -ti deploy/redis-client -c redis-client -- bash
redis-cli -h REPLACE_WITH_REDIS_HOST -p REPLACE_WITH_REDIS_PORT -a REPLACE_WITH_YOUR_PASSWORD
set "hello" "world"
get "hello"

The above should return "world". The redis client is using standard TCP while the Istio sidecar upgraded the connection to TLS in the background. If you need even more validation, go and create another deployment but this time set the annotation to sidecar.istio.io/inject: "false". That will prevent Istio from injecting a sidecar. After you do that you will notice that you can not connect to Redis anymore if your Redis is enforcing TLS.

You can also Check the logs of the istio-proxy to see that Istio is indeed making the TLS connections.

kubectl logs deploy/redis-client -c istio-proxy

In my case the following logs could be seen in the Istio proxy logs:

2020-12-28T21:57:43.055354Z     debug   envoy connection        [C486] write flush complete
2020-12-28T21:57:43.058429Z     debug   envoy connection        [C486] remote early close
2020-12-28T21:57:43.058465Z     debug   envoy connection        [C486] closing socket: 0
2020-12-28T21:57:43.058534Z     debug   envoy conn_handler      [C486] adding to cleanup list
2020-12-28T21:57:43.187286Z     debug   envoy upstream  DNS refresh rate reset for zipkin.istio-system, (failure) refresh rate 5000 ms
2020-12-28T21:57:43.531497Z     debug   envoy upstream  transport socket match, socket default selected for host with address 35.238.219.217:
16222
2020-12-28T21:57:43.531562Z     debug   envoy upstream  DNS refresh rate reset for redis-1425a1d9-google-bc39.aivencloud.com, refresh rate 14
000 ms
2020-12-28T21:57:45.050182Z     debug   envoy conn_handler      [C487] new connection
2020-12-28T21:57:45.050333Z     debug   envoy http      [C487] new stream
2020-12-28T21:57:45.050424Z     debug   envoy http      [C487][S11853156146631931010] request headers complete (end_stream=true):

Summary

You were able to use Istio to do TLS originiation using the sidecar instead of using the egress gateway by just using a DestinationRule and a ServiceEntry. You also validated that TLS origination is working as expected.

Deploying OpenStack on GCP

2020-11-07T22:42:00+01:00

You want private cloud inside public cloud for additional security, improved agility, lower opex and ultimate flexibility? I present you OpenStack running on Google Compute Engine (GCE). I hope you got the joke, if not, let me explain there are no benefits to running OpenStack on GCP. OpenStack on GCP is meant for testing purposes only and this doesn't make sense for a real scenario.

In this blog post, you will learn how to utilize nested KVM inside GCP to deploy an OpenStack environment. The use case of why I did this was for testing the OpenStack K8s Cloud Provider with K8s.

The guide is split up in the following sections:

Creating the GCE VM with nested KVM enabled
Deploying OpenStack using OpenStack Ansible with all in one(aio) node mode
Accessing the environment

1. Creating the GCE VMs with nested KVM enbaled

Let's create a VM called openstack-1 with 32 vCPUs. This VM will be used to run additional VMs that are spawned by OpenStack. The GCE VM itself will run the OpenStack control plane and serve as an OpenStack compute node. Run the following commands:

gcloud compute disks create ubuntu2004disk \
  --image-project ubuntu-os-cloud --image-family ubuntu-2004-lts \
  --zone us-central1-a

gcloud compute images create ubuntu-2004-nested \
  --source-disk ubuntu2004disk --source-disk-zone us-central1-a \
  --licenses "https://www.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx"

gcloud compute instances create openstack-1 --zone us-central1-a \
              --image ubuntu-2004-nested \
              --boot-disk-size 600G \
              --boot-disk-type pd-ssd \
              --can-ip-forward \
              --network default \
              --tags http-server,https-server,novnc,openstack-apis \
              --min-cpu-platform "Intel Haswell" \
              --machine-type n1-standard-32

Now verify that nested KVM is enabled:

gcloud compute ssh openstack-1 --zone us-central1-a
sudo -i
apt-get update && apt-get install qemu-kvm -y
kvm-ok

The output of kvm-ok should show the following:

kvm-ok
# Output should look like below
INFO: /dev/kvm exists
KVM acceleration can be used

2. Deploying OpenStack

Now let's deploy OpenStack using OpenStack Ansible with all in one(aio) mode. Ensure you're still SSHed into the openstack-1 VM, if not run:

gcloud compute ssh openstack-1 --zone us-central1-a
sudo -i

Start a screen or tmux session because deploying OpenStack can take 30 min to 2 hours. Run the following command:

screen

Clone openstack-ansible repo to openstack-1:

git clone https://opendev.org/openstack/openstack-ansible \
    /opt/openstack-ansible
cd /opt/openstack-ansible
git checkout stable/ussuri

Install Ansible on the VM and install all the required Ansible roles:

scripts/bootstrap-ansible.sh

Bootstrap the AIO configuration for openstack ansible

export SCENARIO='aio_lxc_barbican_octavia'
scripts/bootstrap-aio.sh

Create the LXC containers that run the different OpenStack components and install OpenStack:

openstack-ansible playbooks/setup-hosts.yml \
    playbooks/setup-infrastructure.yml \
    playbooks/setup-openstack.yml

Now OpenStack should have been succesfully deployed on the openstack-1 VM.

3. Accessing the environment

The environment is currently only exposed on the internal IP address. The floating IP range is also only routable within the openstack-1 VM. So let's setup a tunnel to the openstack-1 VM to be able to access the newly deployed environment. One way to create a tunnel is to use sshuttle.

On your local machine (laptop, desktop etc), run the following commands to setup the tunnel with sshuttle:

PUBLIC_IP=$(gcloud compute instances describe openstack-1 --zone us-central1-a \
         --format='get(networkInterfaces[0].accessConfigs[0].natIP)')
# note that you may need to add your public SSH key in GCP
sshuttle -r sam@$PUBLIC_IP 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16

Now you should be able to access the web UI on the private VPC IP address of your VM. Get the private address of your openstack-1 VM with the following command:

gcloud compute instances describe openstack-1 --zone us-central1-a \
         --format='get(networkInterfaces[0].networkIP)'

In your browser go to https://$PRIVATE_IP and you should be able to see the Horizon UI.

Creating L2 connectivity between GCE VMs in GCP using VXLAN

2020-09-13T22:42:00+02:00

Cloud providers often prevent you from using L2 protocols such as ARP. These protocols however are heavily used in existing software such as keepalived. This can make it hard for to move certain workloads to the cloud. This blog post demonstrates a method for creating L2 connectivity between Virtual Machines (VMs) running in GCP. The method used relies on VXLAN to create an L2 mesh between all the VMs.

In this blog post, we'll be creating the 2 VMs, named vm-1 and vm-2. The VMs will be launched on the default VPC network. Each of the VMs will have an additional vxlan0 interface, this interface we'll be using the 10.200.0.0/24 subnet.

1. Create the VMs

In this section you will create 2 Ubuntu 20.04 VMs

Let's start by creating vm-1

gcloud compute instances create vm-1 \
          --image-family=ubuntu-2004-lts --image-project=ubuntu-os-cloud \
          --zone=us-central1-a \
          --boot-disk-size 20G \
          --boot-disk-type pd-ssd \
          --can-ip-forward \
          --network default \
          --machine-type n1-standard-2

Repeat the same command creating vm-2 this time:

gcloud compute instances create vm-2 \
          --image-family=ubuntu-2004-lts --image-project=ubuntu-os-cloud \
          --zone=us-central1-a \
          --boot-disk-size 20G \
          --boot-disk-type pd-ssd \
          --can-ip-forward \
          --network default \
          --machine-type n1-standard-2

Verify that SSH to both VMs is available and up. You might need o be patient.

gcloud compute ssh root@vm-1 --zone us-central1-a --command "echo 'SSH to vm-1 succeeded'"
gcloud compute ssh root@vm-2 --zone us-central1-a --command "echo 'SSH to vm-2 succeeded'"

2. Setup VXLAN mesh between the VMs

In this section, you will be creating the VXLAN mesh between vm-1 and vm-2 that you just created.

Create bash variables that will be used for setting up the VXLAN mesh

VM1_VPC_IP=$(gcloud compute instances describe vm-1 \
               --format='get(networkInterfaces[0].networkIP)')
VM2_VPC_IP=$(gcloud compute instances describe vm-2 \
               --format='get(networkInterfaces[0].networkIP)')
echo $VM1_VPC_IP
echo $VM2_VPC_IP

Create the VXLAN device and mesh on vm-1

gcloud compute ssh root@vm-1 --zone us-central1-a  << EOF
set -x
ip link add vxlan0 type vxlan id 42 dev ens4 dstport 0
bridge fdb append to 00:00:00:00:00:00 dst $VM2_VPC_IP dev vxlan0
ip addr add 10.200.0.2/24 dev vxlan0
ip link set up dev vxlan0
EOF

Create the VXLAN device and mesh on vm-2

gcloud compute ssh root@vm-2 --zone us-central1-a  << EOF
set -x
ip link add vxlan0 type vxlan id 42 dev ens4 dstport 0
bridge fdb append to 00:00:00:00:00:00 dst $VM1_VPC_IP dev vxlan0
ip addr add 10.200.0.3/24 dev vxlan0
ip link set up dev vxlan0
EOF

Start a tcpdump on vm-1

gcloud compute ssh root@vm-1 --zone us-central1-a
tcpdump -i vxlan0 -n

In another session ping vm-2 from vm-1 and take a look at tcpdump output. Notice the arp.
```
gcloud compute ssh root@vm-1 --zone us-central1-a
ping 10.200.0.3
```

Summary

You have setup a VXLAN mesh between 2 VMs and can now easily repeat this to more VMs. If you want to have a mesh between more VMs than for each additional VM you would need to run bridge fdp append.

This blog post wouldn't have been possible without Mikal's blog on Setting up VXLAN on Google Compute Engine.

How to enter network namespaces of other containers from a pod in K8s?

2019-12-30T10:02:00+01:00

You might be in a situation where you need to troubleshoot the networking stack on a container where you don't have the tools necessary. Or you might need to figure out which veth belongs to a container. For both these scenarios you will need to be able to get into the network namespace of another container. This blog post describes how to get into the network namespace of another container by running a privileged container on the same K8s node.

At a high-level the following steps are needed:

Deploy a privileged container
Find out the PID of the target container
Use nsenter to enter namespace of target container and relate veth

1. Deploy a privileged container

First, we'll need to deploy a container that uses host networking and has privileges to enter namespaces. Note that this container has full access to node host networking stack and all other containers, potential security concern.

The image that we'll be using is samos123/docker-toolbox, which can be found on GitHub:samos123/docker-toolbox.

Create the file debug-pod.yaml with the following contents:

apiVersion: v1
kind: Pod
metadata:
  name: debug-pod
  labels:
    app: debug
spec:
  hostNetwork: true
  hostPID: true
  containers:
  - name: debug-pod
    image: samos123/docker-toolbox:latest
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 30; done;" ]
    volumeMounts:
      - name: dockersock
        mountPath: "/var/run/docker.sock"
    securityContext:
      privileged: true
      capabilities:
        add: ["NET_ADMIN"]
  volumes:
  - name: dockersock
    hostPath:
      path: /var/run/docker.sock

Create the pod by running:

kubectl apply -f debug-pod.yaml

Verify you can access docker of the host:

docker ps

2. Find the PID of the target container

The target container is the container of which you want to find the namespace and corresponding interface. You can use docker ps --filter name=nginx to list all containers that have the nginx in their name.

Now use the container ID to get the PID of the container:

pid=$(docker inspect --format '{{.State.Pid}}' $containerID)

3. Enter the container network namespace

nsenter can be used to enter the namespace using the PID:

nsenter -t $pid -n ip a

You will notice that there is an eth0@ifX interface inside the container network namespace. The X tells you the interface index on the host network. This index can then be used to figure out which veth belongs to the container.

Run the following commands to find the veth interface:

ifindex=$(nsenter -t $pid -n ip link | sed -n -e 's/.*eth0@if\([0-9]*\):.*/\1/p')
veth=$(ip -o link | grep ^$ifindex | sed -n -e 's/.*\(veth[[:alnum:]]*@if[[:digit:]]*\).*/\1/p')
echo $veth

I've created a script in the following repo: samos123/docker-veth

Custom GCP Cloud Shell image with Terraform and Helm

2019-06-10T13:01:00+02:00

Inpatient people who just want the end-result, please go to: GitHub: GCP Cloud Shell image with Terraform and Helm

Cloud Shell is one of the convenient features of Google Cloud providing you with a secure CLI directly from the browser. The default image contains almost all the tools you could wish for, but in some cases you might need more. In this blog post, you will learn how to create a custom Docker image for Google Cloud Shell that includes the Helm client and Terraform.

At a high-level you have to do 2 things:

Create and publish a Docker image
Configure your custom image to be used in Cloud Shell

1. Create and Publish custom Cloud Shell Docker image

In this section we're going to create new Docker image that's based on the default Cloud Shell image and then publish created image to Google Cloud Container Registry.

Let's start by creating a new repo and setting the project ID where the Docker image should be published:
```
mkdir gcp-cloud-shell-custom-image
cd gcp-cloud-shell-custom-image
GCP_PROJECT_ID=your-project-ID
```

Now with your file editor of choice create a file named Dockerfile with the following content:

FROM gcr.io/cloudshell-images/cloudshell:latest

ENV TERRAFORM_VERSION="0.11.10" \
    HELM_VERSION="v2.14.0"

WORKDIR /tmp

RUN curl https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip > terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
    unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /bin && \
    wget -q https://storage.googleapis.com/kubernetes-helm/helm-${HELM_VERSION}-linux-amd64.tar.gz -O - | tar -xzO linux-amd64/helm > /usr/local/bin/helm && \
    chmod +x /usr/local/bin/helm && \
    rm -f terraform_${TERRAFORM_VERSION}_linux_amd64.zip

Build the Docker image:
docker build -t gcr.io/$GCP_PROJECT_ID/cloud-shell-image .
Push the Docker image to Google Cloud Container Registry:
docker push gcr.io/$GCP_PROJECT_ID/cloud-shell-image:latest
Note: You will need to configure Docker to authenticate with gcr by following the steps here.

2. Configure Cloud Shell Image to use the published image

Go to Cloud Shell Environment settings
Click on Edit
Click on "Select image from project"
In the Image URL field enter: gcr.io/$GCP_PROJECT_ID/cloud-shell-image:latest
Click "Save"

Now open a new Cloud Shell session and you should see that the new custom image is used.

OpenStack Salt MK22 Vagrant-based lab

2016-11-08T13:15:00+01:00

The blog by Sebastian inspired me to try out OpenStack Salt in combination with the new MK22 Reclass model. Instead of using the TCPcloud provided labs I'm using my own laptop beast (ThinkPad W530 with 32GB of memory). For that reason I created a Vagrantfile for the mk22-lab-basic environment to create VMs on my laptop. The Vagrantfile creates the nessecary interfaces and internal networks to mimick the model. At the bottom of this post you can find the Vagrantfile. I've also created a static branch based on the upstream mk-lab-salt-model that I've verified to be working with the instructions provided below. The instructions originally come from Ales Komarek. They have been slightly modified to get them working.

Reclass model used in this blog: gh:samos123/mk-lab-salt-model branch:sam-vagrant-blog-post. The following commits have been created specifically for this post:

fcea636 Remove Horizon contrail integration because of missing pkgs
60af84f Del Mirantis Horizon them because pkg unavailable
cc527df Remove non-used node definitions
f76eab0 Disable monitoring due to resources
e55d0fe Remove ens4 nic dhcp config
5d71986 Change repo to samos123 b/sam-vagrant-blog-post
6222ff5 Add Vagrant file for mk22-lab-basic

The Vagrantfile defines and creates the following VMs:

cfg01: 4GB RAM, used as Salt Master
ctl01, ctl02, ctl03: 4GB RAM, used for Controllers and Contrail
cmp01: 2GB
prx01: 2GB

Each VM that gets created will get assigned the following networks automatically: - eth0/enp0s3: 10.0.2.0/24, NAT interface for internet - eth1/enp0s8: 172.16.10.0/24, host-only network for all OpenStack, Contrail and Data Network - eth2/enp0s9: 192.168.10.0/24, host-only network for Salt to configure hosts

By following below instructions you can bring up an OpenStack cloud (I found it often still requires debugging and tweaking to make it work):

# Manually setup the salt-master
vagrant up cfg01
vagrant ssh cfg01
echo "deb [arch=amd64] http://apt.tcpcloud.eu/nightly/ xenial main security extra tcp tcp-salt" > /etc/apt/sources.list
wget -O - http://apt.tcpcloud.eu/public.gpg | apt-key add -
apt-get update
apt-get upgrade
apt-get install -y salt-master reclass
apt-get install -y salt-formula-linux salt-formula-reclass salt-formula-salt \
                   salt-formula-openssh salt-formula-ntp salt-formula-git \
                   salt-formula-graphite salt-formula-collectd salt-formula-sensu salt-formula-heka
apt-get install -y salt-formula-sphinx salt-formula-horizon salt-formula-nginx \
                   salt-formula-memcached salt-formula-python salt-formula-supervisor
cat << 'EOF' >> /etc/salt/master.d/master.conf
file_roots:
  base:
  - /usr/share/salt-formulas/env
pillar_opts: False
open_mode: True
reclass: &reclass
  storage_type: yaml_fs
  inventory_base_uri: /srv/salt/reclass
ext_pillar:
  - reclass: *reclass
master_tops:
  reclass: *reclass
EOF

mkdir /etc/reclass
cat << 'EOF' >> /etc/reclass/reclass-config.yml
storage_type: yaml_fs
pretty_print: True
output: yaml
inventory_base_uri: /srv/salt/reclass
EOF

git clone https://github.com/samos123/mk-lab-salt-model /srv/salt/reclass -b sam-vagrant-blog-post
mkdir -p /srv/salt/reclass/classes/service
for i in /usr/share/salt-formulas/reclass/service/*; do ln -s $i /srv/salt/reclass/classes/service/; done

apt-get install -y salt-minion
cat << "EOF" >> /etc/salt/minion.d/minion.conf
id: cfg01.mk22-lab-basic.local
master: localhost
EOF

service salt-master restart
service salt-minion restart

# confirm cfg01 is accepted
salt-key 
# check reclass model is fine
reclass-salt --top
reclass-salt --pillar cfg01.mk22-lab-basic.local

salt 'cfg01*' state.sls reclass.storage
salt '*' saltutil.refresh_pillar

salt "cfg01*" state.sls git,linux,ntp
salt-call state.sls salt
salt "cfg01*" state.sls openssh,reclass,sphinx,nginx


# shows there are no nodes added to salt yet
salt-key
# exit back to Host running  vagrant
exit

# now bring up remaining nodes ctl[1-3], cmp01, prx01
vagrant up

vagrant ssh cfg01
# We need m2crypto to be installed else some states will fail
salt '*' pkg.install python-m2crypto
salt '*' state.sls ntp,linux,salt.minion,openssh
salt "ctl*" state.show_top
salt "ctl*" --state-output=mixed -b 1 state.sls keepalived
salt 'ctl01*' pillar.get keepalived:cluster:instance:VIP:address
salt 'ctl01*' cmd.run "ip a"
salt "ctl*" --state-output=mixed -b 1 state.sls memcached,rabbitmq

salt 'ctl*' state.sls glusterfs.server.service
salt 'ctl*' -b 1 state.sls glusterfs.server.setup
# gluterfs.server.setup fails because of issue with formula with older salt minons
# pr is submitted https://github.com/tcpcloud/salt-formula-glusterfs/pull/8
# if you still hit this issue, manually start the volume and rerun state again
salt "ctl01*" cmd.run "gluster volume start keystone-keys; gluster volume start glance"
salt 'ctl*' -b 1 state.sls glusterfs.server.setup

# verify that gluster is working
salt "ctl01*" cmd.run "gluster peer status; gluster volume status"

# install mysql
salt -C 'I@galera:master' state.sls galera
salt -C 'I@galera:slave' state.sls galera
salt -C 'I@galera:master' mysql.status




salt -C 'I@haproxy:proxy' state.sls haproxy
salt -C 'I@haproxy:proxy' service.status haproxy


# For some reason salt's keystone module doesn't work unless you restart the minion
salt 'ctl*' state.sls keystone -b 1
salt -C 'I@keystone:client' state.sls keystone.client
salt "ctl01*" cmd.run ". /root/keystonerc; keystone service-list"

salt 'ctl*' -b 1 state.sls glance
salt 'ctl*' state.sls glusterfs.client
salt 'ctl*' cmd.run 'df -h'
# need to rerun keystone for glance to work
salt 'ctl*' state.sls keystone
salt "ctl*" cmd.run ". /root/keystonerc; glance image-list"

salt -C 'I@nova:controller' state.sls nova -b 1
salt -C 'I@keystone:server' cmd.run ". /root/keystonerc; nova service-list"

# Cinder fails to install some packages
# workaround is to manually install linux-image-extra-virtual package
salt 'ctl*' pkg.install linux-image-extra-virtual
salt -C 'I@cinder:controller' state.sls cinder -b 1
salt -C 'I@keystone:server' cmd.run ". /root/keystonerc; cinder list"

# Neutron
salt -C 'I@neutron:server' state.sls neutron -b 1

# heat 2 states will fail but that's fine as it was created previously. 
# Should still be fixed though, this is the error seen:
# State 'keystone.role_present' was not found in SLS 'heat.server'
#              Reason: 'keystone' __virtual__ returned False
# also if heat-api isn't running make sure the following patch is merged
# https://review.openstack.org/#/c/394599/
salt -C 'I@heat:server' state.sls heat -b 1
salt -C 'I@keystone:server' cmd.run ". /root/keystonerc; heat resource-type-list"



# Several issues with horizon and dashboard
# 1. The package openstack-dashboard-contrail-panels is missing so I've disabled
#    Contrail integration by changing the reclass model. See commit #fcea6366ea50
# 2. The package openstack-dashboard-mirantis-theme is missing so I've disabled
#    mirantis them customization in the reclass model. See commit #60af84f237bab
# 3. The salt-formula-horizon fails when app.plugin is not set in pillar. For
#    this I've submitted a patch in the formula https://review.openstack.org/#/c/394618/
# You may have to apply the patch in point 3 manually ^^ horizon fails for me still.
salt -C 'I@horizon:server' state.sls horizon
salt -C 'I@nginx:server' state.sls nginx

# Install contrail
salt "ctl*" state.sls opencontrail -b 1
salt -C 'I@keystone:server' cmd.run ". /root/keystonerc; neutron net-list; nova net-list"

salt 'ctl01*' cmd.run "/usr/share/contrail-utils/provision_control.py --api_server_ip 172.16.10.254 --api_server_port 8082 --host_name ctl01 --host_ip 172.16.10.101 --router_asn 64512 --admin_password workshop --admin_user admin --admin_tenant_name admin --oper add"
salt 'ctl02*' cmd.run "/usr/share/contrail-utils/provision_control.py --api_server_ip 172.16.10.254 --api_server_port 8082 --host_name ctl02 --host_ip 172.16.10.102 --router_asn 64512 --admin_password workshop --admin_user admin --admin_tenant_name admin --oper add"
salt 'ctl03*' cmd.run "/usr/share/contrail-utils/provision_control.py --api_server_ip 172.16.10.254 --api_server_port 8082 --host_name ctl03 --host_ip 172.16.10.103 --router_asn 64512 --admin_password workshop --admin_user admin --admin_tenant_name admin --oper add"


# Configure compute nodes
salt 'cmp*' state.highstate
salt 'cmp*' state.highstate
salt 'ctl01*' cmd.run "/usr/share/contrail-utils/provision_vrouter.py --host_name cmp01 --host_ip 172.16.10.105 --api_server_ip 172.16.10.254 --oper add --admin_user admin --admin_password workshop --admin_tenant_name admin"

# Temporary workaround for mos packages
salt 'cmp*' cmd.run "find /etc -xdev -type f | xargs egrep -H 'unix_sock_group.*libvirt' | sed -e 's/:.*//' | xargs sed -i -e 's/unix_sock_group = \"libvirtd\"/unix_sock_group = \"libvirt\"/'"

# There is an issue with nova.conf because rabbit_host isn't set in default section
# I've submitted a patch here: https://review.openstack.org/#/c/395161/

# Try rebooting and see if it still works
salt 'cmp*' system.reboot
# For some reason I got an older libvirt from tcprepo instead of mos
# below installs the correct package. Bug filed here:
# https://github.com/tcpcloud/salt-formula-libvirt/issues/1
vim /usr/share/salt-formulas/env/nova/files/mitaka/nova-compute.conf.Debian
# change virt_type from kvm to qemu, bug has been filed about this here:
# https://bugs.launchpad.net/openstack-salt/+bug/1640314
salt 'cmp*' cmd.run "apt-get update"
salt 'cmp*' pkg.install libvirt-bin
salt 'cmp*' cmd.run "ip a"
salt 'cmp*' cmd.run "contrail-status"

# Exit and SSH into ctl01 to test the cloud
exit
vagrant ssh ctl01
sudo -i
source keystonerc
wget http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-i386-disk.img
glance image-create --name cirros --visibility public --disk-format qcow2 --container-format bare --progress < /root/cirros-0.3.4-i386-disk.img
neutron net-create net1
neutron subnet-create --name subnet1 net1 192.168.32.0/24
keystone tenant-list
netuuid=$(neutron net-list | grep net1 | awk '{ print $2 }')
nova flavor-create --is-public true m1.macro auto 128 5 1
nova boot --nic net-id=$netuuid --image cirros --flavor m1.macro testvm1

Below is the Vagrantfile that is used in this blog post:

# -*- mode: ruby -*-
# vi: set ft=ruby :

$script = <<SCRIPT
echo "deb [arch=amd64] http://apt.tcpcloud.eu/nightly/ trusty main security extra tcp tcp-salt" > /etc/apt/sources.list
wget -O - http://apt.tcpcloud.eu/public.gpg | apt-key add -
apt-get update
apt-get install -y salt-minion
cat << "EOF" >> /etc/salt/minion.d/minion.conf
id: {{x}}.mk22-lab-basic.local
master: 192.168.10.100
EOF
SCRIPT

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/trusty64"

  config.vm.provider "virtualbox" do |vb|
     # Display the VirtualBox GUI when booting the machine
     vb.gui = true

     # Customize the amount of memory on the VM:
     vb.memory = "4096"
  end

  config.vm.define "cfg01" do |cfg01|
    cfg01.vm.box = "ubuntu/xenial64"
    cfg01.vm.network "private_network", ip: "172.16.10.100"
    cfg01.vm.network "private_network", ip: "192.168.10.100"
  end

  config.vm.define "ctl01" do |ctl01|
    ctl01.vm.box = config.vm.box
    ctl01.vm.provision "shell", inline: $script.sub(/{{x}}/, "ctl01")
    ctl01.vm.network "private_network", ip: "172.16.10.101"
    ctl01.vm.network "private_network", ip: "192.168.10.101"
  end

  config.vm.define "ctl02" do |ctl02|
    ctl02.vm.box = config.vm.box
    ctl02.vm.provision "shell", inline: $script.sub(/{{x}}/, "ctl02")
    ctl02.vm.network "private_network", ip: "172.16.10.102"
    ctl02.vm.network "private_network", ip: "192.168.10.102"
  end

  config.vm.define "ctl03" do |ctl03|
    ctl03.vm.box = config.vm.box
    ctl03.vm.provision "shell", inline: $script.sub(/{{x}}/, "ctl03")
    ctl03.vm.network "private_network", ip: "172.16.10.103"
    ctl03.vm.network "private_network", ip: "192.168.10.103"
  end

  config.vm.define "cmp01" do |cmp01|
    cmp01.vm.box = config.vm.box
    cmp01.vm.provision "shell", inline: $script.sub(/{{x}}/, "cmp01")
    cmp01.vm.network "private_network", ip: "172.16.10.105"
    cmp01.vm.network "private_network", ip: "192.168.10.105"

    cmp01.vm.provider "virtualbox" do |vb|
       vb.memory = "2048"
    end
  end

  config.vm.define "prx01" do |prx01|
    prx01.vm.box = config.vm.box
    prx01.vm.provision "shell", inline: $script.sub(/{{x}}/, "prx01")
    prx01.vm.network "private_network", ip: "172.16.10.107"
    prx01.vm.network "private_network", ip: "192.168.10.107"

    prx01.vm.provider "virtualbox" do |vb|
       vb.memory = "2048"
    end
  end



end

OpenWRT hourly speedtest maximum bandwidth delivered by ISP

2016-10-11T14:25:00+02:00

Work in progress, place holder, these are the commands required to run the python code that I wrote

opkg install python3-light opkg install python3-email

opkg install python3-openssl opkg install python3-codecs

opkg install ca-certificates

RobotRouter: Bandwidth of Xfinity Internet on 2016-10-11

2016-10-11T00:00:00+02:00

RobotRouter ran speedtest-cli.sh every hour on 2016-10-11 and reported the following aggregate statistics:

Metric	Value
Average Download Speed	88.224583333333325
Average Upload Speed	4.7241666666666663
Min Download Speed	83.819999999999988
Min Upload Speed	4.7
Max Download Speed	99.25
Max Upload Speed	4.75

The raw results can be viewed in the following table below:

2016-10-11 00:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.6 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.944 
    10pct: 9.944 
   Median: 19.508 
      Avg: 21.397 
    90pct: 28.676 
      Max: 57.788

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.317 
    10pct: 9.317 
   Median: 12.462 
      Avg: 12.536 
    90pct: 15.276 
      Max: 16.306
2016-10-11 01:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.62 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.379 
    10pct: 8.379 
   Median: 15.625 
      Avg: 20.844 
    90pct: 32.698 
      Max: 42.024

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.889 
    10pct: 9.889 
   Median: 11.624 
      Avg: 12.468 
    90pct: 15.230 
      Max: 17.286
2016-10-11 02:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  89.38 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.340 
    10pct: 10.340 
   Median: 17.139 
      Avg: 20.383 
    90pct: 29.556 
      Max: 44.748

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.186 
    10pct: 10.186 
   Median: 12.979 
      Avg: 13.438 
    90pct: 17.714 
      Max: 19.982
2016-10-11 03:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.05 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.210 
    10pct: 9.210 
   Median: 18.024 
      Avg: 21.082 
    90pct: 25.847 
      Max: 61.809

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.713 
    10pct: 9.713 
   Median: 13.464 
      Avg: 13.650 
    90pct: 16.105 
      Max: 18.901
2016-10-11 04:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.52 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.179 
    10pct: 10.179 
   Median: 16.973 
      Avg: 20.301 
    90pct: 28.778 
      Max: 41.681

   Upload:  4.7 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.553 
    10pct: 8.553 
   Median: 11.858 
      Avg: 13.694 
    90pct: 18.083 
      Max: 21.372
2016-10-11 05:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.05 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 6.649 
    10pct: 6.649 
   Median: 13.334 
      Avg: 23.934 
    90pct: 47.649 
      Max: 93.557

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.170 
    10pct: 8.170 
   Median: 13.847 
      Avg: 13.958 
    90pct: 16.823 
      Max: 17.033
2016-10-11 06:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.56 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.579 
    10pct: 8.579 
   Median: 14.903 
      Avg: 15.885 
    90pct: 20.722 
      Max: 22.695

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.188 
    10pct: 9.188 
   Median: 12.612 
      Avg: 13.097 
    90pct: 15.554 
      Max: 15.986
2016-10-11 07:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  99.25 Mbps
  Latency: (in msec, 21 pings, 0.00% packet loss)
      Min: 7.658 
    10pct: 9.659 
   Median: 18.485 
      Avg: 18.813 
    90pct: 28.158 
      Max: 46.663

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.554 
    10pct: 8.554 
   Median: 11.557 
      Avg: 11.714 
    90pct: 13.015 
      Max: 15.533
2016-10-11 08:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.67 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.912 
    10pct: 9.912 
   Median: 22.196 
      Avg: 21.745 
    90pct: 28.807 
      Max: 41.498

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.220 
    10pct: 7.220 
   Median: 12.316 
      Avg: 13.181 
    90pct: 15.841 
      Max: 20.503
2016-10-11 09:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  85.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.699 
    10pct: 7.699 
   Median: 15.997 
      Avg: 15.959 
    90pct: 20.044 
      Max: 24.891

   Upload:  4.74 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.598 
    10pct: 9.598 
   Median: 11.525 
      Avg: 12.791 
    90pct: 15.349 
      Max: 19.659
2016-10-11 10:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  89.09 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.980 
    10pct: 8.980 
   Median: 24.391 
      Avg: 28.886 
    90pct: 45.814 
      Max: 75.282

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.819 
    10pct: 9.819 
   Median: 12.681 
      Avg: 13.168 
    90pct: 15.534 
      Max: 18.306
2016-10-11 11:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  86.84 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 5.560 
    10pct: 5.560 
   Median: 17.604 
      Avg: 23.313 
    90pct: 32.759 
      Max: 56.817

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.298 
    10pct: 10.298 
   Median: 11.495 
      Avg: 12.801 
    90pct: 16.287 
      Max: 18.082
2016-10-11 12:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  83.82 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 5.658 
    10pct: 5.658 
   Median: 19.367 
      Avg: 28.371 
    90pct: 58.709 
      Max: 100.927

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.558 
    10pct: 9.558 
   Median: 11.560 
      Avg: 11.980 
    90pct: 13.779 
      Max: 14.816
2016-10-11 13:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.34 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.102 
    10pct: 10.102 
   Median: 17.824 
      Avg: 20.152 
    90pct: 28.898 
      Max: 37.195

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.158 
    10pct: 10.158 
   Median: 11.415 
      Avg: 12.109 
    90pct: 12.857 
      Max: 17.334
2016-10-11 14:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.1 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.930 
    10pct: 9.930 
   Median: 16.275 
      Avg: 23.133 
    90pct: 26.828 
      Max: 113.114

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.626 
    10pct: 9.626 
   Median: 12.182 
      Avg: 13.084 
    90pct: 16.257 
      Max: 17.575
2016-10-11 15:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  89.28 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.374 
    10pct: 9.374 
   Median: 18.034 
      Avg: 20.736 
    90pct: 27.630 
      Max: 40.726

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.093 
    10pct: 8.093 
   Median: 10.717 
      Avg: 11.378 
    90pct: 14.287 
      Max: 16.715
2016-10-11 16:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  91.18 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.583 
    10pct: 7.583 
   Median: 16.894 
      Avg: 18.125 
    90pct: 25.612 
      Max: 35.495

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.301 
    10pct: 10.301 
   Median: 11.662 
      Avg: 12.258 
    90pct: 14.059 
      Max: 18.188
2016-10-11 17:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  85.2 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 5.771 
    10pct: 5.771 
   Median: 11.857 
      Avg: 15.284 
    90pct: 21.350 
      Max: 39.012

   Upload:  4.74 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.892 
    10pct: 8.892 
   Median: 11.706 
      Avg: 12.080 
    90pct: 15.812 
      Max: 18.215
2016-10-11 18:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.94 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.181 
    10pct: 10.181 
   Median: 17.520 
      Avg: 21.894 
    90pct: 30.047 
      Max: 57.163

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.171 
    10pct: 9.171 
   Median: 10.960 
      Avg: 12.551 
    90pct: 15.309 
      Max: 19.540
2016-10-11 19:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.57 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.509 
    10pct: 9.509 
   Median: 20.004 
      Avg: 24.191 
    90pct: 38.143 
      Max: 59.402

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.158 
    10pct: 10.158 
   Median: 12.008 
      Avg: 12.316 
    90pct: 14.076 
      Max: 17.757
2016-10-11 20:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.42 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.689 
    10pct: 9.689 
   Median: 13.848 
      Avg: 25.634 
    90pct: 47.869 
      Max: 116.937

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.984 
    10pct: 8.984 
   Median: 10.750 
      Avg: 11.655 
    90pct: 12.675 
      Max: 18.469
2016-10-11 21:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.41 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.552 
    10pct: 7.552 
   Median: 13.995 
      Avg: 16.615 
    90pct: 23.189 
      Max: 35.197

   Upload:  4.75 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.424 
    10pct: 8.424 
   Median: 10.926 
      Avg: 11.976 
    90pct: 13.996 
      Max: 17.912
2016-10-11 22:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  86.65 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 5.760 
    10pct: 5.760 
   Median: 17.818 
      Avg: 21.742 
    90pct: 34.219 
      Max: 55.317

   Upload:  4.75 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.624 
    10pct: 8.624 
   Median: 10.683 
      Avg: 11.098 
    90pct: 13.299 
      Max: 16.701
2016-10-11 23:01:01 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.13 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.244 
    10pct: 7.244 
   Median: 18.388 
      Avg: 20.252 
    90pct: 28.165 
      Max: 41.986

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.256 
    10pct: 9.256 
   Median: 11.751 
      Avg: 13.059 
    90pct: 15.724 
      Max: 18.900

RobotRouter: Bandwidth of Xfinity Internet on 2016-10-10

2016-10-10T00:00:00+02:00

RobotRouter ran speedtest-cli.sh every hour on 2016-10-10 and reported the following aggregate statistics:

Metric	Value
Average Download Speed	86.536249999999976
Average Upload Speed	4.7145833333333337
Min Download Speed	83.98
Min Upload Speed	4.6399999999999994
Max Download Speed	88.930000000000012
Max Upload Speed	4.74

The raw results can be viewed in the following table below:

2016-10-10 00:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  84.38 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.592 
    10pct: 8.592 
   Median: 16.155 
      Avg: 17.060 
    90pct: 22.004 
      Max: 32.343

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.012 
    10pct: 9.012 
   Median: 12.156 
      Avg: 12.075 
    90pct: 13.822 
      Max: 14.568
2016-10-10 01:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  84.15 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.881 
    10pct: 9.881 
   Median: 24.598 
      Avg: 27.606 
    90pct: 46.843 
      Max: 52.200

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.631 
    10pct: 8.631 
   Median: 11.846 
      Avg: 12.294 
    90pct: 14.689 
      Max: 17.366
2016-10-10 02:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.41 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.800 
    10pct: 9.800 
   Median: 20.548 
      Avg: 22.459 
    90pct: 32.004 
      Max: 51.176

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.966 
    10pct: 9.966 
   Median: 14.108 
      Avg: 14.288 
    90pct: 17.885 
      Max: 20.878
2016-10-10 03:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  84.77 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.490 
    10pct: 9.490 
   Median: 13.660 
      Avg: 17.407 
    90pct: 23.976 
      Max: 41.358

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.699 
    10pct: 9.699 
   Median: 12.250 
      Avg: 12.905 
    90pct: 16.354 
      Max: 19.739
2016-10-10 04:01:01 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  86.17 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.497 
    10pct: 8.497 
   Median: 23.064 
      Avg: 22.267 
    90pct: 30.758 
      Max: 47.491

   Upload:  4.64 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.451 
    10pct: 10.451 
   Median: 11.983 
      Avg: 12.793 
    90pct: 15.078 
      Max: 17.862
2016-10-10 05:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  86.75 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.286 
    10pct: 10.286 
   Median: 19.159 
      Avg: 24.127 
    90pct: 39.139 
      Max: 59.273

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.009 
    10pct: 8.009 
   Median: 11.721 
      Avg: 12.310 
    90pct: 15.551 
      Max: 17.843
2016-10-10 06:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  84.21 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.528 
    10pct: 7.528 
   Median: 16.635 
      Avg: 17.806 
    90pct: 25.786 
      Max: 34.214

   Upload:  4.68 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.515 
    10pct: 10.515 
   Median: 12.479 
      Avg: 13.394 
    90pct: 17.989 
      Max: 18.828
2016-10-10 07:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.93 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.832 
    10pct: 9.832 
   Median: 18.035 
      Avg: 24.029 
    90pct: 29.260 
      Max: 87.664

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.141 
    10pct: 9.141 
   Median: 13.778 
      Avg: 14.121 
    90pct: 16.667 
      Max: 22.430
2016-10-10 08:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.78 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 6.111 
    10pct: 6.111 
   Median: 16.784 
      Avg: 21.448 
    90pct: 31.126 
      Max: 72.453

   Upload:  4.74 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.832 
    10pct: 10.832 
   Median: 11.963 
      Avg: 13.027 
    90pct: 16.389 
      Max: 18.822
2016-10-10 09:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.04 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.897 
    10pct: 9.897 
   Median: 26.201 
      Avg: 26.773 
    90pct: 38.167 
      Max: 67.904

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.500 
    10pct: 9.500 
   Median: 11.446 
      Avg: 12.553 
    90pct: 15.505 
      Max: 18.307
2016-10-10 10:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.444 
    10pct: 9.444 
   Median: 18.635 
      Avg: 24.352 
    90pct: 34.748 
      Max: 64.951

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.197 
    10pct: 10.197 
   Median: 12.499 
      Avg: 13.029 
    90pct: 16.173 
      Max: 19.439
2016-10-10 11:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.64 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.650 
    10pct: 9.650 
   Median: 14.615 
      Avg: 20.982 
    90pct: 28.338 
      Max: 56.729

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.013 
    10pct: 9.013 
   Median: 11.558 
      Avg: 13.471 
    90pct: 18.493 
      Max: 22.434
2016-10-10 12:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.04 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.027 
    10pct: 8.027 
   Median: 16.892 
      Avg: 21.681 
    90pct: 32.636 
      Max: 49.771

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.578 
    10pct: 9.578 
   Median: 10.697 
      Avg: 11.457 
    90pct: 13.614 
      Max: 14.607
2016-10-10 13:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  86.13 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.514 
    10pct: 9.514 
   Median: 18.904 
      Avg: 23.391 
    90pct: 35.112 
      Max: 73.095

   Upload:  4.74 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.740 
    10pct: 7.740 
   Median: 12.058 
      Avg: 12.914 
    90pct: 17.292 
      Max: 21.992
2016-10-10 14:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  84.55 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 5.566 
    10pct: 5.566 
   Median: 14.318 
      Avg: 16.774 
    90pct: 26.801 
      Max: 34.554

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.490 
    10pct: 9.490 
   Median: 12.420 
      Avg: 12.848 
    90pct: 15.366 
      Max: 15.959
2016-10-10 15:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  83.98 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.181 
    10pct: 8.181 
   Median: 17.622 
      Avg: 21.277 
    90pct: 32.401 
      Max: 41.607

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 6.612 
    10pct: 6.612 
   Median: 12.093 
      Avg: 12.920 
    90pct: 16.120 
      Max: 17.202
2016-10-10 16:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  86.31 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 6.248 
    10pct: 6.248 
   Median: 16.636 
      Avg: 21.002 
    90pct: 27.150 
      Max: 78.679

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.059 
    10pct: 8.059 
   Median: 10.309 
      Avg: 11.691 
    90pct: 13.408 
      Max: 18.340
2016-10-10 17:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.36 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.810 
    10pct: 7.810 
   Median: 13.422 
      Avg: 19.757 
    90pct: 34.476 
      Max: 54.694

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.158 
    10pct: 9.158 
   Median: 12.413 
      Avg: 12.739 
    90pct: 15.909 
      Max: 17.396
2016-10-10 18:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.5 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.098 
    10pct: 7.098 
   Median: 14.293 
      Avg: 19.370 
    90pct: 39.550 
      Max: 44.307

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.734 
    10pct: 9.734 
   Median: 11.633 
      Avg: 12.209 
    90pct: 14.658 
      Max: 17.025
2016-10-10 19:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  84.81 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.321 
    10pct: 8.321 
   Median: 16.679 
      Avg: 21.088 
    90pct: 34.552 
      Max: 66.781

   Upload:  4.7 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.899 
    10pct: 9.899 
   Median: 13.824 
      Avg: 14.907 
    90pct: 18.219 
      Max: 21.934
2016-10-10 20:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.62 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.497 
    10pct: 8.497 
   Median: 17.981 
      Avg: 20.725 
    90pct: 32.072 
      Max: 37.254

   Upload:  4.74 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.513 
    10pct: 9.513 
   Median: 11.057 
      Avg: 12.548 
    90pct: 16.286 
      Max: 18.091
2016-10-10 21:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  86.58 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.024 
    10pct: 10.024 
   Median: 17.890 
      Avg: 17.710 
    90pct: 21.433 
      Max: 29.037

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.211 
    10pct: 8.211 
   Median: 11.145 
      Avg: 12.104 
    90pct: 14.745 
      Max: 15.379
2016-10-10 22:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.85 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 5.757 
    10pct: 5.757 
   Median: 22.448 
      Avg: 23.266 
    90pct: 32.600 
      Max: 53.382

   Upload:  4.66 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.859 
    10pct: 9.859 
   Median: 11.503 
      Avg: 12.456 
    90pct: 15.635 
      Max: 17.717
2016-10-10 23:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  84.2 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 6.327 
    10pct: 6.327 
   Median: 12.873 
      Avg: 17.094 
    90pct: 30.386 
      Max: 41.330

   Upload:  4.7 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.742 
    10pct: 8.742 
   Median: 10.882 
      Avg: 12.339 
    90pct: 15.722 
      Max: 20.442

RobotRouter: Bandwidth of Xfinity Internet on 2016-10-09

2016-10-09T00:00:00+02:00

RobotRouter ran speedtest-cli.sh every hour on 2016-10-09 and reported the following aggregate statistics:

Metric	Value
Average Download Speed	87.67375
Average Upload Speed	4.6916666666666663
Min Download Speed	82.67
Min Upload Speed	4.08
Max Download Speed	91.069999999999988
Max Upload Speed	4.75

The raw results can be viewed in the following table below:

2016-10-09 02:41:19 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  91.07 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.196 
    10pct: 10.196 
   Median: 22.423 
      Avg: 28.514 
    90pct: 40.608 
      Max: 76.436

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.020 
    10pct: 9.020 
   Median: 11.358 
      Avg: 11.964 
    90pct: 14.352 
      Max: 16.192
2016-10-09 03:00:17 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  90.23 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 6.590 
    10pct: 6.590 
   Median: 22.767 
      Avg: 23.459 
    90pct: 34.385 
      Max: 50.705

   Upload:  4.69 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.689 
    10pct: 8.689 
   Median: 13.676 
      Avg: 13.996 
    90pct: 17.502 
      Max: 20.042
2016-10-09 03:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.12 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.265 
    10pct: 8.265 
   Median: 17.479 
      Avg: 29.676 
    90pct: 47.984 
      Max: 104.228

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.119 
    10pct: 10.119 
   Median: 12.782 
      Avg: 13.579 
    90pct: 17.296 
      Max: 20.574
2016-10-09 04:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  84.13 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 6.816 
    10pct: 6.816 
   Median: 14.320 
      Avg: 17.758 
    90pct: 26.301 
      Max: 55.582

   Upload:  4.08 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.080 
    10pct: 8.080 
   Median: 12.340 
      Avg: 13.416 
    90pct: 15.812 
      Max: 18.103
2016-10-09 05:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  89.51 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.289 
    10pct: 8.289 
   Median: 18.084 
      Avg: 20.958 
    90pct: 32.680 
      Max: 47.615

   Upload:  4.7 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.674 
    10pct: 8.674 
   Median: 12.021 
      Avg: 12.325 
    90pct: 14.932 
      Max: 17.014
2016-10-09 06:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.47 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.974 
    10pct: 8.974 
   Median: 14.492 
      Avg: 22.652 
    90pct: 40.515 
      Max: 52.104

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.973 
    10pct: 10.973 
   Median: 26.732 
      Avg: 28.330 
    90pct: 37.735 
      Max: 45.379
2016-10-09 06:55:36 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  89.52 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.118 
    10pct: 9.118 
   Median: 16.883 
      Avg: 18.904 
    90pct: 28.476 
      Max: 34.614

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.321 
    10pct: 10.321 
   Median: 12.182 
      Avg: 13.589 
    90pct: 17.335 
      Max: 20.287
2016-10-09 07:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  90.51 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 6.073 
    10pct: 6.073 
   Median: 20.867 
      Avg: 22.278 
    90pct: 28.363 
      Max: 64.259

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.146 
    10pct: 8.146 
   Median: 12.176 
      Avg: 13.054 
    90pct: 17.020 
      Max: 18.884
2016-10-09 08:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  86.82 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 28.831 
    10pct: 28.831 
   Median: 35.451 
      Avg: 38.669 
    90pct: 50.976 
      Max: 56.251

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 26.762 
    10pct: 26.762 
   Median: 29.892 
      Avg: 30.543 
    90pct: 33.213 
      Max: 34.193
2016-10-09 09:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  83.83 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.390 
    10pct: 8.390 
   Median: 13.926 
      Avg: 22.751 
    90pct: 29.910 
      Max: 109.438

   Upload:  4.66 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 6.156 
    10pct: 6.156 
   Median: 10.355 
      Avg: 11.377 
    90pct: 14.874 
      Max: 15.835
2016-10-09 10:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.89 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.436 
    10pct: 7.436 
   Median: 16.372 
      Avg: 16.692 
    90pct: 25.860 
      Max: 33.312

   Upload:  4.68 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.506 
    10pct: 7.506 
   Median: 9.852 
      Avg: 11.091 
    90pct: 14.088 
      Max: 15.684
2016-10-09 11:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.79 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 5.095 
    10pct: 5.095 
   Median: 17.120 
      Avg: 22.101 
    90pct: 26.382 
      Max: 89.227

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.993 
    10pct: 9.993 
   Median: 11.937 
      Avg: 12.857 
    90pct: 15.492 
      Max: 18.931
2016-10-09 12:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  82.67 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.863 
    10pct: 9.863 
   Median: 18.746 
      Avg: 21.070 
    90pct: 30.214 
      Max: 41.320

   Upload:  4.75 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.059 
    10pct: 9.059 
   Median: 14.300 
      Avg: 13.887 
    90pct: 15.543 
      Max: 19.662
2016-10-09 13:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.24 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.119 
    10pct: 8.119 
   Median: 17.618 
      Avg: 22.722 
    90pct: 35.554 
      Max: 64.411

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.160 
    10pct: 10.160 
   Median: 11.762 
      Avg: 12.811 
    90pct: 14.954 
      Max: 22.923
2016-10-09 14:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  89.25 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.666 
    10pct: 8.666 
   Median: 15.211 
      Avg: 19.900 
    90pct: 30.355 
      Max: 45.918

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.420 
    10pct: 9.420 
   Median: 11.250 
      Avg: 12.008 
    90pct: 15.063 
      Max: 16.174
2016-10-09 15:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.568 
    10pct: 7.568 
   Median: 17.917 
      Avg: 21.540 
    90pct: 39.829 
      Max: 53.199

   Upload:  4.74 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.766 
    10pct: 8.766 
   Median: 13.951 
      Avg: 14.179 
    90pct: 19.112 
      Max: 20.085
2016-10-09 16:01:01 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  86.28 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.990 
    10pct: 8.990 
   Median: 13.841 
      Avg: 18.488 
    90pct: 28.590 
      Max: 45.190

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.600 
    10pct: 8.600 
   Median: 11.548 
      Avg: 12.314 
    90pct: 15.494 
      Max: 18.660
2016-10-09 17:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  89.67 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.538 
    10pct: 9.538 
   Median: 21.272 
      Avg: 27.928 
    90pct: 37.575 
      Max: 91.053

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 7.056 
    10pct: 7.056 
   Median: 12.456 
      Avg: 12.964 
    90pct: 16.382 
      Max: 17.997
2016-10-09 18:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  84.6 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.790 
    10pct: 8.790 
   Median: 19.935 
      Avg: 19.595 
    90pct: 27.036 
      Max: 44.024

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.841 
    10pct: 8.841 
   Median: 12.600 
      Avg: 13.497 
    90pct: 18.981 
      Max: 20.482
2016-10-09 19:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  90.14 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.297 
    10pct: 10.297 
   Median: 17.134 
      Avg: 19.934 
    90pct: 28.404 
      Max: 32.049

   Upload:  4.72 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 10.309 
    10pct: 10.309 
   Median: 12.371 
      Avg: 13.626 
    90pct: 17.847 
      Max: 20.235
2016-10-09 20:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  89.13 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 6.647 
    10pct: 6.647 
   Median: 13.560 
      Avg: 18.331 
    90pct: 28.026 
      Max: 57.374

   Upload:  4.74 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.605 
    10pct: 8.605 
   Median: 11.487 
      Avg: 12.560 
    90pct: 17.811 
      Max: 19.245
2016-10-09 21:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  84.93 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.759 
    10pct: 9.759 
   Median: 16.037 
      Avg: 16.425 
    90pct: 21.850 
      Max: 32.179

   Upload:  4.73 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.908 
    10pct: 8.908 
   Median: 11.261 
      Avg: 13.203 
    90pct: 16.999 
      Max: 23.070
2016-10-09 22:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  88.27 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.703 
    10pct: 9.703 
   Median: 16.392 
      Avg: 16.819 
    90pct: 22.102 
      Max: 39.174

   Upload:  4.71 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.804 
    10pct: 9.804 
   Median: 11.788 
      Avg: 12.029 
    90pct: 12.738 
      Max: 16.417
2016-10-09 23:01:00 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (15 seconds in each direction)

 Download:  87.37 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 9.405 
    10pct: 9.405 
   Median: 17.427 
      Avg: 20.987 
    90pct: 32.463 
      Max: 47.838

   Upload:  4.74 Mbps
  Latency: (in msec, 16 pings, 0.00% packet loss)
      Min: 8.915 
    10pct: 8.915 
   Median: 11.732 
      Avg: 12.705 
    90pct: 15.063 
      Max: 16.774

Fuel: Separate Ceph Public from Management using Network Templates

2016-08-27T14:01:00+02:00

This post will demonstrate using Network Templates in Fuel to separate Ceph Public Network from Management Network. By default Fuel combines the ceph public network with the management network. The Ceph Public network is used for communicating from the compute nodes to the Ceph nodes. So in high performant storage networks we would like to be able to isolate this network from any other network such that we can guarantee performance.

Fuel Network Templates allow us separate or combine networks. The general workflow for separating the Ceph Public network is as follows:

Create a new Network Group for the new network
Write and upload a network template that defines how the newly created network should be used
Review and tune the actual deployment settings via the CLI

Demo environment

This blog post is using the following example environment:

5 x VirtualBox VMs
1 x Fuel Master 9.0
1 x Controller node
1 x Compute node
2 x Ceph-OSD
Each VM has 3 NICs: enp0s3, enp0s8, enp0s9
enp0s3 is used for Admin/PXE network
enp0s8 is used for Public/Floating network
enp0s9 is used for Management, Storage, Private and Ceph Public network

Create the new Ceph Public network group

First the newly defined Ceph Public network needs to be created. This is done by creating a new network group via Fuel. In the demo, the nic enp0s9 is used for ceph public and other networks via VLAN tagging. Although in a production environment you could easily put the new Ceph Public network on a dedicated NIC.

The following steps show to create the Ceph Public network with VLAN tag 13:

# First list the network groups to see what node-group is being used
# group-id is usually the same as environment id
fuel network-group
# Create the new ceph-public network
fuel network-group --create --node-group 6 --name ceph-public --vlan 13 --cidr 192.168.251.0/24

Write and upload network template

After having created the new network we use Network Templates functionality to assign the ceph/public network role to our newly created network. Note that the order does not matter. You could also first create and upload template and then create new Ceph Public network group.

The below template is based on the default template with a minor modification to split out the ceph/public network role from the management network.

adv_net_template:
  default: 
    nic_mapping:
      default:
        adm: enp0s3
        pub: enp0s8
        man: enp0s9.10
        priv: enp0s9.11
        stor: enp0s9.12
        cephpub: enp0s9.13
    templates_for_node_role:
        controller:
          - public
          - private
          - storage
          - ceph-public
          - common
        compute:
          - common
          - ceph-public
          - private
        cinder:
          - common
          - storage
        ceph-osd:
          - common
          - ceph-public
          - storage
    network_assignments: 
        storage:
          ep: br-storage
        ceph-public:
          ep: br-ceph-public
        private:
          ep: br-prv
        public:
          ep: br-ex
        management:
          ep: br-mgmt
        fuelweb_admin:
          ep: br-fw-admin
    network_scheme:
      storage:
        transformations:
          - action: add-br
            name: br-storage
          - action: add-port
            bridge: br-storage
            name: <% stor %>
        endpoints:
          - br-storage
        roles:
          cinder/iscsi: br-storage
          swift/replication: br-storage
          ceph/replication: br-storage
          storage: br-storage
      ceph-public:
        transformations:
          - action: add-br
            name: br-ceph-public
          - action: add-port
            bridge: br-ceph-public
            name: <% cephpub %>
        endpoints:
          - br-ceph-public
        roles:
          ceph/public: br-ceph-public
      private:
        transformations:
          - action: add-br
            name: br-prv
            provider: ovs
          - action: add-br
            name: br-aux
          - action: add-patch
            bridges:
            - br-prv
            - br-aux
            provider: ovs
            mtu: 65000
          - action: add-port
            bridge: br-aux
            name: <% priv %>
        endpoints:
          - br-prv
        roles:
          neutron/private: br-prv
      public:
        transformations:
          - action: add-br
            name: br-ex
          - action: add-br
            name: br-floating
            provider: ovs
          - action: add-patch
            bridges:
            - br-floating
            - br-ex
            provider: ovs
            mtu: 65000
          - action: add-port
            bridge: br-ex
            name: <% pub %>
        endpoints:
          - br-ex
        roles:
          public/vip: br-ex
          neutron/floating: br-floating
          ceph/radosgw: br-ex
          ex: br-ex
      common:
# In this example the common network is created for Management and Admin traffic.
        transformations:
          - action: add-br
            name: br-fw-admin
          - action: add-port
            bridge: br-fw-admin
            name: <% adm %>
          - action: add-br
            name: br-mgmt
          - action: add-port
            bridge: br-mgmt
            name: <% man %>
        endpoints:
          - br-fw-admin
          - br-mgmt
        roles:
          admin/pxe: br-fw-admin
          fw-admin: br-fw-admin
          mongo/db: br-mgmt
          management: br-mgmt
          keystone/api: br-mgmt
          neutron/api: br-mgmt
          neutron/mesh: br-mgmt
          swift/api: br-mgmt
          sahara/api: br-mgmt
          ceilometer/api: br-mgmt
          cinder/api: br-mgmt
          glance/api: br-mgmt
          heat/api: br-mgmt
          nova/api: br-mgmt
          nova/migration: br-mgmt
          murano/api: br-mgmt
          horizon: br-mgmt
          mgmt/api: br-mgmt
          mgmt/memcache: br-mgmt
          mgmt/database: br-mgmt
          mgmt/messaging: br-mgmt
          mgmt/corosync: br-mgmt
          mgmt/vip: br-mgmt
          mgmt/api: br-mgmt

Save the above file with the name network_template_6.yaml on your Fuel node. Fuel network template requires that the file name is in the following format: network_template_{{ env_id }}.yaml. If your fuel doesn't match the format you won't be able to upload.

Now upload the the file network_template_6.yaml to Fuel via:

fuel network-template --env 6 --upload --dir .

Review and tune final deployment settings

This last step is to make sure that the Network settings set by Network Templates and Network group match expectations. Download the final deployment settings for the environment using the below command:

fuel deployment --default --env 6

This will create a directory called deployment_6 containing deployment settings for the actual deployment. Open for example deployment_6/master.yaml or deployment_6/10.yaml and verify that ceph/public network role is assigned to controllers and ceph-osd nodes. Next to that make sure that ceph/public is assigned a specific IP address on each node.

Now we're ready to deploy and can trigger the deployment with:

fuel deploy-changes --env 6

If you're lucky enough to not have made mistakes and hit bugs, then you will be running in no time :) This isn't easy to use functionality, without the help of my colleague xarxes I wasn't able to get it working.

Creating a KVM VM for StackLight on Fuel 8.0 master node

2016-06-01T09:31:00+02:00

Creating a KVM VM on the Fuel master node is a nice way to better utilize server resources for small environments that were deployed already without using Fuel compact install or running Fuel as VM itself. We can run Controller, Monitoring or MongoDB as VM on the Fuel node. This blog post explains in detail how to create a KVM VM on the Fuel master node, which can be used for any role defined by Fuel. In this example we assign the StackLight monitoring role to the VM created on the Fuel master node.

Please note that I do not recommend running VMs on Fuel master but instead recommend using the compact install functionality or running Fuel itself as a VM. This guide is meant for people who already deployed Fuel on a physical server which is mostly idle but now want to add StackLight LMA without adding new server resources.

The general steps are:

Install KVM on Fuel master node (Enable CentOS repo, install kvm and libvirt packages)
Create network bridges on the Fuel master node for Admin/PXE and Management network
Create the KVM VM through libvirt using an XML template
PXE boot the KVM VM such that it gets discovered by Fuel
Assign your desired role to the VMs via Fuel and deploy

1. Install KVM and libvirt on the Fuel master

The first step is install KVM and libvirt on the Fuel master node. SSH into Fuel Master node and execute the ollowing steps:

Create the file /etc/yum.repos.d/CentOS-base.repo with the following content:

# CentOS-Base.repo
[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

Now install the required packages from CentOS repo
```
yum install libvirt qemu-kvm kvm
```

2. Create networking bridges for KVM VM

Here we create network bridges on the Fuel master node for Admin/PXE and Management network.

Prevent libvirt from starting dnsmasq. Execute the following command, delete the parts with dhcp and change forwarding to none:
```
virsh net-edit default
```
Create the bridge for Admin/PXE network. This step is not using the standard way to create the bridge, because I didn't want to change the network device name to which the IP is bound. It seems that some containers by default use eth0 and host networking. I don't recommend doing it this way, but it "works".

I created a script which doesn't change eth0, but instead renames eth0 to net0 and then creates a bridge named eth0 and assigns the original net0 ip address to eth0. I tried to use udev but couldn't get it to work, so here we go with a hacky script.

Create a file /usr/bin/create-eth0-bridge.sh with the following content:
```
#!/bin/bash

ETH0ADDR=10.20.0.2/24

# Rename eth0 to net0
ip a flush dev eth0
ip link set eth0 down
ip link set eth0 name net0
ip link set net0 up

# Create bridge eth0 with net0 as interface
brctl addbr eth0
ip link set eth0 up
brctl addif eth0 net0

# Add original eth0addr to bridge
ip a add $ETH0ADDR dev eth0
```

Create a systemd service to launch the script after network was done. Create the file /etc/systemd/system/create-eth0-bridge.service with the following content:

[Unit]
Description=Create eth0 network bridge for Stacklight VM
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/create-eth0-bridge.sh

[Install]
WantedBy=multi-user.target

Now enable the service to be run on startup by executing the command:

systemctl enable create-eth0-bridge

Create the bridge which contains other possible networks. Please note that if you used VLAN tagging then you should add the VLAN trunk port to your bridge. In this example I have a vlan trunk which contains the management network on eth2.

Create the bridge named br-eth2 and add eth2 as port to the bridge with the following commands:
```
brctl addbr br-eth2
brctl addif br-eth2 eth2
```
To make the changes persistent accross reboots create the file /etc/sysconfig/network-scripts/ifcfg-br-eth2 with the following content:
```
DEVICE=br-eth2
BOOTPROTO=none
ONBOOT=yes
TYPE=Bridge
NM_CONTROLLED=no
DELAY=0
```
and the file /etc/sysconfig/network-scripts/ifcfg-eth2 with the following content:
```
TYPE=Ethernet
BOOTPROTO=none
NAME=eth2
UUID=fe45432f-f08a-4537-9851-d53be572aa00
DEVICE=eth2
ONBOOT=yes
NM_CONTROLLED=no
BRIDGE=br-eth2
```

3. Create the KVM VM

In this part we create the KVM VM using libvirt XML to describe the VM. In the XML we specify to boot from network first, which enables us to use Fuel's PXE booting.

First create the disk for our KVM VM with the following command:

qemu-img create -f qcow2 -o preallocation=metadata /var/lib/libvirt/images/monitoring-1.img 300G

Create a file monitoring1-vm.xml with the following content:

<domain type='kvm' id='2'>
  <name>monitoring-1</name>
  <memory unit='KiB'>8388608</memory>
  <vcpu placement='static' cpuset='0-7'>8</vcpu>
  <os>
    <type arch='x86_64'>hvm</type>
    <boot dev='network'/>
    <boot dev='hd'/>
    <boot dev='cdrom'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <devices>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/monitoring-1.img'/>
      <target dev='sda' bus='scsi'/>
    </disk>
    <interface type='bridge'>
      <source bridge='eth0'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-eth2'/>
      <model type='virtio'/>
    </interface>
    <graphics type='vnc' port='-1' listen='127.0.0.1'/>
  </devices>
</domain>

Note you may need to tweak the parameters such as vcpus, memory and bridges according to your actual environment.

Launch and create the VM from the xml file:

virsh define monitoring-1.xml
virsh start monitoring-1

4. Fuel PXE booting and assigning roles

After you have done the previous steps correctly, you should see a new unallocated node pop up. You can now assign any role to this new node.

Running Computer Vision algos on Spark with OpenCV

2016-01-22T16:31:00+01:00

This post shows several computer vision steps implemented on top of Spark. OpenCV is used to extract features on top of OpenStack and Spark MLLib KMeans is used to generate our KMeans dictionary. Then we use Spark and simple vector / matrix manipulation to do coding and pooling.

Workflow implemented using OpenCV and Spark:

Uploading our dataset of images to Hadoop compatible storage as Sequencefile
Running OpenCV feature extraction(SURF, SIFT) code using the SequenceFile of images as input
Running Spark default K-means model training machine learning code on extracted features
Running Feature coding and pooling using our trained K-means model and extracted feature as input
(TODO): Running machine learning algortihm to do classification on the encoded features

Uploading image dataset as SequenceFile to Hadoop compatible storage (Swift)

In the first step we upload our Caltech256 dataset, 30k images totaling 1.2Gb, as SequenceFile to OpenStack Swift. For this I've created a simplistic command line tool to upload folders containing files to be stored as a SequenceFile with key=filename and value=raw_bytes. The tool has been tested with HDFS and OpenStack Swift.

The following commands show how to download Caltech-256 dataset consisting of JPG images. Then upload the downloaded images to OpenStack Swift in sequence file format:

# Download and compile the hadoop sequencefile upload tool
git clone https://github.com/samos123/hadoop-sequence-file-upload
cd hadoop-sequence-file-upload
mvn clean compile assembly:single
#  Download / extract calltech-256 dataset
axel http://www.vision.caltech.edu/Image_Datasets/Caltech256/256_ObjectCategories.tar
tar xf 256_ObjectCategories.tar
# Upload to Swift, this assumes /etc/hadoop/conf/core-site.xml is used to store Swift details
./upload-to-sequence-file.sh 256_ObjectCategories/ swift://spark.swift1/caltech-256.hseq

The dataset is also accessible through Tachyon as we configured it to use Swift as underFS.

Extract SIFT/SURF features using OpenCV on Spark

I've created a simple Spark application that uses OpenCV to extract SURF or SIFT features from an image.

from __future__ import print_function
import logging
import io
import sys
import os

import cv2
import numpy as np
from pyspark import SparkContext
from pyspark.sql import SQLContext, Row


def extract_opencv_features(feature_name):

    def extract_opencv_features_nested(imgfile_imgbytes):
        try:
            imgfilename, imgbytes = imgfile_imgbytes
            nparr = np.fromstring(buffer(imgbytes), np.uint8)
            img = cv2.imdecode(nparr, 0)
            if feature_name in ["surf", "SURF"]:
                extractor = cv2.SURF()
            elif feature_name in ["sift", "SIFT"]:
                extractor = cv2.SIFT()

            kp, descriptors = extractor.detectAndCompute(img, None)

            return [(imgfilename, descriptors)]
        except Exception, e:
            logging.exception(e)
            return []

    return extract_opencv_features_nested


if __name__ == "__main__":
    sc = SparkContext(appName="feature_extractor")
    sqlContext = SQLContext(sc)

    try:
        feature_name = sys.argv[1]
        image_seqfile_path = sys.argv[2]
        feature_parquet_path = sys.argv[3]
        partitions = int(sys.argv[4])
    except:
        print("Usage: spark-submit feature_extraction.py <feature_name(sift or surf)> "
              "<image_sequencefile_input_path> <feature_sequencefile_output_path> <partitions>")

    images = sc.sequenceFile(image_seqfile_path, minSplits=partitions)

    features = images.flatMap(extract_opencv_features(feature_name))
    features = features.filter(lambda x: x[1] != None)
    features = features.map(lambda x: (Row(fileName=x[0], features=x[1].tolist())))
    featuresSchema = sqlContext.createDataFrame(features)
    featuresSchema.registerTempTable("images")
    featuresSchema.write.parquet(feature_parquet_path)

Using the above Spark application we can start to extract features from our image dataset in Swift. As input we provide the sequencefile containing <fileName: String, image: Bytes>. The extracted features we write out as parquet file. The following command extracts the sift features from our dataset:

spark-submit --executor-memory 8g feature_extraction.py sift swift://spark.swift1/caltech-256.hseq swift://spark.swift1/caltech-256-sift1.parquet 100

K-Means Dictionary generation on SIFT features

We can now generate our dictionary of features through Spark's MLLib KMeans algorithm. The below application is used to train our KMeans model using the features generated in the previous step as input dataset.

from __future__ import print_function
import io
import sys
import os

import numpy as np
from pyspark import SparkContext
from pyspark.mllib.clustering import KMeans
from pyspark.sql import SQLContext, Row


if __name__ == "__main__":
    sc = SparkContext(appName="kmeans_dictionary_creation")
    sqlContext = SQLContext(sc)

    try:
        k = int(sys.argv[1])
        feature_parquet_path = sys.argv[2]
        kmeans_model_path = sys.argv[3]
    except:
        print("Usage: spark-submit kmeans.py <k:clusters> "
              "<feature_sequencefile_input_path> <kmeans_model_output>")

    features = sqlContext.read.parquet(feature_parquet_path)

    # Create same size vectors of the feature descriptors
    # flatMap returns every list item as a new row for the RDD
    # hence transforming x, 128 to x rows of 1, 128 in the RDD.
    # This is needed for KMeans.
    features = features.flatMap(lambda x: x['features']).cache()
    model = KMeans.train(features, k, maxIterations=10, initializationMode="random")

    model.save(sc, kmeans_model_path)
    print("Clusters have been saved as text file to %s" % kmeans_model_path)
    print("Final centers: " + str(model.clusterCenters))
    sc.stop()

Start the spark job with:

spark-submit --executor-memory 8g kmeans.py 1000 swift://spark.swift1/caltech-256-sift1.parquet swift://spark.swift1/caltech-256-dictionary

Feature coding and pooling with trained KMeans model

In this step we will use the KMeans dictionary that we trained in the previous step to encode each point of interest to a single cluster. This is done by assigning every row of our x * 128 matrix to a single cluster of the KMeans dictionary. The result is a 1 * k representation for each image utilizing coding and pooling. For pooling we've implemented a simple max and sum pooling method.

The following Spark application implements feature coding and pooling:

from __future__ import print_function
import functools
import io
import sys
import os

import numpy as np
from scipy.spatial import distance
from pyspark.mllib.clustering import KMeansModel
from pyspark import SparkContext
from pyspark.sql import SQLContext, Row

SUPPORTED_POOLING = ["max", "sum"]

def assign_pooling(row, clusterCenters, pooling):
    image_name = row['fileName']
    feature_matrix = np.array(row['features'])
    clusterCenters = clusterCenters.value
    model = KMeansModel(clusterCenters)
    bow = np.zeros(len(clusterCenters))

    for x in feature_matrix:
        k = model.predict(x)
        dist = distance.euclidean(clusterCenters[k], x)
        if pooling == "max":
            bow[k] = max(bow[k], dist)
        elif pooling == "sum":
            bow[k] = bow[k] + dist

    return Row(fileName=image_name, bow=bow.tolist())


if __name__ == "__main__":
    sc = SparkContext(appName="kmeans_assign")
    sqlContext = SQLContext(sc)

    try:
        feature_parquet_path = sys.argv[1]
        kmeans_model_path = sys.argv[2]
        bow_parquet_path = sys.argv[3]
        pooling = sys.argv[4]

    except:
        print("Usage: spark-submit feature_coding_pooling.py "
              "<feature_sequencefile_path> <kmeans_model> "
              "<bow_sequencefile_path> <pooling_method:max>")

    if pooling not in SUPPORTED_POOLING:
        raise ValueError("Pooling method %s is not supported. Supported poolings methods: %s" % (pooling, SUPPORTED_POOLING))

    features = sqlContext.read.parquet(feature_parquet_path)
    model = KMeansModel.load(sc, kmeans_model_path)
    clusterCenters = model.clusterCenters
    clusterCenters = sc.broadcast(clusterCenters)

    features_bow = features.map(functools.partial(assign_pooling,
        clusterCenters=clusterCenters, pooling=pooling))
    featuresSchema = sqlContext.createDataFrame(features_bow)
    featuresSchema.registerTempTable("images")
    featuresSchema.write.parquet(bow_parquet_path)
    sc.stop()

Execute the spark applicaiton with:

spark-submit --executor-memory 8g feature_coding_pooling.py swift://spark.swift1/caltech-256-sift1.parquet swift://spark.swift1/caltech-256-dictionary swift://spark.swift1/caltech-256-bow.parquet sum

Elastic Data Processing on OpenStack with Spark, Tachyon and Swift

2015-11-09T16:31:00+01:00

This post will describe how to configure, build and deploy Spark with Tachyon and Swift as storage. This architecture is meant to be more suitable for running Big Data workloads on top of the cloud such as OpenStack.

Using Swift as storage layer for Spark gives us the ability to utilize the cloud paradigm with Big Data. We can now on demand spin up n-amount of VMs, run a spark job with input data from Swift, then when finished store the result back in Swift and finally when the job is finished, destroy the VMs. This kind of elastic data processing, using Tachyon for fast localized in-memory storage gives you high performance and also elasticity of the cloud.

The general architecture looks like this:

Assumptions:

You have a working Mesos cluster that can run Spark jobs
We have two nodes: mesos-master-1 (192.168.111.54) and mesos-slave-1 (192.168.111.57). We will run tachyon master on the mesos-master-1 node and tachyon worker on mesos-slave-1
OpenStack Keystone and Swift are available at 10.10.10.10
Tachyon 0.8.0 is deployed and configured to use Swift as underfs

The general workflow of this post is:

Installation of Tachyon 0.8.0 with Swift as underFS on the mesos-slaves
Building Spark 1.6.0-SNAPSHOT from latest master branch with Tachyon 0.8.0 and tachyon-underfs-swift as dependencies.
Running an example Spark job which uses Tachyon for input and output.

Tachyon 0.8.0 installation with Swift as underFS

Download and extract Tachyon to the mesos-master and mesos-slave

wget http://tachyon-project.org/downloads/files/0.8.0/tachyon-0.8.0-hadoop2.6-bin.tar.gz
scp tachyon-0.8.0-hadoop2.6-bin.tar.gz mesos-master-1:/srv/
scp tachyon-0.8.0-hadoop2.6-bin.tar.gz mesos-slave-1:/srv/
ssh mesos-master-1 "cd /srv/ && tar xzf tachyon-0.8.0-hadoop2.6-bin.tar.gz"
ssh mesos-slave-1 "cd /srv/ && tar xzf tachyon-0.8.0-hadoop2.6-bin.tar.gz"

On both mesos-master-1 and mesos-slave-1, copy /srv/tachyon-0.8.0/conf/tachyon-env.sh.swift to /srv/tachyon-0.8.0/conf/tachyon-env.sh

mv /srv/tachyon-0.8.0/conf/tachyon-env.sh{.swift,}

Make minor modifications to /srv/tachyon-0.8.0/conf/tachyon-env.sh on both mesos-master-1 and mesos-slave-1:

export TACHYON_MASTER_ADDRESS=192.168.111.54
export TACHYON_UNDERFS_ADDRESS=swift://spark.swift1

The TACHYON_MASTER_ADDRESS is set to the use the external IP of the mesos-master node. The variable TACHYON_UNDERFS_ADDRESS is set to swift://spark.swift1 where spark is the name of the Swift container and swift1 is an arbitary name to specify an Swift connection object defined in tachyon/conf/core-site.xml.

Now in core-site.xml we need to define how to connect to Swift. This is done by changing settings for swift1 provider in core-site.xml. I'm using Keystone to authenticate with Swift and also to retreive the correct endpoint.

I've changed core-site.xml according to my Swift environment and copied it to tachyon/conf/core-site.xml. The following changes were made:

<property>
 <name>fs.swift.service.swift1.location-aware</name>
 <value>false</value>
</property>
<property>
  <name>fs.swift.service.swift1.auth.url</name>
  <value>http://10.10.10.10:5000/v2.0/tokens</value>
</property>
<property>
  <name>fs.swift.service.swift1.http.port</name>
  <value>8080</value>
</property>
<property>
  <name>fs.swift.service.swift1.region</name>
  <value>RegionOne</value>
</property>
<property>
  <name>fs.swift.service.swift1.public</name>
  <value>true</value>
</property>
<property>
  <name>fs.swift.service.swift1.auth.endpoint.prefix</name>
  <value>endpoints</value>
</property>
<property>
  <name>fs.swift.service.swift1.tenant</name>
  <value>spark</value>
</property>
<property>
  <name>fs.swift.service.swift1.password</name>
  <value>test123</value>
</property>
<property>
  <name>fs.swift.service.swift1.username</name>
  <value>spark</value>
</property>

After all config changes are done we can start launching the Tachyon master process on the mesos-master-1 node.

/srv/tachyon-0.8.0/bin/tachyon-start.sh master -f

After successful start of the tachyon master, also start the tachyon worker on mesos-slave-1:

/srv/tachyon-0.8.0/bin/tachyon-start.sh worker SudoMount

You should now have Tachyon running and you can access it via http://192.168.111.54:19999

Building Spark with Tachyon 0.8.0 and Swift

Checkout latest master branch of spark:

git clone https://github.com/apache/spark.git

Apply the following patches to core/pom.xml and make_distribution.sh to use tachyon 0.8.0 and to include hdfs as underfs dependency. Make sure to exclude mockito-all else the build will fail with

diff --git a/core/pom.xml b/core/pom.xml
index 570a25c..98285a0 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -262,7 +262,7 @@
     <dependency>
       <groupId>org.tachyonproject</groupId>
       <artifactId>tachyon-client</artifactId>
-      <version>0.8.1</version>
+      <version>0.8.0</version>
       <exclusions>
         <exclusion>
           <groupId>org.apache.hadoop</groupId>
@@ -287,6 +287,17 @@
       </exclusions>
     </dependency>
     <dependency>
+      <groupId>org.tachyonproject</groupId>
+      <artifactId>tachyon-underfs-swift</artifactId>
+      <version>0.8.0</version>
+      <exclusions>
+        <exclusion>
+          <groupId>org.mockito</groupId>
+          <artifactId>mockito-all</artifactId>
+        </exclusion>
+      </exclusions>
+    </dependency>
+    <dependency>
       <groupId>org.seleniumhq.selenium</groupId>
       <artifactId>selenium-java</artifactId>
       <exclusions>

diff --git a/make-distribution.sh b/make-distribution.sh
index e1c2afd..f676678 100755
--- a/make-distribution.sh
+++ b/make-distribution.sh
@@ -33,7 +33,7 @@ SPARK_HOME="$(cd "`dirname "$0"`"; pwd)"
 DISTDIR="$SPARK_HOME/dist"

 SPARK_TACHYON=false
-TACHYON_VERSION="0.8.1"
+TACHYON_VERSION="0.8.0"
 TACHYON_TGZ="tachyon-${TACHYON_VERSION}-bin.tar.gz"
 TACHYON_URL="http://tachyon-project.org/downloads/files/${TACHYON_VERSION}/${TACHYON_TGZ}"

Build Spark with Tachyon and Swift as underfs:

./make-distribution.sh --name spark-master-tachyon-0.8.0 --tgz --with-tachyon -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests

Add core-site.xml which we used in Tachyon to the Spark tarball.

tar xzf spark-1.6.0-SNAPSHOT-bin-spark-master-tachyon-0.8.0.tgz
cp /srv/tachyon-0.8.0/conf/core-site.xml spark-1.6.0*/conf/core-site.xml
tar czf spark-1.6.0.tar.gz spark-1.6.0-SNAPSHOT-bin-spark-master-tachyon-0.8.0

For some reason Spark also needs core-site.xml with the Swift connection settings even though we already configured this in Tachyon.

Running a Spark job with Tachyon as input backed by Swift

Assuming that in our Swift container we have an object called output.log we can now create a Spark shell session and use

spark-shell
val textFile = sc.textFile("tachyon://mesos-master-1:19998/output.log")
textFile.count()

I've also verified that I could successfuly use sequenceFiles and Spark pickleFiles on Tachyon with Swift by running the Python spark jobs of my other project: Spark Computer vision

Setup IPv6 in Tsinghua on Linux(ArchLinux))

2015-10-08T12:42:00+02:00

Tsinghua university uses ISATAP to provide IPv6 connectivity to students. In Linux you can use the isatapd program to create an ISATAP tunnel based on an IPv4 device.

You can use the isatapd command as follows to establish an ISATAPD tunnel:

isatapd --router isatap.tsinghua.edu.cn

After connecting check that you have received an global IPv6 address with:

ip -6 a

As a last check you need to check your default ipv6 routes. In some cases 2 default routes may get added:

ip -6 route

If there are 2 default routes make sure to delete the route which goes out via your physical device e.g(eth0 or enp2u5). In my case I had to delete one of the default routes as follows:

ip -6 route del default via fe80::5efe:a66f:1501

Archlinux

Archlinux provides an isatapd package, which you can install via pacman -S isatapd. The package comes with a systemd service at /usr/lib/systemd/system/isatapd@.service. You can start the service with: systemctl start isatapd@isatap.tsinghua.edu.cn

Enable SRIOV on OpenStack Juno

2015-07-15T16:31:00+02:00

Update: This blog post is deprecated by the official documentation. Please take a look at the official Networking Guide: Using SRIOV functionality. Patch set that added the official documentation: changeset.

Terms used
Term	Definition
SRIOV	Single Root IO Virtualization. SRIOV allows a PCIe device to appear to be multiple separate physical PCIe devices. What's SRIOV? By Scott Lowe
PF & VF	PF is a physical function. VF is a virtual function. A PF is the physical PCI-e network card. A VF is a virtual network card with it's own PCI address.

This post describes how to enable Neutron SRIOV functionality on Mirantis OpenStack Juno deployed with Fuel 6.1 using Ubuntu 14.04 as host OS. This tutorial assumes you already have deployed OpenStack with OVS + vlan mode for networking. Next to that for the private interface we have selected eth3, which serves both as PF for the VFs and also as private vlan trunk for non SRIOV instances. In our environment we're using the "Intel Corporation 82599" network card which is assigned to eth3. We will create 7 VFs per PF.

I have created an ansible playbook to automate the whole process. You can view this Ansible playbook here: Fuel Ansible SRIOV Automated methods are recommended over manual configuring!

The following steps have to be taken to enable SRIOV manually for reference:

Enabling Virtual Functions in the host Operating System
Whitelist which PCI devices should be used for SRIOV in nova.conf on computes
Configuring Neutron server for SRIOV
Enabling the PCIDeviceScheduler in nova-scheduler
Creating your SRIOV instance

1. Enable Virtual Functions in Host OS

First we need to make sure SRIOV is enabled in BIOS, check for VT-d and make sure it's enabled. After enabling VT-d we also need enable IOMMU on Linux by adding intel_iommu=on to kernel parameters.

vim /etc/default/grub
change "GRUB_CMDLINE_LINUX_DEFAULT="nomdmonddf nomdmonisw"
to "GRUB_CMDLINE_LINUX_DEFAULT="nomdmonddf nomdmonisw intel_iommu=on"

If you added new parameters you need to run:

update-grub
reboot

On each compute node we need to create the VFs via the PCI SYS interface.

echo '7' > /sys/class/net/eth3/device/sriov_numvfs

Now verify that the VFs have been created correctly

lspci | grep Ethernet

We just created the VFs for this session. If we would reboot the node these changes would get lost. So we also add a line to /etc/rc.local to apply these settings on a reboot. Note: The suggested way of making these settings persistent seems to be through sysfs.conf, but for some reason it did not work for me hence the rc.local workaround.

echo "echo '7' > /sys/class/net/eth3/device/sriov_numvfs" >> /etc/rc.local

Verify by rebooting your node that the settings persist.

2. Whitelist PCI devices nova.conf on computes

In /etc/nova/nova.conf add the line pci_passthrough_whitelist={ "devname": "eth3", "physical_network": "physnet2"}, this tells nova that all VFs belonging to eth3 are allowed to be passed through to VMs. Restart nova compute to let the changes have effect service restart nova-compute

3. Configuring Neutron server

Add sriovnicswitch to neutron ml2 conf

sed -i "s/mechanism_drivers =openvswitch/mechanism_drivers =openvswitch,sriovnicswitch/g" /etc/neutron/plugins/ml2/ml2_conf.ini

Find out the vendor_id and product_id of your VFs. Please note this should be the VF not the PF

lspci -nn | grep -i ethernet
87:00.0 Ethernet controller [0200]: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection [8086:10f8] (rev 01)
87:10.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
87:10.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)

In our case the vendor_id is 8086 and the product_id is 10ed. We need to tell Neutron the vendor_id and product_id of the VFs that are supported.

sed -i "s/# supported_pci_vendor_devs.*=.*/supported_pci_vendor_devs = 8086:10ed/g" /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

Neutron also has support for running a special sriov-agent which is able set the admin state. I didn't see any need for setting admin state, so to reduce complexity, we've disabled sriov-agent.

sed -i "s/# agent_required =.*/agent_required=false/g" /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

We now need to add these new config files as parameter to the neutron-server daemon.

vim /etc/init/neutron-server.conf
change "--config-file /etc/neutron/neutron.conf"
to "--config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini"

Restart neutron-server on every controller

service neutron-server restart

4. Enabling the PCIDeviceScheduler in nova-scheduler

On every controller node running nova-scheduler we need to add PCIDeviceScheduler to the filters.

vim /etc/nova/nova.conf
change "scheduler_default_filters=DifferentHostFilter,RetryFilter,AvailabilityZoneFilter,RamFilter,CoreFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter"

to "scheduler_default_filters=DifferentHostFilter,RetryFilter,AvailabilityZoneFilter,RamFilter,CoreFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter"

also add the following line: "scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter"
below the line: "scheduler_available_filters=nova.scheduler.filters.all_filters"

# Restart nova-scheduler
service nova-scheduler restart

5. Launching an Instance with SRIOV ports

After configuring all components we can start trying to launch an instance through the CLI or API. Horizon currently does not support creating SRIOV instances.

Get the id of the neutron network where you want the SR-IOV port to be created.

net_id=`neutron net-show net04 | grep "\ id\ " | awk '{ print $4 }'`

Create the SR-IOV port. We specify vnic_type direct, which means that this a SR-IOV port.

port_id=`neutron port-create $net_id --name sriov_port --binding:vnic_type direct | grep "\ id\ " | awk '{ print $4 }'`

Create the VM specifying that as 1st nic we want to use the previously created sr-iov port.

nova boot --flavor m1.large --image ubuntu_14.04 --nic port-id=$port_id --key-name sam test-sriov

References which deserve credit: * Red Hat Documentation on using SRIOV * OpenStack Wiki: SR-IOV-Passthrough-For-Networking

Create Linux bridge without losing existing connection

2015-06-19T14:25:00+02:00

The dillemma: You're accessing your server via SSH through interface eth1.102 with the ip 10.20.0.2/24 and gateway 10.20.0.1. Now you want to add eth1 to a linux bridge so you can hookin VMs on a vlan trunk.

Solution: Create a script which does everything in 1 go and call this script from screen. Make sure the script is flawless else you still loose your connection.

This is the script I create-bridge.sh used:

# First remove the old vlan device
ip link set dev eth1.102 down
ip link delete eth1.102


# Create the bridge 
brctl addbr br-eth1
brctl addif br-eth1 eth1

ip link set eth1 up
ip link set br-eth1 up

# Add the vlan tagging ontop of the bridge
vconfig add br-eth1 102

ip link set br-eth1.102 up
ip a add 166.111./24 dev br-eth1.102
ip route add default via 166.111.80.10

Now you should first open a screen session(maybe optional) and then call ./create-bridge.sh, which create your bridge and setup the previous interface ontop of the bridge without losing your SSH connection :)

Hadoop on demand with Openstack, Cloudinit and Salt

2014-11-27T13:12:00+01:00

This post will describe the whole process of providing hadoop on demand through Openstack, Cloudinit and Saltstack. We will use Openstack to provision servers on demand, cloudconfig to install salt-master and salt-minion and saltstack to deploy hadoop namenode and datanodes, all fully automated. This makes us able to deploy a 5-100+ node (still need to test 20+) Hadoop HDFS cluster within 5 minutes.

Deploy salt-master with Cloudconfig

We will install and configure salt-master to auto_accept key files from the minions. Please note that this is a security risk.

Create the following master-cloudconfig file:

#cloud-config
apt_mirror: http://mirror.bjtu.edu.cn/ubuntu/

apt_sources:
 - source: "ppa:saltstack/salt"

packages:
 - python-software-properties
 - salt-master
 - git

runcmd:
 - bash /tmp/bootstrap-master.sh

write_files:
 - content: |
        #!/bin/bash
        sed -i 's/^#auto_accept.*$/auto_accept: True/g' /etc/salt/master
        sudo service salt-master restart
   path: /tmp/bootstrap-master.sh
   permissions: "0755"

Now launch openstack instance with: nova boot --image ubuntu_14.04 --user-data master-cloudconfig --flavor m1.medium --key-name sam salt_master. The salt-master that we launched in our case has as ip 192.168.111.66. You can ssh to this machine via ssh ubuntu@192.168.111.66

Configuring the salt master to deploy a hadoop cluster

In order to be able to deploy hadoop on the salt-master we are going to manually configure the salt-master with the official hadoop formula. We could also automate this via cloudconfig, but for sake of demonstration I'm going to list the manual steps here.

Configuring salt to use hadoop-formula and it's dependencies

We are going to use the following formula: Hadoop formula, which has as dependencies the hostsfile-formula and the sun-java-formula. The hostsfile formula is to make all nodes accessible by their hostname/fqdn and the sun-java-formula lets you automatically install java from oracle.

ssh ubuntu@192.168.111.66 # SSH into your salt-master
sudo mkdir /srv/salt
sudo mkdir /srv/salt/formulas
sudo "cd /srv/salt/formulas && 
      git clone https://github.com/saltstack-formulas/hadoop-formula
      git clone https://github.com/saltstack-formulas/hostsfile-formula &&
      git clone https://github.com/saltstack-formulas/sun-java-formula"

cat << EOF > /etc/salt/master.d/file_roots.conf
file_roots:
  base:
    - /srv/salt
    - /srv/salt/formulas/hadoop-formula
    - /srv/salt/formulas/hostsfile-formula
    - /srv/salt/formulas/sun-java-formula
EOF

cat << EOF > /etc/salt/master.d/pillar_roots.conf
pillar_roots:
  base:
    - /srv/pillar
EOF

sudo service salt-master restart

Creating SSH keypairs for hadoop-formula

cd /srv/salt/formulas/hadoop-formula/hadoop/files && ./generate-keypairs.sh

Configuring the top.sls state file to include hadoop and deps

On the salt-master create the following file /srv/salt/top.sls:

base:
  'G@roles:hadoop_slave or G@roles:hadoop_master':
    - match: compound
    - hostsfile
    - hostsfile.hostname
    - sun-java
    - sun-java.env
    - hadoop
    - hadoop.hdfs

By using pillar we can also change the parameters of hadoop. You can check all available parameters that are change-able here: hadoop-pillar.example.

Create the following top.sls pillar file /srv/pillar/top.sls:

base:
  'G@roles:hadoop_slave or G@roles:hadoop_master':
     - hadoop

and the following hadoop specific pillar file /srv/pillar/hadoop.sls:

hadoop:
  version: hdp-2.2.0 # ['apache-1.2.1', 'apache-2.2.0', 'hdp-1.3.0', 'hdp-2.2.0', 'cdh-4.5.0', 'cdh-4.5.0-mr1']
  users:
    hadoop: 6000
    hdfs: 6001

Deploy salt-minions as hadoop nodes

Now that we have configured our salt-master we can start deploying the salt-minions which will either server as hadoop-master(namenode) or as hadoop-slave(datanode).

Launching hadoop-master(namenode) nodes

First create our cloudconfig file used by openstack hadoopmaster-cloudconfig:

#cloud-config
apt_mirror: "http://mirror.bjtu.edu.cn/ubuntu/"
apt_sources: 
  - source: "ppa:saltstack/salt"
packages: 
  - python-software-properties
  - salt-minion

runcmd: 
 - bash /tmp/bootstrap-minion.sh

write_files: 
  - content: |
         roles:
            - hadoop_master
    path: /etc/salt/grains
  - content: |
        #!/bin/bash
        sed -i 's/^#master.*$/master: 192.168.111.66/g' /etc/salt/minion
        sudo service salt-minion restart
    path: /tmp/bootstrap-minion.sh
    permissions: "0755"
  - content: |
        mine_functions:
            network.interfaces: []
            network.ip_addrs: []
            grains.items: []
    path: /etc/salt/minion.d/mine_functions.conf

Then create a new instance using openstack: nova boot --image ubuntu_14.04 --user-data hadoopmaster-cloudconfig --flavor m1.medium --key-name sam hadoop-master-1

Deploying hadoop-slave(datanode) nodes

Create a hadoopslave-cloudconfig:

#cloud-config
apt_mirror: "http://mirror.bjtu.edu.cn/ubuntu/"
apt_sources: 
  - source: "ppa:saltstack/salt"
packages: 
  - python-software-properties
  - salt-minion

runcmd: 
 - bash /tmp/bootstrap-minion.sh

write_files: 
  - content: |
         roles:
            - hadoop_slave
    path: /etc/salt/grains
  - content: |
        #!/bin/bash
        sed -i 's/^#master.*$/master: 192.168.111.66/g' /etc/salt/minion
        sudo service salt-minion restart
    path: /tmp/bootstrap-minion.sh
    permissions: "0755"
  - content: |
        mine_functions:
            network.interfaces: []
            network.ip_addrs: []
            grains.items: []
    path: /etc/salt/minion.d/mine_functions.conf

Now launch 10 hadoop-slave nodes nova boot --num-instances 10 --image ubuntu_14.04 --user-data hadoopslave-cloudconfig --flavor m1.medium --key-name sam hadoop-slave

Let saltstack deploy hadoop

First make sure that the salt-master detected all our hadoop-master and hadoop-slave nodes. Execute the following comand: salt '*' managed.up it should show your 11 nodes.

If all nodes are up and detected you can start deployment of hadoop with the following command:

salt '*' state.highstate

Accessing the hadoop HDFS cluster

After hadoop has been successfully deployed you can go to http://{hadoop_masterip}:50070 and check that 10 live nodes are active.

Now you can login to any of the nodes and save files to hdfs like this:

sudo -u hdfs -i
hadoop fs -mkdir -p /user/hdfs"
hadoop fs -put /tmp/bootstrap-minion.sh /user/hdfs/test.sh

Hope this tutorial was useful!

Notes: You can also configure salt-minion directly through cloudconfig instead.

Neutron Multiple allocation pools single subnet (Solve Fragmented floating ips problem)

2014-11-24T12:01:00+01:00

In a lab environment in the university I only had access to a list of fragmented public routable IP addresses in a single subnet. For example I got access to the following ips 50.15.15.10, 55.15.15.12-15, 55.15.15.17. But I am not allowed to use 55.15.15.11 and 55.15.15.16.

Update: The Openstack neutron API supports fragmented floating ips but not through the CLI. You have to use the Python API like below:

import logging
from neutronclient.neutron import client

auth_url = "http://192.168.33.11:5000/v2.0"

logging.basicConfig(level=logging.DEBUG)
neutron = client.Client('2.0', username="admin", password="password", tenant_name="demo", auth_url=auth_url)

print "".join(neutron.list_subnets())

req = {"subnet": {"allocation_pools": [{"start": "10.0.2.3", "end": "10.0.2.15"}, {"start": "10.0.2.17", "end": "10.0.2.17"}, {"start": "10.0.2.19", "end": "10.0.2.254"}]}}

neutron.update_subnet("d5d48930-7bfb-4f0c-8968-13f8af785868", req)

Old hacky way:

It seems that currently the only way to solve this is to manually change the database bypassing the API. I used the following SQL insert statements to solve my problem. This is assuming that 55.15.15.10 is already part of the pool.

use neutron;
select * from ipallocationpools;
+--------------------------------------+--------------------------------------+----------------+-----------------+
| id                                   | subnet_id                            | first_ip       | last_ip         |
+--------------------------------------+--------------------------------------+----------------+-----------------+
| 78857059-658b-4027-909d-7b8c6d62e52f | c58d4e69-d614-4d05-91e5-95b5cc48b670 | 55.15.15.10    | 55.15.15.10     |
| c2931835-050b-40a0-bfc6-6dcb833517f6 | eda970d4-21c0-4488-85b2-0d3d0000ebb9 | 192.168.111.2  | 192.168.111.254 |
+--------------------------------------+--------------------------------------+----------------+-----------------+

insert into ipallocationpools VALUES (UUID(), "c58d4e69-d614-4d05-91e5-95b5cc48b670", 
                                      "55.15.15.12", "55.15.15.15"); 
insert into ipallocationpools VALUES (UUID(), "c58d4e69-d614-4d05-91e5-95b5cc48b670", 
                                      "55.15.15.17", "55.15.15.17");

After inserting the ips, the floating ips can be allocated automatically using horizon and be successfuly assigned to instances. But please note that this is not officially supported and hacky! I haven't seen any problems as of now, but there probably are some problems with this approach. I just used it for testing.

Side note

Previously in nova-network you could easily add single floating ips or smaller ranges within a subnet. But with neutron the CLI and API's seems to not support this. So after looking at the code I noticed that the data model in fact supports multiple allocation pools for a single subnet:

class IPAllocationPool(model_base.BASEV2, HasId):
    """Representation of an allocation pool in a Neutron subnet."""

    subnet_id = sa.Column(sa.String(36), sa.ForeignKey('subnets.id',
                                                       ondelete="CASCADE"),
                          nullable=True)
    first_ip = sa.Column(sa.String(64), nullable=False)
    last_ip = sa.Column(sa.String(64), nullable=False)
    available_ranges = orm.relationship(IPAvailabilityRange,
                                        backref='ipallocationpool',
                                        lazy="joined",
                                        cascade='all, delete-orphan')


class Subnet(model_base.BASEV2, HasId, HasTenant):
    """Represents a neutron subnet.

    When a subnet is created the first and last entries will be created. These
    are used for the IP allocation.
    """

    name = sa.Column(sa.String(255))
    network_id = sa.Column(sa.String(36), sa.ForeignKey('networks.id'))
    ip_version = sa.Column(sa.Integer, nullable=False)
    cidr = sa.Column(sa.String(64), nullable=False)
    gateway_ip = sa.Column(sa.String(64))
    allocation_pools = orm.relationship(IPAllocationPool,
                                        backref='subnet',
                                        lazy="joined",
                                        cascade='delete')

Openstack Networking Study Notes

2014-08-29T13:42:00+02:00

I recommend these links in the following order:

Docker get private ip address of container

2014-07-08T14:25:00+02:00

In order to access other containers from a container it's needed to get the private IP address of the other containers. The following command is able to get the private ip address of the container.

$ docker inspect --format="{{.NetworkSettings.IPAddress}}" $CONTAINER_ID_OR_NAME
172.17.0.15

Docker Multiple websites/domains on single ip/host tutorial using a HAproxy as reverse proxy

2014-07-07T10:15:00+02:00

This post will describe how to expose multiple docker containers running websites on port 80 using HAproxy as a reverse proxy. This makes it possible to run multiple websites on different domains on a single public ip of the host.

The basic setup is to create 1 container for haproxy which is exposed to the host on port 80. This HAproxy container will forward the incoming HTTP request to the correct container based on the domain name.

Picture drawn with draw.io.

First launch the containers which run different websites. In our example we will use a hello-world php demo container and a wordpress site container.

$ # Run hello world php demo container (test1.domain.com)
$ docker run -d tutum/hello-world
01ec10276761
$ docker inspect -f "{{.NetworkSettings.IPAddress}}" 01ec10276761
172.17.0.26

$ # Run wordpress container (test2.domain.com)
$ docker run -d tutum/wordpress
4d23f10f6b35
$ docker inspect -f "{{.NetworkSettings.IPAddress}}" 4d23f10f6b35
172.17.0.25

Now we need to create our haproxy configuration to configure HAproxy as reverse proxy for our docker containers. Because HAProxy is also running inside a container we need to be able to access the hello-world and wordpress container by their private ip accessible from all containers. We got this IP using the command: docker inspect -f "{{.NetworkSettings.IPAddress}}" $CONTAINERID. Make sure to note down these IPs as they will be used in the haproxy.cfg file.

$ # On the host(not container) create directory containing our haproxy config file
$ mkdir ~/haproxy-config

$ # Create ~/haproxy-config/haproxy.cfg
$ vim ~/haproxy-config/haproxy.cfg
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        user haproxy
        group haproxy
        # daemon

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option forwardfor
        option http-server-close
        contimeout 5000
        clitimeout 50000
        srvtimeout 50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
        stats enable
        stats auth username:password
        stats uri /haproxyStats

frontend http-in
        bind *:80

        # Define hosts based on domain names
        acl host_test1 hdr(host) -i test1.domain.com
        acl host_test2 hdr(host) -i test2.domain.com

        ## figure out backend to use based on domainname
        use_backend test1 if host_test1
        use_backend test2 if host_test2


backend test1 # test1.domain.com container
    balance roundrobin
    option httpclose
    option forwardfor
    server s2 172.17.0.26:80 # This ip should be the ip of hello-world container

backend test2 # test2.domain.com container
    balance roundrobin
    option httpclose
    option forwardfor
    server s1 172.17.0.26:80 # This ip should be ip of wordpress container

$ # Run haproxy and map the host directory ~/haproxy-config to /haproxy-override of the container
$ # See the image README and Dockerfile for info about this override behaviour.
$ # HAProxy is exposed on port 80 because all requests to the public ip should
$ # go to the HAProxy container.
$ docker run -d -p 80:80 -v ~/haproxy-config:/haproxy-override dockerfile/haproxy

The HAProxy configuration could be automated possibly with the use of etcd to store information about services or use a similar method to Automated nginx reverse proxy for docker. The automated nginx reverse proxy didn't work for me though.

Thanks to HAProxy as static reverse proxy for docker containers for the haproxy config file. Although I think it's better to run HAproxy in a container.

Move first word to end of line (Linux/awk)

2014-06-08T20:29:00+02:00

Move the first word of a line to the end of line on Linux using AWK. We first store the first column(first word) in the variable t. Then we set the first column to be empty. At last we add a new column after the last column of the line which is stored in $NF.

$ echo " move_to_end  should stay at the same place" | awk '{t=$1; $1=""; $(NF+1)=t}1
should stay at the same place move_to_end

The idea was from a stackoverflow question which I can't find anymore.

SSH to server behind a firewall via an SSH tunnel

2014-05-10T00:00:00+02:00

A server behind a firewall was unaccesible from my home, but another server which is on the same local network had public SSH access from my home. So we can create an SSH tunnel via this public server to the server behind a firewall.

For example the public ssh server is 8.8.8.8 and the server on the private network that we want to access is 192.168.1.5. Then we can create an SSH tunnel from my laptop at home in the following way:

ssh root@8.8.8.8 -L 2020:192.168.1.5:22

The above command creates an SSH tunnel, where the local port 2020 on my laptop will be forwarded to port 22 of the private server behind the firewall.

After the tunnel is created you can use the following command to directly SSH from your home laptop to the private server with ip 192.168.1.5:

ssh ubuntu@localhost -p 2020

Recursive karatsuba multiplication in Python

2014-05-04T00:00:00+02:00

I'm currently taking the class Algorithm: Design and Analysis of Stanford via Coursera and in the class the Karatsuba algorithm got mentioned. I went ahead and implemented it based on the lecture slides.

The Karatsuba algorithm is a fast multiplication algorithm. It's special because it was the first multiplication algorithm to be faster than the quadratic "grade school" algorithm. Karatsuba algorithm - Wikipedia It also has a funny history, see the history at the Wikipedia article.

My Python implementation using a Divide and Conquer approach:

from math import ceil

memory = [[0,0,0,0,0,0,0,0,0,0],
          [0,1,2,3,4,5,6,7,8,9],
          [0,2,4,6,8,10,12,14,16,18],
          [0,3,6,9,12,15,18,21,24,27],
          [0,4,8,12,16,20,24,28,32,36],
          [0,5,10,15,20,25,30,35,40,45],
          [0,6,12,18,24,30,36,42,48,54],
          [0,7,14,21,28,35,42,49,56,63],
          [0,8,16,24,32,40,48,56,64,72],
          [0,9,18,27,36,45,54,63,72,81],
          [0,10,20,30,40,50,60,70,80,90]]

def karatsuba(x, y):
    """ Recursive multiplication using karatsuba
    x = 10^n/2 * a + b
    y = 10^n/2 * c + d
    x * y = 10^n * ac + 10^(n/2) (ad+bc) + bd
    where (ad+bc) = (a+b)(c+d) - ac - bd
    """
    str_x, str_y = str(x), str(y)
    n = max(len(str_x), len(str_y))
    if n <= 1:
        # Just for fun to not use any multiplications haha
        return memory[x][y]

    str_x = prepend_zeros(str_x, n)
    str_y = prepend_zeros(str_y, n)
    n_2 = n / 2

    a, b = int(str_x[:n_2] or 0), int(str_x[n_2:] or 0)
    c, d = int(str_y[:n_2] or 0), int(str_y[n_2:] or 0)

    ac = karatsuba(a, c)
    bd = karatsuba(b, d)
    ad_bc = karatsuba((a + b), (c + d)) - ac - bd

    # for supporting edge case where n is not a multiple of 2
    n_2 = int(ceil(n / 2.0))
    n = n if n % 2 == 0 else n + 1
    return (10**(n) * ac)  + (10**n_2 * ad_bc) + bd

I also created a testcase to test whether my implementation is correct:

class MultiplicationTestCase(unittest.TestCase):

    def test_small_numbers(self):
        self.assertEqual(karatsuba(5, 5), 25)

    def test_different_size(self):
        self.assertEqual(karatsuba(2, 21), 42)

    def test_different_size(self):
        self.assertEqual(karatsuba(103, 3097), 318991)

    def test_two_digits1(self):
        self.assertEqual(karatsuba(50, 50), 2500)

    def test_two_digits2(self):
        self.assertEqual(karatsuba(19, 21), 399)

    def test_three_digits1(self):
        self.assertEqual(karatsuba(500, 500), 250000)

    def test_three_digits2(self):
        self.assertEqual(karatsuba(223, 321), 71583)

    def test_four_digits1(self):
        self.assertEqual(karatsuba(1234, 4321), 5332114)

    def test_seven_digits(self):
        self.assertEqual(karatsuba(5000000, 5000000),
                         25000000000000)

    def test_random_cases(self):
        for i in range(1000):
            x = randint(1,100000)
            y = randint(1,100000)
            expected = x * y
            result = karatsuba(x, y)
            self.assertEqual(karatsuba(x, y), expected,
                            ("Failed with x: %s and y: %s. "
                             "Expected: %s but got %s") % 
                             (x, y, expected, result))

Using nested tmux over ssh and how to detach a nested tmux

2013-08-15T02:08:00+02:00

Today I logged in to my laptop at home from work which was already running a tmux session, so thought yea just attach cool can continue with what I was looking at at home. I forgot that the keys would be the same ofcourse, and was unable to use the tmux session at my laptop at home. Whenever I would type Ctrl + B + n etc it would do that on my work desktop which was also running tmux. I was lolling oh damn how to get out of that laptops tmux session.

Google to the rescue. Here is the solution:

If you want to access the tmux session inside the ssh session so in my example on my laptop, you have to just use Ctrl + b + Ctrl + b

So if you want to detach the nested tmux session you have to execute the following: (Ctrl + b) + (Ctrl + b) + d.

(Resource: Tmux Tips)

Remove/detach a PCI device linux (kvm VGA passthrgough Nvidia Quadro 2000)

2013-06-09T08:55:00+02:00

Answer:

echo -n "1" > /sys/bus/pci/devices/0000:02:00.0/remove

Where 0000:02:00.0 is related to your pci device address which can be obtained from the command lspci

Related story:
Was working on KVM vga passthrough and encountered problems with the graphics card nvidia Quadro 2000.

This was the error I got when attaching this particular VGA card:
libvirtError: internal error Unable to reset PCI device 0000:83:00.0: internal error Active 0000:83:00.1 devices on bus with 0000:83:00.0, not doing bus reset

After investigating I noticed that 0000:83:00.1 is a Sound compatible controller on the video card. Tried lots of stuff such as disable the drivers of that sound compatbile controller but without much success.

So in the end I decided I should just remove the pci device, but couldn't find how to do it, so just tried some stuff and this is what I came up with. I have no idea if it will break your system or what it does exactly, but after removing the sound compatbile controller I was able to pass the nvidia Quadro 2000 card to my virtual machine :)

Here is the code to remove the pci device using a simple linux command:

lspci -k | grep NVID -C3  
00:83.0 VGA compatible controller: Nvidia Quadro 2000.... (Dont have
the card anymore)  
00:83.1 Sound device: Nvidia Quadro Audo captabiel balbalbal. 2000 GP  
echo -n "1" > /sys/bus/pci/devices/0000:02:00.1/remove

Install PyQt in a virtualenv with Pip

2013-05-30T15:48:00+02:00

Noticed that pip install pyqt isn't working? At least at the time of writing this blog it isn't..

This is the error that I got:
IOError: [Errno 2] No such file or directory: setup.py

It seems they didn't package it well and it's using configure.py to install. So here is the solution to install it by using pip to download the files and configure and install manually.

Solution copied from: http://stackoverflow.com/a/13967084/376445

workon myProject  
pip install --no-install SIP  
pip install --no-install PyQt  
cd ~/.virtualenvs/myProject/build/SIP  
python configure.py  
make  
make install  
cd ~/.virtualenvs/myProject/build/PyQt  
python configure.py  
make  
make install  
cd && rm -rf ~/.virtualenvs/myProject/build # Optional.

Issues I encountered:

Missing development headers for qt:

~/.virtualenvs/scrapy/build/PyQt# python configure.py --verbose  
Determining the layout of your Qt installation...  
/usr/bin/qmake -o qtdirs.mk qtdirs.pro  
make -f qtdirs.mk 
g++ -c -m64 -pipe -O2 -Wall -W -D_REENTRANT -DQT_NO_DEBUG
-DQT_CORE_LIB -I/usr/share/qt4/mkspecs/linux-g++-64 -I.
-I/usr/includ e/qt4/QtCore -I/usr/include/qt4 -I. -o qtdirs.o qtdirs.cpp  
qtdirs.cpp:1:28: fatal error: QCoreApplication: No such file or directory compilation terminated.  
make: *** [qtdirs.o] Error 1  
Error: Failed to determine the layout of your Qt installation. Try again using  the --verbose flag to see more detail about the problem.

Soltution: Install the development headers (Ubuntu 12.04)

sudo apt-get install libqt4-dev

Ignoring every channels join/part/quit messages in IRSSI

2013-05-11T02:00:00+02:00

Tired of all those join part messages?

/ignore \* joins  
/ignore \* parts  
/ignore \* quits  
/save

Create tar archive of changed/added files in a git commit

2013-05-02T02:12:00+02:00

The following command first gets the changed and added files of a git commit and then creates a tar archive of those changed files.

git show {{COMMIT_ID}} --name-status | grep -Ew '^M|A' | awk '{ print $2 }' | xargs tar czf usb-passthrough.tar.gz

Full migration/clone of linux installation to a new system(Without reinstalling)

2012-11-14T13:26:00+01:00

I'm switching to a new laptop, but don't want to have to install Archlinux again. So instead of re-installing I will try to copy all files and hope the system will work. In this blogpost I will describe the steps and issues I encountered while doing so.

Status: Succeeded!!!!

Summary of the plan: Boot from a live usb on the new system, create partitions with gparted, copy all the files over, change /etc/fstab to partition uuids, regenerate the mkinitcpio stuff (Still should read what this actually is).

On the new laptop following steps should be done:

Boot from a Ubuntu live cd and start gparted
Create a / partition (ext4)
Create a /home parition (ext4)
Create a /boot partition (ext2)
Mount the newly created partitions under /mnt/
Copy all files of the old laptop / /home to new laptop /mnt/ /mnt/home using rsync
Chroot into the copied files on the new laptop
Change the /etc/fstab to point to the new partition UUIDs
Install grub on new harddrive and reconfigure Grub
Recreate the ramdisk

Environment:

Old Laptop:
Archlinux 64 bits, 8GB memory, AMD CPU and ATI card
IP: 192.168.1.108
Connected to router via internet cable, WIFI was way too slow to copy files

New laptop: Intel CPU and nvidia video card
IP: 192.168.1.111

Detailed steps:

Boot from a live ubuntu USB on the new laptop and do the following:

Use gparted or fdisk to create the following partitions: /dev/sda1 filesystem: ext2 size: 512mb label: boot (100mb is needed, but had enough space will become /boot) /dev/sda2 filesystem: ext4 size: 30gb label: root(will become root /) /dev/sda3 filesystem: ext4 size: 180gb label: home(will become home /home)

After you have created the partitions successfully, mount all partitions to /mnt

mount /dev/sda2 /mnt
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot
mkdir /mnt/home
mount /dev/sda3 /mnt/home

Now go back to the old Archlinux laptop and copy all files to the newly created partitions:
```
rsync -azv --progress --exclude=/dev --exclude=/sys / root@192.168.1.111:/mnt/
```
This may take a while..................

After everything is copied over to the new laptop, we will have to change /etc/fstab to look as follows:

tmpfs /tmp tmpfs nodev,nosuid 0 0
LABEL=root / ext4 defaults,noatime 0 1
LABEL=home /home ext4 defaults,noatime 0 2
LABEL=boot /boot ext2 defaults,noatime 0 2

Then reinstall Grub on the hard drive:
grub-install --force --target=i386-pc --recheck --debug /dev/sda

Why the --force? I encountered the following error: will not proceed with blocklists archlinux so after reading the wiki it said I could fix this by using --force.
Regenerate grub config:
grub-mkconfig -o /boot/grub/grub.cfg
Regenerate initial ramdisk:
mkinitcpio -p linux
Reboot the system

I also had to update my modules.d/xxx to not load some specific AMD modules for laptop-mode tools as the new laptop has an intel processor. Also has to remove the ati driver and change to intel open source driver:

yaourt -Rs xf86-video-ati
yaourt -S xf86-video-nouveau

The reason I am switching: My current HP 625 laptop has served its purpose for nearly 2 years. The CPU seems to be nearly burnt out, when I opened the laptop few days ago, the CPU was totally black, but still working.

References I used: Archlinux wiki: Migrate installation to new hardware https://wiki.archlinux.org/index.php/Migrate_installation_to_new_hardware Forum post which basically does the same thing https://bbs.archlinux.org/viewtopic.php?id=145025

Only use worker when required on heroku with Django/Python

2012-04-13T09:42:00+02:00

For a mobile project I required a background worker which sents an email with 300 generated QR codes zipped together as attachment. This costs quite some time so we need a background worker to execute this task.

I wanted to achieve the following result:

When the user wants to generate qr codes, Start the celery worker process with scale(1)
Delay the celery task to be executed by the celeryworker
After our task is finished, stop the celeryworker with scale(0)

Why I want this? Well workers costs quite some money if they are running constantly, which is in our case maybe only 1 hour a month. Saving us about \$30 / month.

For more information about how to run a worker check the Official Heroku Django documentation, which shows you how to setup Django with celery. Here you can find how to create the task itself in celery: Django-celery tutorial.

To talk with the Heroku REST API we are using Heroku.py. This API let's us scale and stop / start proccesses with Python. The official documentation tells to do it this way:

import heroku  
cloud = heroku.from_key(settings.HEROKU_APIKEY)  
app.processes['celeryd'].scale(1)  
# now execute our celerytask  
generate_qr_codes.delay(product_id=product.id)

The only problem is that this line actually does not work:

app.processes['celeryd'].scale(1)

It will give an KeyError exception, because the celeryd is not running which is kind of a bug, thats why I want to scale it to 1 in the first place...

So I got stuck on that problem, but after checking out the Heroku Python library I decided to use their internal API as they have an easy way of calling Heroku HTTP resources. It's a quick work around to do what I wanted:

cloud._http_resource(method='POST',
                    resource=('apps', 'heroku_processname', 'ps', 'scale'),  
data={'type': 'celeryd', 'qty': 0})
# qty 0 is scale to 0 processes, if you want 1 process running change to qty 1

I have created an issue in the github repo about not being able to scale a non running process: https://github.com/heroku/heroku.py/issues/10

This is how our celery task looks like:

import logging  
from celery.decorators import task

@task()  
def generate_qrs(product_id):  
    try:  
        y = x + product_id  
        # this is where your code will be that gets executed in the background  
    except Exception, e:  
        logging.exception('Exception occured in the celery task to generate QR codes')  
    if settings.DEBUG == False:
        import heroku
        cloud = heroku.from_key(settings.HEROKU_APIKEY)  
        app = cloud.apps['appname']  
        cloud._http_resource(method='POST', resource=('apps', 'heroku_processname', 'ps', 'scale'),
                             data={'type': 'celeryd', 'qty': 0})
        # the above line will stop the celeryd worker, so we dont have to pay
        # for the woker only when it's actually executing tasks

Using Sendgrid with Django on Heroku (How to find your Sendgrid password on Heroku)

2012-04-11T09:43:00+02:00

While setting up sentgrid I was searching for my password of sendgrid, but coudn't find it. You can get it in the following way:

heroku config --long

Which will show the parameters:
SENDGRID_PASSWORD, SENDGRID_USERNAME

After you get your username don't forget to add it to django settings:

EMAIL\_HOST = 'smtp.sendgrid.net'  
EMAIL\_HOST\_USER = 'app11111111@heroku.com'  
EMAIL\_HOST\_PASSWORD = 'thepassyougotbefore'  
EMAIL\_PORT = 587  
EMAIL\_USE\_TLS = True

Hope it helped somebody

Django Custom View Decorator with Arguments

2012-02-23T17:52:00+01:00

To make my life easier I made a simple decorator which checks if the parameters are present in the request.POST or request.GET and returns a response if they are not.

It accepts a list of parameters in string format and will loop through each parameter to check if they are present in the request.POST.

I noticed there weren't a lot of guides about how to write a decorator for Django which takes arguments. I based my code on this tutorial, it's a really great guide which helps you understand Python decorators at a new level: Amazing insight about Python decorators

Here is the code I wrote for the decorator:

def required_parameters(parameters=('email', 'api_key'), http_method='POST'):  
    """  
    Check if the required parameters are present in the request  
    @param parameters: The names of the parameters that should be supplied  
    """  
    def inner_decorator(fn):  
        def wrapped(request, *args, **kwargs):  
            # check if the user api_key matches  
            for parameter in parameters:  
            if parameter not in getattr(request, http_method):  
                return json_response({'success': False, 'errors': 'Please use the Web API correctly and supply the parameter: '+parameter})  
            # Proceed like normally with the request  
            return fn(request, *args, **kwargs)  
        return wraps(fn)(wrapped)  
    return inner_decorator  
    [/python]

An example of using it in your view:

@required_parameters(parameters=('email', 'username', 'password'))  
def view_which_requires_email_username_and_password(request)  
    bla = 'bla'
    return HttpResponse(bla)

Simple function to return json in Django

2012-02-22T15:53:00+01:00

Just a small code snippet I use when I need to return json, comments are welcomed.

from django.http import HttpResponse  
from django.utils import simplejson as json

def json_response(dict_to_convert_to_json):  
    return HttpResponse(json.dumps(dict_to_convert_to_json),
                        mimetype="application/json")

To use it in your view you just do it this way:

def login(request):  
    return json_response({'success' : True, 'user' : {'name': 'Sam',
                                                      'speciality': 'Django/Python'}})

You could also create a decorator to do this but I think using a simple function is more straight forward.

Chinese New Year 2012

2012-01-22T03:24:00+01:00

The Chinese New Year aka Spring Festival is the most important festival in Chinese culture. All the people travel back to their hometown, because most of them are working/studying in cities far from their own hometown. That means all train tickets are sold the day they get available and plane tickets are super expensive. Luckily Marry was able to order train tickets via her university, so we didn't have to go through the hardship of having to buy the tickets ourself. But there was a little problem with this solution as the tickets will be on somebody's passport, because I she cant order them on my name because I'm not a student on her uni.

So the 18th we took the train, I was really nervous as I also did not have my passport and the train ticket not being on my name. My passport is being hold for extending my visa, and they lately required everybody to show their passport, else you wouldn't be able to enter the train station. But everything went fine in the end, the first security check I just pretended I didn't understand and then they smiled and let me through. After that they only checked the tickets so it was ok.

The train trip itself was pretty decent, could watch outside and there weren't as many people as I expected. I heard horror stories about not being able to move at all in the train. It was hard but not impossible, as people were standing / sitting on the ground because they didn't have a seat. We were sitting together with students from Marrys uni, so we talked with them the whole trip, was pretty fun. Time went quite fast and even managed to sleep for 5 hours.

Here in Fuzhou I'm mostly still working so didn't have time to go to many places yet. Yesterday went to the country side to celebrate new years with hotpot, a big meal and visited some relatives. The country side was quite amazing they had small roads and all houses are different as most of them build the houses them self. Also went to walk around a little and found a nice castle and river. Will go there again in the evening.

My overal impression of Fuzhou is pretty good, I like the atmosphere and the traffic is much better than Beijing. There are also palm trees and many tropical trees, which I like.

Importing a SQL dump in postgres

2011-11-29T13:46:00+01:00

I always keep forgetting how to import a SQL dump into postgres and found it always hard to find the right documentation. So I thought lets share it:

psql -d dbname -U username -f dumpfile.sql

This should also work with files created by pgdump.

Installing fastboot for Ubuntu (Linux)

2011-11-22T14:11:00+01:00

mkdir ~/fastboot  
cd ~/fastboot  
wget http://xt720.ru/Tools/fastboot  
chmod +x fastboot  
sudo ./fastboot --help

Reference: http://forum.xda-developers.com/showthread.php?t=1121420

I tested the above with Ubuntu 11.10, but it should work on any ubuntu version I think.

Trying to install scrapy with pip lxml error: command 'gcc' failed with exit status 1

2011-11-01T14:31:00+01:00

Just tried to install scrapy with pip / easy_install, but when it comes to installing lxml it will give the following error:

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w

In file included from src/lxml/lxml.etree.c:239:0: src/lxml/etree_defs.h:9:31: fatal error: libxml/xmlversion.h: No such file or directory compilation terminated.

error: command 'gcc' failed with exit status 1

The solution
Install Python development package:

sudo apt-get install python-dev

Install the development packages of libxml and libst

sudo apt-get install libxml2-dev  
sudo apt-get install libxslt1-dev[/sourcecode]

Ubuntu 11.10 error while loading shared libraries: libssl.so.0.9.8: cannot open shared object file: No such file or directory

2011-10-29T06:15:00+02:00

A quick solution is to install the libssl 0.9.8 package with the Following command:
sudo apt-get install libssl0.9.8

Thanks to Rob who commented below and provided an answer on Stackoverflow

I just upgraded from ubuntu 11.04 to 11.10 with a fresh install, by keeping my old home folder so not loosing my personal data, but only formatting the whole system partition. After reinstalling my most important development software I still couldn't run django's server. I kept getting the followering error:
error while loading shared libraries: libssl.so.0.9.8: cannot open shared object file: No such file or directory

You will also get this error:
python: error while loading shared libraries: libcrypto.so.0.9.8: cannot open shared object file: No such file or directory

It seems this problem was because the virtualenv was still using the previous python libraries, which linked to a no longer there ssl package, what I did was create a new virtualenv and problem was solved. Anybody know a better way? Is there a way to update virtualenv to a new python version?

Update: Seems it's just the upgrade process is too buggy normally. I really enjoy Archlinux and recommend it to anybody who hates these kind of issues.

Samwich: Sam's tasty sandwiches

2011-09-12T15:35:00+02:00

I went to sell sandwiches on the streets of Beijing. Why you think? Just for fun and experience. I often buy food on the streets here myself, stuff like sushi, pancakes, noodles etc and I really like it. It's quick fun and tasty. So I also wanted to do that.

Link to foto album: https://picasaweb.google.com/118136053313378320646/Beijing2011

Today I first went to Weiduomei 味多美 to buy some Baguettes(stokbrood), that french long crispy bread, we bought 2 for 12元 \~1,20 euro. After that we went to the supermarket to buy tuna fish(12元 \~1,40 euro) in can and olive oil and a plate to put the sandwiches on. After coming back at the appartment I still needed to buy tomatoes and vegetable, which we bought for 4 yuan \~ 0,40 euro. So in total all the ingredients costed 30元 \~3 euro.

When we got all the ingredients around 7 o'clock I started making the sandwiches with tuna, tomato, salad and olive oil, with tomato ketchup. At 8 o'clock we started selling, in the beginning I felt weird all the people had to laugh also at me when they saw a foreigner selling stuff on the street, many people looked at the paper Lindan wrote:Tuna sandwich Big ones 10元, small ones 5元. The first 10 minutes many people looked but nobody buyed, although I was screaming Jing Chang Yu San Ming zi, Da de 10 kuai, Xiao de 5 kuai, still people didn't want to buy it. Some people told me they are full and just ate hehe. But after some time our first customer bought a small one it were 2 girls. After that it went pretty quick within 1 hour we sold nearly all and though it is enough, now we want to eat what's left ourself haha.

In the end we earned 35元, but the costs of all the sandwiches were 30元. We ate 2 sandwiches ourself, so we ate free sandwiches today and still made 5 yuan and lots of fun. Not bad! Gonna do this again next time, but then a little bigger.

The Beijing Bike Tour and China Belly Problems

2011-08-20T14:23:00+02:00

New pics are added to the same album: https://picasaweb.google.com/118136053313378320646/Beijing2011WithBart#

The day before yesterday went on bike tour to some famous lakes in Beijing (Beihai, Houhai, Qianhai, Xihai) and some temple which was closed already :( We biked the following path I created the route on Google Maps for people, who want to try it also: The Route on Google Maps.

View Beijing Bike Tour Wudaokou -> Beihai in a larger map

The bike tour itself was also already pretty good, we didn't use the normal roads but tried to go through the neighborhoods around here. That was pretty fun, the neighborhood we went through seemed a little poor many dirt on the street and many people selling vegetables, also saw a store selling chickens, which were still alive. Actually we were planning to start at Xihai and go down to Beihai, but because we went much faster than expected we ended up at Beihai and went up again to Xihai.

After arriving at Beihai we decided to go in and pay the small entrance fee, which I really hate. Don't wanna pay for some touristic attraction. It will make me feel too much like a tourist. Inside the park it looked pretty good, we went with a small boat to the other side of the lake. On the boat I talked with some Chinese kids and his father was using his iPad to make pictures and videos. We were also on some of those pics :P

Yesterday we went to the Olympic park, a park which they built just for relaxing and beauty. At the entrance of the park there is a big lake with many golden carps, which you can feed and touch if you are quick enough :P We also fed them some Chinese white bread. After it became dark, we went fishing nearby the Olympic park, but didn't catch any :( Last time I went fishing there with Lindan, we catched 2 gold fishes.

Today we planned to go to the great wall, but Bart was a little sick because we traveled too much. Not serious luckily, just little uncomfortable in the belly. The standard stuff you get in China. But I'm there no need to worry, already bought some western food like bread and jam and water. He is already feeling much better. But having a good relax also wasn't that bad for myself also :P

Tomorrow we will even go out to travel again if Bart is totally recovered. We will go to Summer Palace (Yi he yuan) and another famous park nearby called Yuan ming yuan.

The Watercube and it's big waterpark

2011-08-18T04:53:00+02:00

The day before yesterday we went to a big waterpark inside the Watercube in the Olympic Park. We watched some pictures and thought it was really big, but when we came there we noticed it wasn't as big as we thought :P Pictures not only lie with girls, but also with waterparks.

It was still pretty good though, there were 3 pretty high, fast and scary slides and the others were more for kids. There was also the most crazy slide I ever saw. When we saw it outside we already thought wow this is really fast and steep. They first shoot you fast down and then with the speed you gain you will go up again, nearly like a rollercoaster with a loop inside. But that wasn't all when we went upstairs to see the entrance of the slide it got even more crazy. There was a machine which looked like coffin or capsule, which the guy there will open for you and asks you to go in. Then they close the coffin and a womans voice says 3-2-1 and the floor under your feet goes away and you fall down in the slide.

So who would want to take such a slide? Well not so many, also nearly no people toke it so I certainly didn't want to risk my life :P But Bart really wanted to take it and insisted that he would only take it if I took it. So I first interviewed a girl who took the slide and asked her if she felt like she had to puke or got hurt or felt dizzy and such. After the interview was positive I agreed to go with Bart and risk my life haha. I was pretty nervous when the woman said 3-2-1 but when I got out I felt it wasn't that scary. Although I got hurt a little, I have a small wound on my leg now because it went too fast and my leg got burned or something I guess. This is the slide in question: http://waterpark.water-cube.com/english/Attractions_AquaLoop.aspx

After all it was pretty fun, I also never knew there was a waterpark inside the Watercube. But thanks to Bart and his mother, who actually knew there was a waterpark, we went there :) Bart treated us, pretty good :D Today we will go on a bike trip through whole Beijing. Tomorrow probably more about that :)

Bart and his lost Bike

2011-08-11T15:00:00+02:00

Yesterday Bart went on a trip to Tianmen Square, but after coming back I got a call that his bike got lost. So my first thought was must be stolen, but then he said he put it in front of the police post. So I gave it another thought and maybe thought it can still be found if we have look. So Lindan and me went to Wudaokou to check, where we found there are no bikes so maybe the bike shouldn't be put there.

A few moments later Lindan found a sign saying in Chinese characters "You can't put your bike here, you should be responsible for your own acting(禁止停放自行车，后果自负。" Which Bart couldn't understand of course. So Lindan first asked some cleaner or traffic guy, where they would put the bikes, but he didn't know. Then we asked at the police station and they said the put it at a nearby bike parking place. At the bike parking place it was just a matter of finding the bike and paying 2 eurocent, which is the normal price for 1 day parking. So it all ended well. I will also study reading and writing Chinese characters more, seems it's quite useful here.

A New Start in Beijing

2011-08-11T07:07:00+02:00

Pics: https://picasaweb.google.com/118136053313378320646/Beijing2011WithBart

After a tough trip, waiting in moscow for 8,5 hours we finally arrived in Beijing. On the way to the hotel Bart found the traffic here a little rough and dangerous, but actually I found the driver today was driving quite peacefully already haha. Guess I got used to the kamikaze drivers, which cross the roads the way depending on luck whether they live or die. After arriving at the hotel we were greeted with a friendly NiHao(ni hau).

This was the hotel we lived in for 168Yuan/night (17 euro). We also helped Bart to rent a kid's bike for 6 euro for this month. The bike here is pretty handy. We went to bike for half an hour and arrived at the mountain. It was my third time already to go to the mountain. The first time was with Lindan on the bike and 2nd time I went with my old classmates. It's my favorite mountain. Here are some pics: https://picasaweb.google.com/118136053313378320646/Beijing2011WithBart

We also went to the olympic park, which you can see on the pics. Also been there for several times already but I still like it there, always nice and cosy, people dancing, singing and chilling. Next time we will go there in the daytime so we can feed the koi carps there.

Although we did a lot of fun things, we also had to find an apartment because we will be staying here for 1 year. Yea it's definite now. We need to pay the rent for 1 year, because the landlord lives far away and this was how they wanted things done. I first thought maybe it's a cheater, but we already registered at the police so he isn't a cheater. But need to get enough money before the end of the month. He first even said if you dont have enough now we can't rent it out.. So we arranged we can pay the remaining money at the end of the month. I will have to work hard this year I think :P The apartment is really good though, nice living room and good furniture big bed, everything I need.

Tonight will go to the outside swimming pool again. Lindan's swimming skills improved a lot, but still needs practice so she won't die when she falls in the sea. Thanks to Bart's mother we also know we can go to something much better than Tikibad in the watercube(in Olympic Park). We are planning to go there somewhere soon.

How to install Setuptools to a specific python version

2011-07-18T19:34:00+02:00

To install easy_install for a specific python version. I just installed from source and used the python version you want to install setuptools too. I used the following steps in Ubuntu 11.04 with Python 2.5 and Python 2.7 installed. The following commands will install setuptools to python 2.5

wget
http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e  
tar -zxvf setuptools-0.6c11.tar.gz cd setuptools-0.6c11/  
sudo python2.5 setup.py install

The following command installs the pil module to 2.5:
sudo easy_install-2.5 pil

This command installs the pil module to 2.7:
sudo easy\_install-2.7 pil

Sam the terrorist arrived in China

2011-02-13T00:08:00+01:00

It has been some days I guess since I have been away from Hollad. But it has been kind of hectic. I will start from the trip till today.

The trip started early at 6:30, train to Schiphol and than flight to London, wait in London for 5 hours. So I was waiting and got little bored and decided to walk around for a bit and sit at some other places, and watch some episodes on my laptop. Than when I saw I could check in for my flight to Beijing I noticed I forgot my jacket somewhere, so I went to look for it at the places I went before. Because this jacket is my 4-5 year old favorite jacket :D

Then I found it after 10 minutes of searching and thought ah great just get it. Than I noticed there were more securtiy than normally around me. I thought hmm just walk away and at that moment one of them talked to me, asking me if it was my jacket. What is in your pocket? Why did you leave your jacket? What is the flashing light in the jacket? (My bike light, but forgot i took that with me haha)What is your destination? What is your intention? What will you do in beijing? And he kept complaining it was really not good to leave something... he asked me more questions but can't remember all of them.

After a little trouble I arrived in Beijing, took some subway to get to the hotel and meet Marry again :D was really good. We stayed in a hotel for 2 days while searching for appartments. Some of the appartments were so bad haha, with toilets which looked like they weren't cleaned for weeks, but we finally found one for a decent price. A 2 bedroom appartement with a living room, and kitchen for 400 euro a month. House prices are quite high here in Beijing compared to Xiamen. To buy a house also costs like 200.000 euro or more for 60-80m2.

There has also been some snow really fun, because it was Marrys first time to see snow in her whole life. Had some lil snowfights etc hehe :D Too bad didn't make pictures yet, will try to make more. We got a camera just forget to take it...

Edit: Made some pics of the appartment, you can see them here: https://picasaweb.google.com/sammiestoel/BeijingInternship2011Appartment#

Sam Stoelinga

Deploying Stable Diffusion Web UI on GKE Autopilot

Exposing Stable Diffusion Web UI to the internet

Deploying Stable Diffusion Web UI using Docker on GCE T4 VM

GKE Nodepool Add Labels Without overwriting existing labels

GKE list tainted nodepools with a specific taint

3 tips for GKE ML/batch workloads

Top 3 tips for Batch ML workloads

GKE Safely Drain a Nodepool without pod disruptions

How to safely drain the node without pod disruptions?

Why not use kubectl cordon or taint?

Verifying this meets our requirements

Removing sidebar from storefront theme

Creating a woocommerce storefront child theme

Why you should use Wordpress child themes instead of directly editing theme files?

1. Creating a new style.css file with metadata

2. Upload and activate the child theme

Find files with ack and replace string with sed

Python Create UUID from random string of words

Deploying a Weaviate cluster on GKE

1. Creating and configuring the GKE cluster

2. Deploying Weaviate with Helm

(Optional) Restricting access to K8s cluster only

Python Create UUID from integer

Preventing Privileged pods using Pod Security Admission / Standards

Introduction to Pod Security Standards and Pod Security Admission Controller

Preventing privileged pods with PSS

Enforcing the baseline Pod Security Standard

ChatGPT Python: using an unofficial library

1. Installing the ChatGPT Python Library

2. Configuring authentication

3. Using the ChatGPT python library

Trying ValidationAdmissionPolicy aka CEL Admission in K8s 1.26

Creating a 1.26 cluster

Configure the policies

Verify everything works

Deploying K8s on your laptop with minikube

Installing required tools

Deploying K8s on your laptop

Deploying an application to your K8s cluster

Full Text & Vector Search for Firestore with Weaviate

Deploy Weaviate and define Schema

Indexing Firestore Notes automatically on creation

Creating a Firebase Function to search using Weaviate

Conclusion

Security note

GKE GPU timesharing and resource quotas experiment

Creating the cluster and nodepool

Creating the namespaces and resource quotas

Creating a GPU pod in tenant-a and tenant-b

Summary

GKE move system services (kube-dns, calico) to dedicated nodepool

GKE docker registry with HTTP proxy

Creating the docker HTTP proxy script as ConfigMap

Creating the proxy injector DaemonSet

Summary

GKE custom OSS K8s cluster autoscaler

Why?

How?

1. Create a GKE cluster without cluster autoscaler enabled

2. Configure Workload Identity

3. Deploy the K8s cluster autoscaler using helm

Custom DNS entry with KubeDNS stubdomain

1. Deploying CoreDNS with custom DNS entries

2. Configure kube-dns to use stubDomain that points to CoreDNS

Summary

Mounting FUSE without privileges on K8s/GKE

Securing Redis with Istio TLS origination

1. Creating the Redis instance with Aiven or Redislabs

2. Creating the Istio resources

3. Validating that TLS origination worked

Summary

Deploying OpenStack on GCP

1. Creating the GCE VMs with nested KVM enbaled

2. Deploying OpenStack

3. Accessing the environment

Creating L2 connectivity between GCE VMs in GCP using VXLAN

1. Create the VMs

2. Setup VXLAN mesh between the VMs

Summary

1. Creating a new `style.css` file with metadata