Matthew Ruffell

Reflecting on the Azure DNS Outage - A Post Incident Analysis

During my work as a Sustaining Engineer at Canonical, occasionally I get tasked with analysing and fixing high profile regressions that turn into world ending emergencies. I think I have worked on four or five of these cases now, and behind each and every one there is a story to tell, and lessons to be learned.

Today, we will dive into the intricate and complex series of events that caused the worldwide Azure AKS Cloud outage, for systems running Ubuntu 18.04 LTS, which I had the responsibility and leadership to resolve.

So, go brew a cup of coffee or whip up a hot chocolate, and let’s recount the events that happened four months ago, and how we worked to resolve them without causing another world ending event to occur.

Investigating Missing Stack Canaries and Fortify Source on Binaries

Not too long ago, I worked on a fairly interesting case where a user claimed that many of the binaries on their system were missing Stack Canaries provided through -fstack-protector-strong and additionally, many were missing Fortify Source being enabled through -D_FORTIFY_SOURCE=2.

This is most unusual, since these compiler flags, along with many others, are enabled by default for all packages in the Ubuntu archive.

So in this writeup, we are going to investigate this user’s claims, and try get to the bottom of the mystery of the missing compiler hardening options in binaries from the Ubuntu archive. Stay tuned.

Learning How to Write Reactive Charms by Porting our Minetest Charm

It has been a really long time since my last blog post, so let’s fix that by writing a followup post to my popular article on learning to write Juju Charms, where we wrote a simple Charm to deploy a production ready Minetest server, complete with postgresql integration through Juju relations.

Today, we are going to go a step further and delve into Reactive Charms, where we can define and maintain state through flags. Flags let us have a memory of events that have happened in the past, and only run certain functions to “react” to changes in those flags.

Reactive Charms are primarily written in Python, and there are a lot of different submodules that exist to help you develop your Charm. So buckle up, because we are going to take our little Minetest Charm to the next level.

Analysis of the dovecat and hy4 Linux Malware

A few days ago, a case came in which had some rather odd symptoms, such as processes using high amounts of CPU and memory, and running from the /tmp directory.

After asking for some logs, and some samples of the binaries, it became obvious that the system was compromised, and was now running some interesting malware.

In this post, we are going to look into the malware called dovecat, which turned out to be a cryptominer, and hy4, which is a IRC botnet malware dropper.

I’m pretty excited, as I haven’t analysed any Linux malware before, and this is real life stuff pulled directly from a production machine, so it still has its fangs intact.

Let’s get started.

Getting DMESG_RESTRICT Enabled in Ubuntu 20.10 Groovy Gorilla

You might have noticed a small change when running the dmesg command in Ubuntu 20.10 Groovy Gorilla, since it now errors out with:

dmesg: read kernel buffer failed: Operation not permitted

Don’t worry, it still works, it has just become a privileged operation, and it works fine with sudo dmesg. But why the change?

Well, I happen to be the one who proposed for this change to be made, and followed up on getting the configuration changes made. This blog post will describe how it slightly improves the security of Ubuntu, and the journey to getting the changes landed in a release.

So stay tuned, and let’s dive into dmesg.