You might have noticed a small change when running the
dmesg command in
Ubuntu 20.10 Groovy Gorilla, since it now errors out with:
dmesg: read kernel buffer failed: Operation not permitted
Don’t worry, it still works, it has just become a privileged operation, and it
works fine with
sudo dmesg. But why the change?
Well, I happen to be the one who proposed for this change to be made, and followed up on getting the configuration changes made. This blog post will describe how it slightly improves the security of Ubuntu, and the journey to getting the changes landed in a release.
So stay tuned, and let’s dive into
What is dmesg?
dmesg is a command that allows you to view the kernel log buffer. The kernel
log buffer contains a whole wealth of information about system hardware, devices
attached and their allocated memory regions, and error logging for the system.
This log buffer usually lives at
/proc/kmsg, which is what
journalctl or various
syslog programs read from.
If we look at some typical start-up information, it really isn’t too interesting.
Why is restricting dmesg important?
The thing is, the kernel log buffer can sometimes contain all sorts of security
critical information, such as pointers to kernel memory. There has been a large
effort in the mainline kernel for a few years now to remove all instances of
printk("%p"), which leaked raw kernel pointers to the kernel log buffer.
These days, all
%p format strings hash the kernel pointer, so the address
itself is not leaked, but still gives a unique identifier for developers to
look at when doing
However, kernel pointers can still be leaked in other ways, such as if the system suffers an oops, it will print the current kernel stacktrace, as well as provide a copy of register values:
[3191370.893495] WARNING: CPU: 13 PID: 48929 at /build/linux-hwe-FEhT7y/linux-hwe-4.15.0/include/linux/mm.h:852 follow_page_pte+0x6f4/0x710 [3191370.893552] CPU: 13 PID: 48929 Comm: CPU 0/KVM Not tainted 4.15.0-106-generic #107~16.04.1-Ubuntu [3191370.893552] Hardware name: Dell Inc. PowerEdge R740xd/00WGD1, BIOS 2.6.4 04/09/2020 [3191370.893554] RIP: 0010:follow_page_pte+0x6f4/0x710 [3191370.893555] RSP: 0018:ffffad279f7ab908 EFLAGS: 00010286 [3191370.893556] RAX: ffffdc0fa72eba80 RBX: ffffdc0f9b1535b0 RCX: 0000000080000000 [3191370.893556] RDX: 0000000000000000 RSI: 00003ffffffff000 RDI: 800000b9cbaea225 [3191370.893557] RBP: ffffad279f7ab970 R08: 800000b9cbaea225 R09: ffff9359857fd5f0 [3191370.893558] R10: 0000000000000000 R11: 0000000000000000 R12: ffffdc0fa72eba80 [3191370.893558] R13: 0000000000000326 R14: ffff935de09e19e0 R15: ffff9359857fd5f0 [3191370.893559] FS: 00007f68757fa700(0000) GS:ffff93617ef80000(0000) knlGS:ffff964a7fc00000 [3191370.893559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [3191370.893560] CR2: 00007ff92ca7a000 CR3: 000000b7209d2005 CR4: 00000000007626e0 [3191370.893561] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [3191370.893561] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [3191370.893561] PKRU: 55555554 [3191370.893562] Call Trace: [3191370.893565] follow_pmd_mask+0x273/0x630 [3191370.893567] ? gup_pgd_range+0x23f/0xde0 [3191370.893568] follow_page_mask+0x178/0x230 [3191370.893569] __get_user_pages+0xb8/0x740 [3191370.893571] get_user_pages+0x42/0x50 [3191370.893604] __gfn_to_pfn_memslot+0x18b/0x3b0 [kvm] [3191370.893615] ? mmu_set_spte+0x1dd/0x3a0 [kvm] [3191370.893626] try_async_pf+0x66/0x220 [kvm] [3191370.893635] tdp_page_fault+0x14b/0x2b0 [kvm] [3191370.893640] ? vmexit_fill_RSB+0x10/0x40 [kvm_intel] [3191370.893649] kvm_mmu_page_fault+0x62/0x180 [kvm] [3191370.893651] handle_ept_violation+0xbc/0x160 [kvm_intel] [3191370.893654] vmx_handle_exit+0xa5/0x580 [kvm_intel] [3191370.893664] vcpu_enter_guest+0x414/0x1260 [kvm] [3191370.893674] kvm_arch_vcpu_ioctl_run+0xd9/0x3d0 [kvm] [3191370.893683] ? kvm_arch_vcpu_ioctl_run+0xd9/0x3d0 [kvm] [3191370.893691] kvm_vcpu_ioctl+0x33a/0x610 [kvm] [3191370.893693] ? audit_filter_rules+0x232/0x1070 [3191370.893696] do_vfs_ioctl+0xa4/0x600 [3191370.893697] ? __audit_syscall_entry+0xac/0x100 [3191370.893699] ? syscall_trace_enter+0x1d6/0x2f0 [3191370.893700] SyS_ioctl+0x79/0x90 [3191370.893701] do_syscall_64+0x73/0x130 [3191370.893704] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
If kernel pointers happen to be in the registers at the time of oops, they get leaked to the kernel log buffer.
Kernel pointers are valuable to attackers and exploit developers, because they act as information leaks. These information leaks make it much easier to de-randomise the kernel base address and to defeat KASLR. If an attacker is trying to launch a privilege escalation attack against a recently compromised host, they can also use dmesg to get instant feedback on their exploits, as failures will cause further oops messages or segmentation faults. This makes it easier for attackers to fix and tune their exploit programs until they work.
Currently, if I create a new, unprivileged user on a Focal system, they cannot
/var/log/syslog or see system events in
But yet, they are given free reign to the kernel log buffer.
$ sudo adduser dave $ su dave $ groups dave $ cat /var/log/kern.log cat: /var/log/kern.log: Permission denied $ cat /var/log/syslog cat: /var/log/syslog: Permission denied $ journalctl Hint: You are currently not seeing messages from other users and the system. Users in groups 'adm', 'systemd-journal' can see all messages. Pass -q to turn off this notice. Jun 16 23:44:59 ubuntu systemd: Reached target Main User Target. Jun 16 23:44:59 ubuntu systemd: Startup finished in 69ms. $ dmesg [ 0.000000] Linux version 5.4.0-34-generic (buildd at lcy01-amd64-014) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #38-Ubuntu SMP Mon May 25 15:46:55 UTC 2020 (Ubuntu 5.4.0-34.38-generic 5.4.41) [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-34-generic root=UUID=f9f909c3-782a-43c2-a59d-c789656b4188 ro
Strange how an unprivileged user can read dmesg just fine, and yet cannot access any other kernel logs on the system.
The Initial Proposal
I sent a proposal to
ubuntu-devel in June which outlines the above problems,
to gather some feedback and to see if anyone else thinks that this is a good
I suggested that we restrict access to dmesg to users in group ‘adm’ like so:
CONFIG_SECURITY_DMESG_RESTRICT=yin the kernel.
- Following changes to
/bin/dmesgpermissions in package
- Ownership changes to
- Permissions changed to
cap_syslogcapability to binary.
- Ownership changes to
- Add a commented out
# kernel.dmesg_restrict = 0to
Let’s break these down.
Number 1 is how
DMESG_RESTRICT gets enforced, as setting
in the kernel config restricts the kernel log buffer to executables with
CAP_SYSLOG, or root privileges.
Number 2 allows users in the
adm group, also known as “administration”, to
be able to execute dmesg without becoming super user, which means nothing
would change for default users in most systems.
Number 3 adds a easy way for system administrators to disable the change if they want.
I filed a Launchpad bug to document the changes and track the patches I had
Early Responses and Getting the Kernel Config Changed (1)
The security team were +1 with the change:
When I woke up the next day, the strangest thing happened. Phoronix had written an article about my proposal!
This wasn’t expected at all, and it got people talking about the change in forums, instead of it just being silently made and me hoping that no one noticed.
After that, Seth Forshee, from the kernel team, double checked with the security team, and then went ahead and applied the change to the “unstable” kernel tree, since Groovy’s kernel had not yet forked off from it at that point in time.
The kernel commit is:
Commit 25e6c851704a47c81e78e1a82530ac4b328098a6 From: Seth Forshee <[email protected]> Date: Thu, 2 Jul 2020 13:29:55 -0500 Subject: UBUNTU: [Config] CONFIG_SECURITY_DMESG_RESTRICT=y Link: https://kernel.ubuntu.com/git/ubuntu/unstable.git/commit/?id=25e6c851704a47c81e78e1a82530ac4b328098a6
Now that the configuration change was made in the kernel, Number 1 in the list was completed.
Upstream Discussions for Adding CAP_SYSLOG to /bin/dmesg (2)
At this point, things got a bit stuck. I got busy and no one else replied to my
previous posts, so the changes to
util-linux got a little delayed.
I restarted these talks with the below message to
ubuntu-devel, and included
the upstream Debian maintainers to the CC list.
This was successful, and Chris Hofstaedtler, wrote back. Chris asked if this had been discussed before in Debian:
I responded with what I could find, but I also mentioned that I would write
So, I went and proposed similar changes to
debian-devel in this thread:
I got some positive responses, but the most interesting one was from Ansgar:
Ansgar mentioned that if
/bin/dmesg is granted
was opened up to users of group
adm, then any user of
adm could clear the
kernel log buffer by running
$ dmesg --clear.
Now, I had missed this, and it was an excellent catch.
We don’t want to make it easier for anyone to clear the kernel log buffer, since
it can be used to hide an attackers presence, so adding
is a bad idea.
Chris mentions this in his message back:
From there, Steve Langasek also mentioned that it was a bad idea:
and with that, I decided to drop the idea of adding
and changing the group to
That makes Number 2 on the list struck off. It’s a bit of a pity, since it means
that users in group
adm have to write
$ sudo dmesg instead of
Hopefully it won’t be too much of a bother to become superuser to view dmesg.
Time will tell I suppose, and most distros follow this behaviour anyway.
Landing sysctl Configuration Changes (3)
Shortly after the upstream
util-linux discussion ended, Brian Murrary sponsored
my patches to
procps to add some documentation about
and instructions on how to disable it by changing a sysctl variable.
As my description states, if you want to turn off
DMESG_RESTRICT, you can
do so by uncommenting the sysctl string
kernel.dmesg_restrict = 0, and
With this, Number 3 in the list was completed.
That is the story of how
DMESG_RESTRICT was enabled in Ubuntu 20.10 Groovy
Gorilla. We covered how it slightly improves system security by removing an avenue
attackers could use to view leaked kernel pointers, the process of getting all
the separate changes landed, and relevant upstream discussions.
I hope you enjoyed the read, and if you have any questions or comments, feel free to contact me.