Adding a System Call to Linux (4.x)

This semester we have an operating systems course, which requires us to compile the linux kernel and add a system call. However, the tutorial our teacher provided is based on the 2.x kernel; with newer GCC versions, compiling that kernel is basically impossible.

Also, replacing the kernel directly on the host system could cause trouble for users, so none of the operations in this tutorial require modifying the original system. All system call testing is done inside the qemu virtual machine.

If you’re not familiar with Linux, you can use the gedit editor to create and edit files.

The important host environment versions are listed below.

• Kernel version to compile: 4.10.3 (stable release of 2017-03-15)
• Host Linux version: 4.10.2-1-ARCH
• Compiler version: gcc 6.3.1
• qemu version: 2.8.0
• busybox version: 1.26

This tutorial has a limited shelf life. If the future Linux kernel structure differs from this, do not follow it, or contact me to take the tutorial down, so it won’t mislead anyone.

Compiling the Linux Kernel

Downloading the Kernel

• https://www.kernel.org

• Extract after downloading

xz -d linux-4.10.3.tar.xz
tar -xf linux-4.10.3.tar

cd linux-4.10.3         # "kernel directory" in the rest of this article refers to this directory

• Compile and boot

If you use Ubuntu, any problem during compilation can be solved by simply searching online.

make menuconfig         # defaults are fine; trimming options speeds up compilation
make -j4                # plain make is very slow — it uses only 1 core by default

It took me about 15 minutes (CPU: i5-4200M).

Booting Our Compiled Kernel with qemu

What we just compiled is only a linux kernel. It is very powerful, but a kernel alone cannot be called a system. Here we use busybox to build a root filesystem. To make things easy, I provide a ready-made system image here.

Download it here: world.img

• Basic qemu usage (we are in the kernel directory now)

qemu-system-x86_64 -kernel arch/x86_64/boot/bzImage \
        -m 256  \
        -hda ../world.img  \
        -append "root=/dev/sda rdinit=sbin/init noapic"

• Explanation of the parameters

-kernel arch/x86_64/boot/bzImage                # specify the kernel
-m 256                                          # specify 256M of memory
-hda ../world.img                               # use this file as the hda disk
-append "root=/dev/sda rdinit=sbin/init noapic" # kernel boot parameters, defining the root filesystem

After a successful boot it looks like this — a minimal linux system is now set up.

Adding the System Call

Defining the New System Call

• Make sure you are in the kernel directory

mkdir $KERNEL/hello
cd $KERNEL/hello

• Create the file hello.c yourself; its content is shown below (view files with cat)

version_changed: 2019-08-30 Thanks to a junior student for the correction: defining sys_hello in hello.c is problematic. The SYSCALL_DEFINE form should be used instead. This issue is related to CVE-2009-0029; I haven’t studied it deeply. Still, custom system calls had better stay consistent with the kernel’s other system calls.

Please change asmlinkage long sys_hello(int arg0) in the code below to SYSCALL_DEFINE1(hello, arg0)

Adding It to the System Call Table

• Modify syscall_64.tbl

In 4.10.3 this file is at arch/x86/entry/syscalls/syscall_64.tbl.

We add our own system call at the end of the file.

• Modify syscalls.h

In 4.10.3 this file is at include/linux/syscalls.h.

Again we add our system call at the end.

Adding It to the Kernel Build Makefile

• Create a Makefile in the hello directory with the following content

• Modify the global main Makefile

The global main Makefile is the Makefile in the kernel directory, around line 900-something, with this line:

core-y		+= kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/

Now we change it to the following, i.e. adding the hello folder:

core-y		+= kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/ hello/

Recompiling the Kernel

After the work above you can recompile the kernel: just make -j4, no need for make menuconfig. Since the previous compilation left intermediate files, this would normally be fast. Unfortunately, after adding a system call basically the whole kernel must be recompiled — please be patient.

Testing Our New System Call

For a program that uses the system call, we first need to put the program into the root filesystem so it can be executed inside the qemu virtual machine. Therefore, we first mount the root filesystem to a local directory, then add the compiled test program to the root filesystem — that is, copy it into that directory.

Mounting the world.img File on the Host

mkdir world_mount                 # new folder; the program will be copied here
sudo mount world.img world_mount  # mount the image to the folder

As shown below:

Writing a Program That Uses the System Call

• The system call program

The system call number we added is 551, and it takes one int argument, so the function here is syscall(551, 24). If in doubt, refer to the earlier hello.c file — comparing the two makes it easier to understand. The call.c file is shown below.

#include "stdio.h"
#include "unistd.h"
#include "linux/kernel.h"
#include "sys/syscall.h"
#include "errno.h"

int main(int argc, char *argv[])
{
    int ret = syscall(551, 24);
    if (ret != -1) {
        printf("The systemcall %d has been used\n", ret);
    } else {
        printf("Error occured %d\n", errno);
    }

    return 0;
}

• After saving, compile directly

gcc -o call call.c -static

• Copy the program into the root filesystem

sudo cp call world_mount/

• Boot the system with qemu

qemu-system-x86_64 -kernel arch/x86_64/boot/bzImage \
        -m 256  \
        -hda ../world.img  \
        -append "root=/dev/sda rdinit=sbin/init noapic"

The booted system looks like the figure below. Run the program with ./call. Since our system call uses the printk function, the kernel outputs the message, which can be viewed with dmesg — and there you’ll see the printed content.

• The final directory structure looks roughly like this

.
├── call
├── call.c
├── linux-4.10.3
├── linux-4.10.3.tar
├── world.img
└── world_mount

2 directories, 4 files

References

• Implementing a system call in Linux Kernel 4.7.1

• Debugging the linux kernel with qemu+gdb

• Setting up a linux kernel debugging environment

LICENSE:

Copyright © 2016 corvo. Commercial use without permission is prohibited. Please credit the source when reposting.