Here are the commands I used after finishing compiling the source code.
──(kali㉿kali-raspberry-pi)-[~/sources/linux]
└─$ ls
arch certs CREDITS Documentation fs init ipc Kconfig lib MAINTAINERS mm modules.builtin.modinfo modules.order net README.md scripts sound tools virt vmlinux.o
block COPYING crypto drivers include io_uring Kbuild kernel LICENSES Makefile modules.builtin modules-only.symvers Module.symvers README samples security System.map usr vmlinux vmlinux.symvers
┌──(kali㉿kali-raspberry-pi)-[~/sources/linux]
└─$ sudo make modules_install
┌──(kali㉿kali-raspberry-pi)-[~/sources/linux]
└─$ sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/
┌──(kali㉿kali-raspberry-pi)-[~/sources/linux]
└─$ sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/
┌──(kali㉿kali-raspberry-pi)-[~/sources/linux]
└─$ sudo cp arch/arm64/boot/dts/overlays/README /boot/overlays/
┌──(kali㉿kali-raspberry-pi)-[~/sources/linux]
└─$ sudo cp arch/arm64/boot/Image.gz /boot/kernel-ryo-1.img
┌──(kali㉿kali-raspberry-pi)-[/boot]
└─$ sudo vim config.txt
# 64-bit kernel for Raspberry Pi 4 is called kernel8l (armv8a)
kernel=kernel-ryo-1.img
┌──(kali㉿kali-raspberry-pi)-[/boot]
└─$ sudo reboot
And I got this!
┌──(kali㉿kali-raspberry-pi)-[/var/log]
└─$ uname -a
Linux kali-raspberry-pi 5.15.83-v8-ryos-first-kernel+ #1 SMP PREEMPT Fri Dec 16 12:25:46 UTC 2022 aarch64 GNU/Linux
┌──(kali㉿kali-raspberry-pi)-[/var/log]
└─$ cat messages
Dec 16 14:43:01 kali-raspberry-pi pulseaudio[968]: ICE I/O error handler called
Dec 16 14:43:04 kali-raspberry-pi kernel: [ 0.000000] Hello World, I'm Ryo!
Dec 16 14:43:04 kali-raspberry-pi kernel: [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
Dec 16 14:43:04 kali-raspberry-pi kernel: [ 0.000000] Linux version 5.15.83-v8-ryos-first-kernel+ (kali@kali-raspberry-pi) (gcc (Debian 11.3.0-5) 11.3.0, GNU ld (GNU Binutils for Debian) 2.38.90.20220713) #1 SMP PREEMPT Fri Dec 16 12:25:46 UTC 2022
This time I was able to print during the boot up with my customized kernel! I can now try to do some work on the network stack and compile😁.
This time, I modified /arm/arm64/kernel/setup.c as below. You can find the source code here.
void __init smp_setup_processor_id(void)
{
u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK;
set_cpu_logical_map(0, mpidr);
pr_info("Hello World, its Ryo!");
pr_info("Booting Linux on physical CPU 0x%010lx [0x%08x]\n",
(unsigned long)mpidr, read_cpuid_id());
}
Here is the commands that I used.
$ KERNEL=kernel8
$ make bcm2711_defconfig
$ vim .config
CONFIG_LOCALVERSION="-v8-ryos-first-kernel"
$ make -j4 Image.gz modules dtbs
It's completed compiling! Here is what I did afterwards.
$ make
...
OBJCOPY arch/arm64/boot/Image
GZIP arch/arm64/boot/Image.gz
[1] + done make
$ ls
arch certs CREDITS Documentation fs init Kbuild kernel LICENSES Makefile modules.builtin modules-only.symvers Module.symvers README scripts sound tools virt vmlinux.o
block COPYING crypto drivers include ipc Kconfig lib MAINTAINERS mm modules.builtin.modinfo modules.order net samples security System.map usr vmlinux vmlinux.symvers
$ file vmlinux
vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=8d74859bebe9cce14587e1b3272c7b11fe5de611, not stripped
$ sudo make modules_install
INSTALL /lib/modules/5.15.44-Re4son-v8l/kernel/sound/usb/misc/snd-ua101.ko
INSTALL /lib/modules/5.15.44-Re4son-v8l/kernel/sound/usb/snd-usb-audio.ko
INSTALL /lib/modules/5.15.44-Re4son-v8l/kernel/sound/usb/snd-usbmidi-lib.ko
DEPMOD /lib/modules/5.15.44-Re4son-v8l
$ sudo make install
sh ./arch/arm64/boot/install.sh 5.15.44-Re4son-v8l \
arch/arm64/boot/Image System.map "/boot"
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.44-Re4son-v8l /boot/vmlinuz-5.15.44-Re4son-v8l
update-initramfs: Generating /boot/initrd.img-5.15.44-Re4son-v8l
cryptsetup: ERROR: Couldn't resolve device /dev/root
cryptsetup: WARNING: Couldn't determine root device
I'm having trouble installing compiled files. I did some research and boot up process is not what I thought as Rasbarry Pi does not support grub😰. I found the kernel source code for Raspberry Pi here. Another mistake that I found is that I was working on linux/arch/arm. My Raspberry Pi is 64bits machine so the directory should be under linux/arch/arm64. Glad I found this mistake before compilation.
I will once again compile the source code and see how it goes😂.
I modified ./arch/arm/kernel/setup.c as below. You can find the source code here.
void __init smp_setup_processor_id(void)
{
int i;
u32 mpidr = is_smp() ? read_cpuid_mpidr() & MPIDR_HWID_BITMASK : 0;
u32 cpu = MPIDR_AFFINITY_LEVEL(mpidr, 0);
cpu_logical_map(0) = cpu;
for (i = 1; i < nr_cpu_ids; ++i)
cpu_logical_map(i) = i == cpu ? 0 : i;
/*
* clear __my_cpu_offset on boot CPU to avoid hang caused by
* using percpu variable early, for example, lockdep will
* access percpu variable inside lock_release
*/
set_my_cpu_offset(0);
pr_info("Hello world, this is Ryo speaking!\n");
pr_info("Booting Linux on physical CPU 0x%x\n", mpidr);
}
After the above modification, I started compiling the source code as below.
$ cp /boot/config-$(uname -r) .config
$ sudo apt-get install git fakeroot build-essential ncurses-dev xz-utils libssl-dev bc flex libelf-dev bison
$ make menuconfig
No change to .config
*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.
$ make
This is going to take a while😝.
I learned that you can't use printk() during the early stage of the boot up process😅. Here is the output of /var/log/messages.
┌──(kali㉿kali-raspberry-pi)-[/var/log]
└─$ cat messages
Dec 11 02:00:25 kali-raspberry-pi rsyslogd: [origin software="rsyslogd" swVersion="8.2206.0" x-pid="525" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Dec 11 02:27:19 kali-raspberry-pi kernel: [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
Dec 11 02:27:19 kali-raspberry-pi kernel: [ 0.000000] Linux version 5.15.44-Re4son-v8l+ (root@kali-raspberry-pi) (gcc (Debian 11.2.0-19) 11.2.0, GNU ld (GNU Binutils for Debian) 2.38) #1 SMP PREEMPT Debian kali-pi (2022-07-03)
Dec 11 02:27:19 kali-raspberry-pi kernel: [ 0.000000] random: crng init done
Dec 11 02:27:19 kali-raspberry-pi kernel: [ 0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.5
I once again gone through the source code and found that "Booting Linux on" is printed by pr_info() and Linux version 5.15.44-Re4son-v8l+ is printed by decompressor_printk(). I will go for pr_info() as it comes earlier than decompressor_printk()😝.
┌──(kali㉿kali-raspberry-pi)-[~/sources/linux-5.15.44]
└─$ grep -r "Booting" .
./arch/arm/kernel/setup.c: pr_info("Booting Linux on physical CPU 0x%x\n", mpidr);
┌──(kali㉿kali-raspberry-pi)-[~/sources/linux-5.15.44]
└─$ grep printk * -r | grep "decompressor_printk"
arch/s390/boot/pgm_check_info.c: decompressor_printk("Linux version %s\n", kernel_version);
I check my Raspberry Pi kernel version and its 5.15.44.
$ uname -r
5.15.44-Re4son-v8l+
I want to printk during the boot up process so I look through the kernel source code where "Booting Linux on..." is printed as shown below.
┌──(kali㉿kali-raspberry-pi)-[/var/log]
└─$ cat messages
Dec 11 02:00:25 kali-raspberry-pi rsyslogd: [origin software="rsyslogd" swVersion="8.2206.0" x-pid="525" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Dec 11 02:27:19 kali-raspberry-pi kernel: [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
Dec 11 02:27:19 kali-raspberry-pi kernel: [ 0.000000] Linux version 5.15.44-Re4son-v8l+ (root@kali-raspberry-pi) (gcc (Debian 11.2.0-19) 11.2.0, GNU ld (GNU Binutils for Debian) 2.38) #1 SMP PREEMPT Debian kali-pi (2022-07-03)
Dec 11 02:27:19 kali-raspberry-pi kernel: [ 0.000000] random: crng init done
Dec 11 02:27:19 kali-raspberry-pi kernel: [ 0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.5
Seems like "Booting Linux on..." is printed by pr_info(), not printk(). Not yet sure where the first printk() is...
You can find the source code for version 5.15.44 here.
Today I inserted a kernel module which simply prints "Hello Ryo!" under /var/log/messages. Here is the c program.
int init_module(void)
{
//printk(KERN_INFO "Hello world 1.\n");
printk(KERN_INFO "Hello Ryo!\n");
return 0;
}
void cleanup_module(void)
{
//printk(KERN_INFO "Goodbye world 1.\n");
printk(KERN_INFO "Bye Ryo!\n");
}
MODULE_LICENSE("GPL");
initi_module() is used to initialize the module and cleanup_module is responsible for shutting down the module. When you are using printk(), you can give eight 8 levels of different logs at the part of "KERN_INFO". Below is the commands I used for initializing and removing the kernel.
make -C /lib/modules/`uname -r`/build/ M=${PWD} modules
sudo insmod hello_world.ko
sudo rmmod hello_world.ko
Here is the output of the above operation.
Dec 12 14:21:27 kali-raspberry-pi kernel: [ 8233.939103] Hello Ryo!
Dec 12 14:22:33 kali-raspberry-pi kernel: [ 8299.742310] Bye Ryo!
I will start looking into where I can printk() inside the kernel source code during the boot up process!
I've been working on TUN device for a while and thought I should look into how packets are handled within the kernel. TUN device allows us to handle packets above layer 3. The corresponding entry point within the kernel is ip_rcv() whih can be found here. ip_local_deliver_finish() is the function whichis the exit of the layer 3 responsible for passing data to the layer 4 stack based on the protocol found in the ip header. You can find the function https://elixir.bootlin.com/linux/v2.6.20/source/net/ipv4/ip_input.c#L199. After ip_local_deliver_finish(), the data is sent to the function handling layer 4 protocol such as tcp_v4_rcv(), udp_rcv(), icmp_rcv(). Might be fun to customize your own network stack and make a kernel of your own😝.