Lab2: Load Kernel Image

Posted on Edited on In

Rpi3 開機流程

Rpi3’s booting procedure could be roughly divided into 4 steps.

  1. GPU executes the first stage bootloader from ROM on the SoC.
  2. The first stage bootloader recognizes the FAT16/32 file system and loads the second stage bootloader bootcode.bin from SD card to L2 cache.
  3. bootcode.bin initializes SDRAM and loads start.elf
  4. start.elf initializes GPU’s firmware and reads config.txt, cmdline.txt and kernel8.img to start OS.

x86 開機流程

  1. Pre-Pre-Boot

    1. hardware waits for the power supply to settle to its nominal state.
    2. power is sequenced by controlling analog switches
    • Cache, MMU are disabled.
    • CPU in Real Mode. (Addressable memory: 1MB)
    • RAM is empty.

  2. BIOS

    1. Executing BIOS code
    2. POST, load configuration (ex. boot order)
    3. Load Stage 1 Bootloader using the specified boot order

  3. Stage 1 Bootloader

    1. Enable address A20
    2. Switch to 32-bit protected mode (Real mode->Protected mode)
    3. Setup the stack
    4. Load the Stage 2 Bootloader to navigate the file system
    • MBR Keeps executable code and 4-entry partition table.
    • MBR implements the Stage 1 Bootloader which can be used to load the operating system.

  4. Stage 2 Bootloader

    1. Read configuration file (grub.conf in GRUB)
    2. Load kernel initial image (vmlinuz-*) in memory using BIOS disk I/O services

  5. Kernel Startup

    Takes control of and initializes the machine

  6. Init

    base environment initialization

  7. Runlevels/Targets

    Intializes the user environment

Ref: x86 Initial Boot Sequence, 即將換掉傳統 BIOS 的 UEFI，你懂了嗎？（二）

x86 開機流程 (UEFI)

  • Skip Stage 1 Bootloader
  • UEFI boot manager takes control right after the system is powered on
  • Startup files are stored on a dedicated EFI System Partition (ESP)

3rd bootloader

要做一個能載入kernel的bootloader，鑑於kernel可能載入到任何記憶體位置，bootloader要在kernel載入前將自己從0x80000複製到不會被kernel覆蓋的地方。

Spec說要任意指定記憶體位置，理想的流程應該是

  1. 初始化shell
  2. 從使用者取得記憶體位置和kernel size
  3. 如果kernel和bootloader在記憶體上重疊，先搬移bootloader至kernel前方8192bytes(假定bootloader不超過8192bytes)
  4. 開始接收kernel，並驗證正確性
  5. 跳至kernel執行位置

在編譯時觀察bootloader大小

1
aarch64-linux-gnu-readelf -s kernel8.elf | grep _bss_end

得到的記憶體位置減去0x80000就是bootloader大小，例如0000000000081c90代表7312 bytes

Ref: Tutorial 14 - Raspbootin64

1
2
extern char __loader_size;
unsigned long loader_size = (unsigned long)&__loader_size;

存取linker script定義的symbol，e.g.__loader_size

  1. 在程式碼中以extern宣告該symbol，型態為char
  2. symbol只是一個地址，所以存取就要取地址

Ref: Linker Script初探 - GNU Linker Ld手冊略讀

在搬移bootloader後，使用GNU extension的computed goto來計算跳轉的記憶體位置。並預期載入kernel後直接跳至kernel，因此原有的stack就不特別複製，將stack pointer直接移到bootloader前，接收完成後用blr跳轉至kernel，由kernel重新設置stack pointer。

1
2
3
asm volatile("mov sp, %0" : : "r"(__new_start));
goto *(&&_receive_kernel + ((unsigned long)kernel_start - 8192 -
                              0x80000)); // Calculate new address

也因為這樣，接收kernel的部份不能使用到處於stack的變數，畢竟stack是空的，有兩種作法

  1. 以register宣告變數
  2. 特別劃分一塊記憶體當作變數

為了方便採用前者，後續實作checksum時才會採用後者。

Ref: 你所不知道的 C 語言: goto 和流程控制篇、ARM Compiler armasm Reference - BLR、ARM GCC Inline Assembler Cookbook

linker.ld

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
SECTIONS
{
  . = 0x80000;
  PROVIDE(_code = .);
  .text : {
    KEEP(*(.text.boot))
    *(.text .text.*)
  }
  .data : {
    *(.data .data.*)
  }
  .bss (NOLOAD) : {
      . = ALIGN(16);
      __bss_start = .;
      *(.bss .bss.*)
      __bss_end = .;
  }
  _end = .;
}
__bss_size = (__bss_end - __bss_start)>>3;
__loader_size = (_end - _code)>>3;

start.s

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
.section ".text.boot"

.global _start

_start:
    // read cpu id, stop slave cores
    mrs     x1, mpidr_el1
    and     x1, x1, #3
    cbz     x1, 2f
    // cpu id > 0, stop
1:  wfe
    b       1b
2:  // cpu id == 0

    // set top of stack just before our code (stack grows to a lower address per AAPCS64)
    ldr     x1, =_start
    mov     sp, x1

    // clear bss
    ldr     x1, =__bss_start
    ldr     w2, =__bss_size
3:  cbz     w2, 4f
    str     xzr, [x1], #8
    sub     w2, w2, #1
    cbnz    w2, 3b

    // jump to C code, should not return
4:  bl      main
    // for failsafe, halt this core too
    b       1b

shell.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
void loadimg() {
  unsigned char c;
  int load_address;
  register int kernel_size = 0;

  uart_printf("Start loading kernel image...\n");
  uart_printf("Please input 32-bit kernel load address (default: 80000): ");

  // Input load address
  char buf[BUFFER_MAX_SIZE];
  int idx = 0;
  while ((c = uart_getc())) {
    uart_printf("%c", c);
    if (c == LINEFEED)
      break;
    buf[idx++] = c;
  }
  buf[idx] = 0;
  if (!idx) {
    load_address = 0x80000;
  } else if (htoi(buf, &load_address) == -1)
    goto _error;
  load_address =
      ALIGN_uint(load_address, 16); // align load address to a 16-byte boundary

  uart_printf("Please input kernel size: ");

  // Input kernel size in little-endian order
  for (int i = 0; i < 4; ++i)
    kernel_size |= (((int)uart_getc2()) << (i << 3));
  if (kernel_size <= 0)
    goto _error;

  uart_printf("%d\n", kernel_size);
  uart_printf("Kernel image size: %d bytes, load address: 0x%h\n",
              kernel_size, load_address);

  // Relocate bootloader if needed
  extern char __loader_size;
  unsigned long loader_size = (unsigned long)&__loader_size;
  register char *kernel_start = (char *)(unsigned long)load_address,
                *__new_start = kernel_start - 8192;
  if ((unsigned long)kernel_start < (0x80000 + loader_size) &&
      (unsigned long)(kernel_start + kernel_size) > 0x80000) {
    // Relocate bootloader. Assume bootloader size is less than 8192 bytes
    uart_printf("Relocating bootloader from 0x%h to 0x%h\n", 0x80000,
                (int)(unsigned long)__new_start);
    char *src = (char *)0x80000, *dst = __new_start;
    while (loader_size--)
      *dst++ = *src++;
    asm volatile("mov sp, %0" : : "r"(__new_start));
    goto *(&&_receive_kernel + ((unsigned long)kernel_start - 8192 -
                                0x80000)); // Calculate new address
  }

_receive_kernel:
  uart_printf("Please send kernel from UART now...\n");

  // Receive kernel data and validate it
  register int remain = kernel_size;
  while (remain--) {
    c = uart_getc2();
    *kernel_start++ = c;
    if (!(remain % 100))
      uart_printf("\r%d%%", 100*(kernel_size - remain)/kernel_size);
  }

  uart_printf("\nSuccess! Load kernel...\n");
  asm volatile("blr %0" : : "r"(__new_start + 8192));

_error:
  // Shoult not reach here
  uart_printf("Abort!\n");
}

upload_kernel.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
import serial
import sys
import numpy as np
import argparse
from time import sleep

parser = argparse.ArgumentParser()
parser.add_argument("--dev", help="serial device")
parser.add_argument("--rate", help="baudrate, default: 115200")
parser.add_argument("--kernel", help="kernel image")
parser.add_argument("--addr", help="load addressm default: 80000")
args = parser.parse_args()

def main():
    if args.dev and args.kernel:
        dev = args.dev
        kernel = args.kernel
        rate = 115200
        addr = "80000"
        if args.rate:
            rate=int(args.rate)
        if args.addr:
            addr = args.addr
        print(f"Serial device: {dev}")
        print(f"Baudrate: {rate}")
        print(f"Kernel image: {kernel}")
        print(f"Load address: {addr}")
    else:
        print("Insufficient arguments!")
        exit(1)

    try:
        ser = serial.Serial(dev, rate)
    except:
        print("Cannot open device")
        exit(1)

    try:
        with open(kernel, 'rb') as k:
            kernel_data = k.read()
        kernel_size = len(kernel_data)
    except(FileNotFoundError):
        print("File not found")
        exit(1)

    print(f"Image size: {kernel_size} bytes, load address: 0x{addr}")
    ser.write(str.encode("loadimg\n"))
    ser.write(str.encode(addr + "\n"))
    ser.write(kernel_size.to_bytes(4, byteorder = 'little'))

    chunk_size = 16
    chunk_count = (kernel_size + chunk_size - 1) // chunk_size
    for i in range(chunk_count):
        sys.stdout.write('\r')
        sys.stdout.write("%d/%d" % (i+1, chunk_count))
        sys.stdout.flush()
        ser.write(kernel_data[i*chunk_size: (i+1)*chunk_size])
    print("...done!")

if __name__ == "__main__":
    main()

傳輸時刻意加入延遲以展示上傳的進度

Ref: kaiiiz/osdi2020

Hamming Code & Checksum

ARMv8雖然有crypto extension，實際功能視授權而定。BCM2837經過JTAG實測沒有SHA1, SHA2但有CRC，一般使用者也能用lscpu觀察cpu支援功能，這裡就使用CRC作為checksum。

也有不在文件上的功能也能使用，像是理應不存在的PRNG在raspi3-tutorial/06_random就被挖掘出來，但SHA就是真的不支援。

https://www.youtube.com/watch?v=fgHaBP9s7t4

Ref: Bare metal rpi3 programming

[Q] Calculate how long will it take for loading a 10MB kernel image by UART if baud rate is 115200.

理想狀況：10MB/(115200/8)=694 sec

加入checksum及hamming code：10MB/(115200/8)=694 sec