Model Me, Model World

Blayground

RISC-V III Build Toolchain and Linux

After trying a prebuilt image on Zynq FPGA boards, it's high time to compile the full software stack from scratch so as to gain insights into the RISC-V system.

Setup code base and environment

First, fetch rocket-chip, using following shell commands:

#!/bin/sh
# rocket-chip
git clone https://github.com/ucb-bar/rocket-chip.git
cd rocket-chip
git submodule update --init
cd riscv-tools
git submodule update --init --recursive riscv-tests
# go and have a sweet dream

After that, build the tool sets resides in riscv-tools folder. Fire up the script below:

# riscv toolchain
## Dependencies:
## autoconf automake autotools-dev curl
## libmpc-dev libmpfr-dev libgmp-dev gawk
## build-essential bison flex texinfo gperf

## see notes below, don't clone riscv-tools alone
# git clone https://github.com/ucb-bar/riscv-tools.git
# cd riscv-tools
# git submodule update --init --recursive

# add to .bashrc, maybe
export RISCV=~/riscv-toolchain
./build.sh
# go and have a meal
# add to .bashrc, maybe
export PATH=$PATH:$RISCV/bin

Run a test to verify if the toolchain is working:

# test riscv-gcc and spike
vim hello.c
riscv-gcc hello.c -o hello
spike pk hello
##################################
# WARN: as of 2014/12/10
# the latest commit seems to be broken
# when perform
# spike pk hello
# it utters error: Misaligned ....
##################################

The commit of riscv-tools which rocket-chip used, is tested and valid. Newest commit of riscv-tools may be broken. So it's recommended that not to clone a riscv-tools separately. Please use the repository included in the rocket-chip , which points to the right commit.

Update: As ccelio mentioned, this error is fixed by commit @44b1bee

Toolsets included

emulator: C simulator Check if /bin/sh links to /bin/bash. In Ubuntu, it links to dash. The script would fail miserablely. After make -j8 run, one could examine some *.out files in the directory named output. These files are tests performed on the generated. $ROCKETCHIP/emulator/generated-src contains Chisel generated C++ code vsim: Verilog code for simulation on VCS(May be Synopsys is required?). fsim: FPGA-synthesizable Verilog vsim: VLSI-synthesizable Verilog

Chisel code structure*

This part is not elaborated in the publicized documents, several questions still persists. HostIO More details? MemIO Connected to where? How to connect to on-chip DDR? One could post questions to the hw-dev mailing list.

Also, a dive into Makefile may provide much insight into the generation process. If time permits, this topic will be covered in later posts.

Parameterized chip configuration*

It only covered a little on this topic. As the configuration file is written is Scala, which seems an alien language to me.

Compilation and C library concepts*

C library ref link The SYSROOT Concept ref link

We now must instruct the riscv-linux-gcc build process to place our system root files in a place we can get to them. We call this directory SYSROOT. Let's set our $SYSROOT environment variable for easy access throughout the build process. Ensure that this directory is not inside our $RISCV variable.

# define sysroot dir
export RISCVSYSROOT=$RISCV/../riscv-sysroot/

# copy headers
mkdir -p $SYSROOT/usr
cp -r $ROCKETCHIP/riscv-tools/riscv-gcc/linux-headers/* $RISCVSYSROOT/usr

In the event that the kernel headers (anything inside arch/riscv/include/asm/ or in include/ are changed, you can use the Linux kernel Makefile to generate a new set of Linux headers - see here.)

# cd to riscv-gcc and apply patch
curl -L http://riscv.org/install-guides/sysroot-Makefile.in.patch | patch –p1
./configure
make linux INSTALL_DIR=$RISCV SYSROOT=$SYSROOT

Obtaining and Patching the Kernel Sources

Obtaining kernel patch.

cd ~
git clone git@github.com:ucb-bar/riscv-linux.git linux-3.14.17
# git checkout 84ee1c6
# see below

Download the current minor revision of the 3.14 Linux kernel from The Linux Kernel Archives. The -k switch ensures that.gitignore and README files don't get clobbered.

curl -L ftp://ftp.kernel.org/pub/linux/kernel/v3.x/linux-3.14.17.tar.xz | tar -xJk
make ARCH=riscv defconfig
make ARCH=riscv menuconfig
make -j8 ARCH=riscv

Newest commit@863369213f as of writing seems to be broken with kernel 3.14.17 source, compilation terminated showing these errors.

    arch/riscv/kernel/head_64.S: Assembler messages:
    arch/riscv/kernel/head_64.S:81: Error: bad expression
    arch/riscv/kernel/head_64.S:81: Error: illegal operands `auipc t0,%pcrel_hi(1f)'
    arch/riscv/kernel/head_64.S:83: Error: bad expression
    arch/riscv/kernel/head_64.S:83: Error: illegal operands `jr t0,%pcrel_lo(1b)'
    make[1]: *** [arch/riscv/kernel/head_64.o] Error 1
    make: *** [arch/riscv/kernel] Error 2

Checkout out to commit@84ee1c6, everything went by smoothly.

Build BusyBox

curl -L http://busybox.net/downloads/busybox-1.21.1.tar.bz2 | tar –xj
cd busybox-1.21.1
make allnoconfig

CONFIG_CROSS_COMPILER_PREFIX=riscv-linux-, listed as "Cross Compiler prefix" in BusyBox Settings → Build Options

Pack a rootfs

cd into kernel source directory and create a root image

# produce a file with 64MiB zero
dd if=/dev/zero of=root.bin bs=1M count=64
# make it a ext2 filesystem
mkfs.ext2 -F root.bin

# mount the disk image to modify it
mkdir mnt

Following commands requires superuser permission, please understands what they mean, and execute them carefullly.

mount -o loop root.bin mnt # modern kernels usually supports /dev/loop devices

# make sys dirs
cd mnt && mkdir -p bin etc dev lib proc sbin tmp usr usr/bin usr/lib usr/sbin

# copy BusyBox into the mounted fs
sudo cp ../../busybox-1.21.1/busybox bin

Create initialization table /etc/inittab

::sysinit:/bin/busybox mount -t proc proc /proc
::sysinit:/bin/busybox mount -t tmpfs tmpfs /tmp
::sysinit:/bin/busybox mount -o remount,rw /dev/htifbd0 /
::sysinit:/bin/busybox --install -s
/dev/console::sysinit:-/bin/ash

Explanation ref link Warn: comment out line 4 after first boot, symlinks created at the first boot are persist in the filesystem. Or some harmless warning would pop out during the boot procedure.

# NOTE: this symlink refers to relative path, 山口山
ln -s ../bin/busybox sbin/init

# finishing up
cd ..
umount mnt

Return to user permission shell run spike +disk=root.bin vmlinux

Appendix

C library back Newlib is a "C library intended for use on embedded systems." It has the advantage of not having so much cruft as Glibc at the obvious cost of incomplete support (and idiosyncratic behavior) in the fringes. The porting process is much less complex than that of Glibc because you only have to fill in a few stubs of glue code.

These stubs of code include the system calls that are supposed to call into the operating system you're running on. Because there's no operating system proper, the simulator runs, on top of it, a proxy kernel (riscv-pk) to handle many system calls, like open, close, and printf.

The SYSROOT Concept* back

When installing these toolchains, the make system often generates a wide variety of libraries and other files. In particular, building Glibc involves building the run-time dynamic linker and the C standard library (ld.so.1 and libc.so.6, in this case). These, together with header files like stdio.h, comprise the system root, an often-necessary set of files for a fully operational system.

When we built riscv-tools, there was no need for specifying where to install these files, because we assumed we would always be running on the host machine through a simulator; all of the libraries are on the host system. Now that we're running our binaries from within an operating system, we will have to provide these libraries and headers if we want to run dynamically-linked binaries and compile programs natively.

The inittab file back

This script first mounts procfs tmpfs filesystem onto /proc and /tmp. HTIF-virtualized block device (htifbd) is mounted onto root. Then, BusyBox applet symbolic links was installed. Finally, an ash shell on the HTIF-virtualized TTY (console, mapped to ttyHTIF) is opened. (What’s this?) If getty is desired change the 5th line to ::respawn:/bin/busybox getty 38400 ttyHTIF0.

Makefile patch

The patch for Makefile is pasted here, as a reminder, in case of lost or broken link.

diff --git a/Makefile.in b/Makefile.in

index e1744c2..85ec2d1 100755
--- a/Makefile.in
+++ b/Makefile.in
@@ -90,6 +90,7 @@ build-binutils-linux:
 		--enable-tls \
 		--enable-languages=c \
 		--with-newlib \
+		--with-sysroot=$(SYSROOT) \
 		--disable-multilib
 	$(MAKE) -C $@ -j $(MAKE_JOBS)
 	$(MAKE) -C $@ -j $(MAKE_JOBS) install
@@ -121,16 +122,16 @@ build-glibc-linux: build-gcc-linux-stage1
 	cd $@ && $(srcdir)/glibc-$(GLIBC_VERSION)/configure \
 		riscv-linux \
 		--target=riscv-linux \
-		--prefix=$(INSTALL_DIR)/riscv-linux \
+		--prefix=/usr \
 		libc_cv_forced_unwind=yes \
 		libc_cv_c_cleanup=yes \
 		--enable-shared \
 		--enable-__thread \
 		--disable-multilib \
 		--enable-add-ons=nptl \
-		--with-headers=$(LINUX_DIR)/include
+		--with-headers=$(SYSROOT)/usr/include
 	$(MAKE) -C $@ -j $(MAKE_JOBS) cross-compiling=yes
-	$(MAKE) -C $@ -j $(MAKE_JOBS) install cross-compiling=yes
+	$(MAKE) -C $@ -j $(MAKE_JOBS) install cross-compiling=yes install_root=$(SYSROOT)

 build-gcc-linux-stage2: build-glibc-linux remove-native-libs
 	rm -rf $@
@@ -149,7 +150,7 @@ build-gcc-linux-stage2: build-glibc-linux remove-native-libs
 		--disable-nls \
 		--disable-multilib \
 		--disable-bootstrap \
-		--with-headers=$(LINUX_DIR)/include
+		--with-sysroot=$(SYSROOT)
 	$(MAKE) -C $@ -j $(MAKE_JOBS)
 	$(MAKE) -C $@ -j $(MAKE_JOBS) install