Debug using the crash utility tool - Qualcomm Dragonwing Documentation

Crash utility is an open-source tool to debug kernel using a gdb-based command-line interface over RAM dumps.

Prerequisites

kaslr_offset and kimage_voffset values By default, the Qualcomm Linux build has the Kernel Address Space Layout Randomization (KASLR) feature enabled. For the crash utility to work on a KASLR-enabled kernel, you need the values of the kaslr_offset and kimage_voffset parameters, which you can extract from the dmesg_TZ.txt RAMparser output file. The following is an excerpt from the sample RAMparser output file, dmesg_TZ.txt, providing the kaslr_offset and kimage_voffset values. The kaslr_offset extracted is: 0x340c3d320000 … The kimage_voffset extracted is: ffffb40bbf600000
Base address of the kernel binaries (DDRCS*.BIN) When you use PCAT to capture the RAM dump, it also captures the dump_info.txt and load.cmm files. These files contain the base address of the kernel binaries (DDRCS*.BIN).

Download and build the crash utility tool on Linux host

To download and build the crash utility, run the following commands:

git  clone https://github.com/crash-utility/crash

make target=ARM64

make extensions=ARM64

For more information, see crash/README at master · crash-utility/crash· GitHub.

Run the crash utility tool

The following is the sample command to run the crash utility on an Ubuntu host:

./crash <PATH>/vmlinux <PATH>/DDRCS0_0.BIN@0x80000000,<PATH>/DDRCS1_0.BIN@0x100000000,<PATH>/DDRCS1_1.BIN@0x180000000 -m vabits_actual=48 -m max_physmem_bits=48 -m kimage_voffset=0xffffb40bbf600000  --kaslr=0x340c3d320000

Replace <PATH> with the vmlinux RAM dump path.
Extract kimage_voffset, and kaslr from the dmesg_TZ.txt RAMparser output file.
Extract the DDR offsets such as 0x80000000 from the dump_info.txt file available in the RAM dump collected using PCAT.

Sample output:

crash 8.0.4
Copyright (C) 2002-2022  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

NOTE: setting vabits_actual to: 48

NOTE: setting max_physmem_bits to: 48
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=aarch64-elf-linux".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
   <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

WARNING: cpu 0: cannot find NT_PRSTATUS note
WARNING: cpu 1: cannot find NT_PRSTATUS note
WARNING: cpu 2: cannot find NT_PRSTATUS note
WARNING: cpu 3: cannot find NT_PRSTATUS note
WARNING: cpu 4: cannot find NT_PRSTATUS note
WARNING: cpu 5: cannot find NT_PRSTATUS note
WARNING: cpu 6: cannot find NT_PRSTATUS note
WARNING: cpu 7: cannot find NT_PRSTATUS note
      KERNEL: /test/vmlinux  [TAINTED]
   DUMPFILES: /var/tmp/ramdump_elf_PBaLfj [temporary ELF header]
            /test/DDRCS0_0.BIN
            /test/DDRCS1_0.BIN
            /test/DDRCS1_1.BIN
      CPUS: 8 [OFFLINE: 7]
      DATE: Sun Jan  6 05:30:34 +0530 1980
      UPTIME: 00:00:48
LOAD AVERAGE: 1.60, 0.51, 0.18
      TASKS: 532
   RELEASE: 6.6.17-debug
   VERSION: #1 SMP PREEMPT Mon Mar 25 04:52:52 UTC 2024
   MACHINE: aarch64  (unknown Mhz)
      MEMORY: 5.5 GB
      PANIC: "Kernel panic - not syncing: sysrq triggered crash"
         PID: 1419
   COMMAND: "sh"
      TASK: ffff1756f61ea000  [THREAD_INFO: ffff1756f61ea000]
         CPU: 0
      STATE: TASK_RUNNING (PANIC)

Sample command:

crash> man

Sample output:

*              files          mod            sbitmapq       union
alias          foreach        mount          search         vm
ascii          fuser          net            set            vtop
bpf            gdb            p              sig            waitq
bt             help           ps             struct         whatis
btop           ipcs           pte            swap           wr
dev            irq            ptob           sym            q
dis            kmem           ptov           sys
eval           list           rd             task
exit           log            repeat         timer
extend         mach           runq           tree

crash version: 8.0.4    gdb version: 10.2
For help on any command above, enter "help <command>".
For help on input options, enter "help input".
For help on output options, enter "help output".

Sample command:

crash> log

Sample output:

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
[    0.000000] Linux version 6.6.17-debug (oe-user@oe-host) (aarch64-qcom-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Mon Mar 25 04:52:52 UTC 2024
[    0.000000] KASLR enabled
[    0.000000] Machine model: Qualcomm Technologies, Inc. Robotics RB3gen2 addons vision mezz platform
[    0.000000] efi: EFI v2.7 by Qualcomm Technologies, Inc.
[    0.000000] efi: MEMATTR=0x9ccf6018 INITRD=0x9cceaf18 RNG=0x9cce3018 MEMRESERVE=0x9ccea818
[    0.000000] random: crng init done
[    0.000000] Reserved memory: created CMA memory pool at 0x00000000fd000000, size 12 MiB
[    0.000000] OF: reserved mem: initialized node adsp-heap, compatible id shared-dma-pool

Sample command:

crash> p memdump

Sample output:

memdump = $1 = {
table_phys = 4110417920,
table = 0xffff80008087d000
}

For more information about the crash utility, see the following:

​Prerequisites

​Download and build the crash utility tool on Linux host

​Run the crash utility tool

Prerequisites

Download and build the crash utility tool on Linux host

Run the crash utility tool