Friday, December 18, 2009

Miscellaneous Debugging tips

LTRACE
=========================================
from the Man page
ltrace  is  a  program  that  simply  runs the specified command until it exits.  It intercepts and records the  dynamic library calls which are called by the executed process and the  signals  which  are  received  by  that        process.  It can also intercept and print the system calls executed by the program.





STRACE
===========================================
In the simplest case strace runs the specified command until it exits.  It intercepts and  records  the  system  calls  which  are called by a process and the signals which are received by a process.  The name of each system  call, its arguments and its return value are printed on standard error or to the file  specified  with  the  -o    option.



nemesis@nemesis-laptop:~/test_code$ strace ./crashcourse
execve("./crashcourse", ["./crashcourse"], [/* 42 vars */]) = 0
brk(0)                                  = 0x80ba000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7846000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=72786, ...}) = 0
mmap2(NULL, 72786, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7834000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/tls/i686/cmov/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260l\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1319364, ...}) = 0
mmap2(NULL, 1325416, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x626000
mmap2(0x764000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13e) = 0x764000
mmap2(0x767000, 10600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x767000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7833000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb78336c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0x764000, 8192, PROT_READ)     = 0
mprotect(0x8049000, 4096, PROT_READ)    = 0
mprotect(0x3d0000, 4096, PROT_READ)     = 0
munmap(0xb7834000, 72786)               = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++


MTRACE
=============================================
nemesis@nemesis-laptop:~/test_code$ mtrace ./crashcourse
No memory leaks

LibC

nemesis@nemesis-laptop:/lib$ ./libc.so.6
GNU C Library (EGLIBC) stable release version 2.10.1, by Roland McGrath et al.
Copyright (C) 2009 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.4.1.
Compiled on a Linux >>2.6.24-23-server<< system on 2009-10-07.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
For bug reporting instructions, please see:
.

Core Dump : how to enable & Debug

1. find out if core will be dumped.

nemesis@nemesis-laptop:~/test_code$ ulimit -c
0

This means that core will NOT be dumped. ulimit specifies the maximum size of the core file.

2. Now set the value of ulimit to whatever you want it to be

ulimit -c [size]

3.  you can also specify the directory in which core files will be placed
root@nemesis-laptop:/home/nemesis/test_code# echo "/tmp/corefiles/core" > /proc/sys/kernel/core_pattern

4. small program to cause the segmentation fault

###########################################
#include

void func2(void)
{
    int *p = NULL;
    *p = 0xdeadcafe;
}

void func1(void)
{
    func2();
}


int main (int argc, char **argv)
{
    func1();
    int *p = NULL;
    *p = 0xdeadcafe;

}

##############################################


nemesis@nemesis-laptop:~/test_code$ gcc -g -o crashcourse crash_core.c







5. now generate the core file
nemesis@nemesis-laptop:~/test_code$ ./corefile
Segmentation fault (core dumped)

6. a bit more details from the file command [Tells you the program which generated the core dump]

nemesis@nemesis-laptop:~/test_code$ file /tmp/corefiles/core
/tmp/corefiles/core: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from './corefile'


7. gdb [executable_file] [core_file]

##################################################
nemesis@nemesis-laptop:~/test_code$ gdb crashcourse /tmp/corefiles/core
GNU gdb (GDB) 7.0-ubuntu
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /home/nemesis/test_code/crashcourse...done.

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/tls/i686/cmov/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/i686/cmov/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `./crashcourse'.
Program terminated with signal 11, Segmentation fault.
#0  0x080483c4 in func2 () at crash_core.c:6
6           *p = 0xdeadcafe;

##################################################

8. Get the backtrace


(gdb) bt
#0  0x080483c4 in func2 () at crash_core.c:6
#1  0x080483d4 in func1 () at crash_core.c:11
#2  0x080483e1 in main (argc=1, argv=0xbfe81b44) at crash_core.c:17





9. use the "up" and "down" commands to go through the code. (you can't run the code remember :) )

==============================================
UP
==============================================
(gdb) up
#1  0x080483d4 in func1 () at crash_core.c:11
11          func2();
(gdb) up
#2  0x080483e1 in main (argc=1, argv=0xbfe81b44) at crash_core.c:17
17          func1();
(gdb) up
Initial frame selected; you cannot go up.

===============================================
DOWN
===============================================
(gdb) down
#1  0x080483d4 in func1 () at crash_core.c:11
11          func2();
(gdb) down
#0  0x080483c4 in func2 () at crash_core.c:6
6           *p = 0xdeadcafe;


=========================================
NOTE: What if somebody handed you a core file and you don't the crashing process ? Happens a lot of times when the test team would pass you the core file and wash their hands off the issue altogether :)

Answer:
1. from the (file) following command

file [CORE-FILE]


nemesis@nemesis-laptop:~/test_code$ file /tmp/corefiles/core
/tmp/corefiles/core: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from './corefile'


2. just load the core file in gdb with "any" program name.

nemesis@nemesis-laptop:~/test_code$ gdb /bin/ps /tmp/corefiles/core
GNU gdb (GDB) 7.0-ubuntu
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /bin/ps...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
Core was generated by `./crashcourse'.

Program terminated with signal 11, Segmentation fault.
#0  0x080483c4 in ?? ()


GDB is intelligent enough to catch your bluff but the good that comes out of it is that it tells you the correct name of the program that core-d. (GDB can't tell if your bluffing or plain ignorant :))

Sunday, December 13, 2009

GIT Guide --- update

Before your commits make sure you have these variables defined. :)
==================================================
nemesis@nemesis-laptop:~/qemu/qemu$ git config --global user.name "nemesisofstate"
nemesis@nemesis-laptop:~/qemu/qemu$ git config --global user.email "nemesisofstate@gmail.com"


for checkin
========

edit [file]
git add [file]
git commit -m "your-message-here" -s

-s
--signoff

Add Signed-off-by line by the committer at the end of the commit log message.


======================
see your patch

git format-patch origin/master

your messages
===============
git show

gitk ===> graphic tool for git

Send out your patch
================
I use gmail to send my patches ...so u need msmtp for it [http://msmtp.sourceforge.net]. Install the file and configure your ~/.msmtprc file

git-config --global sendemail.smtpserver /usr/local/bin/msmtp

sudo apt-get install git-email

git email-send

e.g
nemesis@nemesis-laptop:~/qemu/qemu$ git send-email 0001-correcting-ARM-CPSR-register-bit-position-comment.patch


HOWTO: Get specific linux kernel headers

On UBUNTU

sudo apt-get install linux-headers-$(uname -r)

Saturday, December 12, 2009

QEMU + U-BOOT

1. Get the U-Boot sources

2. sudo apt-get install qemu-kvm-extras [for qemu-system-arm]
We will compile for ARM (needless to say) target---ARM versatile board (needless again)

[NOTE]: [u-boot-dir]/include/configs contains the default configs for most of the platforms. It almost corresponds to the one in linux kernel sources.

STEPS
=======

1. make versatile_config
2. make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi-

make ....make....make...

[NOTE]: Uboot needs the linux with uboot specific headers in order to boot it. This u-boot specific image can be generated using "make uImage".

However this would need mkimage. Where do we get it from---
Easy:)
---- after running make you can find the mkimage in [u-boot-dir]/tools directory.
---- Or get it from sudo apt-get install uboot-mkimage


3. Launch
sudo qemu-system-arm -nographic -m 256 -M versatileab -kernel u-boot.bin

The above command is good if your testing u-boot as a standalone app. However it would be wonderful to have networking between host and target to boot a linux kernel

sudo qemu-system-arm -M versatileab -kernel u-boot -net nic -net tap -nographic -serial stdio

Don't worry about the error message we will get to it.
===================================================
nemesis@nemesis-laptop:~/u-boot-2009.08$ sudo qemu-system-arm -nographic -m 256 -M versatileab -kernel u-boot.bin


U-Boot 2009.08 (Dec 13 2009 - 00:28:50)

DRAM: 0 kB
## Unknown FLASH on Bank 1 - Size = 0x00000000 = 0 MB
Flash: 0 kB
*** Warning - bad CRC, using default environment

In: serial
Out: serial
Err: serial

===================================================

Build linux kernel Image for U-boot



nemesis@nemesis-laptop:~/linux_kernel/linux-2.6.32$ make uImage
CHK include/linux/version.h
make[1]: `include/asm-arm/mach-types.h' is up to date.
CHK include/linux/utsrelease.h
SYMLINK include/asm -> include/asm-arm
CALL scripts/checksyscalls.sh
MKELF scripts/mod/elfconfig.h
HOSTCC scripts/mod/file2alias.o
HOSTCC scripts/mod/modpost.o
scripts/mod/modpost.c: In function ‘get_markers’:
scripts/mod/modpost.c:1562: warning: ignoring return value of ‘asprintf’, declared with attribute warn_unused_result
scripts/mod/modpost.c: In function ‘add_marker’:
scripts/mod/modpost.c:1982: warning: ignoring return value of ‘asprintf’, declared with attribute warn_unused_result
HOSTCC scripts/mod/sumversion.o
HOSTLD scripts/mod/modpost
CHK include/linux/compile.h
GEN lib/crc32table.h
CC lib/crc32.o
LD lib/built-in.o
LD vmlinux.o
MODPOST vmlinux.o
GEN .version
CHK include/linux/compile.h
UPD include/linux/compile.h
CC init/version.o
LD init/built-in.o
LD .tmp_vmlinux1
KSYM .tmp_kallsyms1.S
AS .tmp_kallsyms1.o
LD .tmp_vmlinux2
KSYM .tmp_kallsyms2.S
AS .tmp_kallsyms2.o
LD vmlinux
SYSMAP System.map
SYSMAP .tmp_System.map
OBJCOPY arch/arm/boot/Image
Kernel: arch/arm/boot/Image is ready
GZIP arch/arm/boot/compressed/piggy.gz
AS arch/arm/boot/compressed/piggy.o
LD arch/arm/boot/compressed/vmlinux
OBJCOPY arch/arm/boot/zImage
Kernel: arch/arm/boot/zImage is ready
UIMAGE arch/arm/boot/uImage
Image Name: Linux-2.6.32
Created: Sun Dec 13 02:29:45 2009
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1535396 Bytes = 1499.41 kB = 1.46 MB
Load Address: 00008000
Entry Point: 00008000
Image arch/arm/boot/uImage is ready



GET READY TO BOOT YOUR LINUX KERNEL
======================================

VersatilePB # setenv ipaddr 10.0.2.16
VersatilePB # setenv serverip 10.0.2.15
VersatilePB # ping 10.0.2.15
SMC91111: PHY auto-negotiate timed out
Using MAC Address 52:54:00:12:34:56
host 10.0.2.15 is alive
VersatilePB # tftp 8000 uImage
SMC91111: PHY auto-negotiate timed out
Using MAC Address 52:54:00:12:34:56
TFTP from server 10.0.2.15; our IP address is 10.0.2.16
Filename 'uImage'.
Load address: 0x8000
Loading: T T T T T T T #################################################################
###T #####################################
done
Bytes transferred = 1535460 (176de4 hex)


PROBLEM: There seems to be a bug in the sense that DRAM setting are not correct for versatile.

Switching to Integrator platform. [build uboot and linux image for integrator]
(uboot config =====> integratorcp_config)

nemesis@nemesis-laptop:~/u-boot-2009.08$ sudo qemu-system-arm -kernel u-boot -net nic -net tap -nographic -serial stdio
[sudo] password for nemesis:


U-Boot 2009.08 (Dec 13 2009 - 04:00:17)

DRAM: 128 MB
## Unknown FLASH on Bank 1 - Size = 0x00000000 = 0 MB
Flash: 0 kB
*** Warning - bad CRC, using default environment

In: serial
Out: serial
Err: serial
Hit any key to stop autoboot: 0
Integrator-CP # setenv serverip 10.0.2.15
Integrator-CP # setenv ipaddr 10.0.2.16
Integrator-CP # tftp 8000 uImage
SMC91111: PHY auto-negotiate timed out
Using MAC Address 52:54:00:12:34:56
TFTP from server 10.0.2.15; our IP address is 10.0.2.16
Filename 'uImage'.
Load address: 0x8000
Loading: T T #################################################################
################################
done
Bytes transferred = 1411276 (1588cc hex)
Integrator-CP # bootm
## Booting kernel from Legacy Image at 00008000 ...
Image Name: Linux-2.6.32
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1411212 Bytes = 1.3 MB
Load Address: 00008000
Entry Point: 00008000
XIP Kernel Image ... OK
OK

Starting kernel ...

qemu: fatal: Trying to execute code outside RAM or ROM at 0xfffff1c2

R00=00000000 R01=00000107 R02=00000100 R03=00000002
R04=00fddfc0 R05=00000000 R06=00008000 R07=00000113
R08=00fddfdc R09=010172c0 R10=010176ac R11=00fddced
R12=0100f750 R13=00000000 R14=00008008 R15=fffff1c2
PSR=600001db -ZC- A und32
Aborted
nemesis@nemesis-laptop:~/u-boot-2009.08$

============================================

So far so good...Uboot passes control to linux kernel but the kernel boot does a crash and burn. Will need to solve this one now....



Sunday, December 6, 2009

Busybox

1. Download the source and patches from the busybox website.

http://www.busybox.net/download.html

The version at the time of writing this is 1.14.1.

2. untar the sources to the directory where you will run the build.

3. patch the sources

patch -p0 <>.patch
e.g patch -p0 <>

4. config the busybox using your favourite config method. I prefer to use xconfig.

5. make your day :) -------- run make

6. once the make has run successful , you would want to install the busybox in a seperate directory which you can later ready to use as the rootfs.

make CONFIG_PREFIX=~/busybox/target install

where I am installing busybox in the /home/nemesis/busybox/target directory

7 cd to the install directory

cd ~/busybox/target

8.
There isn't a great deal there

ls -al would return

sbin, bin , usr and linuxrc (a symbolic link to /bin/busybox)



9. Create the dev, dev/pts, etc, etc/init.d, lib, mnt, opt, proc, root, sys, tmp, var, and var/log directories. Also create the device node for the initial console

mkdir dev etc lib mnt opt proc root sys tmp var debug
mkdir etc/init.d
mkdir var/log

sudo mknod dev/console c 5 1
sudo mknod dev/null c 1 3


10. Mount proc and /dev/pts at boot time
===========================

edit /etc/fstab for these filesystems



11. Shared library and library loader
==========================
We still need the
---------- shared libraries (from your toolchain)
---------- library loader (e.g ld-linux.so.3 in the listing below)


nemesis@nemesis-laptop:~/codesourcery/arm-2009q3/arm-none-linux-gnueabi/libc/lib$ ls
ld-2.10.1.so libdl-2.10.1.so libnss_dns.so.2 libresolv-2.10.1.so
ld-linux.so.3 libdl.so.2 libnss_files-2.10.1.so libresolv.so.2
libanl-2.10.1.so libgcc_s.so libnss_files.so.2 librt-2.10.1.so
libanl.so.1 libgcc_s.so.1 libnss_hesiod-2.10.1.so librt.so.1
libBrokenLocale-2.10.1.so libm-2.10.1.so libnss_hesiod.so.2 libSegFault.so
libBrokenLocale.so.1 libmemusage.so libnss_nis-2.10.1.so libthread_db-1.0.so
libc-2.10.1.so libm.so.6 libnss_nisplus-2.10.1.so libthread_db.so.1
libcidn-2.10.1.so libnsl-2.10.1.so libnss_nisplus.so.2 libutil-2.10.1.so
libcidn.so.1 libnsl.so.1 libnss_nis.so.2 libutil.so.1
libcrypt-2.10.1.so libnss_compat-2.10.1.so libpcprofile.so
libcrypt.so.1 libnss_compat.so.2 libpthread-2.10.1.so
libc.so.6 libnss_dns-2.10.1.so libpthread.so.0


--- that's an aweful lot. Copying them indiscriminately will cause our rootfs to be bloated (something we don't want). So just pick what you need. And And And ofcourse strip these libraries :)

nemesis@nemesis-laptop:~/busybox/target/lib$ ~/codesourcery/arm-2009q3/bin/arm-none-linux-gnueabi-strip *


Thats the minimal set for the rootfs


12. Kernel Modules
=================
Now it is time to get our dear modules.

--- go to the kernel build directory
--- issue the following command : make modules

after it has completed successfully , we need to install the modules in our rootfs. Elementary my dear watson!

make INSTALL_MOD_PATH=/path/to/rootfs modules_install

Once we are out of the debugging phase, we would like to strip the modules to slim down.

cd /path/to/rootfs/lib/modules
arm-none-linux-gnueabi-strip `find . –name “*.ko”`




Root File System

Umpteen ways to get this done :-

1. NFS Mounted. [One of the most popular methods ]

2. Initial RAMDISK (Initrd) ====> Will cover this in more detail in a later post

3. JFFS/JFFS2

4. CRAMFS (Read-only)

We will use busybox -- the veritable "Swiss Army Knife" for embedded systems.


Each of these will be explained in seperate posts, starting with Busybox



Saturday, December 5, 2009

ARM on QEMU

Goal : Run linux on a verstaline board (simulated by QEMU)

why : Because it is not always possible to have the real board available for development.

Prerequisites:
==============

1. QEMU : ofcourse
on debian systems : sudo apt-get install qemu
on RH systems : sudo yum install qemu

2. This one depends on what kind of communication mechanism is used between the host (your development) system and the guest sytem on QEMU. I prefer to use networking.

---- Have DHCP running so that guest system on qemu can use it to get an IP address.
---- host TAP devices used to connect to QEMU. The kernel config of the host should have TAP/TUN networking enabled. The switch for this is CONFIG_TUN.

grep CONFIG_TUN= /boot/config-`uname -r`

This should return you the state of the switch.

nemesis@nemesis-laptop:~$ grep CONFIG_TUN= /boot/config-`uname -r`
CONFIG_TUN=m


nemesis@nemesis-laptop:~$ ls -al /dev/net/tun
crw-rw-rw- 1 root root 10, 200 2009-11-03 11:46 /dev/net/tun


This verifies that the device exists indeed. If not, create it manually.

mknod /dev/net/tun c 10 200


Configure bridge-control
==================
Add the following to a script file

#!/bin/sh
/usr/sbin/brctl addbr br0 ### Add a bridge
/sbin/ifconfig eth0 0.0.0.0 promisc up ### set up eth0
/usr/sbin/brctl addif br0 eth0 ### add an intf(eth0) to the bridge (br0)
/sbin/dhclient br0 ### enable dhcp
/sbin/iptables -F FORWARD

Build kernel
=========

The quickest way is to use the default config file for the versatile platform.
Simply copy the config file from configs directory for ARM arch and rename it as .config

cp /arch/arm/configs/versatile_defconfig /.config

Config the Linux kernel (xconfig, menuconfig or simply using vim to edit..have it your way)

1. Enable ARM EABI support : switch CONFIG_AEABI (part of kernel features)
2. Enable DHCP : Switch CONFIG_IP_PNP_DHCP
3. Enable Universal TUN/TAP driver : switch CONFIG_TUN
4. Enable tmpfs : Switch CONFIG_TMPFS

Now we are ready to make (my day)
Ofcourse ARCH = arm
and CROSS_COMPILE = your_cross_compiler (I use codesourcery and for me it is
arm-none-linux-gnueabi-)

What about the Root File System ?
=======================

We will NFS mount the root filesystem. Build your rootfs.

1. edit /etc/exports ===> Add the following line

/mnt/arm-linux-rootfs / *(rw,sync,no_root_squash)

Here the untarred rootfs is at /mnt/arm-linux-rootfs and it is accessible as rw filesystem from any systems (indicated by *). If static ip-addresses are assigned, use the ip-address of the peer here.

2. restart the nfs service so that our changes come into effect.

/etc/init.d/nfs restart


Boot into Linux on QEMU
======================

qemu-system-arm -M versatilepb -kernel zImage-versatile -append root="/dev/nfs nfsroot=:/mnt/arm-linux-rootfs rw ip=dhcp" \
-net nic,vlan=0 -net tap,vlan=0,ifname=tap0,script=./qemu-ifup


qemu-ifup
=============================

#!/bin/sh

echo "Executing /etc/qemu-ifup"
echo "Bringing up $1 for bridged mode..."
sudo /sbin/ifconfig $1 0.0.0.0 promisc up
echo "Adding $1 to br0..."
sudo /usr/sbin/brctl addif br0 $1
sleep 2



Source : http://fedoraproject.org/wiki/Architectures/ARM/HowToQemu
http://www.aurel32.net/info/debian_arm_qemu.php