The RTLinux DebuggerMichael Barabanov (baraban@fsmlabs.com)Copyright (C) 2000, FSMLabsReleased under the terms of the GPLThis code is partly based on the gdbstubs package (ftp://ftp.gcom.com/pub/linux/src)This package provides debugging facilities for RTLinux threads based on GDB.Currently this works for x86 and PPC versions of RTLinux.	Features- source-level debugging of RTLinux programs using gdb;- simple to use: just one computer is needed (unlike with KGDB or Zentropix(TM)tools that require connecting two computers with a serial line);- no special tools needed; full source code is provided;- crash protection. No more frustrating reboots! (unless you try hard)  The debugger automatically catches all exceptions in real-time programs.  If any of your RT-threads fault, the debugger prints a message and suspends  the thread (see below for more information about this feature)  - you can invoke gdb "lazily", at any time after the fault, to find out where it  has happened, to examine variables, print stack traces - anything that gdb   or ddd can do.- RTLinux thread support!  GDB threads commands are supported  in this release.  After interrupting a running program with ^C, you can use  "info threads", "thread", "backtrace", "up", "down" and other GDB commands  to examine the state of all your RTLinux threads. The DDD can also display  RTLinux threads.- SMP support- it is possible to use any graphical interface to gdb, e.g.DDD -- http://www.gnu.org/software/ddd/xxgdb -- there is a debian package for this very simple and lean GDB front-endinsight -- http://sourceware.cygnus.com/insight/Alternatively, command-line GDB is always possible.--- NOTE: tracing Linux kernel code is not yet supported. Neitheris tracing code that disables interrupts.	Requirements- RTLinux 2.2b or later- modutils 2.3.9 or later (you can check the version  with insmod -V). You can get RPMs of new modutils from  ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/redhat-6.2/i386/RedHat/RPMS  and DEBs from ftp://ftp.debian.org/pub/debian/dists/woody/main/binary-i386/base.- GNU gdb 19990928 or later (gdv -v)	Using the debuggerinsmod rtl_debug.oBefore starting debugging, a CPU exception has to occur in your RTLprogram.  You can use the breakpoint() function defined in the <rtl_debug.h>include file to generate one by hand.Division by zero will also work but is not recommended.Compile your RTLinux module with debugging support.  To enable debugging support, modify CFLAGS to contain "-g", and not "-fomit-frame-pointer". The simplest way to achieve that is to make sure that the CONFIG_RTL_DEBUG is enabled during "make config".Install your RTLinux module, for example:insmod rt_process.oYou should see a message of the following form:rtl_debug: exception 3 in rt_process, thread id 0xc2aa2400; (re)start GDB to debugThen, type gdb rt_process.o or ddd rt_process.oNote: the file name you specify on the command line should correspondto the module that generated an exception.Finally, at the GDB prompt, type target remote /dev/rtf10(if you load the gdb macros described below, just type "dbg").The debugger should stop at the source line that generated the exception.You can then use gdb commands to perform debugging as usual.	Example session (see hello.c in the debugger directory)Tip: if you're in X, use dmesg or xconsole to watch the output from hello.o.# insmod rtl_debug.oRTLinux Debugger Loaded (http://www.fsmlabs.com/)                                 # insmod hello.ortl_debug: exception 3 in hello, thread id 0xc2aa2400; (re)start GDB to debug# gdb hello.oGNU gdb 19990928Copyright 1998 Free Software Foundation, Inc.GDB is free software, covered by the GNU General Public License, and you arewelcome to change it and/or distribute copies of it under certain conditions.Type "show copying" to see the conditions.There is absolutely no warranty for GDB.  Type "show warranty" for details.This GDB was configured as "i686-pc-linux-gnu"...(gdb) target remote /dev/rtf10Remote debugging using /dev/rtf10[New Pid -977672704][Switching to Pid -977672704]start_routine (arg=0x0) at hello.c:2929              for (i = 0; i < 20; i ++) {(gdb) list2425              pthread_make_periodic_np (pthread_self(), gethrtime(), 500000000);2627              breakpoint();2829              for (i = 0; i < 20; i ++) {30                      pthread_wait_np ();31                      rtl_printf("I'm here; my arg is %x\n", (unsigned) arg);32              }33              return 0;(gdb) bre 31Breakpoint 1 at 0xc807f0c5: file hello.c, line 31.(gdb) contContinuing.Breakpoint 1, start_routine (arg=0x0) at hello.c:3131                      rtl_printf("I'm here; my arg is %x\n", (unsigned) arg);(gdb) contContinuing.Breakpoint 1, start_routine (arg=0x0) at hello.c:3131                      rtl_printf("I'm here; my arg is %x\n", (unsigned) arg);(gdb) print arg$1 = (void *) 0x0(gdb) delDelete all breakpoints? (y or n) y(gdb) quitThe program is running.  Exit anyway? (y or n) y# rmmod hello# rmmod rtl_debug	Notes and tips- the .gdbinit file in this directory contains some gdb macros to save you  typing. You can place this file in your home directory or in the current  directory for gdb to automatically load it.The macros are:dbg	start debugging via /dev/rtf10modaddsym module-file-name.o	add symbols from the (currently loaded) module to GDB's symbol tablemodaddsched	attempt to add symbols for the scheduler.	The scheduler symbols are useful for determining where each	thread is blocked. The rtl.mk file should be present in	the current directory.di	disassemble a piece of code around PC- a very useful mode of operation is to simply have the rtl_debug.o moduleloaded to prevent system crashes.If a RT-task performs an illegal operation, the debugger modulewill print a message detailing the module, thread id and the current valueof the instruction pointer.NOTE. To facilitate debugging, all RT-threads stop when there is an exception in any thread.To get more detailed information of the system state, you shouldinvoke gdb module_name.o, and type "target remote /dev/rtf10" at the prompt.You will see where the problem is. You can also examine other threads'state.NOTE. To safely remove the offening module, Ctrl-Z the debugger,rmmod module_name.o, and kill the gdb. Do not type "continue" or "next"at the prompt as this will cause the system to try to execute the badcode again.- to examine internal thread's state (see include/rtl_sched.h,struct rtl_thread_struct), put the following lines at the beginningof the thread code:	pthread_t self = pthread_self();After that, you can examine *self, self->period, etc. For example:"print *self".- after you used "continue" to resume your tasks, you can press ^C toregain control of the debugger. Then you can set a breakpoint in thereal-time code that may be hit after you type "continue"."info threads", "thread", "backtrace", "up", "down" GDB commands are useful in this regard.- if you want to leave your RT-tasks in the running state,delete all breakpoints ("delete"), and use "quit" as the lastcommand to GDB;- you can change the FIFOs used for communicating with the debugger  by specifying fifo=NN on the insmod rtl_debug.o command line. The  debugger uses three FIFOs (fifo, fifo+1, fifo+2). The default is NN==10.- you can specify bp=1 as an insmod parameter to rtl_debug.o. Thiswill cause a breakpoint exception in the debugger. Note howeverthat (due to a bug in GDB's symbol handling) you should alwaysstart GDB with a filename that correspond to the module thatgenerated the exception.- global variables not initialized explicitly (bss) can not be examined in gdb.  This is a bug in GDB. Hopefully it will be fixed in a laterversion. For now, you need to explicitely initialize all global variables.