RTLinux Tracer version 0.2Written by Michael Barabanov (baraban@fsmlabs.com)Copyright (C) Finite State Machine Labs Inc., 1999, 2000Released under the terms of the GNU GPL Version 2.This program package allows tracing of various system and user-defined events(interrupts, taking of spin-locks, etc), in Real-Time Linux. It is useful forfinding out how the system time is spent, where the delays are coming from,etc. 	REQUIREMENTS- RTLinux version 3.0 or later- modutils version 2.3.16 or higher- ksymoops 2.3.4 or higher	INTRODUCTIONReal-time programs can be challenging to debug because traditionaldebugging techniques such as setting breakpoints and step-by-step executionare not always appropriate. This is mainly due to two reasons:- Some errors are in the timing of the system. Stopping the program changes  the timing, so the system can not be analyzed without modifying it's behavior.- If the real-time program controls certain hardware, suspending the program  for analysis may cause the hardware to malfunction or even break (think of  robots).The RTLinux Tracer provides a solution to these and some other problems.In fact, it is sometimes used to debug RTLinux itself!The tracer timestamps and logs the events of interest into a set ofcircular buffers that are flushed by a user-space daemon.The flushing is triggered from your rtlinux program when needed.It is possible thus to find out the sequence of events that lead tosome particular situation. This is an extremely space-efficient approach.There is no need to produce 30MB log files when typically onlyonly a small portion of all events (the latest ones) is of interest.A builtin set of events includes RTLinux interrupt handlers entry and exit,scheduler entry and exit, spin-lock acquiring and freeing, context switches,interrupts disabling and enabling. User-defined events are also allowed.It is possible to attach an arbitrary 32-bit value to each event record.The current value of instruction pointer is also logged with each event.Events are grouped into classes.  It is possible to select logging of events ofany combination of different classes during runtime.	QUICK START0. Enable the CONFIG_RTL_TRACER option when running "make config".Recompile RTLinux.1. Install the base RTLinux modules (scripts/insrtl).Install the shared memory driver: insmod modules/mbuff.o2. Become root, cd tracer; make test3. The rt_process module here is a modified version of examples/measurements/rt_process.c. All supported events are logged;the trace buffer is dumped when a new maximum timing error value isreached. This way it is possible to learn more about what caused alarge scheduling delay (by examining the sequence of events precedingthe delay).4. Sample output:P0    410         rtl_spin_lock  0xc400a1e4   <c4008562>CPU  relative     event name     event data   instruction pointer     timestampThe relative timestamp is measured from the previous event. Due to thisthe first timestamp will be incorrect and should be neglected. The units are nanoseconds. Some possible event names and the meanings of event data:Event name		Event datartl_no_interrupts	saved flags		intercept entry		the irq vectoruser			user-defined (rt_process.c: the max RT-thread delay)rtl_spin_lock		spinlock addressThe full list of events can be found in include/rtl_tracer.h.Note. intercept and local_intercept events correspond to global andlocal CPU interrupts respectively.For the intercept events, the EIP points to the place where the interruptoccurred. For all other events, EIP means the place in the program the routinebeing traced was called from.5. You can use the provided symresolve program on the trace fileto translate the EIP values to symbolic names. Usage: symresolve TRACEFILE.symresolve uses ksymoops(8) to resolve addresses.NOTE. Debugging support has to be enabled in RTLinux for symresolve to work.6. You need to load the shared memmory driver mbuff.o (located in modules/) forthe tracer to work.	PROGRAMMING APIextern void rtl_trace(int event_id, long event_data, void *eip);	This is the main logging function. It does nothing if the rtl_tracer.o module	is not loaded. The list of supported events can be found in	include/rtl_tracer.h. You can also create your own events. 	extern void rtl_trace2(int event_id, long event_data);	This function is the same as rtl_trace, except that it logs the current	instruction pointer.extern unsigned rtl_trace_settracemask (unsigned mask);	Set logging mask (OR'ed event classes). A list of existing event classes	can be found in rtl_tracer.h. You can also define your own event classes.	This function returns the old trace mask value.	Example: to only log scheduling and user events, you can use	unsigned oldmask = rtl_trace_settracemask (RTL_TRACE_CLASS_SCHEDULER | RTL_TRACE_CLASS_USER);	By default all events are allowed (the default mask is set to 0xffffffff).extern int rtl_trace_seteventclass (int event, unsigned classmask);	Set the class of the event.To switch to another trace buffer, and let the user-space daemon processthe current buffer, use the special RTL_TRACE_FINALIZE event with rtl_trace orrtl_trace2. "event_data" and "eip" parameters are ignored in this case.By default, the RTLinux Tracer uses 20 buffers 500 records each. Thismeans the tracer captures 500 events preceding the finalization point of a trace. You can change these limits by editing rtl_tracer.h file(RTL_TNBUFFERS, RTL_TNRECORDS).	CREATING YOUR OWN EVENTS1. Allocate an event number, for example#define RTL_TRACE_MY_EVENT (RTL_TRACE_USER + 1)The total number of events should be less then RTL_TRACE_MAX_EVENTS.2. Set the event class:rtl_trace_seteventclass (RTL_TRACE_CLASS_USER);3. Set the event namertl_trace_seteventname (RTL_TRACE_MY_EVENT, "my event");4. To log your event, you can usertl_trace2 (RTL_TRACE_MY_EVENT, data);---------------------------------------------------------------Please direct all comments and suggestions toMichael Barabanov <baraban@fsmlabs.com>September 16, 1999.Updated for RTLinux 3.0 on August 15, 2000.