<chapter>	<title>Server Operation</title>	<section id="operationalpractices">	    <title>Recommended Operational Practices</title>	    <para>		Operation of a SIP server is not always easy task.		Server administrators are challenged by broken or		misconfigured user agents, network and host failures,		hostile attacks and other stress-makers. All such		situations may lead to an operational failure. It is sometimes		very difficult to figure out the root reason of		a failure, particularly in a distributed environment		with many SIP components involved.				In this section,		we share some of our practices and refer to tools		which have proven to		make life of administrators easier	    </para>	<qandaset>	    <qandaentry>		<question>		    <para>			Keeping track of messages is good		    </para>		</question>		<answer>			<para>			    Frequently, operational errors are discovered or reported			    with a delay.			    Users frustrated by an error			    frequently approach administrators			    and scream "even though my SIP requests were absolutely ok			    yesterday, they were mistakenly denied by your server".			    If administrators do not record all SIP traffic at			    their site, they will be no more able to identify			    the problem reason.			    We thus recommend that site			    operators record all messages passing their site and keep them			    stored for some period of time.			They may use utilities such as 			<application>ngrep 			</application> or 			<application>tcpdump			</application>.			There is also a utility <application moreinfo="none">			    scripts/harv_ser.sh</application> in <application moreinfo="none">			ser</application> distribution for post-processing			of captured messages. It summarizes messages captured			by reply status and user-agent header field.		    </para>		</answer>	    </qandaentry>	    <qandaentry>		<question>		    <para>			Real-time Traffic Watching		    </para>		</question>		<answer>			<para>		    Looking at SIP messages in real-time may help to gain		    understanding of problems. Though there are commercial		    tools available, using a simple, text-oriented tool		    such as <application>ngrep</application> makes the job very well thanks to SIP's textual nature.			</para>		    <example id="usingngrep">			<title>Using <application>ngrep</application>			</title>			<para>In this example, all messages at port 5060			which include the string "bkraegelin" are captured			and displayed</para>			<programlisting format="linespecific">[jiri@fox s]$ ngrep bkraegelin@ port 5060interface: eth0 (195.37.77.96/255.255.255.240)filter: ip and ( port 5060 )match: bkraegelin@#U +0.000000 153.96.14.162:50240 -> 195.37.77.101:5060REGISTER sip:iptel.org SIP/2.0.Via: SIP/2.0/UDP 153.96.14.162:5060.From: sip:bkraegelin@iptel.org.To: sip:bkraegelin@iptel.org.Call-ID: 0009b7aa-1249b554-6407d246-72d2450a@153.96.14.162.Date: Thu, 26 Sep 2002 22:03:55 GMT.CSeq: 101 REGISTER.Expires: 10.Content-Length: 0..#U +0.000406 195.37.77.101:5060 -> 153.96.14.162:5060SIP/2.0 401 Unauthorized.Via: SIP/2.0/UDP 153.96.14.162:5060.From: sip:bkraegelin@iptel.org.To: sip:bkraegelin@iptel.org.Call-ID: 0009b7aa-1249b554-6407d246-72d2450a@153.96.14.162.CSeq: 101 REGISTER.WWW-Authenticate: Digest realm="iptel.org", nonce="3d9385170000000043acbf6ba9c9741790e0c57adee73812", algorithm=MD5.Server: Sip EXpress router(0.8.8 (i386/linux)).Content-Length: 0.Warning: 392 127.0.0.1:5060 "Noisy feedback tells: pid=31604 req_src_ip=153.96.14.162 in_uri=sip:iptel.org out_uri=sip:iptel.org via_cnt==1".			</programlisting>		    </example>		</answer>	    </qandaentry>	    <qandaentry>		<question>		    <para>			Tracing Errors in Server Chains		    </para>		</question>		<answer>			<para>			    A request may pass any number of proxy servers on			    its path to its destination. If an error occurs			    in the chain, it is difficult for upstream troubleshooters			    and/or users complaining to administrators to learn 			    more about error circumstances. 			    <application moreinfo="none">ser			    </application> does its best and displays extensive			    diagnostics information in SIP replies. It allows 			    troubleshooters and/or users who report to troubleshooters			    to gain additional knowledge about request processing			    status. 			    This extended debugging information is part of the warning 			    header field. See <xref linkend="usingngrep"> for an illustration			    of a reply that includes such a warning header field. The header			    field contains the following pieces of information:			<itemizedlist>			    <listitem>				<para>				Server's IP Address -- good to identify				from which server in a chain the reply				came.				    </para>			    </listitem>			    <listitem>				    <para>					Incoming and outgoing URIs -- good to					learn for which URI the reply was					generated, as it may be rewritten					many times in the path. Particularly					useful for debugging of numbering plans.				    </para>			    </listitem>			    <listitem>				<para>					Number of Via header fields in replied					request -- that helps in assessment of					request path length. Upstream clients would					not know otherwise, how far away in terms					of SIP hops their requests were replied.				</para>			    </listitem>				<listitem>				    <para>					Server's process id. That is useful for					debugging to discover situations when					multiple servers listen at the same					address.				    </para>				</listitem>				<listitem>				    <para>					IP address of previous SIP hop as seen by					the SIP server.				    </para>				</listitem>			</itemizedlist>		    </para>			<para>			    If server administrator is not comfortable with			    disclosing all this information, he can turn them			    off using the <varname>sip_warning</varname> configuration			    option.			</para>		    <para>			A nice utility for debugging server chains is			<application moreinfo="none">sipsak</application>,			SIP Swiss Army Knife, traceroute-like tool for SIP			developed at iptel.org. It allows you to send			OPTIONS request with low, increasing Max-Forwards 			header-fields and follow how it propagates in			SIP network. See its webpage at			<ulink url="http://sipsak.berlios.de/">			    http://sipsak.berlios.de/			</ulink>.		    </para>		    <example>			<title>Use of SIPSak for Learning SIP Path</title>			<programlisting format="linespecific">[jiri@bat sipsak]$ ./sipsak -T -s sip:7271@iptel.orgwarning: IP extract from warning activated to be more informational0: 127.0.0.1 (0.456 ms) SIP/2.0 483 Too Many Hops1: ?? (31.657 ms) SIP/2.0 200 OK	without Contact header			</programlisting>			<para>			    Note that in this example, the second hop			    server does not issue any warning header fields			    in replies and it is thus impossible to display 			    its IP address in <application moreinfo="none">			    SIPsak</application>'s output.			</para>		    </example>		</answer>	    </qandaentry>	    <qandaentry>		<question>		    <para>			Watching Server Health		    </para>		</question>		<answer>		    <para>			Watching Server's operation status in real-time may			also be a great aid for trouble-shooting. 			<application>ser</application> has an excellent 			facility, a FIFO server, which allows UNIX			tools to access server's internals. (It is 			similar to how Linux tool access Linux kernel			via the proc file system.) The FIFO server			accepts commands via a FIFO (named pipe) and			returns data asked for. Administrators do not			need to learn details of the FIFO communication			and can serve themselves using a front-end			utility <application moreinfo="none">serctl</application>.			Of particular interest for 			monitoring server's operation are 			<application moreinfo="none">serctl</application>			commands			<command moreinfo="none">ps</command> and			<command moreinfo="none">moni</command>.			The former displays running 			<application moreinfo="none">ser</application>			processes, whereas the latter shows statistics.		    </para>		    <example>			<title>serctl ps command</title>			<para>			    This example shows 10 processes running at a host.			    The process 0, "attendant" watches child processes			    and terminates all of them if a failure occurs in			    any of them. Processes 1-4 listen at local			    interface and processes 5-8 listen at Ethernet			    interface at port number 5060. Process number			    9 runs FIFO server, and process number 10			    processes all server timeouts.			</para>			<programlisting format="linespecific">[jiri@fox jiri]$ serctl ps0	31590	attendant1	31592	receiver child=0 sock=0 @ 127.0.0.1::50602	31595	receiver child=1 sock=0 @ 127.0.0.1::50603	31596	receiver child=2 sock=0 @ 127.0.0.1::50604	31597	receiver child=3 sock=0 @ 127.0.0.1::50605	31604	receiver child=0 sock=1 @ 195.37.77.101::50606	31605	receiver child=1 sock=1 @ 195.37.77.101::50607	31606	receiver child=2 sock=1 @ 195.37.77.101::50608	31610	receiver child=3 sock=1 @ 195.37.77.101::50609	31611	fifo server10	31627	timer			  			</programlisting>		    </example>		</answer>	    </qandaentry>	    <qandaentry>		<question>		    <para>			Is Server Alive		    </para>		</question>		<answer>		    <para>			It is essential for solid operation to know			continuously that server is alive. We've been			using two tools for this purpose. 			<application moreinfo="none">sipsak</application>			does a great job of "pinging" a server, which			may be used for alerting on unresponsive servers.		    </para>		    <para>			<application moreinfo="none">monit</application> is			a server watching utility which alerts when			a server dies.		    </para>		</answer>	    </qandaentry>	    <qandaentry>		<question>		    <para>			Dealing with DNS		    </para>		</question>		<answer>		    <para>			SIP standard leverages DNS. Administrators of			<application moreinfo="none">ser</application> should			be aware of impact of DNS on server's operation.			Server's attempt to resolve an unresolvable address			may block a server process in terms of seconds. To be			safer that the server doesn't stop responding			due to being blocked by DNS resolving, we recommend			the following practices:			<itemizedlist>			    <listitem>				<para>				    Start a sufficient number of children processes.				    If one is blocked, the other children will				    keep serving.				</para>			    </listitem>			    <listitem>				<para>				    Use DNS caching. For example, in Linux,				    there is an <application moreinfo="none">				    nscd</application> daemon available for				    this purpose.				</para>			    </listitem>			    <listitem>				<para>				    Process transactions statefully if memory				    allows. That helps to absorb retransmissions				    without having to resolve DNS for each of				    them.				</para>			    </listitem>			</itemizedlist>		    </para>		</answer>	    </qandaentry>		<qandaentry>			<question>				<para>					Logging				</para>			</question>			<answer>			<anchor id="logging">			<para>			    <application>ser</application> by default logs			    to <application>syslog</application> facility.			    It is very useful to watch log messages for			    abnormal behavior. Log messages, subject to			    <application>syslog</application> configuration			    may be stored at different files, or even at remote			    systems. A typical location of the log file is			    <filename>/var/log/messages</filename>.			</para>			<note>			    <para>				One can also use other <application>syslogd</application>				implementation. <application>metalog</application>				(<ulink url="http://metalog.sourceforge.net/">				    http://metalog.sourceforge.net/				</ulink>)				features regular expression matching that enables				to filter and group log messages.			    </para>			</note>			<para>			    For the purpose of debugging configuration scripts, one may			    want to redirect log messages to console not to pollute			    syslog files. To do so configure <application moreinfo="none">ser</application>			    in the following way:			    <itemizedlist>				<listitem>				    <para>					Attach ser to console by setting <varname>fork=no</varname>.				    </para>