Cable Modem Troubleshooting Tips: Traceroute

The traceroute tool is a valuable aid to network troubleshooting. It also is the most commonly misinterpreted diagnostic tool, capable of raising false alarms when nothing is wrong, and equally capable of showing no problems when quite a lot is wrong.

Before using traceroute, ensure that your network connection is idle (no download or upload), and that there is no background activity which might put traffic on your network connection. Any network traffic on a rate-capped service such as a cable modem will cause spuriously slow timings in traceroute.

To use the traceroute tool in Windows, open a permanent command-line window. In Windows 9x/ME, make the window 50 lines high by giving the command mode con co80,50. In Windows 2K/XP, stretch the window by dragging its bottom margin. Now give the command tracert followed by the DNS name or IP number of the host to which you wish to discover the route. Here is an example:

Each line is labelled with a hop number. Three probes were sent for each hop, and the round trip times (RTT), there and back, for each probe is given in milliseconds (thousandths of a second). If a probe is sent and no reply is received, the RTT is replaced with an asterisk. The IP number of the replying router is given on the extreme right, and if that IP number could be resolved to a DNS name, that is shown too. The final line is for the host given in the command.

Apple Mac OS X 8.x/9.x users can perform traceroutes with utilities such as WhatRoute (on the Mac OS 9.1 CDROM), Interarchy or IPNetMonitor.

How traceroute works

All IP packets are initially sent with a Time-To-Live (TTL) field set to a suitable number. Each time an IP packet passes through an IP router, its TTL is reduced by one. When the TTL reaches zero, the packet is discarded. This strategy is to prevent IP packets circulating endlessly on the internet in the event of a router misconfiguration leading to a circular route.

Each traceroute query is sent with a very small TTL. When the query's TTL reaches zero, instead of being totally discarded, an ICMP TTL-exceeded warning packet is returned to the sending address, where the traceroute program is waiting to time its return. For the first hop, the TTL is initialised to one, three queries are sent and timed. Then the initial TTL is increased by one, three queries sent and timed, and so on. This repeats until the IP number that replies to the query does so with an echo instead of an ICMP TTL-exceeded. This indicates that it is the sought host, rather than an intermediate router.

How traceroute can mislead

Given all these provisos, you might wonder what use traceroute is. Well, it does reliably discover the outward route. But you should be extremely careful about inferring network problems from traceroute reports. Certainly, on its own, traceroute is not a reliable measure of packet loss, as it tries only three times to each hop. However, if indications of packet loss (asterisks) start at a particular hop and continue for every hop thereafter, then this is indicative of packet loss starting at the first affected hop, or the cable leading to it from the previous (good) hop.

Misleading traceroutes to certain NTL servers

Traceroutes to certain NTL servers always look wrong when there is nothing in fact wrong. For instance:

C:\WINDOWS>tracert www.ntl.com

Tracing route to www.ntl.com [212.250.5.110]
over a maximum of 30 hops:

  1    18 ms    19 ms    72 ms  172.16.231.254
  2    17 ms    15 ms    14 ms  cam-cam1-a-fa00.inet.ntl.com [62.253.129.1]
  3    21 ms    13 ms    15 ms  cam-core-a-pos200.inet.ntl.com [62.253.128.133]
  4    27 ms    24 ms    25 ms  lng-bb-a-atm100-808.inet.ntl.com [62.253.187.42]
  5    21 ms    28 ms    24 ms  gfd-bb-a-so-110-0.inet.ntl.com [62.253.188.246]
  6    27 ms    46 ms    24 ms  gfd-dc-a-ge400.inet.ntl.com [62.253.188.154]
  7    23 ms    29 ms    62 ms  gfd-alder-fa10.inet.ntl.com [194.168.2.1]
  8    21 ms    23 ms    20 ms  62.252.0.98
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12     *        *        *     Request timed out.
 13     *        *        *     Request timed out.
 14     *        *        *     Request timed out.
 15     *        *        *     Request timed out.
 16     *        *        *     Request timed out.
 17    17 ms     9 ms    24 ms  www.ntl.com [212.250.5.110]

Trace complete.

The server at www.ntl.com has a defect in its TCP/IP implementation in that it sets the TTL of an ICMP echo to be the same as the residual TTL of the ICMP query. This means that the ICMP echo will be dropped by some router in the return path, until such time as the originating traceroute program has increased the TTL on the query packet to account for the number of hops in the return path as well as the outward path. This fools the traceroute program into inventing hops 9 to 16, which really don't exist: www.ntl.com is in fact directly connected to the router in hop 8. It is characteristic of this defect that one sees the same number of missing hops (8 in this case) as the number of genuine hops before the gap, assuming the return path is the same as the outward one. There is no evidence of any network problem in the above traceroute.

Some Blueyonder servers, such as smtp.blueyonder.co.uk, exhibit the same defect.

Speeding up traceroutes in Windows

In the example traceroute above, the local UBR's private IP address in hop 1 will be slow to appear, because Windows does a reverse DNS lookup on the IP number, but gets no reply from the DNS system, because addresses in the range 172.16.0.0 - 172.31.255.255 and 10.xxx.xxx.xxx are private IP addresses (rather than public internet IP addresses) and are not registered in the DNS system. There will be similar slow responses for all hops with IP addresses in private ranges.

First discover the private IP address of your local UBR: see Finding the UBR address. In this example we shall assume it is 172.19.83.254.

In Windows 9x/ME, copy the file C:\WINDOWS\HOSTS.SAM to C:\WINDOWS\HOSTS. It is difficult to create files with no (hidden) extension after the filename using the Windows graphical interface, so this is best done in an MS-DOS window by giving the commands

In Windows 2000, open the file c:\winnt\system32\drivers\etc\hosts in a plain-text editor.

A suitable plain-text editor is the command line edit. Do not use word-processors such as WordPad or Word. NotePad would be fine provided you know how to defeat its tendency to add a hidden .txt extension to filenames when it saves (the trick, in the Save As dialog, is to quote the filename in double-quotes, e.g. "HOSTS").

where the 172 address you actually use is the one you discovered above. Save the changed file. Make quite sure that it has been saved as a file called hosts or HOSTS with no .txt extension, even when you look at with the dir command. Restart Windows, and try a tracert. You should find that the first hop of a traceroute now looks like:

and appears almost instantly. If other hops in the traceroute are also private addresses (in the ranges 192.168.xxx.xxx, 172.xx.xxx.xxx or 10.xxx.xxx.xxx) you can repeat this process for all such addresses to speed up traceroute results, using invented names for them.

Getting reverse traceroutes

Because a traceroute reveals only the outward path, leaving the return path unknown, investigation of networking problems between your PC and a specific host ideally require a traceroute from that host back to your own PC. For normal users, this is not in general possible. The next best thing is to send your web browser to www.traceroute.org, choose a site close (in terms of internet topology) to the site under investigation, and run a traceroute back to your PC.

Partial reverse routes using ping

It is possible to discover the last 9 routers that packets passed through on their way back to you using the record route feature of the ping command. The options required to select record route vary according to operating system. Here is an example under Windows:

Note that in the above example, three successive ping replies each took a different return route back to me in my ISP's network. This demonstrates how futile it can be trying to make sense of traceroute timings, because return routes can be so different for successive packets.

For Record Route to work, the remote host pinged must support the setting of the Record Route flag in the ping reply: not all hosts support this.

Under Unix-derived systems, the option to request a return route is -R rather than -r, but there might be requirements for additional simultaneous options: check man ping. In some distributions (including Mac OS X), the command ping -R is broken and always gives Invalid argument.

Mac OS X users can request a Record Route using the Trace Route tool of the IPNetMonitorX utility: use the pull down option UDP Trace/ICMP Trace/Record Route.

Cable Modem Troubleshooting Tips