A Networking and System Engineer Blog

Tuesday, September 23, 2008

Understanding Cisco CallManager Traces (CUCM)

Whether you call it Call Manager, Unified Communications Manager or CUCM, everyone agrees that knowing what is going on under the hood when there is a problem is a must. Having had the pleasure of calling Cisco TAC numerous times to find out what happened to a call I finally have the process for gathering traces down. It still takes quite a bit of experience to tell what is going on sometimes and that’s why I’ll still be calling TAC to help decipher the traces. Simply getting the traces saves a lot of time, so I’ll outline the process. The last issue I worked on had to do with debugging ISDN/Q.SIG messages, so I’ll refer to those throughout.

Step 1: Turn on Traces in CallManager
Open up your CallManager Admin page and navigate to Serviceability. Then go to Traceà Configuration, select your publisher and choose the CallManager Service. Make sure Trace is on, and that debug trace level is set to Detailed. You will also see a lot of options with checkboxes. I normally leave these to the defaults, but if you are looking for something specific, go ahead and set these. The ones I am normally sure to check are the PRI trace, MGCP trace and All Phone device traces. Now make sure that the “apply to all nodes” is also checked and click save.

Step 2: Download and Install RTMT (Real Time Monitoring Tool)
This tool is accessed from the normal CallManager Administration page. It can be found under ApplicationàPlugins. Download and install this tool. Make sure you can run the RTMT tool and log in using the admin credentials you use to log into the web admin page. You may see a “select configuration” dialog box. You can cancel this. It is for setting up a default set of windows of CallManager performance data. We just want the traces. Choose ToolsàTraceàTrace & Log Central.

Step 3. Place Your Test Call
Now that you have RTMT loaded up, you need to place your test call or create the condition you are trying to capture. Go ahead and do this, twice even so that you know you can find it. Wait ten seconds or so once you are done with the test so that CallManager has a chance to write the trace files.

Step 4. Collect Trace Files
Now go to your RTMT window that you pulled up in step 2, and double click “Collect Files”. Find CallManager in the list, and click the checkbox for “All Servers” since you may have a situation where a phone is registered to one server and the gateway is registered to another. This is good info to have beforehand so your search for the trace is a bit easier. Click next. The next option is to download system logs. These are useful for system issues, but not necessary for trace file collection. Skip this by clicking next. Now for the file options, choose relative range and go back 5 or 10 minutes. Make sure whatever you choose is sufficient to capture the test call or error condition. Trace files contain a surprising amount of data, of which more is not always a good thing. Choose to zip the files and choose browse to select the local directory you want to download the files to. Now click finish. Depending on your relative range this may take some time.

Step 5. Download and Install Triple Combo
Ok, so you could dig right in and look through your trace files for that test call. That’s what I used to do – until I found Triple Combo. This app helps to filter and display CallManager traces, ISDN Q931, H225, H245 – even CCAPI, Q931 and MGCP debugs straight from your router. It also can decode ISDN facility messages. I haven’t found any easier way to do this.

My particular issue was that I had a Q.SIG trunk from my CallManager environment to a PBX. I had calls going from CallManager to the PBX, and when no one answered, the call got “pulled back” and went to voicemail on the CallManager side. It was not apparent was who was responsible for the Call Forward No Answer (CFNA) behavior. While we had a hunch, we needed the traces to prove it. I was able to get the trace file from CallManager, and found that an ISDN Q.931 facility message was received by the gateway from the PBX over the Q.SIG trunk. I was able to right-click the facility message, choose decode, and saw that the PBX was sending the call CFNA back to CallManager and giving the voicemail pilot number. Now I have the ammunition to go to the PBX vendor and prove that they are the cause for this particular issue.

Now that I (hopefully) have you convinced, install Triple Combo: http://www.employees.org/~tiryaki/tc/ Also download the MT.exe executable. This is required for deciphering the facility messages. http://www.employees.org/~tiryaki/tc/mt.exe
Triple Combo doesn’t install. It is a standalone executable, so make sure that it and MT.exe are in the same directory.

Step 6: Drag Trace Files into Triple Combo
Your trace files are zipped, so you will want to unzip them. The reason for this is because you can use the Windows search feature to look for text in the files if they are unzipped. Instead of searching for a filename, you can search for a word or phrase in the file. This can narrow down which trace file you are going to analyze. The files also need to be unzipped for Triple Combo to analyze them. Once you know which files have the test call, drag them into Triple Combo. In my case, I looked for my Q.931 facility messages, then right-clicked to decode the facility message in the lower viewing pane. There are many ways to filter the output, so I will leave that to you. You’re on your own from here! Kudos to Murat Tiryakioglu for creating this software.

Brad Smith

1 comment:

Unknown said...

is it possible to real time monitoring 1 extension number ?