Cisco TAC Tools - Wireless Debug Analyzer

Introduction

As wireless networks have evolved in the past years it is getting harder and harder to troubleshoot advanced environments where there are a lot of different key players involved: you got the APs and their WLC, the AAA-servers and the clients themselves to keep track of and try to sort out what is going on during times of trouble.

Cisco WLCs has a lot of useful debugging commands that can be run in CLI but trying to fully understand what is going on can be hard at times. Way too many debugging events are poorly “explained” as they fly by the command line and I often find myself having to look up weird messages and status codes afterward.

Cisco themselves, being aware of this issue, has a tool published on their website called Wireless Debug Analyzer. It is listed as a tool used by the Cisco TAC to quickly parse through debug logs/output and make them more readable to the user. The idea is that you save the debugging log of a misbehaving wireless client to a text file and then upload that text file to the tool’s website where it will be processed and “translated” into (hopefully) something better.

You can find the tool at https://cway.cisco.com/tools/WirelessDebugAnalyzer/ (Cisco login required)

According to the very short documentation, all you need to upload is a text file containing the debug output of the command debug client <mac-address>. There is also a short footnote about the tool being able to read some of the debug output from the “debug aaa/mdns/webauth…” commands.

Parsing a successful 802.1X login (debug client <mac> only)

The first test was done using a debug client <mac-address> of a smartphone successfully connecting to an 802.1X SSID using PEAP-MSCHAPv2 as the authentication method. The original debug output was around 400 rows of output and after putting it through the parser we ended up with this:

I’ve highlighted some of the interesting messages where we can see that EAP authentication takes place and is successful based on that the RADIUS server (ISE in this case) permits the access. What I would’ve loved to see here is what kind of access is given (VLAN assignment, ACL assignment, or similar) but with this particular debug it’s as good as it’s gonna get using this tool, even if the original debug output tells us what kind of AAA Override took place (in my case I assigned a particular VLAN to the client).

Parsing a successful 802.1X login (debug client <mac> and aaa)

By adding debug aaa packets to the mix in addition to the debug client <mac-address> command, we do get a lot more debug output which gets translated by the tool when we upload the debug output text file.

This time we get to see which AAA-server (RADIUS) IP address was used to authenticate this client device. For some reason, as you can see, a lot of RADIUS sessions were created during the authentication of this client device (Radius requests/challenges) but ultimately the authentication was successful and an Access-Accept packet was sent to the WLC. However, still no word on other attributes that could’ve been sent to the WLC such as VLAN assignment or ACL assignment.

Parsing a failed 802.1X login (debug client <mac>)

Once again we are only running the debug client <mac-address> command but this time I will deliberately fail the 802.1x authentication by typing in the wrong password.

The information translated for us from this log is pretty weak, as you can see there isn’t much being said about the authentication at all, and at the end of it we are not even being told the authentication failed, only that it WLC/AP asks the device again to identity itself.

Parsing a failed 802.1X login (debug client <mac> and aaa)

By adding debug aaa packets to the mix in addition to the debug client <mac-address> command, we do get some more debug output but most of it is just repeat text telling us the WLC is resending the authentication to the AAA-server over and over again in new RADIUS sessions.

Verdict

The Cisco Wireless Debug Analyzer isn’t as great as it could’ve been, I love the idea of this tool making troubleshooting simpler but it doesn’t seem advanced enough. If anything, it is useful at dividing the debug output into sections even if the “translated” result is not very well-described or not there at all. With the newer WLCs actually running Cisco’s IOS-XE software instead of AireOS which has been used for the past 10 years or so, I don’t see Cisco putting any more effort into making this tool better, but I guess we have to keep in mind this tool was made by a couple of engineers at Cisco so maybe it can’t be considered a “real” product where users can demand new features. The tool is not very useful for troubleshooting failed AAA authentication attempts due to barebone information, it’s better to just look in the live logs of the AAA server if you are able to.

The tool doesn’t seem to be consistent with the information being translated. In some of the cases, I was able to get a translation of which IP address was assigned to the client on successful login, but as you can see above it never happened while I was grabbing these screenshots.