In this article, we take a look at the most common pitfalls when it comes to configuring and troubleshooting 802.1x (EAP) authentications in which certificates are involved one way or the other, and what kind of error messages show themselves in different situations.
Introduction
This article will mostly be a list of common issues related to certificate management and trust within the 802.1 (EAP) authentication flow and some possible solutions that can help you in your ISE deployment. The EAP-TLS method of using certificates to authenticate users and machines has been the industry standard for many years. There is no reason to stick with old username and password credentials for network access, in most cases.
Client is not trusting ISE EAP Certificate in general
12321 PEAP failed SSL/TLS handshake because the client rejected the ISE local-certificate
12520 EAP-TLS failed SSL/TLS handshake because the client rejected the ISE local-certificate
In this scenario, the client trying to connect to the network does not trust the certificate that ISE uses for EAP authentication. The first thing to look at is whether the client’s trusted certificate store contains the Root and Intermediate certificates that issued the ISE EAP certificate.
On Windows, this is done by launching the Microsoft Management Console (run “mmc.exe”) and enabling the Certificate snap-in. Look at the two folders named “Trusted Root Certification Authorities” and “Intermediate Certification Authorities” and make sure the Root and Intermediate certificates of the ISE EAP certificate are installed in each folder, respectively.
Client is not trusting ISE EAP Server Name for 802.1x/EAP
12321 PEAP failed SSL/TLS handshake because the client rejected the ISE local-certificate
12520 EAP-TLS failed SSL/TLS handshake because the client rejected the ISE local-certificate
Same error message as above but a different cause. In this case, the client is trusting the ISE certificate’s root / intermediate certificates but is expecting a certain hostname of the ISE server in its certificate.
Most commonly this happens after migrating to a new ISE deployment that has a different naming than the old one, for example migrating from nodes named ISE01 and ISE02 to new nodes named ISE03 and ISE04, but the clients are still configured to only trust servers whose certificate contains the “ISE03” and “ISE04” hostnames.
Look for the “Connect to these servers…" configuration parameter in your network adapter’s security settings (see “Smart Card or other Certificate” Properties).
Client is not trusting ISE EAP Certificate for 802.1x/EAP
12321 PEAP failed SSL/TLS handshake because the client rejected the ISE local-certificate
12520 EAP-TLS failed SSL/TLS handshake because the client rejected the ISE local-certificate
Another case of the client not properly trusting the ISE EAP certificate, in this scenario, the network settings on the client are not configured to trust the ISE EAP certificate (or more likely the certificate’s Root and Intermediate certificates) for network connections specifically.
This setting is found in the “Smart Card or other Certificate” Properties if you are using EAP-TLS and the “Protected EAP” Properties if you are using PEAP.
In the section called “Trusted Root Certification Authorities,” you must check the Root and Intermediate certificates that issued the ISE EAP certificate.
ISE is unable to check Client’s Certificate against CRL and is configured in “fail close” mode.
Unable to retrieve CRL from the server. This could occur if the specified url is unavailable.
If ISE is unable to verify the authenticating client’s certificate against a Certificate Revocation List (CRL) when configured to do so, authentication might fail if ISE is configured in the “fail close” mode when it comes to certificate verification.
While this is the most secure setting for ISE, it can cause big issues for network authentication if the CRL is unavailable for some reason that is not really related to ISE itself. The server hosting the CRL could be down for the moment or the CRL file might not have been given proper permissions to be read (downloaded) by ISE, or other services for that matter.
While sorting out your CRL file hosting should be of high priority, if you want ISE to “fail open” rather than to “fail close” when a CRL check cannot be performed, head over to Administration > System > Certificates > Trusted Certificates and select your client’s Root/Intermediate (issuing) certificates, click Edit and check the box for “Bypass CRL Verification if CRL is not Received”.
ISE is unable to check Client’s Certificate using OCSP and is configured in “fail close” mode.
12557 User Auth failed because OCSP status is unknown
12993 User Auth failed because OCSP is unreachable
This one is similar to the section above but the OCSP equivalent. If OCSP is configured to verify revocation of client certificates, be careful of the “fail close” mode if you have checked the boxes for “Reject the request if OCSP returns UNKNOWN status” or “Reject the request if OCSP Responder is unreachable” under the Root and/or Intermediate (issuing) certificates settings found under Administration > System > Certificates > Trusted Certificates > select certificate > Edit.
ISE is missing the Client certificate’s Root and/or Intermediate (Issuing) Certificates in its Trusted Store
12514 EAP-TLS failed SSL/TLS handshake because of an unknown CA in the client certificates chain
For ISE to be able to authenticate a client (either the machine or the user itself), ISE must be able to trust the client certificate presented to it by verifying the Root and/or Intermediate certificates that issued the client certificate in question. If ISE does not trust the Root and/or Intermediate certificates in the client’s certificate chain, authentication will fail.
Make sure that the Root and Intermediate certificates of the clients are installed in the Trusted Certificates store, found under Administration > System > Certificates > Trusted Certificates. Import the Root and Intermediate certificates here, and make sure not to miss the important check box described below.
ISE does not trust the Client Certificate’s Root and/or Intermediate (Issuing) certificates for Client Authentication
12514 EAP-TLS failed SSL/TLS handshake because of an unknown CA in the client certificates chain
Same error message as above, but the issue here is that while ISE has the client’s Root and Intermediate certificates in its Trusted Certificates store, they are not enabled to be used for client authentication, in which ISE needs to trust the client certificates to complete the authentication process.
This is easily fixed by navigating to Administration > System > Certificates > Trusted Certificates and editing the Root and Intermediate certificates of the client’s certificate and checking the boxes “Trusted for authentication within ISE” and “Trusted for client authentication and Syslog”.
ISE is missing the complete chain of ISE EAP Certificate in its Trusted Store
In most enterprise cases, the certificate chain used by ISE for EAP authentication contains three certificates: the ISE certificate, the Intermediate (issuing) certificate, and the Root certificate. Before the ISE certificate is imported (installed) into the ISE server, make sure that BOTH the Root and Intermediate certificates are installed before this step.
While you can import (install) an ISE certificate for EAP authentication usages when only one of the Root or Intermediate certificates is installed, you should always install both of the certificates to complete the certificate chain, or you could run into issues in which ISE is not able to present the full chain of its certificate(s), which can cause client errors.
You can also install the full chain after the fact, for instance, if you added the Root certificate to the Trusted Certificates store and then installed the ISE certificate right away without adding the Intermediate certificate first (or vice versa). Double-check the Trusted Certificates store under Administration > System > Certificates > Trusted Certificates to make sure everything looks correct.
Client Certificate is missing ClientAuthentication as Extended/Enhanced Key Usage (EKU)
12519 EAP-TLS failed SSL/TLS handshake because of an unsupported certificate in the client certificate chain
Check whether the client certificate information sent by the client is valid. Check whether the client certificate has expired. Check the client certificate Enhanced-Key-Usage. Check the OpenSSLErrorMessage and OpenSSLErrorStack for more information.
This error is more common with cloud services in my experience, where certificate templates might have to be built from scratch compared to built-in certificate templates in services like Active Directory, where most of the work is already done and most fields are properly set up.
If you are using some other certificate/PKI service than Active Directory, make sure that client certificates issued to endpoints have the Extended/Enhanced Key Usage (EKU) called ClientAuthentication.
Without this parameter configured correctly, ISE might reject the certificate or the client might not choose to present this particular certificate for network authentication. To fix this, the certificate template inside the PKI and re-issue a new correctly crafted certificate.
Client Certificate is missing Digital Signature as Key Usage parameter.
12519 EAP-TLS failed SSL/TLS handshake because of an unsupported certificate in the client certificate chain
Check whether the client certificate information sent by the client is valid. Check whether the client certificate has expired. Check the client certificate Enhanced-Key-Usage. Check the OpenSSLErrorMessage and OpenSSLErrorStack for more information.
Same error message as above and a similar root cause as well. Seen this happen on non-Active Directory Certification Authorities (mostly on cloud-based PKI services) and that is that the Key Usage parameter Digital Signature is missing from the certificate. To fix this, the certificate template inside the PKI and re-issue a new correctly crafted certificate.
Client is sending wrong Certificate to identify its User and/or Machine
22056 Subject not found in the applicable identity store(s)
or more likely:
12514 EAP-TLS failed SSL/TLS handshake because of an unknown CA in the client certificates chain
Depending on your specific environment, there might be more than one user and/or machine type of certificate installed on your device, but only one of the types of certificates is supposed to be used for network authentication towards ISE.
Some enterprise applications install unique certificates (that are only for use within that application) into the Personal certificate store on the device and if you are unlucky, the device will try to use this odd certificate for network authentication, which will most likely fail. Many of these application unique certificates often contain non-friendly names, like the CommonName field consisting of a long string of seemingly random letters and numbers instead of the user or machine’s actual name.
In Windows, certificate selection is generally very good out of the box, but if you find yourself looking at authentication attempts with strange certificate usernames in the ISE Live Log, you might have to tweak the network authentication settings of your devices.
Confirm your settings inside the “Smart Card or other Certificate” Properties and use the “Advanced” button in case you need to specify that only certificates issued by a specific issuer should be used for network authentication.
However, in my experience, using the “Use a certificate on this computer” and “Use simple certificate selection (Recommended)” works great in most cases.
Client Certificate is Revoked
12517 EAP-TLS failed SSL/TLS handshake because of a revoked certificate in the client certificate chain.
A pretty straightforward error message in which the client trying to authenticate to ISE has had its certificate revoked, for one reason or the other. The client will need to be issued a new certificate to be able to connect (securely) to the network again.
While the Root or Intermediate certificates themselves can technically also be revoked, that is a very rare scenario.
Client Certificate is Expired
12516 EAP-TLS failed SSL/TLS handshake because of an expired certificate in the client certificates chain.
While client certificates should be automatically renewed and issued, sometimes you run into issues in which a client certificate has expired and there might not be an easy (end-user friendly) way to renew it.
To deal with expired client certificates, you can configure ISE to allow expired certificates to be used anyway by navigating to Policy > Policy Elements > Results > Authentication > Allowed Protocols > Default Network Access (or whichever you are using) and checking the box for “Allow Authentication of expired certificates to allow certificate renewal in Authorization Policy”.
After this, create an Authorization Profile to make the condition “CERTIFICATE Is Expired=True” and allow the client limited network access to allow for the client to renew its certificate.
Client User certificate does not exist… yet
This issue can occur if you are doing only User authentication (a bad idea in general) or doing double authentication (Machine and User authentication, either combined or one after the other). When a User logs onto a Windows computer for the first time, a certificate for that User will be created as part of the computer’s management policies.
However, this can cause strange situations in which the computer tries to use a User certificate to authenticate to the network before the User certificate even exists.
There are workarounds/solutions for this kind of scenario, the best alternative being to utilize the fairly new TEAP authentication method. You can read more about this issue and possible solutions in this article: Windows Network Authentication Sequence.
ISE has lost connection to Active Directory or LDAP servers
Most ISE deployments depend on reliable connections to either Active Directory or LDAP servers to perform both authentication and detailed authorization based on parameters like group membership or attributes.
If ISE is unable to communicate properly to these servers, users might be unable to connect to the network. There are no good workarounds for this issue, you simply need to keep your Active Directory and LDAP servers up and running at all costs.
In the Active Directory case, you can see an alarm on the front Dashboard of ISE in the Alarms widget.
Active Directory not joined
You can make ISE re-join the Active Directory and run various tests by navigating to Administration > Identity Management > External Identity Sources > Active Directory > select your Active Directory. If ISE does not seem to be joined to the Active Directory, initiate the Join process again.
For LDAP servers, you can also run some tests by navigating to Administration > Identity Management > External Identity Sources > LDAP > select your LDAP server and under the Connections tab, click the Test Bind to Server button to see if you can get any useful error messages in the case of things not working like they are supposed to.
Network Device has not been added to ISE
11007 Could not locate Network Device or AAA Client
This one affects both 802.1x and MAB authentication and is mostly common for wired deployments since switches are usually deployed more frequently than let’s say WLAN controllers. If you forget to add your newly deployed Network Devices to ISE, their requests will not be processed.
Make sure you have all of your Network Devices added by navigating to Administration > Network Resources > Network Devices and look through the list of approved Network Devices, and Add+ more if needed.
ISE Software Bugs
While being a very powerful authentication engine, ISE is not a flawless system. There have been bugs throughout the years in some patches that have completely broken certificate authentication, making you wonder how in the world that patch goes through “quality testing” and no one tested even the most basic features.
One of the most recent big breaking bugs was covered in Field Notice FN74084: ISE Cannot Retrieve Peer Certificates During EAP-TLS Authentication:
22047 User name attribute is missing in client certificate
In this bug, certificate authentication with EAP-TLS did not work at all if EAP-TLS Session Resume and Stateless Session Resume were enabled, which is a very common setting to use since it improves authentication latency and ISE performance significantly.
And there have been plenty of other examples of similar bugs in which these two features have caused things to not work like they are supposed to, unfortunately.
12539 Failed to decrypt the EAP-TLS session ticket received from supplicant
CSCwe62979 - Stateless Session Resume feature does not work with more than one OU subject
ISE sits on other side of a VPN Tunnel/WAN circuit from Network Device causing MTU issues for RADIUS requests
If there is a VPN tunnel or WAN circuit between your Network Devices and ISE Policy nodes, chances are there could be MTU-related issues when RADIUS packets have to be sent from one side to the other. Fragmentation of authentication-related packets can cause major issues and in many cases cause authentications to not work at all.
One case I ran into recently went like this:
User connects to the network (can be either LAN or WLAN), starting the authentication process.
The Network Device sends RADIUS request to ISE server to authenticate the user. The request is small enough to make it through the VPN tunnel with no issues.
ISE gets RADIUS request and runs it through the Policy Sets, finding a matching rule.
ISE sends response back to Network Device.
ISE response is too big in size and is mangled by VPN tunnel’s lower MTU, causing the response to be fragmented or not sent through the tunnel at all.
The Network Device never gets the RADIUS response to accept the authentication and instead restarts the authentication process, and the loop continues.
References
Some good resources for ISE error messages and general troubleshooting and references.
Cisco.com - Cisco ISE Syslogs (list of all error messages)
Cisco.com - How To Troubleshoot ISE Failed Authentications & Authorizations