In this article, we take a look at how you can use an Access Control List (“ACL”) to control your ISE PSN nodes available for RADIUS/TACACS communication and how to set up a testing environment halfway through the ISE deployment upgrade process.
Introduction
When you are upgrading an ISE deployment either using the Backup and Restore method or via the Inline Upgrade method, it is a good idea to test your new deployment halfway through the upgrading process, when half of your ISE servers are upgraded and half remain on the old ISE version.
When the first Policy Service Node (“PSN”) is re-image/upgraded from the previous ISE deployment to the new ISE deployment, and you plan to keep the same IP address after the re-image/upgrade, it is wise to create a scenario where no Network Devices at all can communicate with this new PSN node until you have had a chance to test it. To achieve this, you could use one of the methods below:
Remove the to-be re-imaged/upgraded PSN node’s IP address from all Network Device’s configurations (can be tough if you have a lot of Network Devices).
Block access to the PSN node’s IP address from all Network Devices during the upgrade process (relatively easy regardless of the number of Network Devices). This would be done using an Access Control List (“ACL”) to deny the RADIUS/TACACS+ traffic to the ISE PSN node that is being re-imaged/upgraded.
When the PSN node is finished being re-imaged/upgraded and joined to the new ISE deployment, you need to find a Network Device (a WLC or a switch) that you can force to send authentication requests to your new ISE deployment’s PSN node’s IP address while letting the rest of your Network Devices only talk to the old ISE deployment PSN node(s), which is still up and running.
There are two steps to be considered for this testing phase, assuming you went with option 2 above:
Update the ACL to make only the chosen test Network Device(s) able to communicate with the newly re-image/upgraded ISE PSN node’s IP address.
Still keep blocking all other Network Devices from communicating with the new ISE PSN node’s IP address.
By manipulating permitted and denied traffic this way, all Network Devices except the chosen test Network Device will mark this server as “DEAD”, meaning they won’t send any more authentication attempts to it, while the chosen test Network Device will have no problems communicating with the new ISE PSN node to verify that everything is working correctly.
If you are only using ISE for wireless connections (using a WLC), it is probably easier to schedule a maintenance window where you can test to simply point your 802.1X/MAB/Guest SSIDs towards the new ISE deployment and see if things work, then to fiddle around with ACLs. However, if you also are using ISE authentication for wired connections, going into each of those switches to change which ISE PSN nodes should be active in each switch is a big hassle, making it more convenient to use central ACLs to force the required testing process.
Depending on your specific environment, its size, design, and whether you are using virtual or physical ISE servers, this configuration can in general be applied using an Access Control List (“ACL”) at these places in your network:
Firewall - deny RADIUS/TACACS+ communication to or from the ISE PSN node’s IP address using the firewall ruleset.
Core Switch - ACL applied to VLAN interface or physical interface.
Datacenter Switch - ACL applied to VLAN interface or physical interface.
Depending on your specific network’s design, the permit/deny ACL or firewall rule may be applied in either the inbound or outbound direction to get the wanted result.
The topologies above are only examples, for the rest of the article, we will assume that you are configuring the ACL on the Cisco Core switch (the middle example in the image above), using an inbound VLAN interface ACL on VLAN 110, which is where all the ISE PSN nodes are connected.
If all the Network Devices are connected to the same Management VLAN, one also could consider putting the ACL in the inbound direction on that Management VLAN as an alternative, provided you “flip” the ACLs source/destination ports correctly to properly catch the traffic to be denied/permitted. In this article, however, we will assume that Network Device management is spread out across different VLAN and IP networks, so it would be better to put the ACL on the VLAN interface where the ISE PSN nodes are connected.
Configuring a Service Object Group
If you plan to put your ACL on a newer Cisco switch/routing device, you can use a Service Object Group to group all the PSN-specific ports in one object and reference them all at once instead of making one Access List Entry (“ACE”) for every port/range of ports.
The ports we will be working with and referencing in the ACL are the standard ports for RADIUS/TACACS+:
Ports UDP 1645-1646 = Legacy RADIUS ports
Ports UDP 1812-1813 = New RADIUS ports
Port UDP 2083 = RADIUS over DTLS
Port TCP 2083 = RADIUS over TLS
Port TCP 49 = TACACS+
Create the Object-group for the services using the commands below:
object-group service PSN-PORTS
udp source range 1645 1646
udp source range 1812 1813
tcp-udp source eq 2083
tcp source eq tacacs
With the Object-group now complete, it can be referenced in the ACL to block communication from PSN nodes that are currently in the process of being re-imaged/upgraded.
Since the Object-group itself already states that the ports inside it are source ports, the ACE where it is being used will look a bit “odd” at first sight.
ACL to block PSN ports during node Re-image/Upgrade using Object-Group for Services
As mentioned previously, it is a good idea to block communication to the IP address of the ISE PSN node that is currently being re-imaged/upgraded, so no Network Devices try to send requests to it before it is ready.
The ISE PSN node being re-imaged/upgraded in the example below will have the IP address of 10.10.10.102.
We will permit some of the protocols required for ISE to work properly, like DNS, NTP, and communication to Active Directory Domain Controllers. If you are using special integrations between ISE and other external systems (like DNA Center, SQL Databases via ODBC, etc.), then it’s a good idea to permit those IP addresses/networks similar to sequence “40” below.
ip access-list extended ISE-ACL
! Permit DNS and NTP traffic
10 permit udp any any eq domain
20 permit tcp any any eq domain
30 permit udp any any eq ntp
! Permit Traffic to AD DC network
40 permit ip any 10.20.30.0 0.0.0.255
! Manipulate ISE-PSN to Network Device (any) traffic
100 deny object-group PSN-PORTS host 10.10.10.102 any
! Permit all other traffic
1000 permit ip any any
ACL to allow certain Network Device(s) to talk to new ISE PSN using Object-Group for Services
To build upon the ACL above and allow just one Network Device (either a WLC or a Switch) to communicate to the new ISE deployment, add another “permit” entry just before the general deny rules that start at sequence “100”.
Down below, we will add entry sequence “90” to the ACL, where we permit a single switch (with IP address 10.99.99.50) to communicate to the newly upgraded ISE PSN node in the new ISE deployment. Iif you have several WLCs in your environment, you can use that for testing too.
We can now test RADIUS, TACACS+ and RADIUS DTLS/TLS for this specific switch while the rest are still only communicating to the old ISE deployment.
ip access-list extended ISE-ACL
10 permit udp any any eq domain
20 permit tcp any any eq domain
30 permit udp any any eq ntp
! Permit Traffic to AD DC network
40 permit ip any 10.20.30.0 0.0.0.255
! Manipulate ISE-PSN to Network Device (any) traffic
90 permit ip any host 10.99.99.50
100 deny object-group PSN-PORTS host 10.10.10.102 any
! Permit all other traffic
1000 permit ip any any
Now that the ACL is permitting certain a Network Device(s) to send requests to the newly upgraded ISE PSN node, you may have to force this Network Device to send its RADIUS/TACACS requests to this specific server. Most switches running newer IOS/IOS-XE version, and the new 9800 WLCs, can already use a “round-robin” or “least outstanding” approach to distribute requests to different configured RADIUS servers.
If you are still using the older WLCs (5520/8540 models) running AireOS software, you may need to simply set this newly upgraded ISE PSN node as the first AAA server for a specific SSID to force some RADIUS requests to hit the new ISE deployment. If things don’t look good in the new ISE deployment’s Livelog after a few minutes, you can easily revert the WLC configuration back to what it used to be.
ACL to block PSN ports during node Re-image/Upgrade WITHOUT using Object-Group for Services
If your network device (Core or Service Distribution layer most likely) does not support object groups for all the ISE PSN services, you will need to create a normal ACL instead to deny the interesting traffic (RADIUS, TACACS+, and TLS/DTLS RADIUS).
If you are not using all the related protocols (TACACS, RADIUS (old and new ports), or RADIUS over DTLS/TLS), remove those entries from the ACL to keep it short and simple.
ip access-list extended ISE-ACL
10 permit udp any any eq domain
20 permit tcp any any eq domain
30 permit udp any any eq ntp
! Permit Traffic to AD DC network
40 permit ip any 10.20.30.0 0.0.0.255
! Manipulate ISE-PSN to Network Device (any) traffic
100 deny udp host 10.10.10.102 range 1645 1646 any
110 deny udp host 10.10.10.102 range 1812 1813 any
120 deny udp host 10.10.10.102 eq 2083 any
130 deny tcp host 10.10.10.102 eq 49 any
! Permit all other traffic
1000 permit ip any any
ACL to allow certain Network Device(s) to talk to new ISE PSN WITHOUT using Object-Group for Services
To build upon the ACL above and allow just one Network Device (either a WLC or a Switch) to communicate to the new ISE deployment, add another “permit” entry just before the general deny rules that start at sequence “100”.
Down below, we will add entry sequence “90” to the ACL, where we permit a single switch (with IP address 10.99.99.50) to communicate to the newly upgraded ISE PSN node in the new ISE Deployment. We can now test RADIUS, TACACS+, and RADIUS DTLS/TLS for this specific switch while the rest are still only communicating to the old ISE deployment.
If you are not using all the related protocols (TACACS, RADIUS (old and new ports), or RADIUS over DTLS/TLS), remove those entries from the ACL to keep it short and simple.
ip access-list extended ISE-ACL
10 permit udp any any eq domain
20 permit tcp any any eq domain
30 permit udp any any eq ntp
! Permit Traffic to AD DC network
40 permit ip any 10.20.30.0 0.0.0.255
! Manipulate ISE-PSN to Network Device (any) traffic
90 permit ip host 10.10.10.102 host 10.99.99.50
100 deny udp host 10.10.10.102 range 1645 1646 any
110 deny udp host 10.10.10.102 range 1812 1813 any
120 deny udp host 10.10.10.102 eq 2083 any
130 deny tcp host 10.10.10.102 eq 49 any
! Permit all other traffic
1000 permit ip any any
Now that the ACL is permitting certain a Network Device(s) to send requests to the newly upgraded ISE PSN node, you may have to force this Network Device to send its RADIUS/TACACS requests to this specific server. Most switches running newer IOS/IOS-XE version, and the new 9800 WLCs, can already use a “round-robin” or “least outstanding” approach to distribute requests to different configured RADIUS servers.
If you are still using the older WLCs (5520/8540 models) running AireOS software, you may need to simply set this newly upgraded ISE PSN node as the first AAA server for a specific SSID to force some RADIUS requests to hit the new ISE deployment. If things don’t look good in the new ISE deployment’s Livelog after a few minutes, you can easily revert the WLC configuration back to what it used to be.
Applying the ACL on a VLAN interface
Apply this ACL in the inbound direction on the VLAN interface to which your ISE PSN node belongs or the physical interface where you’re SNS server running ISE is connected. I would recommend applying this list to the VLAN interface to keep all control in one place and not spread out among possible several switches.
interface Vlan 110
ip access-group ISE-ACL in
After the ACL has been applied to the VLAN interface successfully, you will see the automate-tester attempts arrive at ISE in the Livelog (if you have that feature configured on your WLC/switches, which you definitely should!), but ISE will not be able to respond to these RADIUS requests since they are blocked by the ACL. The Network Device will then see that ISE PSN node as “DEAD” because there is no response for their RADIUS requests. The automate-tester function runs fairly frequently (at least on switches where it’s once every 5 seconds), so you should be able to catch these attempts in the Livelog.
During the ISE upgrade process, you may need to adjust this ACL accordingly to match what’s going on at the moment, which PSN is being upgraded next, etc.
Use the “sequence…” command inside the ACL to make sure your entries end up in the correct spot and not just at the bottom of the ACL.
If you find yourself “running out” of sequence numbers that are convenient to use, use the “ip access-list resequence…” command to resequence the entire ACL to create more space between the entries. For example, use “ip access-list resequence ISE-ACL 10 10” to set the first entry to “10” and then increase the sequence by “10” for every entry going forward (20, 30, 40, and so on).
Verification
For IOS, IOS-XE switches, and IOS-XE WLCs, you can use the command “show aaa servers” to see which AAA servers are considered alive (“UP”) and down (“DEAD”).
To make the output a little easier to read, you can pipe the command to simply include only the important parts. To do this, use the command “show aaa servers | include RADIUS:|State:”.