Cisco ISE - Fresh ISE 3.1 Re-image on physical SNS servers - Caveats

In this article, we take a look at what turned out to be a tricky re-imaging process of Cisco ISE 3.1 on SNS servers and how to work around the installation process getting stuck at various stages.

Introduction

This short article will describe several issues I ran into recently while upgrading a Cisco ISE deployment using the re-imaging method, where the re-imaging/installation process would get stuck and never be completed. While browsing the Cisco Support forums and the Cisco Bug Search Tool, I found that many others have run into similar problems while trying to re-image ISE on physical SNS servers, so I thought I’d try to collect some of the information from these threads and bugs and present them together with my own experience of performing many re-images recently.

Scenario

Re-imaging a great number of Cisco SNS-3595-K9 physical servers to run Cisco ISE version 3.1. While the SNS-3595-K9 server models are somewhat old at this point, they are supported to run ISE version 3.1, according to Cisco’s documentation.

The issues presented below seem to also apply to several different ISE versions throughout the years, but this week was the first time I ran into these issues myself as I was upgrading an ISE deployment from version 2.4 to 3.1 (via a temporary 2.7 virtual server since you cannot go directly from 2.4 to 3.1).

The Problem

Re-imaging a physical ISE server can be done in multiple ways, both in-person (that means you being on-site near the server itself) or remotely via a Virtual Mounted DVD.

Using the Virtual Mounted DVD approach did not work at all for re-imaging ISE 3.1 onto the SNS servers. When re-imaging is done remotely, there are 4 different ways to start the re-imaging process, listen below.

Via Java-based KVM Console using the Cisco ISE Installation (Serial Console) option
Via Java-based KVM Console using the Cisco ISE Installation (Keyboard/Monitor) option
Via HTML-based KVM Console using the Cisco ISE Installation (Serial Console) option
Via HTML-based KVM Console using the Cisco ISE Installation (Keyboard/Monitor) option

Neither of these options worked for the ISE 3.1 re-imaging. While the installation process would start, you would always get stuck somewhere during the installation process.

After numerous attempts to re-image ISE 3.1 on a couple of servers remotely, we would always get stuck on these (or similar) installation steps when using the Serial Console option:

Stuck at "Starting Login Service..."
Stuck at "Starting Anaconda NetworkManager configuration."
Stuck at "Starting OpenSSH ecdsa Server Key Generation."
Stuck at "Starting OpenSSH ed22519 Server Key Generation."
Stuck at "Starting Switch Root..."

As for the Keyboard/Monitor option via either the Java or HTML KVM Console, the installation progress went further but in the end, it eventually failed at the step below.

The installation would get stuck at "Downloading packages" forever, or for around an hour before the installation fails and aborts.

cisco-ise-installation-stuck-sns-server-starting-anaconda

While stuck on any of these steps, you can use “CTRL + ALT + F4” to "break out" of the installation progress view to see what is actually happening behind the scenes, but I found the information here to be mostly useless and it was impossible to point out if anything was wrong, since the log spews out a log of information that may or may not be relevant at all.

Use “CTRL + ALT + F1” to get back to the "installation progress view".

The Solution

In the end, the only re-imaging option that made it all the way through was:

Re-image via USB stick using the Cisco ISE Installation (Keyboard/Monitor) option

Using this option, a USB stick with the Cisco ISE ISO image installed onto it as a bootable media using the software LiveUSB-Creator is used as the boot option for the SNS server and then selecting Cisco ISE Installation (Keyboard/Monitor) option. Using this combination the installation process continued to completion without any issues along the way.

After pressing F6 during the server bootup process to get into the boot options menu, look for the name of your USB stick and then select it to start the installation.