Cisco WLC C9800 - Software Upgrade

Note: The description in the article is based on the Cisco Catalyst 9800-L Wireless Controller with Cisco IOS XE Cupertino version 17.9.3. To which Cisco Catalyst 9164I AP access points are connected.

Documentation

Cisco describes the system upgrade options in the official configuration guide. Additional information can be found in the continuously updated document on the recommended version of Cisco IOS XE. Personally, I find the description quite unclear and confusing. Different terms are even used for the same things.

Software and its Versions

Types of Software

IOS XE Software - complete IOS XE software image, i.e., the controller's own software, BIN file, size 1.3 GB
Software Maintenance Upgrade (SMU) - a package (patch) used to fix bugs or vulnerabilities in the current version, BIN file, usually small in size
Access Point Service Pack (APSP) - a package that fixes software bugs in the AP or minor features that do not require changes on the controller, requires AP restart, BIN file
Access Point Device Pack (APDP) - adds support for newer AP models to the current WLC version, BIN file
Field Programmable (FPGA) Firmware - on physical WLCs, we can also upgrade
- ROM Monitor (ROMMON) - initializes hardware and launches IOS-XE, PKG file
- PHY (Ethernet or Fiber) - physical layer, specifically the Shared Port Adapter (SPA) module, PKG file
NBAR2 Protocol Pack - a package that updates the Network-Based Application Recognition (NBAR) engine (application recognition information) of the current WLC version, PACK file

Note: In the case of FPGA Firmware, Cisco always recommends updating to the latest available version. The currently available latest versions are quite old. We can perform the check and installation using the CLI.

A certain IOS XE Software version includes defined features, fixes for older bugs (essentially SMU), support for certain AP models and software for them (WLC and AP must have the same software version), and NBAR2 Protocol Pack for Application Visibility and Control (AVC).

If we want to obtain new features, bug fixes, or support for new APs, we can upgrade to a new version (if available). WLC upgrade always also upgrades the AP and requires a restart. If we have a High Availability (HA) Stateful Switch Over (SSO) pair, the active and standby WLC will switch, and there may be no service interruption. APs can be updated gradually in groups, so some can still serve clients.

It is not always necessary to update the WLC. We can add SMU, APSP, APDP, or NBAR2 Protocol Pack to the existing version. They are only released for certain versions (Extended Maintenance).

Obtaining Software

Installation files can be found on the product page or completely under Support - Download.

We select the C9800 model for which the software types will be offered. In the case of Catalyst 9800-L, we must choose the exact type 9800-L-C or 9800-L-F, otherwise only a small portion of the files will be offered.

Version Labeling

Software Lifecycle Support Statement - IOS XE

Software and its Versions

Types of Software

IOS XE Software - complete IOS XE software image, i.e., the controller's own software, BIN file, size 1.3 GB
Software Maintenance Upgrade (SMU) - a package (patch) used to fix bugs or vulnerabilities in the current version, BIN file, usually small in size
Access Point Service Pack (APSP) - a package that fixes software bugs in the AP or minor features that do not require changes on the controller, requires AP restart, BIN file
Access Point Device Pack (APDP) - adds support for newer AP models to the current WLC version, BIN file
Field Programmable (FPGA) Firmware - on physical WLCs, we can also upgrade
- ROM Monitor (ROMMON) - initializes hardware and launches IOS-XE, PKG file
- PHY (Ethernet or Fiber) - physical layer, specifically the Shared Port Adapter (SPA) module, PKG file
NBAR2 Protocol Pack - a package that updates the Network-Based Application Recognition (NBAR) engine (application recognition information) of the current WLC version, PACK file

Obtaining Software

Installation files can be found on the product page or completely under Support - Download.

Version Labeling

Cisco IOS XE versions are released at regular intervals. The version designation looks like this 17.9.4a. It consists of Major.Minor.Maintenance Release Number, optionally supplemented by a special identifier (in the example a). The Minor version number indicates a significant change in the software and is released 3 times a year. The Maintenance number indicates a release with critical bug fixes (rebuild).

Each Cisco IOS XE version is classified as:

Standard-Support Release (Standard Maintenance) - 12 months support, includes versions 17.1, 17.2, 17.4, 17.5, 17.7, etc., presumably also referred to as short-lived release
Extended-Support Release (Extended Maintenance) - 48 months support, maintenance versions are released more frequently (after 3, 4, 4, 6, 7 months), includes versions 17.3, 17.6, 17.9, 17.12, etc., presumably also referred to as long-lived release

When downloading, versions are labeled with the abbreviation Release Designations.

Early Deployment (ED) - early deployment, includes new features and bug fixes
Maintenance Deployment (MD) - includes bug fixes and ongoing software maintenance

Vulnerabilities and Their Fixes

In the last (approximately) month, a very serious vulnerability CVE-2023-20198 in Cisco IOS XE has been addressed (there have been other vulnerabilities this year as well). Information CSCwh87343 - Cisco IOS XE Software Web UI Privilege Escalation Vulnerability, Multiple Vulnerabilities in Cisco IOS XE Software Web UI Feature and Software Fix Availability for Cisco IOS XE Software Web UI Privilege Escalation Vulnerability - CVE-2023-20198.

In the Cisco IOS XE 17.9 series, the fix is in version 17.9.4a. For versions 17.9.3 and 17.9.4, a Software Maintenance Upgrade (SMU) with the fix is available, which requires a restart.

The problem is if we use the Lobby Admin role to create guest accounts, and due to the vulnerability fix, we upgrade to 17.9.4a. We will likely encounter the bug CSCwh37783 - Catalyst 9800 Lobby Admin Page is not Loading (very poorly described). The only solution now is probably to upgrade to 17.12.2.

Note: For information on vulnerabilities of a specific version, we can use the Cisco Software Checker.

Recommended Version

The document Recommended Cisco IOS XE Releases should provide information on which version is recommended for production use. But I probably don't know how to read it. It contains information about available versions and their SMU (Software Maintenance Upgrade) and APSP (Access Point Service Pack). The only question is how often the document is updated, as several of the latest APSPs are currently missing.

We should probably choose a version from the long-lived release train (which will be the Extended-Support Release) and Cisco will start recommending it only after a certain Maintenance Release (MR). For some versions, there is a note that it is recommended to upgrade to a specific higher version or apply an SMU.

Currently, the document highlights the recommendation

Cisco recommends 17.9.4 + APSP2 + SMU_CSCwh87343 for all deployments

But when we download version 17.9.4, there is information that I understand to mean that version 17.9.4a is recommended

Dear Cisco Customer, If you are not using APSP in 17.9.4, please use 17.9.4a, to obtain fix for CSCwh87343, Cisco IOS XE
 Software Web UI Privilege Escalation Vulnerability, CVE-2023-20273. In case of SMU/APSP installed, please wait until SMU
 for CSCwh87343 is available for 17.9.4

WLC Software Update

Administration - Software Management

We have the option to upgrade the controller by transitioning to a new version using the complete IOS XE image. During the upgrade, we can use features like ISSU or Hitless Upgrade. Or install SMU, APSP, or APDP into the current controller version.

Note: If we have a HA SSO pair, both the active and standby controllers will be updated during the upgrade.

WLC C9800 WebUI - Administration - Software Management - Software Upgr

IOS XE Software Upgrade

Administration - Software Management - Software Upgrade

We can perform the update using the GUI (or of course also in CLI). The following basic options are available. Some switches change the upgrade method and offer additional settings. The description is only brief.

Upgrade Mode: INSTALL (most commonly used mode)
One-Shot Install Upgrade - the entire installation is done at once, the restart is performed automatically, and it does not wait for activation
Transport Type: My Desktop - allows uploading the file via browser (HTTPS)
File System: bootflash
AP Image Predownload - during the upgrade, AP images are preloaded into access points to shorten network downtime after the upgrade
ISSU Upgrade (HA Upgrade) - software upgrade using ISSU
- AP Upgrade per Iteration - the percentage of APs to be upgraded together (in one step)
Hitless Software Upgrade (N + 1 Upgrade) - APs are upgraded gradually, so there is no network-wide outage
Download & Install - uploads and installs the upgrade
Save Configuration & Activate - restarts the WLC with the new software image
After the upgrade, it is necessary to confirm by clicking Commit to make the upgrade permanent. Otherwise, (by default after 6 hours) the WLC will revert to the previous image.
Remove Inactive Files - cleans up old installation files

We cannot combine One-Shot Install Upgrade, AP Image Predownload, ISSU Upgrade (HA Upgrade), and Enable Hitless Upgrade. Activating one option hides the others. Some methods offer similar options or include another method.

Note: In the configuration guide, I did not come across such information, but in the document on recommended versions, there is a note. SMU and APSP require a Network Advantage license, Network Essentials is not sufficient. In the documents Implement 9800 Wireless LAN Controller Licenses: FAQs, Cisco DNA Software Wireless Feature Matrix we can learn even more. ISSU, APDP, APSP, Rolling AP upgrades, Hot Patching, and SMU are part of the Network Advantage license.

Standard Upgrade (with AP Image Predownload)

During a standard upgrade, there will be a network functionality outage for several minutes. All components will restart at the same time. In the case of an HA SSO pair, both WLC and all APs. We can use preloading the image into APs so that copying is not done after the WLC upgrade.

We can always use Download & Install because it does not affect operations. It depends on the architecture, it can take about half an hour. Only when we use Save Configuration & Activate will there be an outage and restart. The WLC will become unavailable after about 5 minutes, and the startup of both takes about 10 minutes.

WLC C9800 WebUI - Software Upgrade AP Image Predownload

Upgrade Process

uploading the image to flash memory (for HA pair on both WLCs) - start by clicking Download & Install
installing the image (adding the image to all members)
preloading the AP image into access points (gradually after a certain number of APs)
activating the image, which will restart the WLC and APs - start by clicking Save Configuration & Activate
confirming changes - by clicking Commit

WLC C9800 WebUI - Software Upgrade AP Image Predownload Status

We can monitor the preloading (Predownload) of the image into APs in the GUI in the details of individual APs. Or in the CLI using the command show ap image.

Note: In practice, I encountered a strange problem. After the restart, the WLC was still unavailable, time was passing, and even after 45 minutes, it did not respond at the WMI IP address. I managed to connect via SSH to the RMI IP address (using a local account). At first glance, everything looked OK, but no outgoing communication to the network was working. I tried pinging various interfaces of both WLCs, and suddenly all communication started working.

Hitless Upgrade

CAPWAP implementation requires that the WLC and AP have the same software version. Therefore, after upgrading the WLC, the AP upgrade follows, causing a network outage.

Hitless Upgrade utilizes high availability (HA SSO) with two WLCs and coverage by N+1 APs. First, the standby WLC is upgraded to the target version. APs are upgraded gradually (moving to the second WLC) using the Rolling AP upgrade feature, which prevents network disruption. Clients are served by neighboring APs when theirs undergoes an upgrade. At the end, the primary WLC is restarted (upgraded), and the APs move back.

In-Service Software Upgrade (ISSU)

ISSU is a procedure for upgrading the WLC image to a newer version while the network continues to forward packets. It helps to avoid network downtime during the upgrade. It can be used to apply Cold Patch SMUs without affecting the active network.

ISSU requires a HA Stateful Switchover pair, i.e., an active and standby controller. Both must be in Install mode. The documentation also states that it is supported for upgrades within and between long-lived major releases (the description is not very clear to me). A compatibility check is performed.

Upgrade Process

uploading the image to flash memory (for HA pair on both WLCs) - start by clicking Download & Install
installing the image (adding the image to all members)
preloading the AP image into access points (gradually after a certain number of APs)
we receive information that the ISSU procedure has started activating the new software, and once the software is installed on the active WLC, it will restart, causing a switchover
upgrading the standby controller - checks are performed, image activation begins, WLC restart
upgrading the active controller - activation and restart, causing a switchover to the standby
switchover to the standby
upgrading the APs - gradual upgrade (restart) of APs according to the specified number
confirming changes - by clicking Commit

WLC C9800 WebUI - Software Upgrade ISSU - Status

WLC C9800 WebUI - Software Upgrade ISSU - Activate

The entire upgrade process is gradual and takes quite a long time. Around an hour for the controller upgrade, and then the APs are updated according to the specified amount (an estimate of 8 minutes was displayed, but it took 43 minutes).

During the upgrade, we can view logs Show logs, and later AP Upgrade Statistics. Before the upgrade, we can check if ISSU is supported and monitor the status during the process (using CLI):

WLC1#show issu state detail
Current ISSU Status: In Progress
Previous ISSU Operation: N/A
=======================================================
System Check                        Status
-------------------------------------------------------
Platform ISSU Support               Yes
Standby Online                      Yes
Autoboot Enabled                    Yes
SSO Mode                            No
Install Boot                        Yes
Valid Boot Media                    Yes
Operational Mode                    HA-REMOTE
=======================================================
Added Image:
Name                                Compatible
-------------------------------------------------------
17.09.04a.0.6                       Yes

Operation type: Step-by-step ISSU
Install type  : Image installation using ISSU
Current state : Added state
Last operation: Activate location standby chassis 2/R0cc

Completed operations:

Operation                              Start time
-------------------------------------------------------
Activate location standby chassis 2/R0 2023-11-19:11:31:49

State transition: Added

Auto abort timer: automatic, remaining time before rollback: 05:49:57
Abort Reason: N/A
Running image: bootflash:/packages.conf
Operating mode: sso, terminal state not reached

Software Maintenance Upgrade (SMU) Patch

Administration - Software Management - Software Maintenance Upgrade (SMU)

SMU is a package that can be installed into the system to fix a bug in a given version. SMU is small, fixing only a specific bug in a particular component. It can be quickly deployed and does not require extensive testing.

IOS XE validates SMU compatibility and does not allow the installation of incompatible ones. SMUs are released only for Extended Maintenance versions. They integrate into a specific maintenance version. They have no prerequisites or dependencies. We can install or uninstall them in any order.

SMU installation can be non-disruptive or may trigger a restart, reload, or HA pair switchover.

Cold Patch requires a system reload during activation. This ensures all processes start with the correct libraries and files installed with the SMU. For a Stateful Switchover (SSO) pair, it can be applied without downtime.
Hot Patching allows the SMU to be effective immediately after activation, without a system reload (it may restart certain processes).

WLC C9800 WebUI - Administration - Software Maintenance Upgrade (SMU)

Installation Procedure

add the SMU file to the device
after uploading, a compatibility check is performed
we see the list of uploaded packages, their status (whether they are active) and type (reload, restart, or non-reload)
- Reload means a controller restart
- Restart means a specific process restart
to install the package, select it and click Activate
to make the installation permanent, we must use Commit (otherwise it will deactivate after the second restart)