The problem
We had built ISE 3.x in parallel with the patched 2.4 environment, which meant every one of the 1,500+ devices in the TACACS estate still pointed at the old one. So did the wireless. The parallel build was the right architecture call, and it came with a bill: nothing moves itself. Every TACACS server statement on every device and every AAA reference on every WLAN had to be repointed on purpose, in a window, with a record.
Repointing all of that by hand, device by device, engineer by engineer, is how enterprises end up with a half-migrated estate and no reliable record of which half. The migration needed to be a process, not a heroic effort.
My role
I built the AAA migration project end to end: the audit scope, the window structure, the verification standard, the documentation, and the automation pipeline that let other engineers run windows without me. I worked plenty of the windows myself too.
The approach
Audit first. Before touching anything, I inventoried every WLAN and every AAA reference across a 900-AP global estate. Early windows captured client counts per SSID, so we knew what normal looked like before we changed it.
The wireless design removed most of the risk up front. Every Catalyst 9800 was pre-staged with the PSNs from both ISE environments, selectable per SSID. Moving a WLAN from 2.4 to 3.x became a selection, not a rebuild.
Two rules governed every window. Verification and rollback steps were written before the window opened, never improvised inside it. And no change does two things: if a task touched AAA and anything else, it became two tasks.
Both rules earn their keep at 2am. When verification is written before the window, nobody debates afterward whether the change worked; the evidence is either there or it is not. When no change does two things, a failure has exactly one suspect.
Verification was not passive. For high-profile users we ran white-glove checks on live calls, watching their authentication flows in ISE in real time, with a ticket and a Webex thread carrying the follow-through until the user confirmed clean.
For the wired estate, the process became software. I built an Ansible pipeline with NetBox as the dynamic inventory, so there was no static host list to drift: devices group by site and role straight from the source of truth. Jinja2 templates render the correct TACACS and AAA configuration per platform, because legacy IOS, IOS-XE, and NX-OS each speak their own syntax. Every run backs up the device config, applies the change, validates it, and arms a five-minute automatic rollback timer that reverts the device if anything goes wrong. Credentials live in Ansible Vault, every window starts with a pre-flight check and a dry run, and on success the pipeline writes the device’s migration status back to NetBox. The inventory tracks its own progress.
What shipped
Runbooks, published to Confluence. Controller inventories that record which controllers exist and which PSNs they talk to. Deployment instructions that read the same on the first window and the fortieth. Per-network-type migration guides, so a wired TACACS repoint and a wireless WLAN move never share a checklist that fits neither. And the guest and DMZ design doc, so the most exposed edge of the wireless network is defined in writing rather than in someone’s memory.
The pipeline shipped as a real repository, not a folder of scripts: Python and Ansible under version control, CI that lints, syntax-checks, and scans for leaked secrets on every change, a clean template branch so new deployments start from known-good, and a runbook that lets an assigned engineer execute a migration with a handful of make commands. Dry run first, always.
The bar for all of it was that any engineer could execute a migration window without me in the room. That is the difference between a project and a process.
Where it stands
1,500+ devices repointed. 190 WLANs moved. Around 10,000 daily wireless clients rode through the cutovers, including the high-profile users who stayed on live calls while we watched their auth flows land on the new environment. Nobody noticed, and silence was the success metric. The runbooks still live on Confluence, which means the next migration starts from page one of a written process instead of from scratch.