We Retired VSS. The Redundancy Model Stayed.

When a WAN aggregation platform hits end of support, the obvious move is to modernize everything at once: new hardware, new redundancy model, new automation hooks, all in the same project. We modernized the hardware. We kept everything BGP actually cared about exactly the same, even though the technology underneath it had to change.

The platform we were leaving

The switches doing WAN aggregation were a pair of Cisco 6840-X-LE-40G devices, running as a VSS pair. Virtual Switching System: two physical chassis presenting as one logical switch to everything around them. Every BGP peer, every spanning-tree calculation, every downstream device treated that pair as a single unit, because it was one, as far as the control plane was concerned.

That platform had been carrying real weight for years. Roughly 150 VLANs spanning data, voice, management, and transit traffic. Over 40 BGP sessions spread across a set of customer VRFs. 32 GRE tunnels out to remote sites and cloud providers. Four WAN carrier links. Just under 100 physical and port-channel device connections riding through it. And it was aging out, with end of support landing before the year was through, which meant the clock on vendor patches and hardware replacement parts was already running.

The replacement was a pair of Cisco Catalyst 9500 switches.

Retiring VSS doesn’t mean retiring the model

Here’s the part that’s easy to get wrong in a project summary: the new Cat9500 pair doesn’t run VSS. It can’t. VSS is a technology built for the Catalyst 6500 and 6800 generation, running classic IOS. The 9000 series runs IOS-XE, and IOS-XE’s answer to the same problem is a different technology: StackWise Virtual.

Different plumbing, same job. Two Cat9500 chassis connect over a dedicated StackWise Virtual Link, and an election process picks one chassis active and the other standby: active handles routing protocols and management, standby stays hot and ready to take over. From BGP’s point of view, that’s the same relationship the old 6840 VSS pair had with everything around it. One control plane, one session per peer, nothing to rebuild.

That single-control-plane design does carry one real risk, and it’s worth naming instead of glossing over: if every link between the two chassis goes down at once, the standby has no way to tell whether the active chassis actually died or just the connection to it did. Left unhandled, both switches would conclude they’re in charge, a split-brain condition StackWise Virtual calls a dual-active scenario, and you’d have two chassis with the same identity both trying to run the network at the same time. The safeguard is dual-active detection: a dedicated path, separate from the main link between the chassis, whose only job is confirming which one is really active. If the two ever lose track of each other, the one that’s actually cut off shuts its own network-facing interfaces down instead of fighting the other for control.

That wasn’t the only redundancy model available on this class of switch. vPC, virtual Port Channel, is the other one Cisco supports here, and it works differently on purpose: two genuinely independent control planes with a synchronized data plane, no single shared BGP process to inherit at all. Plenty of shops reach for vPC on a refresh like this one. We didn’t, because the actual goal was a capacity and performance upgrade, not a control-plane redesign. Rebuilding BGP redundancy at the same time as swapping the underlying hardware means two categories of risk sitting in the same maintenance calendar instead of one. If something breaks at 2 a.m. on a weekend, you want exactly one variable to have changed since the last time everything worked.

So the Cat9500 pair got staged, tested, and brought up as a StackWise Virtual pair, the platform’s own answer to the same job the 6840’s VSS pair had been doing. Every BGP session that used to terminate on “the switch” still terminates on the same logical switch. Nothing downstream, no peer, no monitoring system, no piece of existing automation, had to know or care that the technology underneath had changed at all. The redundancy model’s relationship to BGP wasn’t the thing we were fixing. The capacity was.

Seven windows, one hard rule in every one of them

None of this got touched outside a scheduled maintenance window, and none of it got attempted without a validated rollback plan already written. The cutover ran across seven separate windows, each one scoped narrowly enough that a problem in one category couldn’t take the others down with it.

The new switches got staged and validated in complete isolation first: racked, powered, configured offline, the StackWise Virtual link formed and failover tested before any of it ever touched production traffic. Only after that passed did Layer 2 connectivity go in between the old and new platforms, with the legacy switches deliberately left as the spanning-tree root through that handoff window so the topology didn’t shift out from under live traffic while both platforms were up at once. BGP came next, and not as a hard cutover: local preference got adjusted first so traffic started favoring the new path gradually, with room to watch it happen before anything old was touched. GRE tunnels moved in two batches of sixteen, one batch validated before the next one started. WAN carrier BGP sessions got their own dedicated window. The remaining client BGP sessions across the VRFs got theirs. Only after every one of those held clean did the legacy 6840 pair come out of the network for good.

Every window was capped at four hours. Every window had its own rollback plan ready before it started, not improvised if something went sideways. Every phase ended with a formal validation checkpoint before the next one was allowed to begin.

I led the technical execution on this one, not just the plan on paper, working pieces of it hands-on alongside the rest of the team.

The business case

The capacity jump is the one part of this that doesn’t need my own after-the-fact measurement to back it up. Cisco’s own numbers for the two platform families make the shape of it clear. The 6840-X-LE-40G runs native 10G ports (two 40G uplinks aside), topping out around 240 Gbps of switching capacity per unit, or up to 1.92 Tbps as a VSS pair. The Catalyst 9500 line runs 100G-capable ports and multi-terabit switching capacity even on its smaller members. Whichever exact model ended up in the rack, that’s a real, roughly tenfold jump in what a single uplink or a single chassis can move, not a rounded-up marketing number.

The rest of the case: lower latency, a smaller spanning-tree domain, its own kind of risk reduction, since a smaller domain means fewer things that can go wrong in the same failure blast radius, and higher east-west throughput between key server pairs, the plan being to confirm that last one with iPerf testing run before and after rather than take it on faith. Those were the targets the project got built around, and the cutover itself was scoped for zero unplanned outages across all seven windows, with nothing beyond the sub-10-millisecond convergence blips you’d expect during a controlled transition.

The new platform also opens doors the 6840s never could: VXLAN, segment routing, the building blocks for a software-defined access model, whenever that becomes the next project instead of this one.

The distribution and server-edge layer got its own hardware refresh around the same timeframe, on its own separate project scope and its own separate decisions. That’s a different story for a different post.

The boring choice was the hard one

It would have been easy to sell this project as a full architecture modernization: new hardware, new redundancy technology, new everything, in one announcement. That’s not what happened, and the reason it’s not what happened is the actual lesson here. A hardware refresh and a redundancy-model redesign are two different projects wearing the same maintenance window if you let them be. Retiring VSS meant the technology underneath had to change, StackWise Virtual instead of VSS, because VSS was never an option on this hardware to begin with. What didn’t have to change was BGP’s relationship to any of it. Every peer, every piece of monitoring, every downstream assumption about how that WAN core behaved kept working exactly the way it always had, while the platform underneath it was rated for a tenfold jump in capacity. That capacity jump is the business case the rest of the targets above were built around.

That’s the actual measure of whether a migration like this went well. Not that new switches are racked where old ones used to be. That nothing anyone downstream had to notice, changed at all.