Cisco Network Switches Hit by DNS-Related Reboot Bug

Devices in a number of Cisco switch types experienced significant reboot loops due to a pervasive firmware problem in the internal DNS client service. The DNSC (DNS Client) job was the source of the problem since it mistakenly regarded DNS lookup failures as fatal errors, particularly when trying to resolve “www.cisco.com” or NTP time servers.

From a recent discovery, this has being seen as a complaints that many Cisco switch models are abruptly going through reboot loops after recording fatal DNS client faults.

A firmware flaw in the switches’ internal DNS client service appears to have started treating DNS lookup failures as fatal errors around two in the morning, causing the impacted devices to continuously reboot.

Before restarting, switches affected by the problem are recording fatal errors like these:

The DNSC (DNS Client) task is the source of the fatal errors, which happen when the switches try to resolve “www.cisco.com” and NTP time servers, according to complaints from administrators who contacted the press, Reddit conversations, and Cisco Community forum posts (1, 2).

Network activities are significantly disrupted, according to administrators, since the reboot cycle continues every few minutes.

Every few minutes, the cycle is repeated. A Cisco client shared a comment on Reddit, “This is obviously pretty disruptive and I’m not going to be able to sustain operations like this for very long.”

A concern from sources on a variety of Cisco switch models seem to be affected by the problem, including:

• Cisco CBS250 series (Business model)
• Cisco CBS350 series (including the CBS350-24P-4G) (Business model)
• Cisco Catalyst C1200 series (Catalyst Series)
• Cisco SG350 (Legacy Small Business)
• Cisco SG350X (Legacy Small Business)
• Cisco SG550X series (Legacy Small Business)

Multiple administrators said the outages started at roughly the same time across different networks, indicating the issue may have been triggered globally or by a time-based factor.

The press was informed that Cisco support recognised the problem to at least one client, claiming that it affects CBS, SG, and Catalyst 1200/1300 switches, even though Cisco has not yet made the root cause public.

As of right now, administrators have found short-term solutions that prevent the reboot loops, such as restricting outgoing internet access from switch management interfaces, preventing DNS resolution, and turning off SNTP or time synchronisation.

Even when DNS servers were reachable and operating regularly, several users indicate that turning off DNS configurations prevented the reboot loops. Users confirmed in Cisco Community forum posts that the reboot loops were fixed by eliminating DNS resolution.

Cisco has been approached by members of the press who were carried along for comment, and this page will be updated if further details become available.

Also the recommended workaround is that network administrators have successfully employed the following temporary fixes to halt the reboot loops until a permanent firmware patch is made available: 

• Disable DNS Resolution: Either removing the configured DNS servers or completely turn off the switch’s DNS client service.
• Disable Time Sync: Disabling the time synchronisation settings for SNTP or NTP.
• Limit Internet Access: By stopping the switch’s management interface from trying external fixes, block its access to the outbound internet.
• Static Configuration: Some users discovered that stabilising the devices also involved setting a static IP address without DNS or deleting the default gateway. 

It is recommended that administrators keep an eye out for official bug IDs and firmware updates on the Cisco Security Advisory portal.