C7000 Blade Chassis : Virtual Connect ‘Unknown’

C7000 Blade Chassis : Virtual Connect Ethernet ‘Unknown’

It would appear that my share of HP firmware issues was not satisfied with the recent BL465c G7 issues….

Symptoms

We have six chassis’, two are C7000 G2 and 4 are C7000 G1 – spread across two datacenters. On logging on to the Virtual Connect for one of the G2 chassis I discovered that all ethernet links, shared uplinks and VLAN’s were showing as ‘Unknown‘. All the server profiles were degraded and the Internconnect Modules wereshowing a Communication Status ‘Failed’ warning.

The chassis contains 2x  Flex10 Virtual Connect Modules, 2x 8GB/20-port FC Virtual Connect Modules. The chassis firmware was already running the latest version for both FC and Ethernet, see here.

This started with a single chassis in the morning and affected all by the evening. All of the chassis are running the firmwae version above, although the G1 chassis have HP 1/10Gb VC-Enet and 4Gb FC modules.

Resolution

******** Read this section in full before following the HP suggested fix! *********

HP support advised this was a known issue as per here.

The suggested fix is to remove the DNS server IP address from the Enclosure Bay IP Addressing configuration for the Interconnect Ethernet Modules.

I implemented the suggested fix on a chassis with no production systems, after 10 minutes there had been no impact as you can see from teh OA logs:

Mar  1 16:46:06  OA: admin logged into the Onboard Administrator from 10.144.4.4
Mar  1 16:51:55  OA: EBIPA Interconnect first DNS IP for bay 1 set to  by user admin
Mar  1 16:51:55  OA: EBIPA Interconnect second DNS IP for bay 1 set to  by user admin
Mar  1 16:51:55  OA: EBIPA Interconnect first DNS IP for bay 2 set to  by user admin
Mar  1 16:51:55  OA: EBIPA Interconnect second DNS IP for bay 2 set to  by user admin

This was then implemnted on a further chassis, the remaining G2 chassis with the Flex-10/8GB FC modules:

Mar  1 17:03:09  OA: EBIPA Interconnect first DNS IP for bay 1 set to  by user admin
Mar  1 17:03:09  OA: EBIPA Interconnect second DNS IP for bay 1 set to  by user admin
Mar  1 17:03:09  OA: EBIPA Interconnect first DNS IP for bay 2 set to  by user admin
Mar  1 17:03:09  OA: EBIPA Interconnect second DNS IP for bay 2 set to  by user admin
Mar  1 17:04:39  OA: Internal health status of interconnect in bay 2 changed to Unknown
Mar  1 17:05:08  OA: Internal health status of interconnect in bay 2 changed to OK
Mar  1 17:05:14  OA: Internal health status of interconnect in bay 1 changed to Unknown
Mar  1 17:05:50  OA: Internal health status of interconnect in bay 1 changed to OK
Mar  1 17:06:17  OA: Internal health status of interconnect in bay 1 changed to Unknown
Mar  1 17:06:47  OA: Internal health status of interconnect in bay 1 changed to OK

 

As you can see the VC modules became unresponsive and reset, causing the chassis to lose all connectvity to the network.

On the chassis that did not reset I encountered a further issue today whilst checking a Shared Uplink set, the VC modules reset!

Mar  3 09:39:21  OA: admin logged into the Onboard Administrator from 10.144.4.4
Mar  3 09:44:00  OA: Internal health status of interconnect in bay 1 changed to Unknown
Mar  3 09:44:40  OA: Internal health status of interconnect in bay 1 changed to OK
Mar  3 09:45:00  OA: Internal health status of interconnect in bay 1 changed to Unknown
Mar  3 09:45:27  OA: Internal health status of interconnect in bay 1 changed to OK
Mar  3 09:45:34  OA: Internal health status of interconnect in bay 2 changed to Unknown
Mar  3 09:46:13  OA: Internal health status of interconnect in bay 2 changed to OK
Mar  3 09:46:37  OA: Internal health status of interconnect in bay 2 changed to Unknown
Mar  3 09:47:05  OA: Internal health status of interconnect in bay 2 changed to OK

No changes were made and these modules reset….

We’re arranging a EBIPA/VC reset of the 4 remaining chassis this weekend, out of hours!

You’ve been warned!