I manage a small Cisco Stealthwatch Enterprise 7.0.2 deployment that consists of a Stealthwatch Management Console (SMC), one FlowCollector for NetFlow (FCNF), and one FlowSensor, all virtual. (This deployment started at version 6.9.x, then got migrated to 6.10.x, and then to 7.0.2.)

The deployment had been running well for a long time, but a couple of weeks ago the identity certificates of the SMC and the FCNF appliances expired.

Stealtwatch 7.x uses a centralized appliance management model where all appliances (except the Endpoint Concentrator; should be added in a future release) are managed from the SMC.

When the certificates of the SMC and the FCNF appliances expired, the configuration/management tunnels, which are SSL connections, stopped working. I still could search flows, receive alarms, etc., but I could not manage the appliances anymore. Basically, the appliance status in the Central Management page of the SMC GUI would say “Management channel down” and have a red dot right next to the status:

For the SMC I could still use the “Edit Applicance Configuration” menu option that one gets by clicking in the “Actions” column, but for the FCNF the option was not even available.

Going to the FCNF appliance GUI to try to configure anything did not work either because, while one can log in, the GUI says that the appliance is managed from Central Management and no options are presented.

So basically absolutely no option to manage in Stealthwatch 7.x an appliance with expired certificates if the appliance is centrally managed. Kind of a Catch-22 situation — Central Management cannot talk to the appliance because the certificate is expired (management channel down), and the appliance GUI says that the appliance is centrally managed and does not allow one to do anything.

After trying lots of things, what worked was the following procedure:

1. Remove the appliance from Central Management. To do this click on the three dots in the “Actions” menu and select the option “Remove This Appliance”. After this the appliance will not show up in the Central Management page anymore.
2. For the SMC one is ready to fix the certificate — just go to the appliance management GUI (https://a.b.c.d/smc/index.html) and upload a new certificate via the Configuration -> SSL Certificate screen.
3. For the other appliances there is an extra step — the appliance still thinks it is centrally managed, so one has to fix that first. The way to do that is to SSH into the appliance (username to use to log in is sysadmin), go to the Advanced menu, and select the option “RemoveAppliance”. After doing this one will be able to browse to the appliance IP address (https://a.b.c.d), log in, and fix the certificate via the Configuration -> SSL Certificate screen.
4. Now that the certificates are good, the last step is to re-add the appliances to Central Management. That is done using the Appliance Setup Tool (AST) GUI. For non-SMC appliances the link is https://a.b.c.d/swa/loadAst and for an SMC appliance the link is https://a.b.c.d/lc-ast/. The AST will ask for basic configuration parameters, like IP address of the appliance, DNS and NTP servers, etc. It is a wizard-style tool with multiple screens. The AST on the SMC does not ask for the IP address of the SMC (I am guessing it would ask if the SMC is the secondary in a high availability SMC pair) so after running the AST on the SMC one is done and the SMC will now show up in the Central Management page. For the other appliances, however, the last screen of the AST asks for the address of the SMC. Once one provides that IP address and provides the name of the Stealthwatch domain that the appliance is part of the appliance will be added to Central Management and show up in the Central Management page. After a few minutes the status column should show “Up” for the appliance that just got readded to Central Management (see the above screenshot).

The Central Management feature of Stealthwatch 7.x and later is very convinient but it relies heavily on certificates — if certificates are not valid (like trust is not properly configured or certificates have expired) then something is going to break and there is no easy way that I was able to find to fix it.

Note: when adding new identity certificates to the appliances via the appliance management GUI one also needs to set things up so to the appliance can trust the other appliances it will talk to. This is done by making sure that the right certificate is added in the Configuration -> Certificate Authority Certificates screen.

I believe there is room for improvement in how certificates are managed under this central management scheme in Stealthwatch 7.x and later. Hopefully a future Stealthwatch release will make it easier to recover from certificate issues like expired certificates.

In the meantime, do not let your Stealthwatch 7.x and later certificates expire, and if you catch a certificate before it expires and plan to replace it then cross your fingers that the process to renew the certificates through the Central Management GUI works without issues (I did not have a chance to fix my problem that way so I don’t know how well it works).