Advisory: Sophos Firewall – Appliance goes into failsafe mode when firmware upgrades to 19.0 GA with the reason “Unable to start logging daemon”

Status: On-going 

Overview

If your device is using a configuration previously restored from a Cyberoam backup, and you have NOT regenerated the appliance certificate on SFOS, upgrading to SFOS v19 will result in operation in fail safe mode.

Appliance goes into the failsafe mode with the reason “Unable to start logging daemon” when firmware upgrade to SFOS v19.0 GA. Garner service will go in a dead state.

Product and Environment

Sophos Firewall: upgrading to v19.0 GA

Impact

The appliance goes into failsafe mode after upgrading to SOFS v19.0 GA if the device had the “Appliance certificate” generated with “md5WithRSAEncryption”. The appliance certificate generated in Cyberoam devices uses a weak signature algorithm (MD5) that is NOT supported for appliance certificates in SFOS v19.

Symptom

1. The appliance goes into the failsafe mode with the reason “Unable to start logging daemon
   To find out the root cause of the failsafe mode, do as follow:

• Access Sophos Firewall via SSH or console cable.
• Select Device Console and press Enter.
• Run the command “show failure-reason” and press Enter.
  If affected, you should see the following message:
  failsafe> show failure-reason
  Unable to start Logging Daemon.

1. Garner service becomes dead.
   To check garner service status, run the command “#service -S | grep garner”
   If affected, you should see the following message:
   garner DEAD
2. The following error is shown in garner.log:
   ERROR May 24 12:01:34Z [4152300608]: SSL_ERR: Error loading certificate to OpenSSL

If all three symptoms are matching then appliance is affected with this issue.

How can I identify whether appliance is affected with this issue before upgrading to v19.0 GA?

Check the Signature Algorithm of the Appliance certificate by running the following command on the advanced shell:

      openssl x509 -in /conf/certificate/ApplianceCertificate.pem -text -noout

If the output shows the signature algorithm as “md5WithRSAEncryption“, DO NOT upgrade to v19 before regenerating the appliance certificate.

Resolution

Please check two feasible workarounds for this issue.

Rollback to the previous version

In Sophos Firewall, go to Backup & firmware > Firmware. You can see the previous version under Firmware. If you want to roll back, click the button for Boot firmware image for the previous version.

Verify the Signature Algorithm of the Appliance certificate

Note: Regenerating the appliance certificate will have some impact. Make sure to read the section “Impact of the regenerating appliance certificate“.

On 18.5.MR3 or previous versions, verify the Signature Algorithm of the Appliance certificate by running the following command on the advanced shell:
openssl x509 -in /conf/certificate/ApplianceCertificate.pem -text -noout

If the output shows the signature algorithm as “md5WithRSAEncryption“, DO NOT upgrade to v19 before regenerating the appliance certificate.

To regenerate the appliance certificate from the UI,

1. Go to SYSTEM > Certificates > ApplianceCertificate.
2. Click Apply on the “Regenerate certificate manage” section.

Already affected by the issue?

If you are already affected by this issue (running 19.0.x in Failsafe mode), do as follow:

1. Rollback to 18.5.MR3
2. Regenerate Appliance certificate from the UI
3. Run “openssl x509 -in /conf/certificate/ApplianceCertificate.pem -text -noout”
   The output should show “Signature Algorithm: sha256WithRSAEncryption”
4. Migrate to SFOS v19.0.x by downloading the latest firmware (do not do firmware switchover to 19.0.x)

Impact of the regenerating appliance certificate

The regenerated appliance certificate must be replaced for the features dependent on the appliance certificate:

1. SSLVPN remote access: Regenerating the appliance certificate results in remote users being unable to connect via VPN to the Sophos Firewall.  Have the remote VPN user(s) re-download their client configuration package from the user portal to fix the issue.
2. SSLVPN site-to-site server-side SFOS: Regenerating the appliance certificate results in a connection failure of the SSLVPN client. Download the SSLVPN server configuration and import it again on the SSLVPN S2S client-side firewall.