Rackspace Hosted Exchange Outage Due to Security Incident via @sejournal, @martinibuster

Rackspace hosted Exchange suffered a catastrophic outage beginning December 2, 2022 and is still ongoing as of 12:37 AM December 4th. Initially described as connectivity and login issues, the guidance was eventually updated to announce that they were dealing with a security incident.

Rackspace Hosted Exchange Issues

The Rackspace system went down in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be resolved.

Customers on Twitter reported that Rackspace was not responding to support emails.

This has been quite the day with #Rackspace. Every hosted exchange client has been down for 14 hours or so. Support isn't reading/responding to tickets. Updates are unhelpful.

I am concerned now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7

— Joe Sinkwitz (@CygnusSEO) December 2, 2022

A Rackspace customer privately messaged me over social media on Friday to relate their experience:

“All hosted Exchange clients down over the past 16 hours.

Not sure how many companies that is, but it’s significant.

They’re serving a 554 long delay bounce so people emailing in aren’t aware of the bounce for several hours.”

The official Rackspace status page offered a running update of the outage but the initial posts had no information other than there was an outage and it was being investigated.

The first official update was on December 2nd at 2:49 AM:

“We are investigating an issue that is affecting our Hosted Exchange environments. More details will be posted as they become available.”

Thirteen minutes later Rackspace began calling it a “connectivity issue.”

“We are investigating reports of connectivity issues to our Exchange environments.

Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their email client(s).”

By 6:36 AM the Rackspace updates described the ongoing problem as “connectivity and login issues” then later that afternoon at 1:54 PM Rackspace announced they were still in the “investigation phase” of the outage, still trying to figure out what went wrong.

And they were still calling it “connectivity and login issues” in their Cloud Office environments at 4:51 PM that afternoon.

Rackspace Recommends Migrating to Microsoft 365

Four hours later Rackspace referred to the situation as a “significant failure”and began offering their customers free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the problem and could bring the system back online.

The official guidance stated:

“We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any further issues while we continue work to restore service. As we continue to work through the root cause of the issue, we have an alternate solution that will re-activate your ability to send and receive emails.

At no cost to you, we will be providing you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice.”

Rackspace Hosted Exchange Security Incident

It was not until nearly 24 hours later at 1:57 AM on December 3rd that Rackspace officially announced that their hosted Exchange service was suffering from a security incident.

The announcement further revealed that the Rackspace technicians had powered down and disconnected the Exchange environment.

Rackspace posted:

“After further analysis, we have determined that this is a security incident.

The known impact is isolated to a portion of our Hosted Exchange platform. We are taking necessary actions to evaluate and protect our environments.”

Twelve hours later that afternoon they updated the status page with more information that their security team and outside experts were still working on solving the outage.

Was Rackspace Service Affected by a Vulnerability?

Rackspace has not released details of the security event.

A security event generally involves a vulnerability and there are two severe vulnerabilities currently in the wile that were patched in November 2022.

These are the two most current vulnerabilities:

An advisory published in October 2022 described the impact of the vulnerabilities:

“An authenticated remote attacker can perform SSRF attacks to escalate privileges and execute arbtirary PowerShell code on vulnerable Microsoft Exchange servers.

As the attack is targeted against Microsoft Exchange Mailbox server, the attacker can potentially gain access to other resources via lateral movement into Exchange and Active Directory environments.”

The Rackspace outage updates have not indicated what the specific problem was, only that it was a security incident.

The most current status update as of December 4th stated that the service is still down and customers are encouraged to migrate to the Microsoft 365 service.

Rackspace posted the following on December 4, 2022 at 12:37 AM:

“We continue to make progress in addressing the incident. The availability of your service and security of your data is of high importance.

We have committed extensive internal resources and engaged world-class external expertise in our efforts to minimize negative impacts to customers.”

It’s possible that the above noted vulnerabilities are related to the security incident affecting the Rackspace Hosted Exchange service.

There has been no announcement of whether customer information has been compromised. This event is still ongoing.

Featured image by Shutterstock/Orn Rin