Rackspace hosted Exchange suffered a catastrophic interruption starting December 2, 2022 and is still continuous as of 12:37 AM December fourth. Initially referred to as connectivity and login concerns, the guidance was eventually updated to announce that they were dealing with a security incident.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the issue was, much less an ETA of when it would be dealt with.
Clients on Buy Twitter Verification reported that Rackspace was not reacting to support e-mails.
This has actually been rather the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours or two. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace client independently messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Not sure the number of business that is, however it’s significant.
They’re serving a 554 long delay bounce so individuals emailing in aren’t aware of the bounce for several hours.”
The official Rackspace status page provided a running upgrade of the interruption but the preliminary posts had no information aside from there was an outage and it was being examined.
The first authorities upgrade was on December second at 2:49 AM:
“We are examining a concern that is affecting our Hosted Exchange environments. More details will be published as they become available.”
Thirteen minutes later Rackspace began calling it a “connection concern.”
“We are examining reports of connectivity concerns to our Exchange environments.
Users may experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their email customer(s).”
By 6:36 AM the Rackspace updates explained the ongoing problem as “connectivity and login issues” then later that afternoon at 1:54 PM Rackspace announced they were still in the “examination stage” of the outage, still attempting to determine what went wrong.
And they were still calling it “connection and login issues” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
Four hours later Rackspace referred to the scenario as a “substantial failure”and started providing their consumers totally free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the problem and could bring the system back online.
The official assistance mentioned:
“We experienced a considerable failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any more concerns while we continue work to bring back service. As we continue to work through the origin of the concern, we have an alternate solution that will re-activate your ability to send out and receive emails.
At no charge to you, we will be supplying you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 till additional notice.”
Rackspace Hosted Exchange Security Occurrence
It was not up until nearly 24 hr later on at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was struggling with a security event.
The announcement further revealed that the Rackspace professionals had actually powered down and disconnected the Exchange environment.
“After more analysis, we have actually identified that this is a security event.
The known effect is isolated to a part of our Hosted Exchange platform. We are taking necessary actions to assess and protect our environments.”
Twelve hours later on that afternoon they updated the status page with more info that their security group and outside professionals were still dealing with resolving the outage.
Was Rackspace Service Affected by a Vulnerability?
Rackspace has not launched details of the security occasion.
A security occasion typically includes a vulnerability and there are two extreme vulnerabilities presently in the wile that were patched in November 2022.
These are the 2 most current vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Demand Forgery (SSRF) attack enables a hacker to check out and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an enemy is able to run malicious code on a server.
An advisory released in October 2022 described the effect of the vulnerabilities:
“A validated remote assaulter can perform SSRF attacks to escalate benefits and carry out arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the attacker can potentially gain access to other resources via lateral motion into Exchange and Active Directory environments.”
The Rackspace failure updates have not shown what the particular problem was, only that it was a security event.
The most existing status upgrade since December 4th stated that the service is still down and customers are motivated to migrate to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in dealing with the event. The accessibility of your service and security of your data is of high significance.
We have actually devoted substantial internal resources and engaged world-class external expertise in our efforts to decrease unfavorable effects to consumers.”
It’s possible that the above kept in mind vulnerabilities relate to the security incident affecting the Rackspace Hosted Exchange service.
There has actually been no statement of whether client details has been jeopardized. This event is still continuous.
Included image by Best SMM Panel/Orn Rin