When news of the far-reaching Spectre and Meltdown exploits first hit, Simon Margolis, cloud platform director for managed service provider Sada Systems, braced for impact.
Margolis worried that customers were about to be walloped thanks to the sweeping hardware vulnerabilities that until now, had laid dormant in the heart of all major operating systems. However, public cloud giant Google had already covered its tracks by taking the necessary remediation steps, and the impact was barely noticed by users.
“To be honest, I freaked out when I first heard about it,” Margolis said. “But it was really encouraging to hear that Google was the team that discovered Spectre and Meltdown and had patched all their internal systems already.”
Of the three leading cloud providers, Google received the highest scores in a recent CRN survey investigating how vendors responded to the widespread Spectre and Meltdown vulnerabilities.
Sada Systems, a Google Cloud Premier Partner, received notice from Google within a couple of hours of the vulnerabilities coming to light with mitigation instructions. Google, Margolis said, was the first provider to come out with information on the vulnerabilities and what to do about it.
“Google has always been very quick to announce anything security-related — they really don’t have an ego that gets in the way when it comes to serious issues, and this situation was no exception,” he said.
CRN conducted an online poll of 190 members of the CRN Channel Intelligence Council, a panel of solution providers representing the broad channel ecosystem in North America. In the survey, solution providers ranked the vendor responses to the Spectre and Meltdown vulnerability issue on a scale of one to five, with five being the top mark, or “excellent.”
Google received the highest overall ranking among cloud providers, with 17 percent of respondents saying Google had an “excellent” response and 27 percent rating its response as “good.”
By comparison, Microsoft Azure received an “excellent” ranking by 13 percent of respondents, and 27 percent of respondents ranking the cloud provider’s response “good.”
Amazon Web Services (AWS) following in third place, ranked “excellent” by 8 percent of respondents and “good” by 17 percent of respondents.
Partners said that Google’s strong response out of the gate did not come as a surprise; the internet giant has always been very security-minded. Google’s own Project Zero — a team of security analysts employed by Google tasked with finding zero-day vulnerabilities — first discovered Spectre and meltdown and disclosed the flaw to chipmakers Intel, AMD, and ARM.
Soon after, Google notified more companies, including Amazon and Microsoft, who would also need to patch their cloud platforms. In the end, dozens of competing IT companies collaborated to keep the bugs secret for long enough for many of the industry-leading vendors to respond with updates for their operating systems.
By the time news of Spectre and Meltdown came out publicly, the exploits were mostly uneventful for Agosto, a Minneapolis-based Google partner.
“We didn’t have any of our infrastructure clients impacted, and Google was really diligent and handled all of [the mitigation efforts]; we really didn’t have to manage that process at all,” according to Aric Bandy, president of Agosto.
Six days after Google’s CTO office first published a blog post explaining Spectre and Meltdown, Ben Treynor Sloss, vice president of engineering for Google, posted a follow-up blog on January 11 explaining how the cloud provider identified and put patches in place to address the exploits, which he called the “most challenging and hardest to fix in a decade.”
He also shared how Google was protecting its cloud customers from new vulnerabilities without impacting performance.
“By December, all Google Cloud Platform services had protections in place for all known variants of the vulnerability. During the entire update process, nobody noticed: we received no customer support tickets related to the updates. This confirmed our internal assessment that in real-world use, the performance-optimized updates Google deployed do not have a material effect on workloads,” Sloss said. “In sharing our research publicly, we hope that this can be universally deployed to improve the cloud experience industry-wide.”
While Google was quick to respond to Spectre and Meltdown, Los Angeles-based Sada Systems also immediately sent out its own notice to customers filling them in on the vulnerabilities, as well as initial client-side remediation steps to take. All of Sada’s end customers today have completed the necessary updates, and are no longer at risk, Margolis said.
“Between what Google and we sent out [to customers], no one reached out to us for further instruction. It ended up not really being an issue,” he said.
While Google may have been first, partners say that cloud competitors AWS and Microsoft both also did a commendable job of communicating with partners and end customers, giving both groups a head’s up on the issues, as well as recommendations to help mitigate any potential impact to users.
“Microsoft did a pretty good job and so did AWS, but [AWS] kind of handled it in their own, laid-back approach by directing everyone to their blog, so I thought Microsoft was a little bit better in terms of leveraging their channel,” according to one Microsoft and AWS solution provider partner that asked to remain anonymous.
Microsoft Azure declined to comment on its mitigation efforts or its response to partners regarding Spectre and Meltdown.
AWS did not respond to CRN’s request for comment before publication.
Considering the “awful and wide-ranging nature of vulnerabilities,” all three cloud providers handled the situation very well, said Paul Vallee, president and CEO of Pythian, an Ottawa, Ontario-based Google, AWS, and Microsoft partner.
“Given the scale of the problem — and it was pervasive — and the difficulty of the solution, I really liked what I saw regarding the reaction of the three major cloud providers,” Vallee said. Pythian’s customers on AWS, Google and Microsoft cloud platforms are all up to date with the appropriate patching and updates.
Because Google’s team first discovered the vulnerabilities, which were revealed to the public in January, the company had a head start patching and updating its environment; it started in September, Vallee said. “As a result, there were no user-visible consequences to Google customers, and that includes performance consequences,” he added.
It stands to reason that security is a top priority for a massive target like Google, which owns one of the world’s largest network, with millions of endpoints, and customer assets with more than a billion active users. The internet giant’s cloud platform consists of homegrown technology because Google has elected to build its own servers, software stack, and operating system.
“I do believe they are leading the charge in security … everything in their environment is proprietary, and I think that is instrumental to Google being able to properly secure and run a network of their size at scale while not compromising speed,” said Agosto’s Bandy.
However, the internet giant’s security efforts often reach outside of the company. Google’s Project Zero team is constantly on the hunt, “hacking for good” to continually search out and identify any vulnerabilities so the internet remains a safe place for all companies and end users, Bandy said. “It’s almost unfortunate that the market doesn’t understand how much emphasis Google puts internally on their approach to security.”