Rapid Response: SolarWinds: Urgent lessons from a cyberattack, w/ CEO Sudhakar Ramakrishna

The very best founders I know are brilliant at building systems. They connect teams, they remove bottlenecks, and they eliminate single points of failure. And yet When it comes to their own wealth. Most are running a disconnected stack. A tax accountant here and a state attorney there, a wealth manager who doesn't talk to either one of them.

Creative planning was built to fix exactly that. One integrated team of tax professionals, state planners, investment specialists, all coordinated by a dedicated wealth manager who sees your full financial picture and keeps every piece working together. Proactive tax efficiency, state strategy, investments all under one roof. Creative planning where wealth works together. Learn more at creative planning dot com slash masters of scale.

Global hiring is such a headache when you're building fast, but companies like Eleven Labs are scaling effortlessly with deal. They quadrupled their workforce in twenty twenty five. Without hiring an HR team. Build your global team with deal. Visit deal.com slash MOS and start expanding your business today. That's D E L

dot com. Slash M O S. Hey, folks, Jeff Berman here. I am thrilled to share some of the new names who will be joining us at this year's Masters of Scale summit. This may be our biggest stage yet. Reed Hastings, Meredith Whitaker, Van Jones, Amjad Masad, and more.

Will be there with us October 20th through 22nd in San Francisco. If you're building something great, or you want to build something great, We want you there with us too. Join us at masters of scale dot com slash apply twenty six. That's mastersofscot com slash apply. twenty six.

We were the victims of one of the most sophisticated A tax which has been attributed to a nation state. In every one of these attacks you can always learn something that had you Done those things could have either dissuaded somebody or prevented Every enterprise must look out and protect its infrastructure, its data sources, and do the very best to train its users to protect its assets.

At the same time, it is very difficult for any one company, no matter how many resources we have or how large we are. to be fully protected. Now that we have been thrust into this spotlight, we have to galvanise the industry and speak up and actually actively contribute to that. That's Sadakaramakrisna, CEO of SolarWinds, the tech provider that became synonymous with cybersecurity risk late last year after a sophisticated attack compromised its software, potentially infecting thousands of organizations.

I'm Bob Safian, former editor of Fast Company, founder of the Flux Group, and host of Masters of Scale Rapid Response. I wanted to talk with Sadr because in our increasingly digital world, cyber risk has become an ever present threat to scaling, whether you're a startup, a brand name platform, or even a government. Sadaka joined the company soon after the breach was discovered, thrust into a crisis that included leadership challenges, technology challenges, and business challenges. His experiences illuminate what you can control and what you can't, and how putting in place what he calls a framework can clarify your actions. He also discusses the brand hurdle SolarWinds faces and the ways in which he's trying to turn the negatives of the crisis into an opportunity.

Cybersecurity needs to be anticipated, Sadaka says, because no matter who you are, it's gonna find you at some point one way or another. I'm Bob Safian and I'm here with Sadakar Ramakrishna, the CEO of Solar winds. Sadaker, thanks for joining us.

Thank you, Bob. Thanks for having me. So you've been CEO for less than a year, but it's been an eventful year. You were announced last December, and between the time You were announced and you're taking the post in January, the company became aware that it had been a victim of a cyber attack, and not just any cyber attack. It was described as one of the most sophisticated and complex ever. Solar winds provides IT software to thousands of places and suddenly all of them were

potentially exposed. And so SolarWind's moved from Relative obscurity to becoming this sort of household name, although not necessarily for what any business or CEO would hope for. So When you come into a situation like this What do you do first? This wasn't what you expected to be handling when you agreed to do the job.

Absolutely, Bob. As you said, this was an eventful phase of my Korea and my life. You don't really prepare for these types of situations, but our collective experience over the years allows us to create a set of guiding principles. And if you have the humility to continue to learn and iterate on those

then you can start making progress. So that's the approach that I took. Well you had some Experience in the cybersecurity area before coming to SolarWinds, you had been most recently the CEO of Pulse Secure, which provides Secure access.

How do you know what to do and what order to do it in when something like this arrives? So First and foremost, I did look at the security posture of Solomon's. Like Many CEOs who come into a new situation, you have already met the team, you've understood the situation, and you start building your 90, 100 day plans.

In my case, I had to basically throw everything out the window and reset my plan because what needed to be done at that point in time Given the event. is taking care of your employees and taking care of your customers. That was the primary and I would say the only obligation. My previous experience dealing with cybersecurity issues definitely did help me address this. You look at what were the security investments where they commensurate for a company our size and scope. Where there any obvious deficiencies and so on.

And the way I would describe it is Solomon's investments in security as well as The tools we were using. to establish a security posture. Way consistent. But

the industry average and in some cases better than industry average. So it wasn't so much a negligence or a deficiency. We were the victims of one of the most sophisticated And I would say patient. a tax which has been attributed to a nation state.

But if I'm hearing you sort of the standard uh Solar winds was Up to par for the industry. But still hadn't been good enough. To protect from this attack.

If I can be so bold as to say If a determined nation state Is after you they can probably create an attack and breach through anything. So it is less to do with

one specific company's resources or posture. But equally I'll be the first one to say that in every one of these attacks You can always learn something. That had you Done those things.

could have either dissuaded somebody Or prevent it. But this is the nature of our industry. This is the nature of the security industry. Because if we had a blueprint for having no breaches at all, uh we would probably all be adopting. Yeah, you mentioned Nation State, the US government's cyber and infrastructure security agency. SISA, I think is the acronym part of Homeland Security apparatus.

They've attributed the a attack to state sponsored actors out of Russia. Yeah. You didn't know that right away, or did you know that right away? No, we did not know that right away. At the same time While Solivens provides mission critical applications And capabilities to customers. I don't characterize us as a classical security company, especially as a company that does security investigations, for instance.

That's not our uh strength or sweet spot. So we had to rely on outside experts. To be able to do some of that work. There were initial reports that like thousands of companies of your clients could be affected and then was revised to like fewer than a hundred

Let me actually set some context on that, Bob. the way we came up with the initial estimate of the eighteen thousand, which is what we were reporting. was because of Our data that said eighteen thousand customers approximately downloaded that piece of

Self. Right. This was a routine software update. Exactly. That they downloaded. So that is the largest possible set of customers that could be impacted. But as you know, in these cases what happens is customers sometimes download them but don't deploy them, in which case

There is no harm done. In some cases customers deploy them. But configure it in such a way that the software is not able to connect back into the internet. In which case again the malware cannot do anything. So if you start sifting through those

Eventually it came out to be an estimate of less than hundred. Some said sixty, but let's just say less than hundred. Our focus was Let's assume all eighteen thousand were impacted. Touch each one of them, make sure that they were updated and upgraded. And worry about

the rest after that fact. I think a lot of businesses sort of assume that. The government Or prior to this, maybe assume that the government was protecting them from state sponsored actors like this. And this sort of rip the veil off of something that makes

A lot of businesses smaller and larger than yours anxious about What's my responsibility to worry about and what's gonna be policed by others? How do you think about that question about where the responsibility falls and what we've learned from this? My view on this and this is a view that I've held well prior to joining Solivence is that Every enterprise

Must. look out and protect. It's infrastructure, its data sources. And do the very best. To train its users, meaning its employees.

To protect its assets. At the same time, I also believe that It is important for us to be part of this community. You probably have heard me use the word community vigil as it comes to security. Because the simple matter is that threat actors

have to be right once to breach through. We have to be right every single time to protect ourselves. And especially when a threat actor is a nation state actor. It is very difficult for any one company, no matter how many resources we have or how large we are. To be fully protected.

So in that world we have to have a very tight partnership. of transparency and collaboration both amongst the community As well as with the authorities and the regulators. So you mentioned Sisser earlier in the conversation. And

That is a body which we are actively working with to support. for two way collaboration and communication. There's been some news. Of late of regulators. Asking for more disclosure from companies about security. Not necessarily from uh

cybersecurity point of view, more from a financial markets point of view, but it sounds like the more sharing of this information that there is, the easier Chance that we're gonna be able to identify patterns. Most definitely. At the same time, the point you made about the regulators is a key one. I think more disclosure is definitely important. But it should be done in an environment where

Victims don't feel shameed about coming out. Or The discoveries are not done with the intent of leading to punitive measures. That's not to take away accountability and responsibility from enterprises such as us. We have a obligation responsibility and have to take accountability.

But if we are constantly worried about who is gonna sue us. or who is going to be punished for coming out and saying what we think will be better for the larger community. then we are going to be hesitant about doing it. And this fear of liability will have people resist being as transparent as they might be. I believe so. I believe so. And I

believe there are some progressive senators I like Senator Warner. who are talking about how do I indemnify you for coming out early and speaking About these types of issues. In February you testified in Congress alongside Microsoft President Brad Smith and FireEye CEO Kevin Mandia. When you're asked to testify in Congress, you could feel like everyone's looking at you that idea of blame or shame.

Was that part of that experience? Yes, at some level, Bob, but my focus going to the Senate was to directly and transparently communicate what Solovins had done and was committed to doing. This was a great opportunity for three great companies to come out and talk about what we can do for the future. As you were preparing for that testimony at the same time you were preparing for your first earnings call.

Working with a team that was new to you and you were new to the team. How did you approach that leadership challenge? So the approach that we took was first and foremost when you deal with an issue like this you gotta have a framework. of how you have to solve this problem. The framework also has to be fungible because you learn new things every day and you have to adapt to it. But there has to be a framework.

The second part of it is the transparency associated with it, with employees, customers, your partners on what's happening, what do you know? What is next? So You have a framework You have

Transparency. You have to work with a great sense of urgency. Because you gotta act and start making progress. But equally you have to demonstrate a sense of humility as you go through this urgency because nobody knows how to solve these problems. It's framework transparency.

communications Humility. And doing it with a sense of calm across the board. And so I would say we were

leveraging all of those principles across the team. We call ourselves Solarians, by the way. And having them on my side Helped me a lot.

in doing my job. This concept of a framework. Can you explain a little bit more like what the framework is and why it's so important? Definitely. So the framework I used here is called Secure by Design, and I'll take you back to the comments about Was there anything deficient, was the investment enough, and so on. There's always things that you can do to improve.

I'm an engineer I've built software just like you build quality software, you build secure software. So the framework had three key pillars. one was how do we improve the infrastructure security and infrastructure posture of solemn's better than what it ever was before. How do we make it best in class? Two is given the unique nature of the supply chain attack. How do we protect and secure our build systems, software build systems, to the next level?

And the third is can we innovate in the bill systems themselves? Such that it makes it difficult, if not impossible, for a threat actor to break into your supply chain. So those were the three pillars. So the reason for that framework then is if I look at any particular employee in the organization, they should be able to relate to I'm contributing to that pillar and these are my actions. So

independence is very, very important because without independent you cannot act with a sense of urgency. But then interdependents is also extremely important. Because without that, you cannot support a broader mission or a broader framework. So independence and interdependence kind of became part of the fabric and the value system of the business. Can you give us an example of a track you were going down and then you learned something and you adjust it?

More than I can count. When you have an issue like this, Bob, the first thing that you have to be open to is establishing Hypothesis. Could this have happened? Did that happen? So one of the very early hypothesis when you are in a global company with presence all over the world is Could this be a insider job? And if it is, then you act on it.

In that vein, for instance. We came up with multiple hypothesis. I had the unique vantage point of being an outsider coming in. So I was not biased, so to speak, by anything that was happening inside. So I came up with my set of hypotheses. The team had some, we pooled together. And we said we'll go after every one of these

And figure out where does it lead us. And many led us to nowhere. So we had to keep Reinventing or recreating a hypothesis. When you've built substantial wealth through your business, it's often tied up in a single equity position.

The upside is real, but so is the risk, and knowing when to act isn't always obvious. Creative planning works with business owners to build a strategy around concentrated equity. When to diversify, how to manage tax risk, and how to protect what you've spent years building. Creative planning where wealth works together. Learn more at creative plating dot com slash masters of scale.

Hey listeners, Bob here. If you listen to Rapid Response on Masters of Scale, you may be missing half the show. Because every Friday we release a second rapid response exclusively in the rapid response feed. The guests and topics are just as compelling and timely from Ford CEO to NASA's administrator to the lessons from The Devil Wears Prada. It takes about 10 seconds to find, just search rapid response wherever you listen to podcasts and hit follow to make sure you never miss an episode. I hope to see you there. Humans will never be more intelligent than AI.

Those were great at AI and those that went out of business because they weren't. How do we build a future? That is human centered. I'm Rana El Calyubi. And on my podcast Pioneers of AI, we answer that question and so many more. As an AI scientist, entrepreneur, and investor, I know what it takes to build AI that works for everyone.

Every week, I sit down with the pioneers shaping our future. And we take you behind the scenes of the AI that's transforming our lives. Find pioneers of AI wherever you tune in. Before the break, we heard SolarWind CEO Sadaka Ramakrishna recount how he responded to a high profile cyber attack. Now he talks about how the episode impacted SolarWind's brand and business. He offers practical advice to other leaders about how to best protect yourself from cyber risk and how to respond when you do find yourself in the crosshairs. I know there are different labels that have been used to describe this particular attack. The government said that it shouldn't be labeled

Solar winds, but of course that was the dominant association in the media and the marketplace. From a business and a brand point of view. What can you do about that? Did you think about Changing names. So that has been suggested to me, Bob, and I had the freedom to do that.

Even from my board. When I think about brand Brand is a function of My people My products or offerings.

And my customers meaning what value I deliver to them. Yes, name is significant, but those are the three things that we should really be focused on. There are a few reasons why I chose not to think about changing the brand or the name of the company immediately. One is the most critical thing here was addressing the issue at hand, which is the security breach and the safety of the customer. segments.

So by changing the name and spending time and money doing that We are distracting from the job at hand. And potentially Coming across as swiping an issue under the rug. So that is fundamentally opposed

to the approach of saying we're gonna be transparent. Collaborative and communicative. Two is Focus all your energies on the problem at hand and not try to deviate from it. It was painful, I will tell you. Because it should not have been called the Solovins breach. As you said, even the government agrees with that.

Would I wish that there was a more generic name attributed to it? Yes. But I would say that Bye. Serving the m primary obligation we have in the long run.

I'm confident we'll come out better. And just a quick side question just to clarify for some of our listeners, this term supply chain attack that we have used. Can you explain what that term means?'Cause it's not specifically about a traditional view of a supply chain, but more of a technology supply chain. Yes, and more specifically in this case Software supply chain. In a traditional supply chain. let's say physical parts, let's say you're assembling a television set.

You are taking electronics from different sources. Putting it in a supply chain, and then assembling it. Towards the end. And shipping off of finished good. The

process in software supply chain is very similar in the sense that you take various of software or code as it's called and you compile them. So compilation is similar to assembly. And after you compile it, You ship it. So What a supply chain attack is, is from the time you collect all these pieces of code

To the time you compile them. If a threat actor where to inject let's call it another piece of code into that. And you inadvertently compile it and ship it. then in a sense the malware becomes part of what you actually shipped.

And so that's what a supply chain attack is. And because your own distribution is so broad that malware is getting it to lots of places. Absolutely. And I've been asked why did You think you were picked.

Why were you the victim? And I have been describing it to some degree as this is the price you pay for ubiquity. meaning we are deployed in over three hundred thousand customers and So we can be a very large target. So for instance if you look at breaches that uh are reported, or I should say security vulnerabilities that are reported in the industry, Microsoft ranks very high in the list.

I attribute that to their ubiquitous nature. Not because of necessarily deficiencies. In what they do. So that's kinda the connotation here. Yeah, if you want to have impact you go to the places that have the broadest reach and the biggest impact.

I was asking about the brand name and the noise around that Are there any positives to the notoriety that that's come to Solar Winds? I mean the company's certainly more widely known, if not for the ideal thing. Like, are there any business opportunities that can be unlocked from a crisis like this? Yes, as long as we stay true to what we are trying to work towards. Which is first and foremost

Learn from this and improve. Many customers Since the time of the breach have expanded with Solomon's Because of how we are approaching The secure by design tenants.

The fact is, many of my customers are also producers of software. So they appreciate that if it can happen to us, it can happen to them as well. They want to protect themselves ahead of time, so to speak. But it also allows us to serve them on a broader scale and as a result enjoy better business success as well. Through the

pandemic certainly through this era we're in all companies are more and more dependent On technology. In some ways every company has to be a technology company today. A lot of the folks listening today are business leaders, some of smaller startups maybe dependent on technology remote access in ways they never have before.

How Can we learn from this? The first thing I would focus on is awareness. inside the company. Security awareness.

Security training and behavioral training of the employees. more than any technology that we can deploy. That has to be a priority. Fishing attacks and spear fishing attacks are

one of the most common ways in which threat actors gain access To your environment. the percentage of users that are clicking on synthetic attacks is still very, very high, as high as forty percent. in an enterprise. We did a recent IT trends report where we interview

various professionals in the IT sector. What are the trends, what are the concerns. Lack of trained personnel. Always. seems to be at the top of the list. And so we just need to keep training our people better and better and better.

As you describe this need for Better security hygiene. Say. That existed before this attack, though, right? Like so is it that there's not that much different that people should be doing today than they would have been doing a year ago, even though all this New activity has come to light. I unfortunately think the basics still remain the same, Bob.

And this is a more of a human and uh behavioral aspect, which is as long as it doesn't hit me I continue to believe it won't hit me. Right. And this is true in jobs and in economies and unfortunately true in security as well. What's next?

For solar winds? Do you know the point where you say okay Crisis over, we can go back to like Are regular planning? Or Does the impact of this change your business forever, like indelibly?

I would say it did change it indelibly. But Hope For the better. In

twenty twenty one, we basically established one primary goal, which was Support our customers come back online and customer retention is our number one priority. And so eight months or two quarters plus into this journey I am pleased to say that That has

Panned out. Equally We have ambitious plans of Continuing to build. On our capabilities.

Customer environments are becoming more and more complex, as we all know. Cloud and deployment of cloud both Pre COVID and post COVID. are accelerating that creates Many more

areas there could be security issues, management issues, monitoring issues for customers. And in this world Customer budgets are not increasing commensurate to their needs and their complexity. So what can we do as Solo Winds to support them in those needs? And so that requires us to continue to deliver powerful solutions to address those multitude of needs. But do so in a simple fashion that increases the productivity of our customers.

So I'm not hearing in your voice like Oh, if I'd known this was happening, I would never have taken this job in the first place. No. Like this is not what I signed up for. No, it was definitely not what I signed up for, but I look at it as An opportunity, as I like to say, an opportunity to learn, an opportunity to serve, and an opportunity to grow. So what do you feel like is at stake for solar winds now? What's at stake for Solomon's now is

Taking the obligation of being Thrust into the limelight. I would say Solomon was perfectly happy not being in that limelight and simply focusing on customers for ever and ever. But now that we have been thrust Into this spotlight.

We have to be more transparent, be more collaborative. Calvanize the industry around These topics. And make it okay to speak up about those. and actually actively contribute to that.

So speaking of contributions the innovations that we are doing in building better and more robust supply chains. We could use that as proprietary information, but we have decided that we're gonna publish that broadly. for the benefit of the broader community. So that is an obligation that has got a certain cost involved in it.

And so my commitment is that we will continue to support that going forward. And if I'm a a business leader or a CEO who's listening to this and I have just learned or I'm hearing about a potential breach at my organization, what's my roadmap? Like what are the things that I should be doing? When I'm faced with that. I'll go back to the basics and that, Bob. And the unfortunate fact is if you're a business leader, It's probably more likely that you have to deal with this issue than less likely. No matter what your level of preparation is. So you might as well start thinking about

a framework similar to what we did with Secure by Design. Two is I would be relentless in my communication with my customers and my partners and my employees in terms of What happened, what do you know, what are you doing, and what's next? There's no reason to

Hide. From those facts and getting out there is more important than anything. Three is Leverage others. And they could be competitors, they could be unrelated, but it didn't matter.

So making this a community vigil community event. And learning from others and applying to your context. And then be humble. There is no such thing as I won't be breached because I'm too secure or too smart.

You could be breached. And when it happens, be humble, learn from it, adapt. And act with a sense of urgency. Well, Sadakar, this has been really fascinating and instructive. Thank you so much for spending time with us. Thank you, Bob. It was my pleasure.

Masters of Scale Rapid Response is a wait what original. The show is recorded remotely using sanitized audio gear. I'm your rapid response host, Bob Safian. Host for Masters of Scale is Reed Hoffman. Our executive producers are June Cohen and Darren Triff. Our supervising producer is Jay Punjabi.

Our producers are Jordan McCloud, Christina Gonzalez, and Marie McCoy Thompson. Our music director is Ryan Holiday. Original music and sound design by Daniel Nissenbaum and the Holiday Brothers. Audio editing by Keith J Nelson, Steven Davies, Andrew Nalt, and Mike Gallagher. Mixing and Mastering by Aaron Bastanelli.

Special thanks to Emily McManus, Sarah Sandman, Kelsey Capitano, Tim Cronin, Charlie Menessis, Adam Heiner. Anna Pizzino, Ben Richardson, Mina Kurosawa, Saida Sapieva, and Colin Howard. Become a member of Masters of Scale to get access to a year's worth of courses and content on the Masters of Scale Courses app. Find out more at masters of scale dot com slash membership. Visit masters of scale.com slash rapid response to find the transcript for this episode, and be sure to subscribe to our email newsletter.

What you see above is a preview of the first minutes. One unlock costs 3 credits and covers this episode forever: full segment and word-level timestamps on this page, plus .txt, .srt, .vtt and word-level JSON downloads, as many times as you like.