UnHacked

Justin: 00:14

Welcome everybody to episode 34 of unhacked. Guys, unhacked, like I say every week, it's a bit of a deliberate misnomer because the truth is most of these breaches, about 97% of this stuff, we could prevent if we get ahead of it, if we're doing all the things, if we're following the formulas that we talk about. But once you've been hacked, you can never truly get unhacked. And so, that said, today's gonna be a really interesting, very special episode. I am really excited to be here because we have, we you know what?

Justin: 00:47

I'm gonna save that introduction. Our our regulars, we've got our, Mario and Brian who are here every single week. We've got Virender who's here, you know, when he feels like it. And then, we've got a special guest today, Robert. So, in in order, Mario, Brian, Birender, tell us who you are, what you do, and who you do it for.

Justin: 01:06

Mario, take it away.

Mario: 01:08

Mario Zaki, CEO of Mastech IT. We are a managed service provider in, New Jersey, located right outside of Manhattan, and, we do everything IT for clients in, in the tri state area. I've been in business for 20 years, and I have a lot of happy raving fans.

Justin: 01:31

Alright. That's a good good to hear. Good to hear. Brian.

Bryan: 01:35

Okay. Bryan Lachapelle with B4 Networks . With us. Brian. Hello?

Bryan: 01:41

Can you hear me?

Justin: 01:45

Brian's brand studio is a little bit, he he went back in time a little bit, I think. He's broadcasting from the eighties.

Bryan: 01:52

How's that?

Justin: 01:55

Nope. Yeah. Go ahead. We'll we'll we'll work with it. We got nothing.

Justin: 02:02

Alright. Oh. Brian, drive past your real office, and we're gonna we're gonna continue. Birender, go ahead. Give us an introduction.

Barinder: 02:11

Barender Hans with Red Rhino. We're based just outside of Vancouver, BC here, and, we are a managed IT service provider that focuses on enabling our clients to do their best work. So anything typical MSP, cybersecurity, as well as, business transformation using technology.

Justin: 02:29

Alright. And I I kinda skipped myself. I'm Justin Shelley, CEO of Phoenix IT Advisors. We do work in the Dallas metro and then also out in the west, Northern Nevada, Utah, Idaho. And we kinda lead with compliance.

Justin: 02:41

That's, we talk about this formula week after week, and that is how we handle our clients as well. We we measure against the industry standards. We create a plan for you, and then, we just work, week after week, month after month, quarter after quarter to implement that plan. Alright. Now saving the best for last.

Justin: 03:02

We are here with Robert Cioffi. Did I do it right, Robert? I I mean, I tried. You gave us the Italian lesson beforehand. I think I still screwed it up.

Justin: 03:10

But, Robert, tell us about yourself, a little bit about you, a little bit about your company.

Robert: 03:15

Yeah. Robert Cioffi. I'm the CTO and cofounder of an MSP based in Yonkers, New York. We are, name of the company is Progressive Computing. We are just north of Manhattan in Westchester County, if you know where that is.

Robert: 03:33

We've been in business for 32 years. Our story is probably much like a third of the MSP landscape. Right? Been been doing this for decades, and, we've been through just about every iteration and permutation of what an IT company looks like, started in application development in the early nineties, moved to networks, moved to, break fix, moved to managed services, cloud, cybersecurity. It's the same story that you've probably heard a 100 times that you probably even experienced yourself.

Justin: 04:05

Alright. Alright. Brian, it looks like you're back. Let's let's try that one more time. Tell us about yourself.

Bryan: 04:12

Yeah. Brian Lachfeld with B4 Networks based out of Niagara, Ontario, Canada. We provide managed services and computer support to businesses throughout the

Justin: 04:21

area. Alright. Alright. Okay, guys. So in today's episode, with introductions behind us, technical difficulties hopefully behind us, We are gonna talk about something that might potentially make all of us a little bit uncomfortable.

Justin: 04:39

What happens if your IT company gets breached? And more specifically, in this case, what happens if one of our vendors gets breached? I mean, this this can go all the way up and down what we call the supply chain. And the bottom line is this stuff does happen. We try with everything we've got to prevent it, but what happens when, all of our efforts fail and the bad guys do get in?

Justin: 05:05

So, that is what we are going to address. We are all gonna be a little bit, uncomfortable today talking about this, then but we're going to wrap it up with the formula that we always talk about and the formula that we have broken down in great detail this October being cybersecurity awareness month. Alright. Guys, let's go ahead and jump in, and we are we're gonna start with a little bit of background. Well, all of us are familiar with a company called Kaseya.

Justin: 05:35

A lot of us have used or do use some of their products. And, Brendar, why don't you, why don't you tell us a little bit in just layman's terms, who is Kaseya, what do they do, and why does it matter to IT companies?

Barinder: 05:49

Sure. Yeah. Kaseya is, I would say a juggernaut in the managed IT service provider, space, But in IT in general, through their own products as well as acquisitions, they've got a a large suite of products where they provide tools to IT departments and managed service providers to run the IT ecosystem. So they have tools to manage their tickets and endpoints to allow technicians to work. They have what's called, and for purposes of our conversation here, a remote monitoring management tool.

Barinder: 06:21

It's a piece of software that controls your computer. You install it on every computer in your IT ecosystem, managed service providers to rely on this, and it does a lot of very good work. And then they also have IT documentation and a whole bunch of other, products in their suite of, in their portfolio.

Justin: 06:36

Okay. And I mean, you said a key thing. Right? Like, we put the software on our clients' computers. And with this software, we have the power to do pretty much anything we want or need to do to support them, to take care of them.

Justin: 06:49

It's a very useful, very powerful tool. In the right hands, it does good things. In the wrong hands, well, we're gonna dig into that. Okay. So, Mario, I'm gonna turn this to you a little bit.

Justin: 07:02

Let's just talk about what what exactly is an MSP and maybe a little bit about the evolution of why we call ourselves MSPs today versus when I started, I was a computer repair guy running around with a toolbox, a screwdriver, electric screwdriver no less. I thought it was so cool to have that thing back in the 19 sixties or whenever I got started. Alright, Mario. Tell us a little bit about what MSP is.

Mario: 07:25

So, MSP stands for managed source provider. You know, everybody has heard us use that term, you know, a 1000000 times. So, essentially, it transformed from being the IT guy or, you know, the break fix guy to a company, that will manage your entire IT IT infrastructure, and it could be everything from your IT to your phone system to your website and stuff like that. And, essentially, it's not what it used to be where what like, when I started was when somebody, you know, is having a problem, they call, and we would go with our little handy dandy screwdriver or little, you know, floppy disk or thumbnail. And, you know, a lot of times, we would have to say, okay.

Mario: 08:12

Yeah. We have to wipe out the whole thing and and fit you know, and start over. Now, MSPs are responsible for, you know, managing the entire network from, you know, protecting it with cybersecurity, you know, doing backups, doing, you know, proactive maintenance, and, you know, using tools like, you know, like what Kaseya provides, like, when they're just that, you know, that can automate a lot of the stuff that, you know, a person an IT person or a warm man shop would be going traveling from, you know, office to office, you know, cleaning up temp files and, you know, running, like, this cleaner and stuff like that. A lot of now the MSP is responsible for all of this, including unlimited support. And a lot of the times, you know, support is included, unlimited, like, remote access or phone support.

Mario: 09:09

So, essentially, they're now benefiting by having the network running smoothly versus calling. You know, when there's a problem, you know, we fix it, and then, I'm sorry, Brian.

Justin: 09:23

I know. He just keeps flipping himself sideways, upside down. He's like, Brian's at the circus, but he's here with us. So it's okay.

Barinder: 09:30

He he got hacked.

Bryan: 09:31

It wasn't intentional. I promise.

Mario: 09:38

So essentially, it it benefits them to have everything smooth, protected because they get paid no matter what every month per computer versus in the old system only getting paid when when there's a problem, they fix it, then they send you a bell.

Justin: 09:55

Alright. Now I I have to because you you said floppy disk, and you sent me down like, oh my god. You sent me back decades. Pop quiz. Everybody here, I need an answer.

Justin: 10:07

When you first started your career in the world of IT, what was the media of choice? Mario, you go first. What was it? What were you using? Which kind, though?

Justin: 10:17

Puppet. That that's too vague. Which one? Were they

Mario: 10:20

the

Justin: 10:21

5a quarter, 3a half. Were they 8 inch? Were you using the 8 inch floppy disks?

Mario: 10:25

No. No. I wasn't using 8 inch. I I was 5a quarter.

Justin: 10:29

Okay. Okay. You're OG, Barinder?

Barinder: 10:31

When I started my IT career, I was 3a half. But my very first computer was

Mario: 10:36

5 and

Barinder: 10:36

a quarter. I I I went to a friend, got him to copy a game for me, and I used that to to to install on my computer, and it was half an hour. It took at least. At least. Yeah.

Barinder: 10:49

Brian, what were you using?

Justin: 10:50

Alright. When I started my career, I believe it was 3 and a half.

Bryan: 10:54

But when I was in college, it was definitely 5a quarter. But before that, it was a tape, a cassette tape that I had to put into this, like, tape player to load the program, and that took

Justin: 11:08

Damn. Long I think it

Bryan: 11:09

was a Tandy or something. I can't remember. Long time ago.

Justin: 11:13

Yeah. Yeah. Okay. So far, Brian's winning. Robert, your turn.

Barinder: 11:18

Well, I'm

Robert: 11:18

a little mad at Brian because he kinda stole my thunder. But, career wise, 5 and a quarter inch. But prior to that, my first computer was a Texas Instruments 99 slash 4 a with extended basic cartridge, and I used to save my programs to a cassette recorder, that I bought from RadioShack. So, if that doesn't make you feel old,

Barinder: 11:41

I don't know what cards.

Robert: 11:45

Yeah. Yeah.

Justin: 11:46

No punch card. Really? We definitely missed the was it called a Zip drive?

Barinder: 11:50

Mhmm. Drive.

Robert: 11:51

Yeah. Yeah. Well, those are later.

Barinder: 11:53

Those are They were they

Justin: 11:54

were later, but that was, like, the the cool stuff. That was the cool stuff that came out. Windows 3.1 was on, 5 and a quarters or 3 and a halfs, and it was, like, 20 of them or some Windows 95. I remember that one was just, like, stacks of of flunk disk.

Barinder: 12:09

Okay. Imagine how imagine how much data was breached back in the day because none of those disk none of those zip drives were encrypted. It's just like all this free floating Oh, yeah. Oh, yeah. Yeah.

Justin: 12:19

But it was hard to access because we were we were running on the sneaker net back then. Do you guys remember the sneaker net before Ethernet? Yeah. Pull that 5 and a quarter inch discount, run to your neighbor, and, okay. Oh god.

Justin: 12:34

The the evolution that we live in. Brian, if your, if your 19 eighties technology is keeping up with us, will you tell us a little bit about what happened on July 2, 2021?

Bryan: 12:48

Alright. So just taking a couple steps back before that day, Kaseo's noticed or notified that there was, some some bugs in their code that allowed an intruder to get in, or to be able to exploit it. I believe there was 7 or 8 different, vulnerabilities that were reported to them by a researcher. And, despite having patched, I think it was 4 out of the 7 or 8, they didn't patch all of them, and, they got breached by a zero day exploit or at least what it was claimed as zero day exploit. I I feel like if they got heads up and heads up notice, it wasn't quite a 0 day exploit.

Bryan: 13:25

But that's the extent of of what I understand, that happened that day.

Justin: 13:31

Okay. Let's talk about what a 0 do it oh, jeez. I can't even talk. 0 day exploit is. What does that mean?

Justin: 13:35

Well, exploit is. What does that mean?

Bryan: 13:40

Well, typically, when, when and as vulnerability is is found, it's usually announced, to let people know that it exists. But before that, usually, their their the the the company is told privately that it exists, and then it's announced to the general public that it exists, hopefully, after the patch has been released to let people know, like, hey. You guys need to patch this now. Usually, 0 day means that the day it's announced is the day that that it's breached or and or before that. Like, it's an unknown vulnerability that was taken advantage of.

Justin: 14:11

Okay. Talk a little bit about supply chain attacks.

Bryan: 14:20

Okay. So and this applies. It's not just with regards to our our vendors, but also our customers' vendors as well. A supply chain attack typically comes from, one of your vendors or, in our cases, our vendors. Our vendor gets breached, and in turn, they could potentially breach all of our customers.

Bryan: 14:41

In the case of a client, and I use I use, like, some of our clients as examples, they have a lot of vendors that will remote into their systems to, you know, fix their piece of machinery or their piece of software, where every time you do that, you're you're opening up the doors to your entire network to their IT, security. And so if they're breached, you have a possibility being breached in turn because they can then utilize their systems to get into your system.

Justin: 15:07

Okay.

Barinder: 15:08

22 2 recent just a recent example of a supply chain attack attack that was in the news, not related to software, is, the walkie talkies and the pagers that were blowing up in Lebanon. So that was

Justin: 15:20

Oh.

Barinder: 15:20

A a a a a a a somebody, got into that supply chain and basically on demand was able to cause them to explode when they wanted to. Like, that's something that was in the news, not necessarily related to our technology here, but just a visible Same concept.

Justin: 15:35

Yeah. Same concept. Easier to understand too. Very good point. Yep.

Justin: 15:38

Very good point. Yep. Yep. I think there's

Bryan: 15:40

a situation way back in the a couple couple years ago too where, they were injecting or putting in some some chips inside of some used or what was claimed to be used Cisco equipment and put it up on eBay and or Amazon and that they they also had backdoors into those. So, I guess that could technically be a supply chain attack.

Barinder: 15:59

No. Absolutely. No. I remember that.

Justin: 16:03

Alright. So, guys, I don't know about you, but I have literally had nightmares of my, my technology getting attacked, getting breached, and and the ramifications of that because, like, I I said this before. I did not get into IT because I like cybercrime. I didn't do it. I didn't do this on purpose.

Justin: 16:23

I'm not a criminal justice major, but here we are. We are we are literally fighting crime, organized crime, serious crime, with with little well, almost no regulation, very little required training, and it is the reason that I insist on doing these podcasts because it keeps me sharp. It keeps me aware. But this, what we're talking about today, is literally something that I have nightmares about. And, Robert, you got kinda forced into this where, I mean, honestly, I think that most of us, if it happened to us, would kinda try to hide it from the public.

Justin: 17:00

But here you are as a willing and outspoken victim. So from, like, genuinely from the bottom of my heart, thank you for being on here. We we reached out. Thank

Barinder: 17:12

you, Robert.

Justin: 17:12

Yeah. Brian suggested it. I'm like, I don't know. Will he really do it? And, then I find out that not only will you come in here and talk to us, but you're kind of on a traveling circuit

Robert: 17:21

Right.

Justin: 17:22

And talking about it. So, this is the burning question for me. What was the emotional impact? Because we talk a lot about financial impact, technology impact. I've seen it in person a little bit, but I had a client get breached way back when.

Justin: 17:39

This was when I became like, oh my god. I'm not an IT or a computer guy. I'm a cybersecurity guy. And this was a husband, wife team. They almost got divorced over it.

Justin: 17:48

They almost lost their business over it. They fired me over it. Like, none of this. Nothing good came out of this. So tell me from your perspective, what was the emotional impact as as you went through this?

Robert: 17:59

Well, that right there is the crux or the epicenter of my story that, while this is a very highly technical story that happened to a bunch of technology companies, including our own

Justin: 18:14

Right.

Robert: 18:15

The story that I have to share is the human side of this, the emotional, the psychological impact. So thank you for starting with that.

Bryan: 18:24

Mhmm.

Justin: 18:24

Before I

Robert: 18:24

go there, I just wanna focus on 2 words that you used. One that's been used repeatedly, and I don't at the risk of sounding, preachy or, or professorial here, or persnickety. The word breach has a legal connotation to it.

Justin: 18:43

So what happened

Robert: 18:44

to us was, an attack. There was an incident. There was no breach, because breach, does imply, some legal liability, especially with, regulation and compliance and notification and things like that. Lawyers can explain that way better than I can. But that was one thing that during our attack, I learned very, very quickly, don't don't use the b word, because that can open up some, legal trouble for you if you're ever

Justin: 19:13

I'm taking notes.

Robert: 19:15

Yeah. Yeah. And I know. Listen. You know, and I think you said it also very well that this industry, you know, I joke around that my mother who's in it, you know, an immigrant from Italy, didn't get past the 6th grade because there was no 7th grade in her small little town that she grew up in, in the mountains of Italy.

Robert: 19:34

She could print a business card and be an I you know, be an IT consultant tomorrow. There's no barriers to entry. Right? I am way overdue for a haircut. The girl who cuts my hair needs a license in New York State to cut my hair.

Robert: 19:45

But none of us need a license or any kind of certification to do it do what we do.

Justin: 19:50

So crazy.

Robert: 19:51

The world is a little upside down for us, but that sounds like a whole another episode. So, so I'm gonna leave that there. But I wanna focus on a really important word, getting closer to your question about the human, or emotional impact that it had upon me, and that is the word victim. I wanna pause there for a second for your listeners to really let that word sink in, and then I wanna explain why I do what I do because one of the first emotions that I felt was severe shame. I mean, I was beside myself.

Robert: 20:29

How can I, the expert, the person who is given the stewardship of protecting all of these companies, how could I somehow and maybe this is Catholic guilt speaking and not reality? How could I allow such a travesty, an injustice, a crime to take place upon my watch? And, what I needed to do was to come to terms very quickly with the fact that I was a victim, and everybody else who was, involved or touched by this crime was a victim of that, particular, criminal act. So one of the mantras that I use repeatedly is that there is no shame in victimhood. And I wanna repeat that.

Robert: 21:20

There is no shame in victimhood. So if anyone should ever undergo any sort of traumatic experience, be it a personal, issue or be it a business event like what we went through, you need to kind of suppress those feelings of shame because you are a victim and, victims, need help, and it's okay to say that. Right? I had to come to terms with me speaking vocally about this very quickly, to understand that that my victimhood did not make me less of a human, did not make me less of a professional, and did not diminish the capabilities and the integrity of my company. But everybody else in the world wants to put their head in the sand or to stay quiet, right, and muzzle people.

Robert: 22:12

Let's not talk about this. And, honestly, that was one of the first thoughts that went through my head was I don't like, let's just shut up, fix this, and, like, hope to god that this doesn't make the 6 o'clock news.

Barinder: 22:25

Right.

Robert: 22:27

But, so there I was on July 2, 2021, and there's a lot of details. We could spend hours talking about what happened, and how I found out, but it was it was brought to my attention that our phones were ringing off the hook. Everybody's phone was blowing up. Email blowing up. Text messaging.

Robert: 22:48

We came to realize very suddenly that all of our customers were completely ransomware because a cybercrime, element had broken into our Kaseya VSA server and used that, tool in which we use to manage all of our cost customers, use that tool to carry out a malicious deed, right, to install that ransomware on everybody's computers, by the way, including our own, including the very I'm speaking to you from right now. Right? That computer, I still have today. Right? It was reformatted and rebuilt and blah blah blah.

Robert: 23:24

But, you know, that's what happened to me, on July 2, 2021. I watched, proverbially you know, and I hate you know, I say this all the time too. You know, people a lot of people suffer from the Hollywood syndrome. Right? You watch too much TV and movies and, you know, the movies and television aren't an accurate description of reality.

Robert: 23:46

But there are some things that I can draw upon Hollywood for. One of them is this notion of, you know, your life flashing before your eyes. Right?

Justin: 23:54

Yeah.

Robert: 23:55

That's what happened to me. I thought about my entire career, 28 years being running this company, thinking about all the relationships that I built, the the the the security that I built for my family, and, you know, my business partner, all of our employees, the trust that we had in so many people that we had built with over those decades, I watched it all evaporate. Right? Right.

Justin: 24:21

Yeah.

Robert: 24:22

And for about 10 minutes, I was completely frozen, staring into the abyss, contemplating my own mortality.

Justin: 24:29

Mhmm. Right?

Robert: 24:30

And even when I talk about this, the emotions are always, there. They're omnipresent. Right? They won't ever go away, but it helps me. It's a bit little bit cathartic to talk about it because if I can change somebody's perception about what they need to be doing in their business, now whether that's a business owner who's nontechnical or a technical person who's in charge of managing technology.

Robert: 24:55

Either way, if I can move people, toward doing something better for themselves, then I'm taking that really horrible experience that we went through and turning something positive out of it.

Mario: 25:10

So what what happened next? Like, you you you know, this you you I guess, your your team was the one that notified you Yeah. About it.

Robert: 25:19

Yeah.

Mario: 25:19

And what did you what happened? Like, did you gather them all together? Did you have a game plan?

Robert: 25:24

Yeah. I mean, we spent, my operations director and I, spent about, 10 or 15 minutes, in this room that I'm sitting that I'm speaking to you from right now. This is my desk in my office, and he was, you know, right at my, shoulder here, off my left shoulder. We were digging into systems trying to figure out which way was up, what the hell happened. After we realized that we were, I mean, screwed is really the only way I could explain, the feeling, just this utter and complete annihilation of our business.

Robert: 26:00

I knew that we had to do something, but I didn't know what. So what I did is I, I I thought about some historical figures. I'm a pretty big history buff. I thought about Winston Churchill, who said when you're going through hell, keep going. Right?

Robert: 26:19

He said a lot of things about, his leadership in England during, the war against the Nazis, and that was one of the lines that he used, as many many, inspirational quotes from him. But then I challenged another world, or I channeled another World War 2 figure, and that was general Patton. Right? I turned into general Patton and just started barking orders at people because it was the only way I could contain my emotions is to just become stoic. And, you know, with this rock hard exterior, like, just all business and, and I

Justin: 26:54

Correct me if I'm wrong, but this is what your team needs in that moment. You don't get to meltdown.

Robert: 26:59

No. That's correct. I was melting down on the inside, Justin. I was

Justin: 27:03

But you can't.

Robert: 27:04

I was a puddle of emotion, and, you know, fearing for, the the, financial well well-being of myself, my business, my partner, our employees, as well as our families. Right? But here I had to be sort of tough. And so, you know, I started barking orders at people. You go downstairs, get everybody at help desk, get their asses upstairs.

Robert: 27:29

I mean, these are words that I was using.

Justin: 27:30

Mhmm.

Robert: 27:31

You, tell everybody to get the hell off the phones right now. Put them down, get them in the conference room. I quickly realized I was being a little too So I remember going up to one individual who was on the phone with a client, and I said I tapped him on the shoulder, and I said, when you get a chance and the conversation, you come into the conference room. Right? I started to tone it down a little bit.

Robert: 27:55

But, you know, I would be a liar if I said I had a prepared speech. I had been down this road before, I was experienced in these sorts of matters, I was trained in crisis management, or had bullets flying over my head in multiple theaters of, of operation and in various war like, none of that existed. Right? Here I was just like, holy shit. I need to figure out what the hell I'm gonna do and what the hell I'm gonna say.

Robert: 28:23

And my mind was a blank, other than, you know, general Patton sort of taking over here. But we sat in the conference room. Slowly, people started to walk in and, I don't know. Something just clicked. Something snapped in my head.

Robert: 28:38

And I remember sitting there looking at my business partner, and I remembered the foundation of you know, some of the foundational pieces of our culture and our company. We're an EOS shop here if anybody knows what entrepreneurial operating system is. Yep. So the concepts of core values aren't really that foreign to anybody even though if you if you don't know EOS. But I just recited our core values.

Robert: 29:02

Alright? And I read through all 4 of them 1 by 1, and drew upon them for both comfort and inspiration to let everybody know that we're in a serious situation here. But together, you know, we were gonna get through this. Gave them some things to say and do to clients. And then, I mean, I'm giving you a very short version of the the of the moments that, I spent in the conference room there.

Robert: 29:30

But, essentially, just, you know, got them back to their desks so they can start handling, the influx of calls and and just managing the questions of what the hell just happened here.

Justin: 29:42

I love that you went to core values. I absolutely love that. I mean, like, companies have core values, but this is how you know that you actually live them.

Robert: 29:49

Yeah. Love

Justin: 29:50

that. So, Brendon, I

Robert: 29:51

hope they were very, applicable to us. Yeah. They were they were not immediately, understood how applicable they were. But, I mean, some of them really stood out. I mean, one of our core values is team.

Robert: 30:04

Together, we get it done. And so, you know, what a huge rallying cry that was.

Justin: 30:09

Yeah.

Robert: 30:10

Commitment, determined to do what it takes. Like, hey. Like, we're not talking about fixing a printer commitment. We're talking about, like, turn the world upside down commitment.

Justin: 30:21

Grab a

Robert: 30:22

shovel and start going. Right? Another one was, humble confidence. Right? Service without ego.

Robert: 30:28

So we have to lot have have a lot of confidence in that moment, but we also had to have a lot of humility because people were pissed. Right? There were people who are super upset by this, and we needed to, you know, handle that with kid gloves. And then finally, respect is our 4th core value, respect always, and remembering that we just have to have a lot of respect for each other, our customers, our end users, our families, our vendors, anybody who would come to help us. Respect was something to help us keep grounded throughout that entire event.

Robert: 31:02

So we used them in every way that we could and reminded each other of them throughout the next, specifically 17 calendar days to get us through most of the recovery.

Barinder: 31:15

Robert, that sounds like an absolute nightmare of a day that, like Justin was saying earlier, keeps us all up at night in the MSP industry.

Robert: 31:22

They truly

Barinder: 31:23

Yeah. Like, how how how law yeah. And rightfully so. If it doesn't keep you up at night, then you should get out of the MSP industry. Right.

Barinder: 31:28

But but so how long was it before you But but so how long was it before you figured out it was VSA was a source of the breach? Did it matter?

Robert: 31:37

Yeah.

Barinder: 31:37

Like, what was the rest of your day like? Who did you need to call or what your like, your clients are probably still calling you directly at this point. Like, who are you talking to? So

Robert: 31:44

there was a lot of confusion with communications. They were happening bidirectionally, across multiple mediums. Right? Like I said, there was tickets coming in, emails, phone calls, text messages, like, you know, landline calls, cell cell line calls, like, everything. I think somebody even reached out to me on social media, right, just to say, what the hell is going going on here?

Robert: 32:07

You know, our instincts took over pretty quickly. As soon as I heard, because my operations director was the one who approached me first to tell me that all of our customers were ransomware. And, of course, that's just a completely ridiculous statement. So I rejected his, you know, untruths to me and inserted my own reality that that's an impossibility. How could that happen?

Robert: 32:32

Right? There's no way that that could happen. And, after he started to rattle off customer names 1 by 1 and he and I still have a little debate over this as to whether or not, he intentionally picked the largest ones to talk about first. After he got to about number 15, I was like, okay. Stop.

Robert: 32:51

Now you have me convinced, yes, all of our customers are ransomware. I instinctually knew that it was Kaseya VSA. Mhmm. What else could it have been?

Barinder: 32:58

It can't be anything else.

Justin: 32:59

Yeah. There's no

Robert: 33:00

more nothing else that it could have been. And, like I said, we came into my office, and I had 2 of my top engineers who are already inside VSA investigating, looking what was going on. It was our first hunch, and it was the right hunch, that that's where the attack took place. My one of my engineers saw the script, running, but it had already been processed on almost every single computer that we managed. So he caught it right at the very tail end, and it was like, I can't even stop this.

Robert: 33:36

It's the damage is done. So, so that's kinda how we knew. It was pretty obvious that it had to have been that, and looking for the evidence or at least the the surface level evidence wasn't all that very difficult, to uncover.

Barinder: 33:56

So, like, what was your first call? Is it to a lawyer? Are you talking to clients?

Robert: 34:01

Yeah. What was our first call? So while I may sound, put together and, coherent right now, that day was anything but put together or coherent. There was a lot of chaos. And I made a number of I mean, after the conference room, conversation and dismissed everybody, I was just I was calling everybody that I knew.

Robert: 34:27

I I was my first instinct, which was wrong, by the way, and I'm very public about, telling people about the things that I did wrong. I started to call my friends that I knew that had security experience. I started to reach out to other names that I knew of that I knew that were security experts that somehow, magically, they were gonna, like, wave a magic wand and make land somewhere go away. It was I don't wanna say it was stupid thinking, but it was, maybe a little too wishful. Mhmm.

Robert: 35:00

And it took a number of phone calls before, my director of finance, who does not have a technical bone in her body, she came knocking on at my door saying, you know, hey, Robert. Do you think we should call our cyber liability insurance?

Justin: 35:17

And

Robert: 35:17

if you really wanna laugh, that policy went into effect on Monday, June 28, 2021, 4 days before the attack.

Justin: 35:24

Oh, shit.

Robert: 35:28

And I'm like I looked at her, and I'm like, thank you for your brilliance.

Mario: 35:32

And after all

Robert: 35:34

this is done, you could take me out into the parking lot and flog me for not thinking about this faster. But, yeah, as soon as you activate your as soon as you, enter the claim, you get assigned. And now here's this word, breach. You get assigned breach counsel, which is a terror I've yelled at every single lawyer that calls themselves a breach counsel. It's a it's a terrible title, and they all admit to it.

Robert: 35:58

But that's what they're known as, breach counsel lawyers who are experienced in handling cyber events. And then from there, depending upon your policy, you could be assigned crisis management, forensics, even a negotiator. We had negotiator capabilities. In fact, a negotiator had gone as far as to, follow the ransom instructions to see what the next steps were, which I could have, you know, done just by looking at the ransom note myself. But he actually went to the, dark website and, you know, looked at the the page for instructions and whatnot.

Robert: 36:38

That was something that we did not pursue. But the best resource that I would advise anybody who's going through something like this is to make sure that they have counsel. You really I mean, I hate hiding behind lawyers. And it was not hiding behind lawyers, but was getting a lawyer to sit shoulder to shoulder with you to make sure that you're navigating a situation like this very carefully. There were things that we could have done wrong had we not had that assigned counsel.

Mario: 37:09

And then what happened after that? Did you, like, did you start trying to recover? Did you try to figure out if Kaseya was, you know, had a fix for it?

Justin: 37:20

Or or Yeah.

Robert: 37:20

Or Yeah. I was really I mean, I think, I was in a little bit of denial and sort of in that wishful thinking state for about 24 hours, maybe less. You know, it was probably far less than that. It was probably sometime that evening where I just realized there is no undo here. Right?

Robert: 37:40

For all you techies out there, control z on your keyboard. Right? There's no there's no control z to undo the mistake that you just made in Word or Excel. There was no way to kinda reverse this to install a piece of software that, like, you know, cleaning up a virus. It that your systems are encrypted.

Robert: 37:59

Without that decryption key, you are way up the the stream without a paddle.

Justin: 38:05

I will. I wanna I wanna interject a little bit because there is a particular I won't name names, but there is a particular, called an antivirus company. You know, we we all know them as something more technical than that. But who who has that claim that they do have a rewind button in the case of ransomware attacks, and I don't remember which episode we were talking

Robert: 38:27

about this. They may have a rewind button, but they don't have the ability to decrypt.

Justin: 38:32

Well, what they do though is there is about 12 pages of fine print that say what has to be in place for it to work. And, basically, it's impossible. It's not gonna happen. So there's a lot of false sense of security Yeah. In, you know, thinking that we've done x y z, and we're now protected.

Justin: 38:48

The reality is, like I said, once you get hit, god, coming back to me is tough.

Robert: 38:52

Read read the fine print. If that were the case, then I'm sure every one of our clients and every IT department in the world would have that technology in place.

Justin: 39:00

Right. Right.

Robert: 39:01

Now there are certainly things that you can do to, ensure that, after boom goes off. Right? Right of boom, like I say. For all of you military people out there. After a right of boom, you have to be able to respond and recover, properly.

Robert: 39:19

And sometimes technology can help with that, and sometimes it's just good old, human plans and human decisions and actions. But

Justin: 39:30

about law enforcement? Do you have to get law enforcement involved?

Robert: 39:33

Yeah. So the I wanna make sure that I got, Mario, you asked the question about, right, about, like, you know, what sorta happened next, but it's okay. I'm gonna kinda try to loop all this together. Well, one of the things that I did, early was to file an IC three report at the FBI's website. Right?

Robert: 39:50

Fbi.gov, Internet Crime Complaint Center is the I c three. And then within, like, an hour and a half, I had an FBI agent in my ear asking me questions because it became very, quickly known, and widely known in the community, and in the law enforcement arena that, you know, this was a widespread international attack. This wasn't just progressive computing being singled out. If that were the case, then we probably wouldn't be having this conversation. You probably would not know me.

Robert: 40:27

Well, maybe. I don't know. I can't I can't say

Justin: 40:29

that, but

Robert: 40:30

but it certainly would be be a different story. But in terms of recovery, no. One of the things that our, our attorneys had advised us is to get a proper forensic analysis first. So, and and I'm not plugging here. Believe me, when I name a name, it's simply because out of the I feel that they did the right thing by me.

Robert: 40:54

We were in the process of, testing Huntress on about a 100 different, systems here, which is a small segment of our, you know, population of of users or machines under management. And, you know, they found out about what happened super fast. I'm gonna make a long story short and just say I'm on the phone with a lot of their people there, including their CEO, Kyle Hanslovan. And they essentially offered to take our Kaseya log files and do the analysis. Right?

Robert: 41:26

Now granted, they got some good publicity out of the work that they did, but it was very helpful to us to understand the nature of the attack. Right? So, Mario, if you were ever to, you know, be in a situation like that, like many of people like you and I who are, you know, smart, experienced, people in this industry, your gut reaction is recover and recover now. But it's the wrong thing to do, especially when you have clients that are breathing down your neck, right, and yelling at you and and threatening you with lawsuits or bodily harm. Nobody threatened me that way.

Robert: 42:02

But certainly lawsuits, it felt like they were threatening me, but nobody actually did that, because they know better. But it and seriously, I I like to joke about this because it's such a serious topic that the humor helps me get through some of the rough parts. But, to the point, and that is that you've gotta make sure that you understand the nature of the attack before you can do a recovery plan. Because what if because the first thoughts that went through my head, how long were these guys in our system? Mhmm.

Robert: 42:36

Were they there an hour, a day, a month, a year? Did they exfiltrate or copy data out? Did they put in a 100 backdoors? What like, what did they do? What do they know?

Robert: 42:48

What do they know? What do they have? These were all giant fuzzy questions that I had no answers to. So it wasn't until Huntress completed the forensic analysis on Saturday afternoon, Saturday, July 3rd, and, Kaseya did an independent review of our same log files and turned around an answer to us on the afternoon of Sunday, July 4th, that is when we knew 2 independent analysis confirmed, particularly for us at 10:49 AM, a threat actor, using a commandeered AWS web server had launched their attack against us specifically. Now that data that I just provided you and that timeline was different for every single customer that was attacked.

Robert: 43:40

And when I mean customer, I mean a Kaseya customer, me being a Kaseya customer. But at 10:49 AM, the attack started upon us, and the 80 clients that it affected for us, which was a 100% of our clients, that had Kaseya VSA installed, were all ransomware between 10:49 AM of on July 2, 2021, and the encryption process ended somewhere between 12 and 12:30 PM, depending upon what speed of processors, how much memory, how fast disks are. If you really think about it, all of us guys who converted everybody to SSD drives, right, like, they're super fast. Right? They're great performance, especially in laptops.

Robert: 44:25

Well, guess what they do? They They

Justin: 44:26

help the bad guy.

Robert: 44:28

Very, very efficiently. Right? Bad guys use good technology

Justin: 44:31

into their Mhmm.

Robert: 44:33

So it wasn't until Sunday afternoon that in consulting with our attorneys that we had a recovery, a plausible recovery plan in place. We knew that we could start restoring servers sometime anytime before 10:49 AM. In our particular case, we made a decision to go back to 8 AM even though we had some 9 and 10 AM snapshots because our systems back up hourly. We decided to go back to 8 just because we were too paranoid. Right?

Barinder: 45:08

Mhmm. Mhmm.

Robert: 45:10

So I hope that kinda answered your question. We didn't really start our full recovery efforts until Monday, July 5th, which was Independence Day Observed. We We took on 3 clients that day because we did not have a battle tested and battle hardened recovery plan. And we needed to kinda figure things out. So we decided that day we're just gonna attack 3.

Robert: 45:38

Or I should say go take care of 3, not attack. We're not attacking our own customers. And, and then started to ramp that up on Tuesday, Wednesday, Thursday. Right? And that's also where the story takes an interesting turn about how we, were able, to, to get to so many customers.

Robert: 45:58

And I know 17 calendar days doesn't seem like a short amount of time, but that's, like, super fast.

Justin: 46:04

Yeah.

Bryan: 46:04

Well, for the scale of

Justin: 46:05

an attack, it is. I I mentioned this the only story I have with this was a client of, I think, 10, 11 computers counting the server, and it took us almost 3 weeks. Yeah. And we did a longer story, but yeah.

Robert: 46:19

Yep. We did 250 servers and about 22100 endpoints in 17 calendar days.

Barinder: 46:25

Jesus. Robert, I understand that community came to your aid. Like, obviously, like, you know, Huntress was helpful here. Anybody else that that was able to help in this scenario?

Robert: 46:37

Yeah. I mean, you know, it's it's worthy to note that our, disaster recovery, partner, Acxiom, was a massive help to us, providing us both technical resources and, like, direct line instant response, like phone call like, skip the, skip the help desk queue kind of thing. ConnectWise was super, super helpful to us. Pax8 stepped up a bit as well, Huntress. But then there was the, IT Nation evolved community.

Robert: 47:05

We've been a a member of a peer group, which some of you may know as HTG since 2010. It was then bought by ConnectWise, rebranded as IT Nation in somewhere around the 2019, 2020 time frame. And all of our friends that we had made over that decade plus started getting wind of what happened. So we had other MSP owners, flying as far away from Santa Barbara, California, Austin, Texas, Minneapolis, the middle of Kansas, Massachusetts, Florida. People came in from everywhere, not just, business leaders, but also bringing some of their best technicians along with them.

Robert: 47:50

Wow. We have a math problem here. Right? How do we, as an MSP, who if you really think about, we're sorta like a, you know, like a time share. Not everybody can show up for service on the same exact day.

Robert: 48:04

Not everybody could show up to the whole, you know, to the time share unit and vacation on the same exact day. Right? And so we can't service all of our customers simultaneously. No MSP can. And if you think that that's if you're gonna argue with me on that one, we'll have a big There's no way that you can it's not we're not designed that way.

Justin: 48:25

Mhmm.

Robert: 48:26

Even an IT department for any size company, enterprise or all the way down to a smaller company, your IT staff can't handle every single user needing service now. It just doesn't work that way. Right?

Justin: 48:42

Right.

Robert: 48:43

So we had to figure out, and it was pretty easy to figure out. We just needed as many bodies who are willing to help us as possible. All in all, we had about 27 different companies help us either locally by coming in or helping with some of our remote locations across the US, or just or just helping by doing, remote work from wherever they sat.

Barinder: 49:10

Wow. That I I love this community. What what a great story.

Justin: 49:14

Yeah. Yeah.

Robert: 49:15

And listen. And I don't I don't have a lot of experience outside of the MSP community as a as a business owner or an operator. But, you know, I'm glad that you said that, Birender, because I don't know if other industries are as fortunate as we are. The camaraderie and the fraternal type of nature that we have. I tell the story sometimes.

Robert: 49:38

I'm at a swim meet, watching my daughter swim, because she's on a competitive team. Actually, both of them were. And I was sitting in the lobby of, like, this y, it was a YMCA in Greenwich, Connecticut. And I was catching up on some work, and I was drawing some flowcharts out. And the guy some guy sitting next to me looks over my shoulder and he goes, are you in the IT business?

Robert: 50:02

Right? Turns out, I'll say my competitor, a comp an MSP in New York City. Like, you know, instead of it like, it was it turned into such, like, a friendly conversation. Like, you run into each other in the strangest places or at conferences, and we're very giving of each you know, to each other. We're very there isn't a lot of, you know look.

Robert: 50:24

Mario and I are kinda neighbors in a way. Right? Yeah. We'll never cross paths or even if we do, we'll be like, okay. So, like, there's so much business around.

Robert: 50:34

We'll help each other. I know we would. We we we're just seeing each other now.

Barinder: 50:38

Absolutely.

Justin: 50:39

Yeah. Yeah. So

Bryan: 50:43

I have a question. After all of that happened, I I mean, I know that, you know, if something like that happened to me, I would probably be looking at another provider. Did you lose a lot of business as a result of this?

Robert: 50:54

So very strange phenomenon happened. I got sued. I got personally, and verbally abused and attacked, and we had gift baskets sent to us. So

Justin: 51:09

God.

Robert: 51:09

I had one customer tell us about 8 months after the attack, he said I was sitting in their conference room leading them through a strategic business review, you know, an annual assessment and review with them. And they said, you know, we thought about firing you, but then we decided, that that would not be a good idea for two reasons. Because number 1, the way you guys responded and handled yourselves, with such grace under such enormous pressure. And 2, who would we rather have on our side as our MSP other than an MSP who's lived through that, that has walked through hell? Right?

Robert: 51:46

Now I'm not trying to take away anything from any other MSP. That's not the point. But because I got so beat up, at least I get the claim that I've lived through it. And it should something like that ever happen again in any form that I've got the experience, we have that experience. We know what it's gonna be like.

Robert: 52:07

So we did take a pretty big haircut over time. There were just some people who were bent on blaming us. We had one customer say, we don't wanna fire you, but we have to because our largest customer, doesn't like what happened, so they're forcing us to fire you. They they basically told them fire your MSP, or we're not doing business with you anymore, which is a little bit of horse crap. But Mhmm.

Robert: 52:34

But that's, you know, that's life and that's business. Right? You gotta just, you know, hold your head up high and just keep moving along.

Justin: 52:44

Crazy, man. So law enforcement. I, I mean, I I I'm intrigued because, like I said, this isn't my world. At least it wasn't supposed to be. But you did have to get law enforcement involved.

Justin: 53:00

But here's the problem. Most of these guys hide behind, well, they hide. Right? In other countries, they hide behind cryptocurrency. They hide.

Justin: 53:10

We can't usually get them. There's not much Right. What what was law enforcement able to do other than tell you, hey. Sorry, Robert. You're fucked.

Justin: 53:17

You know?

Robert: 53:18

Yeah. Well, that's they said it in a much nicer way.

Barinder: 53:21

Actually, they don't even think

Justin: 53:22

that for at least that.

Robert: 53:26

I didn't really expect a lot out of them because I were, well informed the way this works. Right? Like, I'm thinking to myself, why am I even filling out this FBI Right. Point other than just to report myself for what I just instinctually felt I needed to do it. And by the way, we encouraged all of our customers to do it, believe

Bryan: 53:49

it

Robert: 53:49

or not, at the request of the FBI. And so I complied with that request to compel everybody to do so. Some did, some didn't. Can't tell you how many did and how many didn't. And while I was in communication with them, receiving updates, there was a conference call or 2 I sat in on, got another call from another agent.

Robert: 54:12

Basically, I'm retelling the story yet again, like, 2 2 or 3 days later. I kinda wrote it off and thought, alright. Like, you know, like you said, my attacker is probably some faceless, nameless person in some other country with 0 extradition to the United States. And I'm never even gonna know the name of my attacker. Well, as some of you who have followed the story know that in October of that year, an individual, a 23 year old Ukrainian, by the name of Yaroslav Vasinski, decided to go back to school in Poland, which is where he was a student, and decided to travel from his home in the Ukraine, into Poland.

Robert: 54:59

When he crossed the Polish border, interpol and the Polish police apprehended him at the border. Why? Because the FBI had been tracking him. They knew who he was, and he was apprehended at the request of the FBI. What the FBI would later tell me, because I've actually met, some of the people involved in the in the case in the background, what they would later tell me is they flew there.

Robert: 55:26

He was very belligerent. Basically, like, f you. You're not gonna get me. I'm going back to my country. Screw you.

Robert: 55:36

And then 5 months later, he was extradited to the United States and was put in a Dallas, federal prison, and charged with something like 11 counts from the Department of Justice. Right.

Justin: 55:49

It

Robert: 55:50

was my understanding, because I, like I said, I had some personal direct contact with some people at the DOJ and the FBI. They entertained my questions. It was very gracious to me. And for that, I'm very, very grateful that I was I had at least a little tiny taste of the entire process. But I asked them, like, so 5 months.

Robert: 56:12

Was that, like, a long time? And they said, no. If you know anything about extradition, from country to country, it's potentially a multiyear process. It was lightning fast. Mhmm.

Robert: 56:22

The, the Polish authorities were incredibly helpful. So if you are part of the Polish law enforcement, and you're listening to me, thank you. I don't know how to say that

Mario: 56:35

in Polish.

Robert: 56:35

Hopefully, you can understand my English. Appreciate that. It's nice to have allies, in the world. So and another suspect or another coconspirator was indicted, and he is living in South Central Russia. So unlikely that that person will, ever be, served justice, but Yaroslav not only plead guilty, but was sentenced to 14 years.

Robert: 57:03

I was present for his sentencing. I

Bryan: 57:06

got some

Robert: 57:07

impact statement, in the Earl Cabell Federal Building in Dallas, Texas at 1100 Commerce Street. Some things I won't forget, before judge Schoeller on the 16th floor. Fact check me. It's all correct. There were assistant US attorneys present, FBI agents present.

Robert: 57:27

There was, Yaroslav was, there in an orange jumpsuit and shackled at the wrists and ankles. There were US Marshals, attending to him, probably separating me from him. Mhmm. You know, I kinda guess, but, you know, listen. I have a lot of reasons to have such enormous hate in my heart.

Robert: 57:48

But here's the other, I think, surprising thing. Because people are like, you know, did you wanna, like, jump over the thing? Yeah. I mean, I'm, yeah, I'm human. I a part of me wanted, you know, to dismantle him, you know, limb you know, appendage by appendage, slowly by by, by the way.

Robert: 58:05

But, like, here I am looking at this child.

Justin: 58:09

Right. That was my thought. 22.

Robert: 58:12

Get get this image out of here. What do you think about when you hear the word hacker? Right? Hollywood, hooded, faceless, like, dark, like, dark room. You know?

Robert: 58:21

Like, you know, it's this it wasn't some evil person. It was this child in that room. He looked like he was 14 years old. And while I don't care for him as an individual, I sorta had a weird sense of pity Yeah. That he kinda got caught up in something that he shouldn't have.

Robert: 58:46

And while I think some of his statements to the judge to get his sentence reduced were full of crap, And I don't, you know, I don't, I think he should serve all 14 years and wish he could serve more. But I think it was just, you know, part of the actions of stupid 22 year old. What I'd really like to see are the leaders of Revill Corp. Right?

Barinder: 59:12

Mhmm. Right.

Robert: 59:13

Those are the makers of the ransomware, software that he was using, to carry out his dastardly deed along with his coconspirators, plural. I'd like to see those people brought to justice. Yeah. Absolutely. Evil bastards.

Mario: 59:30

Yep. I I completely agree. They they never helped with any of that stuff. I

Robert: 59:37

don't know. What do you mean? I'm sorry.

Mario: 59:39

Were were were they able to have provide any type of help in the in the process?

Robert: 59:44

You mean the FBI?

Mario: 59:45

No. The the the people that caused the the the software issue, to say.

Robert: 59:51

No. I mean, you know, if you if you read up on the root news stories, what happened was, about a week into the events, apparently, Biden called, Putin, and it was some White House to Kremlin phone call, basically to say knock the shit off. Right? Because they're because Reeveil has ties to the topmost parts of the, Russian political, you know, of of the government. It's widely known that the the the leader of, Revill Corp I I can't remember his name or I probably can't even pronounce it, but he's, I don't know.

Robert: 01:00:29

He's probably in his thirties. His wife, his father-in-law is the head of the FSB, which is the former KGB or was. So had direct ties to, you know, the topmost parts of the, Russian government, is widely known to drive one of his 5 Lamborghinis around Moscow doing donuts around police. You know, the guy operates with impunity there. So is he state sponsored?

Robert: 01:00:56

That's kind of a gray statement, but it's pretty much I mean, it's it's the same equivalency. Right? You know, those are the people that need to be, brought to justice. Right? But, you know, they'll never leave or they'll never set foot in a country where the US can get their hands on them.

Justin: 01:01:16

When whether they're state sponsored or not, something that's interesting in in other countries and maybe some to some extent our own, but, these guys pay off the government. They it's it's a line item on their p and l where they pay for protection to do this stuff. So sponsored or not, they're definitely protected in a lot of cases. So we are fighting we're fighting we're fighting big, big names. We're not we're not fighting 22 year olds.

Justin: 01:01:44

We're fighting way higher than that, unfortunately.

Robert: 01:01:46

So so so this is the point here. Like, I saw I finally got to meet my attacker face to face. I got probably within about maybe 10 or 12 feet of

Justin: 01:01:55

them. Right?

Robert: 01:01:56

Like I said, I'm describing to you this, you know, essentially, this this person that looks like he's a 14 year old boy to me. Right? And, you know, here I am thinking, you know, he is the just very, very tip of the spear here. There is so much more to that spear Yeah. Than, you know, him.

Robert: 01:02:13

And it's and I would really love to get my hands on the rest of that spear, not that very, very tip. So for those that don't understand the way this stuff works, it's very much like a legitimate franchise business here. Right? So if I wanted to open a Dunkin' Donuts, I could go make an application to corporate to, you know, to get a franchise license. Right?

Robert: 01:02:37

And I would have to pay money to Dunkin' Donuts to use their brand and their product and blah blah blah. Right? It's not a lesson on franchise business models. But the same parallel exists within ransomware as a service models, meaning some company or some group of individuals, and I use the word company literally, some company, which group of individuals, writes the software, puts up the web page, has all of the processing capabilities to process crypto payments. Right?

Robert: 01:03:07

And then when they collect, they give a kickback to the franchisee. Right? And so that's what this kid was caught up in

Justin: 01:03:14

Right.

Robert: 01:03:14

Was how do I make some money using Rivo Sode Nokibi. Right? And, apparently and I don't know what the story is. I wish I did, but I'd be really curious to know how did he find out about that 0 day exploit. We were fully patched on our Kaseya VSA server.

Robert: 01:03:33

We had the very latest security patch installed exactly 1 month before our attack. I have proof of this and everything. I know we were fully patched. There were still some holes, and he walked right through that stuff. So, yeah, I know there's a you know, there's so much in this story.

Robert: 01:03:52

It's hard to kinda get it all out coherently.

Mario: 01:03:55

Yeah. Yep.

Robert: 01:03:56

I know I'm jumping around a little bit.

Bryan: 01:03:58

Al, that's

Justin: 01:04:01

I I

Mario: 01:04:01

know you created a nonprofit as a result of this event. Yes. Can you tell us about that?

Robert: 01:04:07

Sure. So, it was August of that summer. So about, 5 or 6 weeks after what had happened, and I was reflecting. I was getting philosophical, on what had just happened and, taking, taking stock and account of, who I owed my life to, and then realized something that the magic of the community coming to our aid was like the writers of Rohan, right, turning the turn the tide of battle in the Lord of the Rings just when all hope seemed completely lost to hear the writers of Rohan show up and turn the tide of that battle around. Right?

Robert: 01:04:45

It was really a pivotal moment in the book. And that's the way I felt was such my, you know, my heart went out, in gratitude to all these people. And then I realized, like, most MSPs in this world aren't as well connected as I am, have not put the investment of time into going into conferences, joining peer groups, making tons and tons of friends. Many of us, unfortunately, sort of stay stuck in our own little worlds. Right?

Robert: 01:05:17

I might know 1 or 2 others, but I don't really know anybody. That's the norm. And then I thought if one of those guys got popped, how would they be able to handle this? They probably wouldn't be able to. So that was the genesis of my idea, and I registered the domain name immediately, msp911.org, with the idea that if you ever get hit, that you would have sort of that giant red emergency button or break glass in case of emergency.

Robert: 01:05:45

Now we're in a little bit of flux with MSP 911 because we, I ended up, and I'm gonna make a long story short here. I ended up meeting some folks over at CompTIA, and we, ended up joining forces with the idea. We branded it as the CompTIA emergency response team, but we're in the process. And for all very good reasons, and everybody is still very friendly over this, We're in the process of spinning that off, and becoming our own entity because there are some constrictions based on CompTIA's legal charter and nonprofit status that we can't accomplish as part of our mission. So it's better that we, operate as a stand alone.

Robert: 01:06:30

So that's gonna get rebranded as something called Cyberrise. The website, I believe, still points to the Comtea landing page. So if you went to it today on, you know, in October of 2024, it probably is still gonna land on a CompTIA page. But soon, we'll have that redirected so that, it will we will be able to take inbound requests for help. And, it is a 100% volunteer led and driven organization.

Robert: 01:06:59

There are many of us, who are, inspired by some of the things that I talk about and my experiences as well as some other victims that I met, that are all willing to step up and sort of take a rotation of being on call. So that if somebody gets hit, at least they've got somebody to help them, sort of coach and sort of, you know, psychologically kinda get through that and pointing them in the right direction or or or or or being a resource to them to know who to call, what to do, what not to do, things like that. Right?

Barinder: 01:07:34

Hello? I just went I I just went to the website. This is you got some good people, knowledgeable people from the community on this. I am. Thank you for starting this.

Robert: 01:07:43

Yeah. I am humbled to be, joined by some real, some extremely talented and caring people out there. These are the best of the best out there. Matt Lee, Jason Slagle, Chris Lair. I mean, I'm just naming a few.

Robert: 01:08:01

I mean, everybody whose names this, is really no joke. These are the best of the best out there.

Barinder: 01:08:08

So what's next for you, Robert?

Robert: 01:08:11

Yeah. That's a funny I

Barinder: 01:08:13

I hear I hear I hear, this was this was hard, but you're now tackling something even harder. Is there a book coming?

Robert: 01:08:22

I'm working on a book.

Barinder: 01:08:24

Okay.

Robert: 01:08:24

I'd really like to get that, completed before the end of the year. That's my promise to myself. It's difficult because they take a lot of personal time, including weekends to work on it. I'm at about 75,000 words, which is a lot of words. I probably have about another 10,000 to go before I'm all done.

Robert: 01:08:44

So it's not gonna be one of your, you know, typical little pamphlet books. It's a it's it's a story. Right? It's it's and it needs to be told, and I'm trying to write it so that this is a, it's a business story and it's a human story. It's not a it's not, you know, like an MSP story.

Robert: 01:09:04

Right?

Barinder: 01:09:05

Mhmm.

Robert: 01:09:05

I'm I'm hoping that what I can do is influence the business community, through awareness, and through some storytelling. Mhmm. And I don't know. Beyond that, you know, maybe there's something that can be done legislatively. I you know, laws aren't gonna prevent people.

Robert: 01:09:25

You know? You're not supposed to spit on the sidewalk or, you know, jaywalk, yet people still do it. Right? You're not supposed to rob banks that you know? I was just I passed, I passed the middle school that I went to, growing up.

Robert: 01:09:37

I was going to my mom's house the other day, and I kinda laugh because there's a sign on the telephone pole that says, you know, drug fee free school zone. And I'm like, is that really gonna keep out the I don't mean I don't mean to sound, you know, sarcastic here, but, like, that's my viewpoint. Like, law's a lot like, you know, they're they could be helpful. So I'm kinda hoping to see what more can be done through advocacy, and spreading the spreading the word. And like I said, if I can affect a few handful of people to just do better and be better, then mission accomplished.

Robert: 01:10:12

And if I can put together an organization that can help victims during their worst day, then then I've, you know, I've also affected the world in a positive way. I wanna make sure I leave this planet on very good terms with, humanity.

Justin: 01:10:28

Amazing. I love the organization, Robert. God. I just I I have had kind of visions or fantasies of my own to have some sort of a community wide MSP community wide set of standards. Yeah.

Justin: 01:10:42

That's kind of there there are enough resources out for that, but this, having having somebody you can call in in that moment is phenomenal because I'll tell you what. You know, I've I mentioned before that I've had literal nightmares about this situation. I've been through it on a very small scale, long time ago, but it haunts me. You know? And so I thanked you once.

Justin: 01:11:05

I'm thanking you again for being here. One of the most frustrating parts of, you know, our job is that we can talk about how to prevent these breaches, but we don't spend a lot of time on ground 0.

Robert: 01:11:17

And We don't.

Justin: 01:11:19

Yeah. You My my follow-up my go ahead. If you have something to say, because I'm kinda winding this up.

Barinder: 01:11:23

Please. Okay.

Robert: 01:11:24

Hit me with your follow-up. Yep.

Justin: 01:11:27

Every every week, week after week, as we go and we we break down different breaches that we read about in the news, and we, you know, we we throw our theories and our, you know, if you did this, then would that have happened? But we come down to and I have to give you a little bit of background. I used to work for Loomis Armor. Carried a gun, protected other people's money with my life. Terrible job.

Robert: 01:11:47

Should hire you.

Justin: 01:11:49

In in the gun training, one of the things they taught us is that you and you kind of just alluded to it. We're not going to stop crime. Yeah. We are not going to stop people from getting killed protecting other people's money, but what we can do is make it happen to somebody else. So the lesson that I try to learn every time I dissect a breach is how could we make this happen to somebody else?

Justin: 01:12:09

And, Robert, that's the hard question. Could you, knowing what you know today, could you have made it happen to somebody else, or can you advise MSPs on either better preparation, or, you know, better protections, better protocols? What is your key takeaway from having gone through this?

Robert: 01:12:29

So, and it wasn't an immediate key takeaway. Well, one one part of it was, and that's the community element. Right? That's the sort

Barinder: 01:12:38

of the

Robert: 01:12:39

I have 3 pillars that I talk about, and that's one of the pillars is that community is the missing element to our cybersecurity stacks. Usually, when I do my if I if I'm doing a keynote speak or a presentation at a at a conference, I'll usually say, hey. Look. You know, there's a lot of wonderful cyber, tools, you know, vendors out in the in the vendor hall in the pavilion. Yes.

Robert: 01:13:02

Go talk to them all. Figure things out. But but tools alone are not going to protect your business or your customers. You need to invest in the community. Right?

Robert: 01:13:13

You saw what the community did for me, and you saw now what I'm trying to put back into the community. Community is that missing layer that we don't really talk about. A good friend of mine, Wes Spencer, very well known in our space, when I first met him and told him about the story, it was actually about a month after it happened. He was flabbergasted. He comes from the banking industry where he said there is no way that my counterparts at other banks would have come in to help me, recover if we ever had a problem, unheard of in some other industries.

Robert: 01:13:48

But in the MSP space, everybody's like, in their hands, rubs them together, rolls up their sleeves, and says, how do I help? Right? Let's go get these guys.

Justin: 01:13:56

Yeah. Yeah.

Robert: 01:13:57

So we're just a strange bunch. I don't know.

Justin: 01:13:59

I mean, right now, I'm looking at that page saying, how do I sign up? How do I get involved?

Barinder: 01:14:03

How do

Justin: 01:14:03

I be a part of this? So Right. Exactly agree.

Mario: 01:14:05

The same way.

Robert: 01:14:06

I listen. I knew instinctually. Like, even if I, like, didn't have a good story to tell, like, instinct all I need to do is put it up there, and you'd be like, I don't know who this Robert is or what his story is. But, yeah, that's super interesting because it speaks to us. So the the other thing I would highly encourage everybody, in the technical space who's listening to this is you really need to be following some sort of framework.

Robert: 01:14:34

Mhmm. I figured that out within about 6 months after our attack. We were largely dependent upon vendors and their guidance about how to shape our security services and protection for our customers. I realized that I needed an an unbiased framework to help shape what that should look like. Now we landed on CIS controls.

Robert: 01:14:56

I'm not here to necessarily promote them. I believe in them. I really like what they do. And I've modeled our, practices to follow as closely as we can to that model. Now if you're a NIST fan or if you're working for mainly government clients and you gotta do something like CMMC or any of the other trillion acronyms that we're drowning in, well, then go do that.

Robert: 01:15:23

Right? But do it consistently and do it, unforgivingly. Like, you need to be persistent with it. Just do it. Follow a framework.

Robert: 01:15:33

Stop pulling solutions out of your own hat and thinking that you're the smartest cyber guy in the room because you are not.

Justin: 01:15:40

Well, like I said, we're not we're not battling amateurs here. We're not as much as like, you got to see that 22 year old kid, but that's not our enemy. No. I mean, we are fighting the smartest, the most brilliant people with resources, people with financial backing, with government backing. I mean, this is a huge, huge, huge problem.

Robert: 01:16:01

So Justin, have you heard the term dark PE? I heard that about 2 years ago. No. It's private equity. Right?

Robert: 01:16:07

Oh. We all know private equity. Right? Yeah. Right?

Robert: 01:16:10

This is dark private equity. It's

Justin: 01:16:12

Oh, shit.

Robert: 01:16:13

People with money who are like, hey. I wanna get in on this cybercrime gig. Let me go find some some gang some, some hacker gangs. Right? Threat actors.

Robert: 01:16:23

Well, they don't think they're threat actors. Let's go, find some of these smart, you know, business folks that are attacking other people, and let me, fund their operation. Right? Dark PE. Did you know that cybercrime combined, all the money that they collected, last year or the year before, I don't remember what the exact year was, was $8,000,000,000,000.

Robert: 01:16:45

And if you add that up as a GDP comparatively, the US, number 1 at 27 trillion, China at 17,000,000,000,000, 3rd largest GDP in the world would be cybercrime at 8,000,000,000,000. The 4th largest, I think, was Germany at 4.6 and then Japan at 4.2. So

Justin: 01:17:03

Holy shit.

Robert: 01:17:05

It is massive business. And if you think if you're listening to this and think that it's just a bunch of kids messing around out of their basements, think again. You are so hopelessly wrong.

Justin: 01:17:20

I I god. I'm and, like, I'm almost speechless. I will say that I I am I am thrilled to have had this conversation with you. It confirms what, you know, the Yes. This group here, what we talk about week after week, which is use the industry standards, best practices, a framework, whatever you wanna call it, but use published standards to protect.

Justin: 01:17:42

And I break it down into 3 buckets just for so we can bite it off and chew it easier. Protect your technology, protect your data, protect your people, wrap it up with a good cybersecurity insurance policy, have your policies and procedures in place, and and that's what we can do to fight this battle. What thoughts on that? Would you add anything to that?

Robert: 01:18:05

I mean, no. I mean, you've just you've said what I've been saying just maybe, you know, coming at it from a different, level, but, I guess the only thing I would add to that is, again, the emphasis on the community piece. And what I mean by that is go with an inquisitive mind. I used to be egotistical when I joined a peer group back in 2010. I thought my shit didn't stink and everybody else sucked because I'm this whippersnapper from New York, and we're the best.

Robert: 01:18:32

And and I'm learning that people in the middle of nowhere in Kansas were, like, way outshining me, so it had nothing to do with your geography, and it didn't even have anything to do with education. Mhmm. It just had a lot to do with some good common sense and, you know, some some really smart people out there. What I'm getting at is this, is I don't care how brilliant you are. There is something for you to learn.

Robert: 01:18:57

And Yes. Do that by checking your ego at the door, go to conferences, make friends, ask questions, share like hell, right, and you'll get back a hundredfold.

Mario: 01:19:10

I love that 100.

Justin: 01:19:11

Like I said, I'm I'm taking notes as fast as we can do this. If if and I know we're we're a little over time for what we normally spend here, but, Robert, if you have a few more minutes, what I'd like to do is just kind of open this up, to to each of our panelists here. Do you guys have a final question for Robert? Robert, if you're okay with that, we'll do that, and then we'll go ahead and wrap up. And this will conclude our cybersecurity, awareness month 4 part series.

Justin: 01:19:39

So Mario, Brenda, and then Brian, if you guys if you have a question, if you're not, that's fine. We can we can thank Robert for being here and move on. But if you have a question for him, this is your chance, and then we'll wrap. Mario, why don't you go ahead and go first?

Mario: 01:19:55

Yeah. So I my question is, what did what happened with Kistea? Like, what did they tell you? Did they, you know what was their I'm sure you've had a conversation with them after that. What what what did they do?

Robert: 01:20:10

Well, I mean, if you, if you were watching, you would have seen that they made some very public statements about, their desire to help out their partners. Let me put it this way, Mario. I don't think that we necessarily got the help from them that I was expecting, but I'm not here to disparage Kaseya in any kind of way. And not because of some legal contract because I have none with them, but by me, because I get this question a lot. Like and some people will and I know you didn't phrase it that way, but some people don't have a high opinion of them and try to ask me that question in that way to get me to kinda, like, bite on something.

Robert: 01:20:57

And even if I did feel those things, it takes away from the story. So did we get help from them? The short answer is no. Not much. I mean, the help that we got was from the the vendor and peer group community.

Robert: 01:21:12

Like I mentioned, that was 99.9% of the help. And we've, you know, we've we had to move on from Kaseya VSA because, politically, I had a lot of customers who were emailing me, some of the negative press about that. You know, they knew they knew about the like, yeah. Well, you know, Microsoft knows about a lot of 0 day exploits, but I don't see you firing them. So, but I I could not to say about Good point.

Justin: 01:21:43

Yeah. Alright, Brenda.

Barinder: 01:21:47

So many questions, but, something that might be a benefit to any listeners is just what, social platforms are you connected with? What's the best way for people to reach out to you if they have follow-up questions? I'm sure long

Robert: 01:21:58

as well. Yeah. And I'm very, I'm a pretty open person about our story, because like I said, it I think it improves the world by me sharing as much as I can. I am a LinkedIn denizen. I probably spend too much time on LinkedIn.

Robert: 01:22:15

That's the absolute best way to find me. It's linkedin.comforward/inforward/r choffee. That's r c I o, double f as in fox, I. Absolute best way to connect with me. If you decide to send me a note or a connection request, please reference the podcast because you'd be shocked at how many, connection requests I get, and I don't know why people are reaching out to me.

Robert: 01:22:43

So some context, please, before you send me an invite. But happy to connect and continue the conversation there.

Justin: 01:22:50

That's a great question, Brenda.

Barinder: 01:22:52

They all they all they all wanna sell you something.

Robert: 01:22:56

Yeah.

Bryan: 01:22:56

Some of some

Robert: 01:22:57

of them listen. And some of them turn into that, and it's like, I'll just politely, you know, kinda just turn the conversation the other way. Look. I'll answer

Justin: 01:23:04

questions for

Robert: 01:23:04

you, but,

Barinder: 01:23:04

like, I

Robert: 01:23:04

I'm, yeah, I'm I'm full. I I forgot to mention this earlier, Robert. Have you heard about

Barinder: 01:23:09

my new AI software? Oh, yeah.

Robert: 01:23:16

You, it's a good thing that we're remote from each other because as soon as we start talking about AI powered something, I'm like, come over here.

Justin: 01:23:27

Brian, I think you're our you're our last one. Go ahead.

Robert: 01:23:29

Cut the marketing, speak out, please. Let's get that

Justin: 01:23:32

blast back. Right? Yeah. So I

Bryan: 01:23:36

don't have a question, but I did wanna say, you know, Robert, I've I've I've met you at conferences. I've I've watched you present on stage about various topics, and I just wanted to thank you for taking the time and effort to come out today and and chat with us and and the rest of our our, our potential clients and prospects and just and and what you do for the community in general. Thank you very much for

Robert: 01:23:55

I appreciate that, Brian. It's, comments like that that just give me the fuel and fire to keep going. Because first first couple of times I did it, I wasn't really sure, like, am I impacting people? Do they like to hear this stuff? I I mean, I'm not saying I need to hear this for my ego, although that helps a little bit.

Robert: 01:24:14

But my point is this is if I get no feedback, then I don't know if the things I'm playing are resonating. Because I could do this. Right? We're recording this podcast. And if I get 0 people reach out to me, I I just don't know.

Robert: 01:24:29

Did anybody listen? Did they think I was a buffoon? Were they motivated, inspired? I have no idea. But if you think I'm a buffoon, then please keep your comments to yourself.

Robert: 01:24:39

And if you're inspired, or at least humor it slightly, a kind little note, will brighten my day, so I would appreciate it.

Justin: 01:24:50

Alright. Thank god. A couple thoughts to wrap up, Robert. I we've been, you know, 34 episodes. I'm not saying we're Joe Rogan or anything, but, you know, we've been doing this for a little while.

Justin: 01:25:01

Uh-huh. And there's this episode and there's one other episode that have really impacted the way I look at what we do here, both on the podcast and what I do in my business, and maybe even what I do in my personal life. So thank you again sincerely for being here. If there is one change we could make in in community, as you say, in in maybe just the business world at large when we're talking about cybercrime, if we could in fact label the victims of this crime as victims instead of as perpetrators, that would be phenomenal. Yep.

Justin: 01:25:34

Unfortunately, that's not the case. The ones who get hit with these attacks are the ones who get dragged through the mud, and that is tragic.

Bryan: 01:25:41

We need

Robert: 01:25:42

to have more empathy for victims and remember, you know, their you know, it it even if it is their fault to an extent. Right? Like, I oh, I left I left my keys in the car, and the car unlocked overnight, you know, but still somebody stole the car. Right. But, like, you know, stuff happens.

Robert: 01:25:59

Like, if you think you've got a perfect environment, you know, give me give me 24 hours, and I'll prove you wrong.

Justin: 01:26:05

Yeah. Absolutely. So So anyways, yes. Thank you. You have my full support.

Justin: 01:26:12

I imagine I'm speaking for everybody here on on what you're doing moving forward. I will read your book when it comes out even though it's 330 pages based on my quick Google math, and I have ADHD. So that will be I'm I'm telling you, if I tell you I'm gonna read your book, that is a huge thing because I don't read a lot of us.

Robert: 01:26:27

But Well,

Justin: 01:26:28

may may

Robert: 01:26:28

may maybe I'll read it to you.

Justin: 01:26:30

Do audio. Please do an audible version. Please

Mario: 01:26:33

do audio. That's

Justin: 01:26:35

it's anyways, everybody, thank you for being here today. Robert, again, can't thank you enough. And we're gonna go ahead and wrap up. Guys, the 4th in our series for October, the cybersecurity awareness month on the formula for protecting your business. So thank you all for for being here, and we will see you next week.

Justin: 01:26:54

Take care.

Mario: 01:26:55

Thank you very much, sir. I appreciate it. Thank

Robert: 01:26:57

you. Alright. Peace.