Site icon Easy Prey Podcast

The Update That Broke America with Gabe Dimeglio

Easy Prey
“Any time you introduce change, you are also introducing risk.” - Gabe Dimeglio Share on X

Many industries are reliant on software and if the software becomes corrupt or an update fails, it may require hands-on support. Do you have your infrastructure set for repair and recovery?

Today’s guest is Gabe Dimeglio. Gabe is a 20-year veteran of information technology and security for private and public sector organizations. He is a results-driven leader, specializing in security services and solutions for mission-critical, complex enterprise platforms. His expertise includes strategic consulting services, risk analysis/risk mitigation, and compliance. 

Mr. Dimeglio serves as Vice President & Executive Advisor, Security, Office of the CTO at Rimini Street. He is responsible for oversight of the GSS organization that provides tailored consulting and advisory security services to prospects and clients, in collaboration with Rimini Street sales, client engagement, and retention functions.

“I think that there’s going to be a rapid rise in people being compromised by the use of new technologies.” - Gabe Dimeglio Share on X

Show Notes:

“We have to get it right every single time. An attacker only has to once.” - Gabe Dimeglio Share on X

Thanks for joining us on Easy Prey. Be sure to subscribe to our podcast on iTunes and leave a nice review. 

Links and Resources:

Transcript:

Gabe, thank you so much for coming on the Easy Prey Podcast today.

Thank you for having me.

Can you tell myself and the audience a little bit about your background, who you are, and what you do?

Sure. Gabe Dimeglio, GVP and GM of what we call Rimini Protect at Rimini Street. Bottom line is that I have, over the years, built out a global practice of essentially cybersecurity solution and compliance solution providers, services, binaries. We do full implementations, managed services of security solutions, but they're really specifically tailored primarily to enterprise software, so ERPs, CRMs, databases, web services, and all that fun stuff.

How many years? Or do you not want to say how many decades?

Yeah, definitely. I'll put it this way. I've been with Rimini Street for 14 years in November, so long enough.

You're an outlier. You've been at your job for more than two years.

I know.

Maybe we'll talk about that in a little bit, not about the 14 years, but the two-year thing. Before we start talking and dive into our topic today, the audience knows that I really want to try to de-stigmatize people who've been a victim of cybersecurity fraud or criminal activity, help them to know that they're not alone, and they don't need to be ashamed. I always ask my guests, have you been a victim of a cybersecurity incident, a scam, or a fraud?

Yes, and of course more than once. Let's start with the basics here. Back when cell phones were not particularly popular out there in the market yet, it was really just technologists walking around with them, so to speak; that was my first experience. I fortunately was never one that fell for the prince from Nigeria or anything along those lines, but I did fall for a phishing campaign many years ago, actually. It was our own, though, fortunately, when we implemented a specific solution here at Rimini Street. That was probably 12 years ago.

The phishing methodologies that were being used by those malicious actors were not particularly advanced really at that point, certainly nothing like today, but the solution that we implemented actually did really have some foresight. You get an email from the CFO that says, “Hey, I need you to go look at this thing and find out what's going on.” You click on it, and boom, it looked like it was from him. It wasn't until I realized later that, “Oh, there's my little notification. Dink. You failed.”

It's amazing how much the phishing and the scams have evolved from the Nigerian fax days or the letters in the mail days. It is so much more sophisticated now.

Generative AI is a problem, quite frankly. I think that there's going to be a rapid rise in people who are compromised from the use of some of those technologies until people really figure out a great way to train and educate the… Share on X

It absolutely is. It's actually pretty terrifying in some ways looking at, I believe, the direction that this is headed with some of these recent technologies, let’s just be honest. I don't necessarily want to go too deep on AI. To me, people are talking about it all the time, but generative AI is a problem, quite frankly. I think that there's going to be a rapid rise in people who are compromised from the use of some of those technologies until people really figure out a great way to train and educate the masses on them.

It'll be a lifelong cat-and-mouse game.

It will be. Yes, unfortunately.

Let's jump in. We're recording this episode in August. Not too long ago, there was a worldwide critical infrastructure patch-related disaster. Let's talk about that. Can you give an overview of what happened?

Crowdstrike issue, yeah. I was in California when that happened. I live in North Carolina. You can imagine, I was a little concerned about catching my flight the next day.

What actually happened? The problem really came down to some poor coding. Somehow there was effectively a signature update. If you think of an update of any antivirus rules, historically, that basically contained a bunch of zeros. There was a problem with no pointer reference, not to get too technical. Bottom line is the way that solution is actually embedded into Windows itself, the agent is effectively a kernel driver.

In the Windows space of the world, and this is why we think about it in general, but from Windows’ perspective, you got kernel mode and user mode with Microsoft products to have a great program around certification of anything that's going to interact at that kernel level.

Most of the time, the vendors who develop drivers that would interact at that level—naturally like your video card driver or something along those lines—go through these extensive certification testing processes, and the updates are not particularly frequent. But every time that they provide an update, they go through that same testing. There's a certificate, effectively, that's issued for that kernel mode interaction, that driver.

In the way that CrowdStrike works, they have the same thing. They're fully embedded into the kernel, but the software updates themselves, actually, the sys files inject into those running processes in the kernel.

The idea of having kernel mode versus user mode is that the user mode gives you the space for failure. If there's going to be a code-related crash, a logic problem—whatever it may be—the user space can throw up, and it splashes a nice little box. It says, “Hey, this thing just crashed; go restart it.”

When you have a kernel mode driver that does that—the blue screen of death as we've called it for years, affectionately—is the best option that is out there. It basically frees everything now. The reason being is that the risk of corruption of the rest of the data associated with those processes is too high in order to attempt to just restart the function.

By the way, this affects Linux and every other platform out there, too. The approach that they take is the system halts right now before there's the potential for damage. Basically, their approach is injecting in the kernel. By the way, make no mistake, it gives them great visibility into everything that the system does, into processes and activities that otherwise you may not have such visibility and control over.

To further complicate things, there's a term. It's the boot mode. There's a bootload code, boot mode driver. Effectively, you can't restart the platform as the full Windows operating system with that thing effectively disabled unless you go into safe mode, where there's the absolute minimum set of drivers. That's what happened.

One of those updates was effectively faulty.

Yeah. In fact, it's interesting because the source code's out there on Twitter. It's been analyzed and assessed by a ton of different developers out there. Probably the best explanation and example of it I saw was a gentleman who's retired from Microsoft, who spent a lot of his career debugging blue screens of death.

Again, there's been a thousand people commenting on code. This is interesting with those code reviews. What people have found was that, “Oh, this thing's been broken for a long time. It just took this one particular straw that broke the camel's back to actually cause this event to transpire.”

Just to put it in perspective, over the course of a year, how many signature updates are there for CrowdStrike?

Several a day, sometimes. That's interesting. Think about this for a second. If you look at how CrowdStrike works, they're basically side-loading these updates continually that are then processed by this kernel driver. It doesn't change that often because that does require going through that certification process or the testing and evaluation.

The idea is if they could side load these updates into this thing, then it's the fastest way to get the most comprehensive protection. That is the thought process. This is interesting because many moons ago, the CEO of CrowdStrike was the CTO at McAfee. They had a very similar problem back in 2012–2013, something along those lines, but the interesting thing is that McAfee learned from that and they changed their approach.

If you look at how Trellix does it today, they actually have also injected into the kernel, but all of their updates are provisioned and run in the user space. They can provide updates multiple times a day. There's no need to go through a certification process, but those updates effectively aren't truly interacting at the kernel level the way the CrowdStrike updates do.

The way that they actually roll out changes is very different from the CrowdStrike approach. Again, they learned, “Hey, if we roll it through North America first, out of testing, out of the QA lab when we release this into production, they tend to do it in waves around the world.” That also gives them the opportunity to identify any early indicators of a problem or compromise. It's different approaches for different organizations.

Is this the first time that CrowdStrike has had an issue that has taken out so many systems?

So many systems? Yeah, this is a major problem.

I'm just trying to think through the math here. How long have they been around pushing updates like this? Because we're a thousand a year, let's say, can we get it for 10 years?

CrowdStrike, I remember when they were a young company many years ago, and they were doing stuff that was pretty revolutionary and they exploded, and they have captured a massive portion of market share. It's interesting because I have a client right now that actually has been in recovery phases from a ransomware attack on some phishing campaign. Nothing we support, but it doesn't matter. Things start laterally moving through an organization, and they're running Linux.

There are a lot of options around endpoint detection and response solutions or anti-malware solutions in the market for Windows-based platforms. There aren't many for Linux. A lot of people like to say things like, “Well, that's because Linux isn't susceptible.” Wrong, that's not true. There are things that affect these platforms as well as I just saw, for example. I would have historically said, “Hey, you need to look at CrowdStrike.” They could utilize their capabilities there, and the answer would have been, “Great idea.”

My response to them when they're saying, “Hey, we need help finding an EDR platform so that we can scan these systems before we bring some of these back online.” I said, “Believe it or not, I know you've had this conversation, but CrowdStrike is a great solution. They can't run some of the other options for various reasons.” The response was, “We're not interested in CrowdStrike right now.” The interesting thing we're seeing is that across the board, there's a lot of reputational damage there.

I guess my reason asking the math questions was this is one mistake out of 10,000 updates. That's a really low error rate.

Yeah, their track record's incredible. Frankly, most of these security companies have unbelievable track records. In fact, one of the leaders on my team tells everyone under his arm of the org, and he'll share it with anyone in our team. It's like, “Look, we have to get it right every single time. An attacker has to be right once,” and it's true.

We remind the team on a pretty regular basis just because it's like, “Hey, stay pumped up. Stay motivated. We're doing some of the most important work out there. We understand the value of it, and our clients understand the value of it.” It's good. It helps to keep people motivated to remember it.

Is the tradeoff with these platforms the speed at which the protection is put out versus the reliability or the potential downtime impact?

Yeah, that's interesting. The reason they do this side-loading, as I mentioned before, is because they can roll out these updates very quickly and effectively process them as comprehensively as possible across the system. Again, despite the track record, that is a high risk to automatically apply those updates based on any vendor just go apply this update thing. There's testing that needs to be completed.

In the context of CrowdStrike, it's interesting because you can't just really disable auto updates like it's a part of the feature. In order to do that, you actually have to manipulate the uninstall and maintenance service. There are other ramifications that come with that. Most people just have it on autoupdate, obviously.

What do people with critical infrastructure do? We know the people that are trying to take advantage of system issues. We don't trust this major platform provider. The answer can't be, “Well, we're just going to leave our systems more vulnerable.”

The answer is test anytime you're going to introduce a change. Here's the thing that's interesting. I've learned this through our core business at Rimini Street, which is most of this software, especially the stuff that we support, big enterprise platforms, they've been around for 30-plus years, and they are very reliable. What you might think is interesting, a lot of people say that's legacy software, whatnot, whatever. People build integrations to them all the time. ERPs are specifically portable for that. You have that capability, but this is reliable stuff that's proven.

Anytime you introduce change, that's where you introduce risk. Really, if you think about it in general, the ones and zeros, the binaries that are running will continue to operate the way they're expected to. They don't wear out. What causes problems? You've either introduced a change of some type, or you've changed the data either way that you've now introduced a state that it was not tested against or caused some kind of an issue.

What causes problems? You've either introduced a change of some type, or you've changed the data either way that you've now introduced a state that it was not tested against or caused some kind of an issue. -Gabe Dimeglio Share on X

We see everything under the sun, as you can imagine, in that regard. If you change the data, that's one piece. Frankly, very rarely changing the data of a system does anything. It's generally going to be around updating how the platform works, whatever it may be.

Was one of the difficulties with CrowdStrike with this particular crash and the blue screen of death is that it required hands-on every single box?

Exactly. This is an interesting thing. This issue requires you to boot into safe mode, remove the sys files that are associated with that update 291—whatever it was—out of the sys32 drivers folder. You have to boot into safe mode to do that. When it restarts, you're golden, and it's just because of that one update. Once you remove that update that's side-loaded into the driver, you're good.

But in general, not everyone's running KVM technologies. I'm not talking about kernel-based virtual machines. I'm talking about the KVMs with actual wires and plugs that all of your hardware is plugged into, and you can SSH into a keyboard effectively and monitor that's plugged into that.

That is a tool that we use and other organizations use all the time that enables you to make changes from the boot sequence up on a box and manage that remotely very efficiently.

But a lot of organizations that are running these Windows servers that were specifically affected by this don't necessarily have that, so it took a lot of them a long time to get people to drive to different facilities, airports, whatever, and get hands on the keyboard on those boxes to just roll through the change. It's a logistical nightmare, frankly.

I think if even the co-location where I have my servers, they do hands-on support and that's part of the contract. But I know there are two guys in the NOC center, and there are 50,000 servers in that building. Even if the NOC is like, “Hey, we'll go out and do it for you, but you're number 3000 in line. We'll get to it in two weeks.”

Exactly, and it's brutal. I'm going to tell you seriously, I remember one time I was on a dark fiber project that was doing encryption stuff. I had to go to two IBM data centers—one was in New York, and one was Connecticut or something. Both of them had one person there, and it was the security guard. That was it; nobody else there.

I'm like, these are big buildings, all the offices are vacant. You're just like, “Dude, literally you have to come to this and do everything yourself to this facility. What happens if your connection goes down? What happens if power problems occur and you have to go swap out equipment? There's no one to do it for you. Just go figure it out.”

I wonder, was the airline industry more affected by this because now they can't get their people to the data centers to reboot the machines, because the flights are delayed?

It's amazing how these things compound on top of each other. When you really think about it, there are these dependencies. Were those accounted for in the risk assessment that was done when you actually selected these controls? Probably not.

No one thought that all the airlines are going to be down, so our people can't get to the data centers to restart the boxes that we need to fly the people out to.

Exactly. It's funny because that being said, hey, look, man. I, myself, and several of my colleagues here are ministry or pilots. Make no mistake, we saw some of our pilot friends out there flying people around for those exact purposes and little bitty airplanes because if you've got to basically go anywhere up and down the East Coast, you can do that in a day. Crossing the country takes longer, but there was a whole lot of shuttling going on.

It's really interesting. We're joking about this with the airline sector. How many other sectors are reliant on the same thing? If there was an update that failed in a different transportation sector, it would have the same impact. Who knows, if we're not supply line specialists, how many other industries where that same thing could happen, where it goes sideways? It affects everybody.

Exactly. That's the thing. I do like the fact that, in my role and with our company, we're pretty much vertical agnostic. We have clients in every industry. I'll put it this way: Our clients are running enterprise software. Great, that's our vertical. But even with that, I do have such limited exposure to the day-to-day operations sometimes of these IT centers and these services.

You think about the energy sector. You think about gas and pipeline distribution, electrical distribution. They have similar challenges. I have clients that have databases that are on oil tankers for crying out loud. If that thing goes down, they have to fly someone out on a helicopter to meet the tanker. Come on.

If you can, without disclosing clients and industry things, the energy verticals and whatnot utilities, are more of their stuff air gapped?

Yeah. Anything that's critical infrastructure, ideally, is going to have to be segmented off on its own environment. The way you think about it is OT (operational technologies). All that stuff is generally going to be a separate network environment, where it goes through a clean environment, if you will, for testing prior to introduction into the OT environment.

That's ideal. That is the way we think of things. We want to protect them because in those environments, similar to manufacturing environments, there's a lot of old tech as far as PLCs, SCADA devices, and all this kind of stuff.

For example, if you're running a Windows system that does actually control a SCADA device or some series of controllers, and it gets a CrowdStrike update and all of a sudden the machine tanks because of that, now you're no longer controlling valves. You're not measuring anything around those processes anymore. Those are the areas where it's critical.

Even, quite frankly, let's be honest. If you think about our military industrial complex, especially around nuclear weapons. Those systems, the circuits, they're all dedicated, isolated, completely segmented ecosystems of technology, we’ll say.

I don't think we're doing this on the nuclear side, but what does terrify me around manufacturing and even critical infrastructure is more and more people are effectively selling their solutions. The solutions are cloud-managed. That's dumbfounding to me.

I don't understand why people are buying into that because we see cloud breaches all the time. Failures of vendors to properly configure and harden the controls that they are responsible for hardening, putting critical infrastructure up against that where you really do have minimal visibility into those controls, the efficacy, and everything from personnel management, you name it, that keeps me up at night thinking about the ramifications there.

We already have enough challenges. Quite frankly, we are well aware that third parties will say nation state-sponsored programs have basically compromised every American company that exists. It's like the old FBI director said. He said that every single company out there has been compromised, and for those that haven't, they just don't know they've been compromised. That's the reality, and that's where we should be baselining those kinds of critical assets from a protection perspective.

It's like the old FBI director said. He said that every single company out there has been compromised, and for those that haven't, they just don't know they've been compromised. -Gabe Dimeglio Share on X

Work from the assumption that things will get compromised. What happens if these things get compromised? How are you going to manage that as opposed to if they get compromised?

It's when, not if. That's the bottom line.

I think I have a couple of guests, and I have talked about that. I've said, “Have you been a victim of a cybersecurity incident?” One of the persons was, “Not knowingly. I'm not aware of one, but that doesn't mean that it hasn't happened.” In likelihood, he was like, “Yeah, it probably has happened that I just wasn't aware of it.”

Yeah, I love that. It's the same thing. We have thousands of clients. I tell people here and there, I'm like, it's important to distinguish that our track record, we never had a client that's been compromised while under our support and security. I’ve got to retract that and say, “OK, not the things that we were supporting.” Yes, those things do happen.

Quite frankly, organizations that are very immature and in segmentation, they always have it the worst. No mechanism of managing a blast radius because they're just this big L2 flat broadcast network. It's like, “Dude, you don't know what a nightmare it's going to be when it happens. You need to go ahead and start figuring out how to segment these environments effectively.” You see it here and there and try to tell people, but you can only lead a horse to water, as they say.

What do you tell people who run small businesses or medium-sized businesses, where they may not even have a dedicated security team?

The amount of companies that have one dude who just wears 20 hats—he’s the server guy, he's the VoIP guy, and security, it depends on the business they're in, what they're doing, and what data is it that they have that's important.

For anyone that small that goes, “Hey, dude, you've been in this space forever; can you give me some advice here?” Understand what it is that you contain that's of value to anyone, yourself included. If you're just one person and a dog in a shop, you're making widgets, and all you're doing is taking orders and payments over Venmo here and there, maybe it's not that big of a deal.

It doesn't really matter for you unless you are using a CAD system, and you've got drawings that are there. It's about the thing that you invented, where you're going to eventually get the funding to file a patent for. Someone steals those CAD drawings and starts making them in China tomorrow. Those dreams are gone. You have to figure out what's important to you and everyone else, and then you can start figuring out what you're going to do around protecting those things.

In general, when you start getting into these bigger shops, as they start to grow and evolve, everything is just subscription-based. Everything out there is subscription. You are dependent on these vendors. Most people don't have the money to put in server rooms and all this nonsense. I say nonsense because it's a lot to manage, you're going to have people.

Let's be honest, the skills gap that's out there right now is crushing most businesses, much less these smaller businesses. They can't compete financially. They can't afford the skilled workforce because they're getting paid top dollar by financial firms and basically big companies with big budgets.

Is this a good field for people to go into?

I cannot think of a better field in the world. If you are in fact interested in and intrigued with any of the various domains that exist within this space, that's the nice thing about it. If you like tech at all, it's broad and wide as far as the different things that you can learn and work on.

Quite frankly, if you come from more of a business background, it's still broad and wide on the compliance, policy management, standards, and auditing side. There is something for everyone for sure, and that's not going to change. As far as I can read into the future, the dealies ever.

It's going to be a constant arms race, at least, for the foreseeable future.

It is. Not to say anything inappropriate, but I remember being at DEFCON many years back. It was the funniest thing. Some dudes were walking around with a shirt that says, “Security: the world's second oldest profession.” It's true in one regard, but it's hilarious.

This was at a time where we were starting to see maturity in security programs, finally, with some form of consistency. Let's be honest, 20 years ago, that just wasn't a thing.

At best, security was an afterthought.

Yeah. There was no way you were going to get in and have a seat at the table at the C-level. It was challenging to say the least.

Are you finding that it is easier now in working with your clients? Cybersecurity does have a seat at the C-level table now?

I can't think of a client that I've worked with in the last several years where cyber was not one of the highest priorities to the executives, including the board. -Gabe Dimeglio Share on X

I can't think of a client that I've worked with in the last several years where cyber was not one of the highest priorities to the executives, including the board. Gardner made a prediction. Anyway, it was 2023 cyber predictions, I think, around compliance.

Basically, one of the predictions was that somewhere shortly after 2025, a certain percentage of every company in the world that at least is big enough to have a board, the board will have responsibilities around cyber and risk as a part of their role in leading the growth of that company.

I'm seeing that nonstop. I can't tell you how many times a week I hear it. “We've talked to the board about this and we've made this. The board's approved X, Y, Z,” and so on. That is a thing that I just didn't envision happening 20-some years ago.

Is it easier from your perspective for these companies now, assuming they've got the profit margin, but they have a reasonable budget for cyber now?

I would say that in general, security for the last couple of years has been the thing that, no matter what you can get budgeted for, if there's a real problem that needs to be solved, it's quantifiable in the way that business leaders think and understand. You can't just go and say, “Oh, man. Geez. We don't have this technology and someone's going to hack us.” It doesn't necessarily mean that much.

Make no mistake, they read the news too, and they aren't concerned about these various hacks. But if you go to those leaders and explain that, “OK, based on our risk profile, as it sits right now, we have this probability that this thing could occur, and here's the impact of the business if it does, and for this investment of X dollars, I can actually lower the probability and reduce the blast radius or the impact if it were to happen.”

You just keep showing, “Here are your options. Go ahead. Here's your sushi menu. Pick an option. What would you like? But here's the results every time you pick one of those things, and these are measurable.” Then they understand it.

People were not having conversations from that perspective long ago. They are today. As a result, as long as you do that, as long as you're having a business-level conversation, you can get funding for basically anything you need because there is nothing more expensive than shutting down the business.

Ask Delta.

By the way, I want to give them props because, again, as I mentioned, I was in California and I had to get home. I have to say those crews were amazing. They were literally calling people on cell phones, bridging calls on cell phones, trying to figure out who's running the flight crew on a piece of paper. The fact that they got as many flights out as they did in the timeframe, like seriously the next day, when I was heading home—I was only delayed for like three hours. I just couldn't believe it; they were incredible. I’ve got to give them props.

When you said bridging cell phones, I'll age myself for the audience. I thought back in those days that if you had two phone lines in your house, you'd take the two phones, turn the speaker to mic and hold them together, and let the two people talk to each other. “Are you guys done talking yet?” “Nope.” “OK.”

Exactly.

Anybody under 40 has no idea what I'm talking about.

Yeah. Let's think about this. It's like the concept of the old party lines too. Basically, almost no one under 50 understands or knows, at least if you didn't grow up in a rural environment, you have no clue what a party line was.

As we wrap up here, any parting thoughts or advice for the audience? Or, “Hey, I forgot to say this, and I really want to say this”?

This is a great conversation. I enjoyed it quite a bit. I'll say this: Change control is everything. We constantly are enhancing technologies and capabilities. Digital transformation initiatives are just everywhere. If you're not enhancing and creating new apps and widgets, you're sliding down the market share. Your competition is out there roasting you.

People are trying to understand how to balance these investments in cyber versus enhancements to the business, new developments, social, and everything else, but don't ever let someone convince you to shortcut, fully understanding what the technologies are, the implications of those technologies, and what they can have if you were to make a change without really thoroughly assessing what is the impact of the business potentially with this change if everything goes wrong. What's your rollback play?

This is all basic stuff from a change-management perspective. But the fact, to me, that organizations allow anything like we saw with CrowdStrike, to just make these changes into a system that are not tested or vetted, and we see this with antivirus software every day.

I don't understand that. I do understand the need to get things rolled out expeditiously. But quite frankly, I know of organizations and businesses that have labs where those changes are pushed right to them. As soon as they can validate, functionality is good, and then they have an approved policy, that update file is pushed, and then everybody gets it. But you've got to figure out how to actually control, manage, and really validate those changes appropriately. Otherwise, you're going to end up in this exact scenario and have a lot of problems.

Gabe, if people want to find you online, where can they find you?

I'm on LinkedIn. Generally, I don't do a lot of other social stuff unless you're into hunting groups, motorcycles, airplanes, and that kind of stuff. In general, just find me on LinkedIn with Rimini Street. That's about it.

Awesome. Gabe, thank you so much for coming on the podcast today.

Thank you. I quite enjoyed it. It's nice to be here.

 

 

Exit mobile version