35 Years in Cybersecurity: Randy Marchany’s Biggest Lessons

In this episode of the Cybersecurity Mentors Podcast, we sit down with Randy Marchany, longtime CISO at Virginia Tech University, to break down the biggest lessons he’s learned from 35 years in cybersecurity and over 50 years in IT.

From building one of the earliest hands-on cybersecurity training labs to leading incident response and mentoring the next generation of professionals, Randy shares what actually matters if you want to succeed in this field.

This conversation goes far beyond a single story—it’s about mindset, experience, and what separates good cybersecurity professionals from great ones.

Come hang out with us in the Cybersecurity Mentors Skool community. It’s free to join.

Transcript

Cold Open And Big Quote

Randy 0:00

And then uh a week or two later I get another email from from Sands, and this time they're asking for uh to call for proposals. And and they said that if your proposal for a talk gets accepted, they would waive the registration fee. So Ron and I we looked at each other and we said, Well, we don't know shit about about computer security, you know, but golly, you know, and then Ron said, Well, I guess we could talk about what happened to us.

Speaker 3 0:31

Could you teach me?

Speaker 0:34

Then learn fly.

Speaker 3 0:39

I know what you're trying to do.

Randy 0:43

I'm trying to free your mind, Nia. But I can only show you the door. You're the one that has to walk through it. What is the most inspiring thing I ever said to you?

Speaker 0:54

Don't be an idiot. Change my life.

John 1:04

All right, welcome back to another episode of the Cybersecurity Mentors Podcast. And on today's episode, we have Randy Marchaney, a longtime security practitioner and CISO for Virginia Tech University. And um, and Randy and I have connected different ways over the years and just stayed in touch with questions that I would have. It's like, hey Randy, what about this? What about that? Because he's usually been there and done that, um, which is great. And Randy, you want to introduce yourself?

Randy 1:36

Sure. Um, thanks for having me. My name is Randy Marchaney. I'm the CISO here at Virginia Tech in Blacksburg, Virginia. Um, I've been the CISO for uh 15 years now. Um and uh this summer uh in July uh I'll be retiring from this role as the CISO. Um I'm also the director of the Virginia Tech IT Security Lab, which is uh um affiliated with the security office here at Virginia Tech. So I I keep my uh my uh hand in the research side of the house with uh graduate students and undergraduate students, um, creating something like a what we call a teaching hospital. Um I've been in the in the IT world uh here at Virginia Tech. This is my 50th year uh of uh long, long history of IT here at the university. Um been doing cybersecurity stuff for about 35 of those uh 50 uh years. Um I'm uh I found the SANS Institute uh uh in its infancy. Um I'm in SANS instructor number two. Um and uh currently now I guess I'm the the longest running uh SANS instructor. Hal Pomerance was instructor number one, but uh he retired from SANS a couple years ago. So um, you know, uh if you can't if you can't beat him on merit, you just beat him on longevity. Um but it's it's been a lot of fun um uh here. Um uh, you know, and uh as you'll see, uh uh helped I had a hand in in a a lot of things. Um Virginia Tech and myself, we're charter members of the Center for Internet Security. And uh there were a number of other uh projects. Um uh the U.S. Cyber Challenge, which was uh founded uh by Alan Powler from SANS, to um um uh basically set up uh what I call a sports uh tournament type format where um we ask people to participate in the CTF, and the top scorers of those CTFs get invited to uh a week-long camp that that we set up with instructors all over the place. Other things uh in Virginia we have a consortium of the public universities, uh the 15 of us, plus the Virginia Computer uh Community College System, and I'm one of the founders of that. Um then from the educational side of the house, um I was part of the team that that uh um created the uh Virginia Cyber Range and uh and its twin, the U.S. Cyber Range, and that's a platform that allows K-12 community college and um higher eds in Virginia to actually have hands-on cybersecurity classes, and you don't get screened at by your local IT people for having hackers on your network. And then uh uh on on the side notes, uh I was telling uh you, John, that I was a grad assistant volleyball coach. I used to play a lot of club volleyball in my younger years, and then um I'm a musician. I play uh I played in a band for uh uh 38 years, and we put out a bunch of albums, bunch won a bunch of awards, and still play music here in this area. Um Blacksburg is pretty rich in in uh traditional music events around here, so it's been a lot of fun.

Building A Cyber Teaching Hospital

John 4:56

Yeah. Man, there's a lot there, which is which is great. Um I think the first note I wrote down was the the the lab, the teaching hospital piece of that. Could you talk about that a little bit? I I really I really like that idea, and and we do something not like a teaching hospital, but with our SOC, right, with the students. But yeah, could you talk about how that Virginia Tick uh teaching lab is set up?

Randy 5:21

Sure. So um in uh in 2002, 2003, my CIO uh came over and asked if if we could build a lab uh where we could uh get uh teach students hands-on, uh give them some hands-on experience in cybersecurity. And so um my wife is a is a physician, and so um I know when she went through the residency and and that type of stuff, it was you know, that concept of a teaching hospital is kind of what what kind of popped in my head when he started talking about it. And so uh uh in that era, we we started we started recruiting, we started recruiting uh um uh you know uh undergrad students uh to to uh come in and um do an independent study. So the the hook was to get them, give them some course credit, and and you know, that way they could they could they could uh we could justify uh them working for us because we didn't have a budget at that time for wage and and things like that. And so over the years, as as the security office, I was able to add more full-time staff, um, um, we were able to do the same thing with students. We were we were very fortunate. Um the graduate school here at the university uh allots uh uh a certain number of of uh graduate research assistants, GRA slots, to various departments. And um when I was looking for uh you know grad students and I was like, well, how am I gonna fund them? Uh it turned out that we had like these four GRA slots that were unused, that nobody in the in our division was had claimed them. So I said, well, I'll take them. And um, and then that that really uh it was what got us on the on the the board, so to speak, with uh with the on the graduate side of the house. Um we were very lucky uh also that um the Army has has a program called Advanced Civil School where officers um can uh uh they apply to the Army to to be in the program, and if they get accepted, uh then basically the Army says um you know apply wherever you want to apply to, and if you get accepted, uh that becomes your next post. And so um, you know, you're if you're working on a master's degree, uh you'll be posted at that university for two years, and if you're w working on a PhD, you'll be posted there for three years. Well, a lot of other schools didn't want to do the PhD thing because you know, we have to have PhD people for five years or more, right? And and uh uh Dr. Joe Trant, who was in the computer engineering department, um, he was my advisor when I was in grad school. Um you know, he he and I went to the grad school and we said, can people do, you know, is it okay in the grad school if somebody does a PhD in three years? And they kind of laughed at us, like, you know, they they said, well, you know, we've we've had uh, you know, one or two people, uh former president of of uh of the university who was, you know, at the genius level, he did his PhD in two years, you know. And they said, as long as he meets the requirements, which was you know, published papers and and the and the coursework, he said, as long as they meet the requirements, we don't care. Um, and so with that, you know, we we were able to get uh a number of army officers uh uh come through the the lab. So uh history-wise, um again, the stud the both the army guys and and again this is in the you know mid-2000s, the 2000 to 2000, you know, uh uh 17, 18 range. Uh so some of the early army officers, um, you know, they transitioned to uh to Cyber Command as it was just starting up then. Um one of one of my favorite little stories about that is is uh we had um probably graduated um uh eight or nine army officers in the PhD program. And after they finished at Virginia Tech, they went to uh be instructors at West Point. And so the you know, the joke that I used to say was, you know, three quarters of of their uh electrical and computer engineering department came from our lab. Um but uh you know, so uh in in uh we were able to graduate um, you know, um 14 um PhD students, uh 15 master's students, and we got three cybersecurity patents out of the work that that uh the students did. Um and in addition, you know, so now some of those guys, they're CISOs, uh one of them's a CISO for a major credit card company international and sporting goods uh uh companies and and that type of stuff. So so they've all farmed out, and we've you know, we've we've tried to get more students to come in. Um budgeting cov COVID put a real hit on us, uh, and it's taken us a couple of years to recover from that. You know, it's kind of like once you get that pipeline going, and it's just to get that pipeline going again.

John 10:40

Yeah.

How The Teams Work Together

Randy 10:40

But the whole idea was to to have you know um them do kind of a residency, if you will. Uh we tried to get um tried to get students when they were juniors and and seniors, um, you know, uh in that era of freshmen and sophomores. Well, no, you you have to learn how to walk before you can run. I mean you know. Um but but after their sophomore year, we we would, you know, we would try to to to get them and get them some hands-on. Um nowadays, um, you know, my office is organized. Um, I have uh four teams um uh of uh uh people. We've got uh uh 13 people in in the security office here now. Uh I have a red team that does pen testing for for uh university applications and websites. So we we basically pen test ourselves internally. Um and they also do uh security architecture consulting uh there. So that's the red team. The blue team is the traditional team. They um they look at our sensors and you know they're they've they're the core of the incident response team uh there. Uh I have a green team which does uh risk assessments and risk management stuff and uh third-party risk uh management, if you will. Um uh we would uh they work closely with the university procurement, so you want to buy some software, um you you know, you go through the university procurement process, but one of the checklist items is they have to it has to be reviewed by the green team. And they will query the vendor with the HECVAT uh, you know, that that we we use and those type of vendor questionnaires, depending on the sensitivity of the of the uh data that's gonna be handled by the vendor, uh, you know, that type of stuff. So um, and and they are doing our risk management stuff. We're using uh the Salty Clouds ISORA uh risk uh uh GRC platform. And so they're coordinating and doing all that type of stuff there. So that's that's the green team. And then I have a one-person purple team that uh interfaces with our internal SOC. Um and uh we we also have an external SOC. We're using uh Indiana University's OmniSOC. So he's the liaison person between the external SOC and that. So the students um uh what we're trying to do is um, you know, if we get them in the junior years, we um we we ask them, you know, what do you want to be? Well, everybody wants to be a hacker. Well, you know, okay, got it. Uh, you know, but but like like Ed said, you know, when he was talking to you, is you know, for us to be good defenders, we have to know how to attack. And so uh, you know, we we we say, okay, here if you're interested, here's um, you know, work on with the red team on some stuff, um, and then we're gonna transition you to the blue team. Well, now that you know how to break into something, or you know, you you can try, how do you defend against that? How do you detect it? Um uh, you know, that type of things. And then um uh the business information technology uh uh people in the College of Business, they have done a tremendous job with their business students that are interested in IT, and they're actually teaching classes on teaching their business students uh that are interested in auditing and that type of stuff, IT auditing. You know, they're teaching them how to use basic tools like uh like NMAP and and things like that. So um so we have a couple of them that that you know come over to work with us. So um, you know, we're trying to to rebuild the budgets after COVID and and you know get get them on as wage, but um, you know, for instance, I've got uh two two undergrads that um uh they're working with our our Splunk team uh and with the with the green team and they're building dashboards for us. Um uh you know, how many uh high risk systems uh that the green team has has you know categorized, how many of them are actually sending logs to our to our our central log system. So it it's um it's not where it was number wise before COVID, but it's getting there. And and to me that's uh something that you know I I think is gonna help them out a lot.

John 15:00

Absolutely. Yeah. And and we see it, we started the the student SOC in 2016, and it's just it was just a game changer for us and for me, um, being able to immediately see how much of an impact they could make and help protect and detect. Um, and then last year we started with our red team, we started the red team, and then we started it with the idea of a similar bringing in the students as well, and just a force multiplier. That's really that's really the easiest way to describe it is if you know you could have your team, but then now imagine you could multiply that out to resources that you couldn't afford to to bring in, you know, full time. Um and they and it's a win, you know, it's a win-win, right? They get they get that real experience that's tough to get until you get it out a lot of times. So they get it in college and then they go back to class, but then they come back, and then they go on summer internships in different with different companies, and they bring that skill set back. So each year you keep them, the better they are. Um, and then ultimate the ultimate goal is if we have a vacancy, is to try to keep them here. You know, which is which has worked out for us uh several times. You know, it doesn't always work out, but it does work out occasionally because then you've got a chance to see how they do and how they operate, and you get to know them, they get to know you, you can you know their strengths and weaknesses, um, and then they can they can hit the ground, you know, running immediately. Um some of those students have left and then they came back, which is great. You know, if they come back, not everybody comes back, but uh you know, if I can trick them to come back, then um then keep them here as long as I can. That's that's just win-win for us too. So yeah, it's it's really been beneficial.

Randy 16:50

You guys have the thing with like uh faculty and staff can take a class um under like a tuition waiver for you know for free, if you will.

John 16:57

Yep, yep.

Randy 16:58

Yeah, yeah. I um I I use that a lot with with the uh as a hook, you know, try because uh you know, as you and I know, we can't afford to match a lot of what industry can pay these guys. But you know, I said, hey, if you know if you're interested in getting a master's degree, uh you can do this and and not not get a student loan.

Speaker 3 17:17

Yeah.

Randy 17:17

Uh, you know, um, and um, you know, you have to take classes part-time, so you know, three years instead of two years, but hey, if I keep you for three years, that works for me.

John 17:28

Yeah. Um we've we've had several go through our MBA program and do that exact same thing. So yeah.

Randy 17:34

Yeah, use that as a hook. Yeah. And I mean you know, the other nice thing is sometimes they, you know, I get in a way, I guess you and I, because we've been doing this for so long, you know, we we sort of get jaded sometimes in our approach, and then they come in and ask a completely, you know, to us a left field question, and then we go, God, why didn't we think of that?

John 17:53

Exactly. Yeah, I I I think it is it's unique having young minds with a different perspective that haven't been in this so long, and they're just able to Yeah, they throw those those those uh left field uh balls at you, and you're like, Man, that's a good idea. Yeah, let's look at that. So being open to that.

The First Hack And Trust Mistakes

Randy 18:14

I would always get the question, you know, so well, will this work? And I go, I don't know, let's try it. That's what research is, right? We got a question, let's see if it, you know, if it doesn't work, well, we know that that approach doesn't work, you know. But I I just like, you know, uh uh I have a I have a a little you know thing here just says dare to suck, you know. I mean, you a lot of times people are just afraid to hit that hit that key, you know. And I said, you know, as a system in, because I I, you know, uh, geez, I started off as an IBM systems programmer, uh, then I then I went to uh Intel 8080s, you know, nowadays what we call Internet of Things, we were doing data acquisition stuff. And then from that I transitioned to uh BACS systems. I was a BAC system, you know, uh manager for a while and then transitioned to Unix um and you know did that. But I said, you know, in all of those scenarios, uh, you know, we blew up systems um and we had to recover, uh, you know, and and so it's like, you know, if you if you have a good backout plan, you know, any good system in nose, you don't do anything unless you have a good backout plan. Um, you know, and that's one of the things I I try to, you know, uh let them know. I said, look, you know, um failure, yeah, nobody wants to see that. Um, but if you can recover from the failure, you know, that that's a that's a plus. And that's something that later on, because we all know systems are going to fail sooner or later. Um, but you know, knowing that you have the confidence that you can rebuild a system that you burned down to the ground, uh uh, you know, is is is that's a just a confidence builder. And and and you know, you you you you don't panic as much as you would if you had never done it before.

John 20:30

Yeah. Um well let's go back. You sp you spent you talked about the your history with the system side of things. And then I was reading what you had on your on your side about the first hack that you experienced. I think it was like 1991. So you want to talk about that?

Randy 20:47

Yeah, that was um uh so at that time, you know, um uh this was in uh 19 in the uh 91, I think it was. Um, but in the late 80s and early 90s, um, you know, uh we had interactive computing for students, but it was on a mainframe. And so we'd have this one big you know mainframe, and we had a couple hundred uh, you know, user IDs for students taking classes, they would all log into this mainframe. So at that time, uh there were three um I'll call them main Unix mainframes on campus. Um I worked in the computing center, which was a central IT for the university. Computer science had uh a mainframe of their own, and then uh electrical engineering, which then became computer engineering, they had uh a Unix mainframe of their own. So literally there were six of us on campus there because each one of, you know, each we had two people working on uh on each one. There were six of us on campus who knew anything about Unix, and so we formed our own little you know support group. So um with that, uh, you know, um, and again for for the Unix diehards, because in that era there were only three machines and there were six admins, we became our own little backups for each other. And so we gave each other our login connections using the R commands, right? So uh, you know, you log into one machine, you were trusted to log into the other machines without. Presenting a password. That was a very common practice at that time. And so what happened was the computer science sysadmin, she discovered somebody, a user ID on her mainframe for a professor that she knew was on sabbatical and knew was not, you know, using their account. And so she killed the account and the account came back, and then she killed it again, and then another account came back, and then all of a sudden her machine started acting really funny. And again, the technology at that time, the main consoles were hard copy consoles. So, you know, we used to feed it the fanfold paper and and uh and and we would have a hard copy of what was going on. And that really is what kind of uh you know broke it open for her because she's looking at the logs and she sees that this this uh particular user ID had logged in after she killed the other one and had SU'd to root. And then things started, you know, host not file not found, file not found. So that's how she knew that she was in in trouble. Well, because we we had this shared you know arrangement with the R commands, uh the hacker, when he broke into her machine, he bounced over to my machine and and hadn't done anything on my machine. Um and and the way I discovered it was um uh I started looking, you know, when when she said something's going on over here, um uh uh, you know, we the both of us on the E on the EE side and and then on my side, we we started looking at our logs, and I noticed that that she had uh her user ID had logged into my system like at four in the morning. Now, you know, like any other, like any good system, and you know the habits of your of your users. Well, she never comes in. She was not a night out. She would log in at 8 a.m. at 12.01, she'd you know, log out to go to lunch, she'd come back at 1.05, at 430, 4:45, she was gone, right? Never that thing. So I saw this login at four in the morning. So I sent her a note and I said, Hey, welcome to the to the vampire club, you know. And uh, and she goes, What are you talking about? I said, I saw you logged into my machine. I guess you were trying to recover some of your files. And and she goes, Wasn't me. And that's what Yeah. Oh crap.

John 24:57

Yep.

Randy 24:59

So it it took us, uh, so the the hacker was was pretty persistent. You know, we we'd we'd rebuild systems and we'd see them banging on on our doors again. Um, and again, this was in the you know 1991. Uh Alec Alec Moffat had just come out with the password guessing program called Crack. And and we found that we found the program uh you know out on the net, and we started running it against our user IDs. And of course, in that era, you know, everybody's password was guessed uh, you know, because we picked stupid passwords. Um so as we found you know user IDs with this, um 48 hours later, we would see all these login attempts to those user IDs with the old password. And we're like, wait a minute, you know, this guy's grabbing our password files because you could do that in that era. Uh because the password file in Unix, right, it was read-only. Right. Um, but it it was before shadow passwords were were you know invented, if you will. Um so the encrypted password string was inside the the Etsy password file. And and so what we realized was that he what he did was he would he pop popped into our machines, copied the Etsy password file to his machine, and was running crack on it to get to get user IDs, and then he would come back in and log in with the right credentials.

John 26:31

Sure.

How SANS Started For Randy

Randy 26:32

Yeah. It took us two and a half months or so to clean it up, and you know, finally, you know, he he went away. Um a lot of the he was coming in through MIT at the time, had a machine, a server out on the net that they left wide open to people to be able to, you could log into that machine, and then from there you could log anywhere else to any other machines that were on the internet. Um and and they did that as a service, you know, they were pretty adamant at that time about not restricting it. But this kid um, who it turns out was in Oregon, found that site and was using that machine to log into all these other uh you know machines around the world. Um so it took us, it took us um, you know, like I said, two or three months to clean up the mess. And then we all decided, you know, um uh my backup, Ron Jarrell, uh he and I, we we looked at each other, we said, we don't want to do this again. Let's see how we got in. And we started looking to see what type of computer security books were out there, and the only things we could find were all cryptography books and things like that. And then um uh uh Cheswick and Bellavin came out with a firewall uh book and and uh and we said, oh my god, this is you know, this is kind of what we need. And then out of the blue, I got this uh email uh from uh this startup company called the uh Sands Institute. And uh in in that era, uh Sands uh Sands would uh so nowadays well SANS calls them summits. They're like two-day conferences with you know 45-minute speeches, it's like everybody does a talk. Well, that's what SANS was like in that era uh in the 90s. It was it was these two-day conferences, people would do these talks. And they were doing a a uh there it was an advertisement for a conference, computer security conference, and they had featured speakers, and and the featured speakers were the authors of the rare of the few books that we found. And uh, you know, I said, well, and it was in Washington, D.C., which is I grew up in the DC area, it was a four and a half hour drive for us from you know to there. And I said, Well, this is great. I'll I'll you know, let's go to this. Um and I went and I asked my boss, you know, hey, you know, I'd like to go to this conference. And he looked at it and he said, Can't go, we don't have the money.

John 29:06

Um so that sounds familiar.

Randy 29:10

Yeah, it sounds real familiar, right? So, so oh, but you know, God, but we really need to go. He said, sorry, we just don't have the money to send you. And I said, But you know, I got family there, you don't have to all he said, I said, don't have it. Um so you know, I was uh we're we're cleaning up, we're trying to figure out, you know, um how to keep this guy out. And then uh a week or two later, I get another email from from Sands, and this time they're asking for uh to call for proposals. And they and they said that if your proposal for a talk gets accepted, they would waive the registration fee. So Ron and I, we looked at each other and we said, Well, we don't know shit about about computer security, you know, but golly, you know, and then Ron said, Well, I guess we could talk about what happened to us.

John 30:00

Yeah.

Paper Logs To Time Based Security

Randy 30:01

And and, you know, again, you know, sometimes uh, you know, uh, I'd rather be lucky than good, you know. Um and and and we were too dumb to realize that that nobody talked about breaches back then. Because I've, you know, they were they well, if we talk about breaches, then the hackers are going to come back to us, which is the exact opposite of what would happen in that era. If we talked about a breach, the hackers would know that we knew how to find them and they'd stay away from us, right? So we submitted this proposal. I still remember it. The title was called Anatomy of an Incident. And uh so we submitted it and it got accepted. So I went to my boss, I said, Hey, um, you know, we we're doing this talk, uh, you know, uh, oh well, you know, you can't can't really, you know, don't go into don't name names, you know. We uh said, Yeah, I got it. So we um so um we went up there, did the talk, and Alan Power, it was a standing room only, um, and uh, you know, we said, This is what happened to us. And, you know, and I I made a joke, I said, y'all are laughing at us now, but when I get to the to the root causes of what happened, you're gonna be taking notes, you know. And the audience laughs, you know, and I said, Yeah, it was, you know, poor password selection, we weren't logging, you know, blah, blah, blah, and people were taking notes. Um, and so then after the talk, Alan Power came up to us and he said, Hey, that was a really, you know, very good talk. Would you like to work on some projects with us? And he said, Sure. We wanted to learn about computer security. Here's a guy that was offering us a chance to do it. So we we jumped at it, and the rest is history. I I wrote the uh uh I I did a bunch of talks, but um, I wrote the uh the first uh half day class. So at the time I was doing some outside consulting with um with the big firm in New York. The guy had heard me do a talk uh for a tech users conference, and I was I was I had built this a small uh three-hour course on teaching auditors how to audit Unix systems. Um and you know, again, auditors in that era, they were financial auditors, didn't know anything about technology. And so um I had done a talk at SANS, one of the SANS technical conferences, about that. And Alan came up to me and said, Hey, um, you know, I've got a bunch of auditors that are asking for this. Can you build a course, a half-day course? I said, Yeah, I can do that. Um, and I did, and it was a full house. And then he said, Can you make it a full-day course? You know, yeah, and then can you make it a two-day course? And then this was towards uh 98, 99, when Sands was starting to build the six-day courses, you know, what uh 401 and and and what became 401. And so um they asked uh, you know, if I could if I could build a six-day version of the course, and I at the time I said I just don't have the time to do six days, but I'll work with whoever. And Dave Holzer, who's uh uh Sands fellow now, um, you know, he he joined SANS, and so he and I we wrote the first um uh uh um what became the 507 course, the into the auditing course. Um and then uh, you know, uh Sands uh uh uh at that time was um you had the sort of the instructional wing, the the guys that were teaching the classes, and then you had sort of the research wing, and that was Alan's Alan's thing. You know, Alan would say, uh, we're getting a bunch of people together, um, you know, and we we want to see if we can come up with a way to to to uh categorize the attacks that we're seeing, you know, and and uh that you know uh and then and then once we got that list, then we want to figure out ways to defend against it, you know, and again most common attacks. And that became the basis for the Center for Internet Security. That was a SANS spin-off project. Um, you know, the US Cyber Challenge, the thing I was mentioning here, that was a SANS spin-off project. Um uh he asked us uh here at Virginia Tech if I could put together a team to build a 2,000 question pool for C, C, C, Pearl at the time for the scripting language, um and uh on how to write secure code in those languages. And we did, and that became what he did was then he went out to software vendors and he said, Your your software developers don't know how to write secure code. And uh here, here's an exam. You don't have your guys take it, no, no strings attached, you can do whatever you want. And the average test scores were like in the 30s out of a hundred. You know, so from that they built the secure coding courses that SANS used to, you know, teach. So, so I spent a lot of time on that research side of SANS, and it was, you know, and everything that I learned there, I just brought back to tech. Yeah. Um and then one of the programs uh I guess for me that I'm sort of most proud of is is um was the SANS training thing that the program that we did for Edus and for state and local governments, um, where uh you know SANS um basically gave everybody, if you were any edu, K through 12, community college or higher ed, state or local government, you got a 60% discount on the on the SANS classes. We hosted it at uh on campus. We did it once a year, and we'd have 250, 300 students come uh to tech to take a SANS class. And uh we we were able to do that for 18 years. Um we we had to we had to stop again. COVID kind of blew us blew it up. Uh in fact, when they were just starting to shut down for COVID, that was the tail end of the last class that we were able to do on site. Uh we did a couple afterwards that were you know remote, uh, but then we ran out of money to fund people uh uh at the university. Uh but uh that program I think was really, really good because it it it got training out to people around the country that needed it. And and you know, everybody I I think it just sort of you know raised the level for everybody else.

John 36:12

Yeah, I I took advantage. I didn't go to the one on site, but I definitely took advantage of the discounted uh courses with Sans. Yeah um and it and it really was helpful and my and my team did the same. Um super helpful, right? Just very in-depth, very professional. Um, one thing you mentioned that I I just circled was paper logs. So legit, you went back through paper, reams of paper. Man, I'm trying to get this picture for people who who are thinking about investigating incidents and they're like, wait a minute, you went through paper to find bad activity? Yeah.

Randy 36:56

Hang on, I'll be I'll be right. Let me pull this up.

John 37:00

Yeah, yeah. You got one, hopefully. Oh yeah. Oh man, there it is. The green and white ribbon paper. Very cool.

Randy 37:13

This was, you know, the line printers at the time, they would print out all this stuff, and and and yep, I still have this. I'm I'm you know, cleaning out my office since I'm getting ready to retire. And I've I've found those. I was like, oh my God.

John 37:27

So imagine trying to, you know, you guys probably weren't thinking about finding bad activity, but when you came across something weird, now you're having to flip through that that fanfold paper to find, wait a minute. Okay, this is weird. This is weird. Yeah.

Randy 37:42

Yeah. And and then, you know, but but later on, but you know, it it became it became uh it was a good incentive to start writing scripts because you know we had the log files in electronic form, and then you know, it it would print out on the main console. Um and then we just you know started writing scripts to say, okay, let's filter this down. And then we printed out you know what we filtered filtered out. But but yeah, I mean, uh, you know, we it'd be a boxes of paper that would go through that whole thing. And you know, floppy drives the whole whole nine yards. Um, you know, it in that era in '91, it it would take, I mean, uh CS had a really you know uh heavyweight machine. They had uh they had a one gig drive that we thought, man, you could put all the knowledge of the world on one gig, you know. Um but it took them almost um a day and a half to recover the system. Uh you know, when when the hacker started deleting stuff, she had just literally finished uh a backup, but it was on nine-track tapes. And and you know, this is one of those things that we learned um, you know, even if we have up-to-date backups, we're we're gonna be restricted by the data transfer rate of the backup media, right? You know, so if if we have a gig of you know data, but we only have something that can do you know three megabytes an hour, you know, do the math, right? I mean, yeah. And that taught us a lot about you know, making sure, yeah, network backups, they're great, but if you only have a uh you know DSL line, it's gonna take you a while to get that that data, you know, uh restored to the machine. Um Wynn Schwartau wrote a great book, a little book called Time-Based Security. Um it's uh it's uh you know, kind of a small book, like um, you know, about not that thick, but in there, uh one of the biggest things he talked about was was he did it quantitatively. So he would say, literally, um, you know, take a stopwatch. How long does it take you to do something? You know, click. And so one of his little equations was this E equals D plus R. So E is the amount of time you're exposed, and that's equal to D, the amount of time it takes to detect the attack, plus R, the amount of time it takes to react to the attack. And so what he was suggesting as a scenario was, you know, I'm sitting, I'm the sysadmin, I'm sitting here at my desk, I've got my dashboard up, and I see a you know flash, you know, bang, something detect. Well, how long did it take to detect that? Well, if it's an automated thing, maybe 250 milliseconds, right? Um and so in his example, he was talking about antivirus software. So let's say it takes 250 milliseconds to detect a virus and then to quarantine it, uh, let's say it takes another 250 milliseconds, right? So for half a second, you're exposed. Uh before, you know, you you you kill the attack. That's if the that's if the virus is known, the signature is known to the antivirus software, right? If the signature is not known, it's infinity and you're exposed, you know, forever. So I've I've always kind of used that as a as a measuring stick. It's it's you know, for incident response. How long does it take us to detect something, and then how long does it take us to react? And and in, you know, uh a simple way of doing it, you know, as a as a Unix system in or Windows Sysemin, we we have configured our machines to not allow us to log into uh uh the root account on a Unix system remotely, right? You you you don't do that. You come in under your own account and you sudo you know up, right? Well, you know, if if in in 98 we had an attack that I it was a Friday evening, and I got a call from the data center, always on a Friday, and uh, I got a call from the data center, and they said, Hey, um, looks like one of your servers is hacked. Okay, great. So I'm at home, I d I try to dial in, I can't dial in, my account's locked out. My regular account's locked out.

Speaker 3 42:05

Yeah.

Randy 42:05

I'm going, and I'm going, I haven't had enough beer that I, you know, I'm blowing my password. So, but what did that what did that do? That forced me to drive to the data center. That counts as our time. So, you know, for me it's a 20-minute drive to the data center, so that's 20 minutes that we add to the reaction time. And I got there, and and now I can log in as root because I'm at the console, and that's where I see, yeah, I am getting hacked, right? And as I'm working on that machine, my backup system, she comes around the corner, and this Friday evening, right? And she we're looking at it, what are you doing here? She said, Well, I got a call that said one of our other servers got hit as well. And I said, Well, you know, you know, it's Friday. Didn't you go home? She said, No, I tried to log in from home, but my account was locked out. So what they had done was they didn't care what our passwords were, they locked us out.

John 42:59

Because you couldn't respond.

Randy 43:01

So we couldn't respond. And so that added to the reaction time. So that's why I'm not a big fan of account lockouts um, you know, these days. Um, but uh, but yeah, so you know, little things like that, you know, we we we learned. And and um uh you know uh but that E equals D plus R for me, that's that's just and and it it's um it's a way that I can actually frame it to my executive management and the board.

John 43:29

Yeah. No, I I like it. I I've not heard that or heard of that book, so I wrote it down. I'm gonna I'm gonna take it out. It it you know it applies to to everything, so yeah.

Randy 43:38

Everything. You know, and his scenarios like, you know, okay, the attack happens because I'm I'm the system in I'm here at my desk, the attack happens that I'm at lunch, the attack happens that I'm I'm at home, the attack happens that I'm at a conference, you know. So and and factoring in all those things, and then you know, uh uh it's just a way to break it down.

Traits That Make Great Responders

John 43:59

Yeah, um if you if you think about it, um if most most of the attacks, not most, but a lot of attacks happen overnight or when everybody's going home, then it's always adding that extra recovery time because uh who's on the keyboard?

Randy 44:16

Yeah, who's on the keyboard?

John 44:17

Yeah.

Randy 44:18

And and now with AI agents, you know, acting both as as you know, attackers and defenders that that time gets compressed even more.

John 44:26

Yep. Um well I was just thinking about you know because you've been in security so long and you've seen just that that evolution of attack and response. Like what do you what have you seen like the the the folks that make the best incident responders security professionals is there anything that traits wise that you would say man these things that I've seen kind of stand out of you know I I kind of use this this analogy. It's not a legit analogy but I always thought like a a um a submarine radar operator would be an amazing incident response analysis because they're just watching right and they're looking for the weird things and the pings and and things like that. Like I've never met a submarine operator but I would love to ask them what they do on the on the daily job. Right? Those kind of like the things that I I've also talked to others about like there's this investigative mindset that where when something happens the person the people that do a good job of this they have those reps but really what they do a great job of is is they can sift through the noise that's out there and and get to closer to the true signal in a quicker manner. And maybe that's just because of experience but maybe it's there's something to it where they have that thought process that they can jump through what most people might get stuck in the just the noise. I don't know. What do you think about that?

Randy 45:54

Oh no I I I think I think you're dead on right with this one. I I've I've always maintained that I'm I'm looking for people who have an aptitude uh for things and and and some of the advice so I'll I'll give you this uh an example I I had a student here um down the uh he was here before covet um the the kid wrote ctfs he he um participated in ctfs and he always would score in the top you know five percent he uh uh taught himself uh Linux he was a great Linux system uh you know uh manager he was a phi beta kappa he was um uh uh he's nationally ranked as a chess master um got into cyber worked in our lab and and did some really cool stuff but he would not be able to get a job in the federal government not because of any criminal thing that he did but because his major was English and the civil service people for cybersecurity jobs they want people with computer engineering degrees and computer science degrees and nothing else so you know so that was one of those things that I just kind of went this is a total waste that he works for a pen test company now and he's doing really well he's you know rising up the ranks but what the aptitude thing is you know he had that he would just look at something and and go you know he he was stubborn about it he would just keep plugging at it and and seeing something's odd here let me let me see if I can get down to the root of it I I was really lucky because one of my uh one of my instructors when I was an undergrad here at Virginia Tech was a guy by the name of IJ Good um and Ij Good um quintessential British guy mathematician super smart much later I found out in the 70s I actually did an independent study for him but much later I found out that he was one of the codebreakers of Enigma and he worked for Al Turing. Wow and and I I found that out because I was reading a book about a you know Enigma and and they're they're talking about you know Turing's team and there's IJ Good you know so so when I talked to him and I and I asked him and and I said uh you know uh and this is you know in the in the 90s when I when uh before he died and and I said uh said you know you guys were first order mathematicians there's no doubt about it but you had thousands of people that at Bletchley that worked on stuff I said what did you look for? And he told me it this actually showed up in the movie the imitation game but what he told me back then was he said we look I we recruited crossword puzzle champions. Yeah and and I said really and he said why and he goes number one they don't let go of a problem so they're they're very persistent with it and number two they have this in this ability to sort of you know in uh extrapolate some information from snippets of information you know like 13 character word starts with AT ends in O N. Oh attenuation you know I mean that that type of stuff and he said that's what they were looking for in that in that thing. And and I I just that stuck with me you know in my career here when I'm looking for students uh I mean some of my most talented students are not CS people. Yeah um you know in fact they're they're very creative they're they're musicians they're they're English majors they're they're you know business people of a wide variety and and so and and there are a lot more of those people than you than you know you would think you know everybody oh they've got to be special no no there are a lot of people that that like to do that and so sometimes I I you know I I would ask a question at an interview and I'll just say um you know you guys play board games or you know when you were a kid or you know did you do puzzles and if they you know oh yeah yeah I I like puzzles or stuff like that. Okay. To me that's uh you know that's a plus thing. Because I can teach you know uh I can teach them the the the you know the the basics of cybersecurity and then the skill sets you know it's some people have a knack for race car driving some people have a knack for you know running some people have a knack for flying I mean yeah yeah uh you know if they got if they got the the will and the determination to learn it then then you know take advantage of it but this you know and my degrees are in CS and in and in computer engineering um but I'm I'm like you know that there are people out here that can do much you know they they would be great at it.

Advice For New Cyber Pros

John 50:34

Yeah I agree I we had one student who was a psychology major but was technical and could could could combine the two and just was a great leader for some of our student teams doing competitions because he wasn't so overly technical he could combine the two right and really help lead and speak the language and understand the technical but also really move and help the team do what they were trying to do which was always impressive. So definitely I call it geeks who can speak I love it. Yep. Yep so on the I'm just thinking about um your advice for those we talked a little bit about mentioning this you know everything's shifting all the time but it's always staying the same in some way but is there any specific advice you would give or different advice you'd give today for those that are wanting to get into cybersecurity?

Randy 51:34

Um oh yeah I mean you know number one don't be afraid to make a mistake I didn't become a cybersecurity expert because I'm a genius or anything like that. I became a a cybersecurity expert because I got the crap hacked out of me in the early days you know I mean and and you know what what what would happen though it it's like a self-fulfilling prophecy right you know I got hacked the first time we spend a couple months trying to recover from it and then three or four months later someone else in in the university gets hat hit with the exact same attack and then we go over and go oh look over here look over here look over here and they think oh my god these guys are geniuses and we're going man I'm glad I wasn't the only one that got hit that way but but you you use each experience as as a learning thing and and so you build up that so you know and and like I said we're very fortunate you know I know there's businesses that if you screw up you're fired and that that you know tends to tighten people up but um you know you're gonna make mistakes and and learn from the mistakes you know um uh you know I touched a live wire one time when I was a kid and and you know uh you know my dad said bet you'll never do that again you know that's right so you know that it's it's that fear of failure I think that's that that really I've seen in a lot of students especially the nowadays is is is you know uh it's ever you know they're so used to everybody doing stuff for them that that said well did you do this? Well no you know why not well you you know uh the answer the answer's not in the back of the book well no the answer's not going to be in the back of the book tell me what you think um yeah so you know that that's the first thing the the the second thing I I would say is embrace the technology um you know now we're in transition points when you know 30 years ago when when I was really fortunate to to come in um when there weren't any standards when they there wasn't any you know things like that and ironically some of the you know some of the standards that I see now in the federal government and that I I look at some of those words and I go hmm that's something that we wrote you know with SANS you know 30 years ago um but embrace the technology I mean AI uh you know I've been playing with with with it the in the last year and you know I started off using uh uh the agents you know Claude and and chat GPT to write drafts of policies and standards that I had to create because I'm I'm always terrible at writing a first draft right um and then you know now I'm now I'm I'm you know branching off and I'm saying okay I'm writing this article um here's the parameters um you know rip it apart uh if you were a reviewer what what what are the bad parts what are the the good parts of it you know and and give me suggestions so I'm starting to use the the the chatbots as as um you know something to help me improve something and and so you know embrace the technology um things you know the technology changes so much in in our in our world um you you've got to do that and um the other thing that the so that's the that's the second thing I would say and the third thing I would say is uh get a life I mean you know don't just stand there and do stuff with computers and nothing else cross-train I'm a big fan of cross-training and this comes from my volleyball coaching days you know we had the athletes doing strength conditioning and all that but in the summertime we'd say look if you're gonna play volleyball in the summertime go ahead and do that but play outdoors play beach you're you're gonna use a whole different muscle set um and and you know it it you you you learn from little things so like for me you know I've I've people say I'm a good public speaker I'm okay I think but where did I learn that I I learned that really from from playing in the band uh you know we we were very fortunate in the band that I I played in we were together 38 years we put out nine albums one bunch of individual uh independent versions of the Grammys stuff like that but we would play in front of five people or we would play in front of 2000 people and we had a you know we would tell stories our shows where we tell a story about a song and all that so I learned how to do public speaking you know that way bring that skill back now to the technology side of the house um you know um uh you know so if you have a hobby whatever skill set you learned in that hobby maybe being maybe being very meticulous maybe being very meticulous about uh you know putting together stuff like that that's a skill set that you can apply to work so you know cross-train and and and it does not matter if you're a knitting or you're a mechanic or you're a carpenter or you're a coach uh coaching I I always I think that's one of the best skills to become a manager.

Music Career And Closing

John 56:41

You know I've I I've you know think about it you know when I was coaching uh uh in grad school I was a coaching uh uh at the tech women's volleyball team right you got a bunch of elite athletes all of them who've been starters and now I got to pick six of them to be the starters and how do I keep the others motivated right and and how do I how do I create training programs so that they improve well those skill sets that I learned as coaching going I you know I went to coaching clinics and I thought oh I'm gonna learn diagrams and all this stuff that was a small part of the coaching clinic a lot of the coaching clinics was how to deal with the athletes how to motivate them find out what their strengths are and their weaknesses are well that's a skill set that that has helped me to to be I hope uh you know a good manager for my team so you know dare to suck I mean learn from your mistakes you know and embrace the technology and cross-train uh that would be my three three uh pieces of advice no that's great yeah and we didn't get to dive into the music part but that like you mentioned you're a musician you had on here that you you guys wrote the an original theme song for one of MPR's uh audio shows yeah that's very cool um so check check out I would get these re I would get these royalty checks for five dollars and twenty two cents and and I was like well that's nice you know and and it was always a code they never said what it was for it was always a code and I never bothered looking up the code and and then I found out later on that it was for it was the original theme song for the NPR program uh World Cafe. Very cool. Well Randy this has been great thank you for taking the time and and really I mean I love diving into the history there's so much there that you have that um I I want to learn from so just thanks so much for being here.

Randy 58:34

No problem. Thank you for having me. And this is a great thing that you're doing to talk to people like me and and Ed and others.

John 58:41

Yeah I think I think you know getting that word out there and for people that may not know your story um and trying to spread it I think is great. So thanks again and and with that that's it. We'll see you.

Speaker 58:54

See ya. Thank you for tuning in to today's episode of the Cybersecurity Mentors Podcast.

John 58:59

Remember to subscribe to our podcast on your favorite platform so you get all the episodes. Join us next time as we continue to unlock the secrets of cybersecurity mentorship.

Speaker 59:08

Have questions, topic ideas, or want to share your cybersecurity journey join our school community the Cybersecurity mentors where you don't have to do this alone. Connect with us there and on YouTube. We'd love to hear from you. Until next time I'm John Hoyt and I'm Steve Higgeretta