Holiday in the Sun’s Austin Nichols on Kissing Mary-Kate Olsen at 15 – ryan

Actor Austin Nichols says that playing the love interest of Mary-Kate Olsen’s character in the 2001 movie, Holiday in the Sun, was something he was initially “concerned” about — largely because he was an adult and she was just 15 years old.

Speaking on the April 15 episode of Drama Queens — a podcast hosted by One Tree Hill alumni Sophia Bush and Robert Buckley — Nichols (who also appeared on One Tree Hill) spoke about the film, which was one of his first roles.

Nichols played Griffen Grayson in Holiday in the Sunwhich centered on twin sisters (played by Mary-Kate and her sister, Ashley Olsen) who are forced to spend their winter break on a family trip to the Bahamas. Griffen is a fellow vacationer who befriends the twins on their trip.

“I was definitely nervous because Mary-Kate was 15, and I was, I think, 18 or 19, or maybe even 20. I don’t remember, but I was too old, and it felt weird, and it felt wrong. And, you know, I looked very young, so watching the movie, I don’t think you notice,” Nichols, now 44, said on the podcast. “I’m real tall and skinny and, like, nerdy, but I don’t think you can tell there’s a big age difference, but for my brain, was like, she’s 15, this is not, this is weird.”

He continued: “So I remember talking to producers and I remember talking to even her dad was there and … I was like, ‘Is this okay with you guys?’ And they were all like, ‘Oh, don’t worry about it.’ Nobody was you know, nobody seemed concerned, and I was very concerned, but it turned out fine. It was all very G-rated.”

Acknowledging that the Olsens were “children” at the time of filming, Nichols said the project was one that may have initially seemed “silly” but was ultimately embraced by audiences.

“It’s one of those things … where, when you’re young and you’re trying to make your career as an actor and do all these cool projects, when you have this one that you feel is maybe a little silly, I had to give that up years ago because people love it,” he said.

He continued: “It’s only paid me back a million times with love. Though that’s another lesson I’ve learned from doing what we do is, you know, sometimes it’s the same thing. Sometimes those things that you think maybe aren’t, like, super cool or hip — a lot of people love them.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

Nichols wasn’t the only actor to get his start on Holiday in the Sun. Notably, the film also starred a young Megan Fox.