Keith Bynum is looking back on some of the challenges he faced during his tenure as an HGTV star — and the familiar faces he says had his back in his darkest hours.

The Bargain Block star and his co-host and partner Evan Thomas shared that their show had not been renewed after five seasons in a candid post on Instagram in June. Now they’re speaking out in their first interview since the surprising news broke.

“Ninety-five percent of it was very smooth sailing,” Bynum told PEOPLE via Zoom on Monday, Sept. 8 ahead of Bargain Block‘s series finale on Sept. 10. “We had a fantastic team on the ground [while filming Bargain Block] that we got along with. They were our best friends.”

Another HGTV series, however, was more difficult for the designer to navigate.

Bynum and Thomas appeared on two seasons of the competition series Rock the Block, in which teams of HGTV stars renovate identical houses in a series of room-by-room challenges. At the end of each season, the homes are appraised and the team with the highest property value wins.

“There were several times with Rock the Block that I butted heads with a certain producer to the point I had called and asked around if I had screwed my career up,” Bynum recalls, claiming he felt like the person had “it out for me.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

Three people who he says always had his back were Rock the Block host Ty Pennington, Married to Real Estate star Egypt Sherrod (whom he competed against in Rock the Block season 3) and Farmhouse Fixer co-host Jonathan Knight.

“There have been many times that Egypt or Ty or Jonathan, like all these people, have kindly offered us help and guidance, condolences,” Bynum says, revealing that “Egypt was my shoulder to cry on for a little bit.”

Sherrod and her husband and co-star Mike Jackson shared in June that they learned their show had been canceled while they were on a family vacation, and Knight wrote that same week that he learned the network had canceled his show for “reasons I’m still processing” while he was preparing to perform in Las Vegas with New Kids on the Block. HGTV has not publicly addressed any of their statements.

Pennington made his feelings known by commenting on Sherrod and Jackson’s joint Instagram post: “You know, in general I thought I had [a] decent gauge about what TV executives wanted and the last few years have completely shattered that,” the longtime host wrote.

Related Stories

‘Bargain Block’ Stars Reflect on Their HGTV Run and Explain Why Fans Shouldn’t Feel Sorry for Them (Exclusive)

HGTV Lost Almost Half Its Viewers in the Last 8 Years: Reports

With Sherrod, in particular, Bynum says he leaned on her in “those moments you realize that you’re going through something that almost no one else in the country could understand except for this small group of people. And you know, there are some that are amazing and some that are, like, I’m probably not gonna answer that call.”

Bynum and Thomas say they’re grateful for the unlikely camaraderie they found among the other impacted stars. Around the same time, Israel “Izzy” Battres also said his show had been canceled after one season, and sources confirmed to PEOPLE that Christina Haack’s Christina on the Coast and Tarek and Heather El Moussa’sThe Flipping El Moussas, wouldn’t be returning to the network.

While they look ahead to what’s next for them, both as business owners and potentially on other platforms, Thomas says, “We’re proud of what we’ve done here and how we’ve grown.”