“If you’re going to marry and you’re going to marry for eternity, you should start out in love,” Kody said, comparing his failed marriages to his successful journey with Robyn

Neither Kody Brown nor his family could have predicted how his instant romantic connection with fourth wife Robyn Brown would change the trajectory of the future.

During Sunday’s Sister Wives: One on One special, the Brown family patriarch said he doesn’t think he’s “hit rock bottom” despite experiencing ongoing anger issues amid the “unraveling” of his family unit.

“Something about the struggles in the family just made me kind of go, Is this what real life is like? Is this what real marriage is like? Is this how it is?” he told host Sukanya “Suki” Krishnan. “I’ve been thinking about this. Like, all I do when I’m not working, it’s on my mind,” he continued. “It’s rolling through and I’m replaying and rehashing discussions and rehashing my life and how I started this family out and everything.”

Kody, 54, lost wives Christine Brown, Janelle Brown and Meri Brown within a span of 14 months. He now only remains married to Robyn, 45.

Reflecting on his experience with polygamy as a whole, Kody said, “I was so anxious to be living the principle of plural marriage that I was not vetting who I was marrying to be compatible with me.”

“The blame lays on one thing and one thing only. If you’re going to marry and you’re going to marry for eternity, you should start out in love,” he explained. “I thought it was acceptable. I thought I could make up the difference. I thought [about] everything in this situation with all three wives, ‘This isn’t about being, like, in love. This is about growing a family together.'”

But Kody noted that the “gospel requires us to love each other,” though he “never really actually suffered in a fit of passion in this place.” He then claimed to have had a very different dynamic with Robyn compared to his other wives. “I fell in love with Robyn. I never fell in love with anybody else. I was just choosing to be in that covenant of love with them,” he suggested, despite now-ex Janelle, 54, and Robyn claiming otherwise while speaking to the host separately.

“He’s grown and changed, and I’ve grown and changed, and I think maybe he does have more affinity, or more connection, with Robyn, and that’s OK,” Janelle said. “But I think it’s unfair to judge our early years with that same perspective because he was not the guy back then he is now.”

Janelle added, “I don’t think we had this grand passion, and maybe Robyn is his grand passion. I don’t know. But what we had was good.”

According to Kody, “There was a safety and a vulnerability that I had never experienced in my life. I met her and I started weeping for joy and didn’t stop for months.”

“Let’s just say the two of us had a click whereas I was working for years with these others to get them to get this place,” he claimed. “Never had the click.”

As Robyn reacted separately to Kody’s claims about him loving her and not his other three wives, she said she “did not know that until like just recently.” But she was also skeptical of those suggestions. “It still makes me wonder because I wonder if it’s just the rewriting of history, since there seems to be a lot of that,” she said.

Robyn also commented on the challenges of concealing her successful romance with Kody from the other wives.

“I did my best. I don’t know what else I’m supposed to do, but I did my best to not have it be something that was painful for them,” she said. “I constantly was suppressing and hiding my relationship in any kind of connection I had with Kody.”

But the wives were still well aware of Kody’s stronger feelings for Robyn. Christine, 51, even recalled the “yucky day” she realized Kody had fallen in love with Robyn — and how that changed the dynamics of his other relationships going forward.

“At that point, Kody and I were already really, really struggling. So, that didn’t help,” she said. “It broke my heart. That was one of my first moments of heartbreak was that moment. I was just sitting at the edge of my bed and watching him get ready to go on a date with her before they were married. He’s like, ‘I don’t know what’s wrong with me.’ And I said, ‘Oh, you’re in love with her.'”

Christine, who is now married to David Woolley, said of Kody: “He looked at me, he’s like, ‘I am, I am in love with her.’ And then he got really giddy. … It hurt. It hurts so bad. It’s so bad because you realize at that point, everything now is going to change.”

Next, Kody later told the host, “When I met Robyn, it was, you know, like first time ever being in love. So that changed the dynamic, I think. And that’s not anybody’s fault.”

“It was a call for us to put more effort into the other relationships, in my mind. But that became a grind,” he added. “It was constantly, constantly trying to improve your love and prove yourself and and and like, kind of doing this dance constantly. And some years were good because the dance was working really well. And what I’m talking about is these struggles in relationship when we don’t have the basis of passionate love.”

Meri, on the other hand, does believe Kody has a “soul connection” with Robyn. However, she believes his prioritization of that bond should not have been put ahead of the entire family.

“I know that he’s happy with her and I know that they have this soul connection. He speaks of that often,” Meri said. “I’m glad that he has that with her. I think everybody should have that. I think it’s very unfortunate that the rest of the family had to go by the wayside to get that. I think that he has, in his life, had the capacity to love and care for multiple of us, right? Like, we’ve got a big family. I think he has a capacity and for whatever reason, he’s not doing it.”

Agreeing that Robyn is Kody’s “soul mate,” Christine concluded: “I think him marrying her, I saw what a real marriage could be. I saw him with his soul mate. I saw, ‘Oh, I guess he does have it in him to help out. I guess he does have it in him — just not with me.;”

“I just think that maybe her relationship with Kody was so very different from the get-go,” Janelle also shared. “Maybe it was hard for him because it really felt like he more and more pulled away from wanting to be a plural husband.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Sister Wives airs Sundays at 10 p.m. ET on TLC.