“It would absolutely devastate me if he tried to leave me,” Robyn Brown said of Kody Brown during Sunday’s ‘Sister Wives’ episode, later adding, “I know he would regret it”

The family fallout is beginning to take a toll on Kody and Robyn Brown’s marriage.

During Sunday’s episode of Sister Wives, the couple separately addressed the overall impact of the ongoing drama, sparked by Christine Brown’s departure and Janelle Brown’s recent fight with Kody, and how their marriage is potentially at stake.

Kody, 54, met up with his brother-in-law Nathan — who is married to Meri Brown’s sister but was also was previously polygamous — to have a heart-to-heart about how he can best navigate this tough road ahead. In doing so, the Brown family patriarch questioned whether he can even “manage” monogamy.

Kody also acknowledged how hard this period has been for his now-last-remaining wife Robyn, who is losing sister wives amid his various divorces.

“It’s not just the husband that’s being left. Robyn’s feeling the ultimate rejection,” Kody told the cameras as Robyn, 44, said in a separate confessional, “I’m dealing with some depression and some mourning, some loss and grieving. And it’s a very difficult heartbreaking situation.”

As Kody’s brother-in-law warned that he’s now “going to have to face the darkest parts” of himself in this new chapter, Kody said: “It’s weird, I am! And I don’t like who I am.”

Kody then told the cameras he has an “urge” not to be “mean as hell,” which is something he’s “seriously fighting.” He now feels that because he “failed” with three of his four marriages, he should now “just be the Devil I am.”

“This is stuff that I should just be keeping to myself,” he continued. “I don’t want Robyn to see this. I don’t want her to know how dark I’ve gotten.”

Kody added, “I just have this gut thing that’s like I’d like to run away from my life. You know, just escape. Just change my name, disappear and be somebody else. I just … I hate my life right now.”

Addressing her own perspective, Robyn said she’s still “invested” in plural marriage since it’s something she’s longed for from a young woman. She said Kody’s questioning of polygamy “feels a little bit like a deal breaker of our marriage,” adding, “This is not what our marriage agreement is.”

“It would absolutely devastate me if he tried to leave me,” she continued. “I mean, I know he has these thoughts. I mean, you can’t make anybody stick around or stay.”

Of Kody possibly leaving her, Robyn said: “I know he would regret it.”

“I just know it’s from pain,” she added. “It’s not because he’s a selfish or a bad person. He’s just in pain.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.