Who argued that it is ok?
Now I will argue that it is probably indeed the lesser of two evils for those who have tried very hard to improve things to no effect particularly when they have children, their spouse has a medical or mental condition etc. . . that doesn't mean those in that situation are necesarily saying it is ok, but that it is the lesser of two evils.
For me it is like my experience of divorce, I'm not arguing that it was a fantastic wonderful thing, but it was best thing to do and the lesser evil.
So in the chain of argument here, you are still back in the Victorian era. To start working through the above thought experiment you first have to graduate from an earlier one.
Namely, you have to come to terms with the reality that almost everybody cheats. It is not deviant, or aberrant behavior. It is in fact the behavior of the majority of people, always has been, and always will be. Most of those who don't cheat only fail to do so because they lack good opportunities. There have been some genetic studies on this question that have provided strong evidence that this is so, that our DNA mixes more rapidly than monogamy would allow, and surveys of the percentage of children who are related to their father showing the same thing, further re-inforced by surveys of peoples attitudes showing a great willingness to cheat, given the right opportunity.
So you have to ask yourself, if you have a law that is violated by everyone, is that good law? Similarly, if you have a supposed moral code that is in fact routinely violated by most people, is that actually a good moral code?
In short sooner or later you're going to paint yourself into one of two corners. You can adopt the position that the majority of people on the planet are wrong and evil, and that pretty much the whole world is immoral. This is a Platonic conception: The moral idea comes first, derived from some first principles of logic, without regard to reality, and then we check to see how reality is doing, and find reality is imperfect. It's a rejection of life in favour of a purer idea. If that's what you want to do then I guess you can then go live the life of a hermit somewhere.
Alternately you can take the view that behavior that is so widely accepted is in fact acceptable, and prefer objective reality over the Platonic ideal. This is a more pragmatic, realistic, and empirical way of viewing reality, and a view that embraces life, celebrates life, values who and what we are, as opposed to who and what some abstract notion says we should be.
This shift from the Platonic ideal to the empirical reality is widely seen as one of the hallmarks in the shift from the classical to the modern, from the Victorian era to the present day. You appear to have chosen a pre-modern worldview that rejects life in favour of an ideological or theological ideal; I've chosen a modern outlook that celebrates life itself in all its complexity above all else.
I recognize that rejection of life is a valid choice, and theologically speaking, often one advocated by a variety of the world's religions. But I choose life, not rejection of life.
For those of us who choose to embrace life, then the thought experiment about cheating on your spouse, versus cheating with your friend's spouse, becomes an interesting question.