On the surface it may look that simple, but many times married couples have a lot more complexity to the situation than just that.
I've been married almost 17 years, and with my wife for over 20 years. Through all that time, we've always struggled with differing sex drives. Pretty much mine's always been there full tilt, and her's not so much. Sex when we do have it is great, it's just never very often (maybe once a month, 2 months sometimes longer). Sex for her just has never been a priority. I discussed my feelings about it but all it's ever ended up in was a huge fight and no resolution or change. For years I lived with it without ever straying, but last year I'd had enough. Almost 20 years of barely getting any wears on a man and I just reached that point and started hobbying a little over a year ago. Why should I have to beg and plead just to get scraps of sex here and there? Do I feel guilty? No. Strangely though, I feel guilty that I don't feel any guilt over it if that makes any sense lol.
Should the marriage end? No, kids are involved and it would be far too harmful to them. My wife and I don't fight constantly, so we're civil enough at home.
Years ago I would have said the exact same thing ASN, but after being in the situation for so long, my opinion's changed.