We feel the same way. We both kind of want to wait until our children are a bit older. I just have 1 child, he has 3. I dont think there is aby easy way.
From my point of view I think life is too short. Everyone deserves to be happy. My wife deserves to be with someone who makes her happy, and if I am just unhappy with myself and my situation I doubt I am making her happy.
Dont be jealous. Its hard not being with the person you want to be with. Even family weekends away cut deep.