I'm no car detailer, but with practice, you will get quicker. Always try to wash your car in the shade, not in direct sunlight (or early morning or late afternoon). You will find that you won't end up with water marks and streaks. And try to dry the car quickly after rinsing with a large microfiber drying towel. You can get a reasonable one from halfords (own brand) for £5 or large supermarkets. Still using my halfords one which I have had for ages. I always put the wash mitts and drying towel in the washing machine after use, so they are like new for the next wash. I know my techniques are not perfect, but like you I have a fine limit on how much time I will devote to car washing. You will also find that if you can one weekend find the time to apply a wax or polish, that washing gets even easier the next time round. I started off with turtle wax wet n dry ( for convenience ) and got good results and was quick and easy to apply. I now have gone slightly (only just) up market with other products (autogylm), but that's only because I have got a good routine and 2 hours max is what I spend, usually every 2/3 weeks.
If you are always washing exterior and hoovering interior at same time, try saving time, by doing each activity on alternate weeks.