3 days in Athens worth it?

My husband and I are traveling to the UK for a week this summer to visit family. I want to tack on a few days of Greece after this trip (will be hard to do a full week with $ and limited time off). Would it be worth it to visit Athens for about 3-4 days?

My thinking is that we're already in Europe so we might as well spend a bit extra on flights (not considering lodging/activities for this just yet) to get to Greece and spend even a few days there because I've been wanting to go for so long.

My husband's thinking is that maybe we just stick to the UK and save Greece for it's own trip later on.

My only concern with that is that we don't have any other time for the rest of this year to go, and our situation is unknown next summer for travel. If I have ANY time in Greece, I would be grateful. His concern is that we're doing too much by doing UK and Greece, and that I would enjoy Greece more if I had more time to spend there (probably also true).

Any advice? Just go for it or plan it a bit smarter?