clothing - Is it permissible for a man to be seen without a shirt?

In many parts of the United States, it is socially acceptable for a man to be without a shirt while working at a physical activity. Examples might be cutting grass or digging a hole in the summertime. This takes place out of doors and so is generally in the view of the public.

What are Jewish perspectives on this behavior? The head might still be covered by a hat in this situation, but it is not as modest as you would often expect an observant Jew to be. Is it permissible? Would it matter if the shirtless activity was inside ones house instead (say washing dishes)? Were standards on this different at different times? For instance, I have read that in the middle ages, yeshiva students would swim naked in rivers (cannot lay my hand on where I read this right now).