Is it true that men are only attracted to beauty?

There is a guy I work with that is married to a woman who is nothing to look at very homely. We got to be friends and I show him attention he flirts with me and teases me harmless. He gets really nervous around me because we are both married I think. I am not anything to look at I am overweight and really don't find myself attractive at all. I think he likes my "mind" and company. I enjoy his humor and he says his wife hates it. He is a Gemini and I am in a Libra which is bad news because we are so much alike. My friend says men only act on ual attraction he has never once made a p at me. She says he is attracted his is talking. If it is not that he is living out his desire for me in his head. Once he said to me the 1s thing to go is a mans desire then it is the memory. He just turned 42 2 weeks ago. The wife treats him crappy. Could he be living out his desire for me in his head because he knows I am happily married? Can I see how he feels without asking?