For religious reasons, I am not permitted to touch males, other than young children, outside my immediate family (father, brother, son, or husband). I've never had a problem once I explain this, while taking a clove with my fingers and saying, "But the offer was sweetly made, and I hope that you will instead accept a drink (or some food or just conversation) instead, kind sir/madam." One or two have seemed disappointed, but no one has seemed offended, thank goodness.
The first time someone offered me the cloven fruit, I had no idea what it was for. I smiled and said a flowery version of "No thanks, I'm not hungry, but it was very generous of you to offer." It was hours later when I found my friends again and had them explain why the person had walked away looking sad. I felt bad about it, but what can you do? If a person hasn't the courage to make an offer that's understandable even to a newbie, how does the newbie know what's on the table?