In my life honesty is paramount; I don't know of another way of living, but that is not to say that it's an easy way to live for myself or those around me. In fact, I am sure it isn't. Lying keeps the world civil and functional.
As a famous TV doctor says almost every week, "Everybody lies." We lie to others to get something we want. We lie to our selves and call it rationalizing. We lie to the people we care about to be nice. Lying to others and to oneself is a survival mechanism. As Jack Nicholson said "You can't handle the truth," and most people cannot. Truth, factual truth, is often unpleasant. "Yes, you are being stupid. "Yes, you do look fat in those pants." "No, you are not a good person." "No, it is wrong you cheated on your spouse and that does make you a terrible person." These are thing no one wants to hear.
Honesty as the best policy is an ideal. It works in a perfect, utopian world. It works in a world where people are fundamentally honest with themselves. That would be the key people admitting to themselves who they are truly. This of course is incredibly difficult and painful, hence why people don't do it. Once honesty is achieved within it is easier to manifest outward and to accept it from others.
Honesty has always appealed to me because it is how I want to be treated. The unspoken social games and contracts annoy and anger me. I find the, unnecessary complication to life. They require more work then I want to do. So to simplify my life I try to be honest about everything, particularly with what I think and how I feel. I have often felt much of the trouble people have in their lives is due to a lack of honesty. It always seemed to me that the majority of people portrayed on TV in couples counseling are there because they are not being honest with each other or with themselves.
Still, I have my doubts. I believe the world would be a better place if more people were honest, but at the same time lies are too seductive. The human psyche is too fragile to handle honesty on a constant basis. White lies and small courtesies are too engrained into human behavior. People love to play games and add complexity to situations. Whenever you have more then 2 people in a room a hierarchy and thus politics come into play. Thus telling someone what they want to hear becomes much more valuable than honesty. So, no, honesty is probably not always the best policy.