There are masters and amateurs alike who adore the game of chess. Likewise, there are others who despise those sixty-four squares. Such people's dislike of the game does not translate into a lack of value, however.
Chess is, before anything else, a game. As a game, it offers players a chance to enjoy themselves in thought and competition. It can serve as a social pasttime (as with the old men in the park), a reason to keep in touch (people played chess by mail long before the internet was dreamed of), or a silent diversion from a boring lecture (those magnetic sets are really handy in college). For anyone who enjoys the game, it is no more a waste of time than any other form of entertainment. Games play an important role in daily life, providing an outlet from stress and an escape from the mundane. Chess does this quite well for its devotees, for players can be caught up in it for hours.
For the person who dislikes chess, it obviously has less entertainment value. Even then, the game may hold value for them. Particularly in youth, chess is recognized as an excellent developmental tool. It hones the skills used in math and logic reasoning. It is no coincidence that people who are good at math have a natural tendency to excel at chess. For children who are taught the game while their minds are in the formative stages (before and into the teenage years), there is a notable benefit. Obviously this benefit is more pronounced when the child plays regularly, and makes an effort to develop strategy. It is the strategic thinking ahead, and not the memorization of moves that creates the benefit. In the end, chess will be a waste of time only for the person who neither enjoys it nor makes an effort to play well.
Intelligence is not a commodity. When it is used, it is never used up but increases with experience. As such, playing chess can never be a waste of intelligence. As discussed above, chess is an intelligence builder, focusing on math and reasoning skills. It is unlikely to improve a person's singing or literary skills, so it only applies to specific realms of intelligence, but it will never cost brain cells. (Binge drinking is the way to achieve that.)
Chess even serves as an artistic inspiration. Countless sculptors have carved their own unique pieces, some according to the traditional medieval motif, others delving into realms of fantasy, history, or nature. Chess sets have been fashioned from wood, plastic, paper, stone, glass - in short, there is no end to what creation may be inspired by this one game.
With the advent of computers, graphics designers had the chance to get involved as well. They set their creative genius to work creating artistic pieces, animated pieces, and even clever battle sequences as the pieces duel one another. Beyond the game itself, chess is helping to develop intelligence in computing. No chess tournament made bigger headlines than when Deep Blue (a chess-playing supercomputer) took on Grand Master Kasparov in a series of games. The developmental work in designing those algorithms represents a great contribution to programming - all inspired by chess.
Chess offers rewards in pleasure, socialization, distraction, thinking skills, art and computers. As such, it can neither be considered a waste of time, nor a waste of intelligence. That doesn't mean that everyone has to like it. It just means that they have to find a better excuse.