To me the answer lies in several factors content, technique and legality. Graffiti art, in my mind, has different criteria then other forms of painting. Now I would like to point out that in no way am I an expert in graffiti art. I am not familiar with any movements and I know only 1 artist through family ties. However, I do know a thing or 2 about art.
Now as I said earlier graffiti art is subject to slightly different criteria. Unlike other media, there is a predisposition in this country to view it as not art. So where fine art has more liberties, perhaps even too many, graffiti art has to be careful. For example if an artist put a word or phrase on a canvas questions would not be asked, but in graffiti some would debate that artistic merit. Also if you were to take work like Jackson Pollack and applied it to graffiti media, chances are people would want it painted over rather then praise it.
Technique to me is another criterion in making a judgment, as a professor of mine once told me "Good art looks good." I have seen graffiti artists do things with paint cans that I can't do with a brush at a thousandth the size. I've seen some truly fantastic design and composition work done that boggles my mind. I know that these people see space in a way that I just do not, mind-blowing in their intricacy with perspective that I couldn't even comprehend attempting. When the technique is poor though, perhaps it might still be art, but if it is bad art more questions are asked then if it is good.
Finally there is legality. The content might be fantastic and executed flawlessly but if it is in a place it is not supposed to be then it is still defacement. When art is public a lot of permission must be given, a sculptor cannot just make a piece and drop it off on a sidewalk or in a park. If they could, then I would be littering the landscape with my work. Public art is always subject to approval, whether it is from one person i.e. the land/business owner or a committee of some kind. No matter who it is though, some form of permission is necessary. So to me legality is a major issue and the dividing line between art and not art.
I should state again that in no way am I the authority on graffiti art, but then again no one is really the authority on any art. Art is a subjective experience. If the piece though meets my personal criteria then I deem it art, but that's just me.