Whenever we used to think of object detection we tend to think of applying approaches of Neural Networks, but it turns out to be a tough task without it. I tried some features like Edges, Contours, Color, Contrast, Brightness, etc to recognize the object, but starting from this level of searching would take a large lump sum of time if we consider scaling, orientation, luminous conditions, etc. "Object Descriptors" are one of the wonderful things I discovered while solving this problem. Object Descriptors encode the information of the object which can then be further used for comparisons. This is all that I needed.

I tried SIFT (Scale Invariant Feature Transform),SURF (Speeded Up Robust Features) and HOG (Histogram of Oriented Gradients) for this, but HOG seems better to me. We can also use combination which compliments HOG, like LBP (Linear Binary Pattern).