Medical image analysis has become one of the most popular subjects in computer vision research in recent years. Cancer detection/classification from medical images is a topic of vital importance because early detection of cancer may allow patients to receive proper and timely treatment, significantly increasing their rates of survival. Over the past two decades, researchers have demonstrated the feasibility of automating cancer classification through a sequential pattern recognition process from effective image pre-processing to useful feature extraction to building an accurate classification model with supervised learning techniques. Pixel-level textual and shape features (known as “handcrafted” features) such as color variation patterns, edges, lines, circles and ellipses have been successfully extracted by using sophisticated algorithms and deployed in computer-aided diagnosis (CAD) systems. Over the last five years, deep learning neural networks such as Convolutional Neural Networks have been developed. Such deep learning neural networks tend to provide an “end-to-end” solution for pattern recognition and have achieved impressive performance results for various applications of computer vision. More recently, deep convolutional neural networks have started to appear in CAD systems. In this presentation, we will review some of the major aspects of CNN and handcrafted feature extraction for lesion classification from medical images. The presentation will query the recent technical development in the literature with regard to cancer recognition and detection. The presentation eventually leads to some questions for future investigation and research.