Abstract- Deep Learning as a part of machine learning area that has proven to solve many problem in the real world such as object recognition or detection. One of popular deep learning method is Faster Region-based Convolutional Neural Network ( Faster R-CNN). Faster R-CNN proposed on integrated objects in a single image. Even tough deep learning is powerful for objects recognition or detection, it would still be a problem for devices due to the need for a large memory and computation. In this paper, we propose to reduce the number of filters and nodes in the convolutional and fully connected layer to 50% to make it feasible for implementation in a mobile environment and compared it with the original model. Second, we use structured Sparsity Learning (SSL) in the convolutional layer to regularize Deep Neural Network (DNN) structure with group lasso. Third, we use Ristretto framework to convert floating point to 8 and 16 bits fixed point to represent weights and outputs of the fully connected layer. Our result shows that filter and node number reduction lowering memory storage down to 4.16x and successfully trained on NVIDIA Jetson Tegra TXI Development kit as mobile environment emulator. Ristretto successfully condense a model to 16 or 8 bits with error tolerance~1% but has better accuracy from 0.83 to 0.84 in k-5 and from 0.84 to 0.86 in k=10 and accelerates computations time 2.27x faster than the original convolution layer without SSL.