In this paper, we consider the problem of reconstructing the full layout of an indoor room from a few RGB images. Taking an arbitrary number of images as input, a novel deep Convolutional Neural Network (CNN) architecture is proposed to learn the wall segmentation and features for matching walls across different images in a unified manner. Unobserved areas are then completed by leveraging the information from large-scale datasets. To the best of our knowledge, we are the first to deal with the problem of generic room layouts including the non-cuboid and even non-Manhattan building architectures using RGB images. Additionally, we can produce a holistic layout reconstruction of the indoor room with around 4 images while SfM-based methods usually fail due to the sparsity of input data.