Driven by the exponentially increasing amount of mobile video traffic, caching videos closer to the end users has become an appealing solution to reduce the traffic through the backbone network while improving users’ perceived quality-of-experience (e.g., better video quality and reduced service delay). This research interest has been gaining lots of momentums due to the emergence of smart Access Points (APs), which are equipped with large storage space (several GBs). To address the “small population” problem involved in the prefetching at the edge, we propose to prefetch videos to APs ahead of users’ requests via tensor learning: We first adopt the weighted tensor model to mine the hidden semantic pattern to characterize both users’ preference for different types of videos and the dynamic video popularity over time; Then, based on the resulting low-dimension matrixes generated by the tensor factorization, we adopt an exponential smoothing model to capture the temporal pattern to predict users’ propensity to unwatched videos; Finally, based on the predicted video popularity, we proactively replicate videos from the original CDN server to the APs at the edge. Through trace-driven simulations, we show that the proposed prefetching solution can outperform the baseline algorithms: compared with the SVD-based prefetching strategy, our design achieves a better hit ratio (e.g., surpassing about 10%) and accuracy (e.g., surpassing about 15%); compared with the history based strategy, our design also have about 40% (resp. 20%) improvement in terms of hit ratio (resp. accuracy).