Taxi demand prediction has recently gained increasing attention because of its impact on enabling intelligent transportation systems in a smart city. With the increasing availability of ride-sharing and e-hailing services such as Uber and FreeNow, large-scale taxi demand data can be collected in real-time. An accurate prediction model can help pre-allocating taxis to better meet demand at lower operating costs while reducing traffic congestion and air pollution. Improving the demand prediction by utilizing this data remains a critical problem to be solved, though. Recent advances in traffic prediction jointly model the complex spatial and temporal dependencies in the data by applying deep learning techniques, which outperforms both traditional forecasting methods as well as deep learning architectures modeling them separately. However, all known studies are concerned with the prediction of the taxi pickup demand (TPD, i.e. the number of passenger pickups in an area during a time interval) while none of them predicts the actual taxi trip demand (TTD, i.e. the number of taxi rides between the pickup location and the dropoff location during a time interval). Therefore, we explore the TTD prediction problem by adjusting a state-of-the-art TPD prediction model and develop three different end-to-end architecture frameworks for TTD prediction. Experiments on a large-scale real-world data set show that similarly good results can be obtained for the TTD prediction problem as for the TPD prediction problem. We propose the best-performing Multi-Channel Deep Spatial-Temporal Network (MCRDST-Net) to be used for the task. The MCRDST-Net uses local convolutional neural networks (CNN) and long short-term memory (LSTM) to process multi-channel matrices generated from the TTD values of the trips starting in the neighborhood of a pickup location and ending in the neighborhood of a dropoff location.