Predicting the quaternary structure of protein complex is an important problem. Inter-chain residue-residue contact prediction can provide useful information to guide the ab initio reconstruction of quaternary structures. However, few methods have been developed to build quaternary structures from predicted inter-chain contacts. Here, we introduce a gradient descent optimization algorithm (GD) to build quaternary structures of protein dimers utilizing inter-chain contacts as distance restraints. We evaluate GD on several datasets of homodimers and heterodimers using true or predicted contacts. GD consistently performs better than a simulated annealing method and a Markov Chain Monte Carlo simulation method. Using true inter-chain contacts as input, GD can reconstruct high-quality structural models for homodimers and heterodimers with average TM-score ranging from 0.92 to 0.99 and average interface root mean square distance (I-RMSD) from 0.72 Å to 1.64 Å. On a dataset of 115 homodimers, using predicted inter-chain contacts as input, the average TM-score of the structural models built by GD is 0.76. For 46% of the homodimers, high-quality structural models with TM-score >= 0.9 are reconstructed from predicted contacts. There is a strong correlation between the quality of the reconstructed models and the precision and recall of predicted contacts. If the precision or recall of predicted contacts is >20%, GD can reconstruct good models for most homodimers, indicating only a moderate precision or recall of inter-chain contact prediction is needed to build good structural models for most homodimers. Moreover, the accuracy of reconstructed models positively correlates with the contact density in dimers.