Seismic exploration is sometimes unsuccessful in some areas with difficult geological and topographical conditions such as complex mountainous area. Experiments show that the joint application with seismic and non-seismic methods especially electromagnetic (EM) is an effective way to ensure high quality prospecting. This paper discusses the theory and algorithm of three dimensional forward modeling and inversion of controlled-source electromagnetic method (CSEM) with topography, which is tested to be accurate in theoretical data inversion. The paper consists of two parts: 3D forward modeling and non-linear conjugate gradient (NLCG) inversion. For 3D forward modeling, the finite difference based on scattered fields is implemented, and the obtained apparent resistivity in far field is similar with that of magnetolluric (MT) method on a typical model. Therefore the forward modeling algorithm is suitable and accurate. On the other hand, NLCG is used in the 3D inversion and initial solution optimization is adopted, which can efficiently reduce computation time consumption. Numerical experiments illustrate that the distribution of anomalous bodies can be well recovered by the proposed inversion and the initial solution optimization can dramatically reduce computation time consumption.