To revolutionize computational chemistry within simulating and analysing molecular systems. A principled framework that captures the relative 3D information and long-range interactions is needed. In this work, we propose a generic framework to capture these interactions, known as the deterministic point graph network (DPGN). It provides a unified interface to interact with 3D graphs on different levels of bonds, angles, torsional and long-range effects. We then leverage this framework to add multiple structural improvements to propose the geometric message passing scheme (GMP) to realize DPGN. We demonstrate the benefit of the proposed changes in ablation studies. Finally, we validate this model by presenting results beating the predecessor MPNet on six properties on the QM9 data set, whereas one of them is state-of-art. Additionally, we demonstrate the models' ability within the field of molecular dynamics, where it beats state of the art on \(50\%\) of the targets on in MD17 data set.