Although quartz (α-form) is a mineral used in numerous applications wherein radiation exposure is an issue, the nature of the atomistic defects formed during radiation-induced damage has not been fully clarified. Especially, the extent of oxygen vacancy formation is still debated, which is an issue of primary importance as optical techniques based on charged oxygen vacancies have been utilized to assess the level of radiation damage in quartz. In this paper, molecular dynamics simulations are applied to study the effects of ballistic impacts on the atomic network of quartz. We show that the defects that are formed mainly consist of over-coordinated Si and O, as well as Si–O connectivity defects, e.g., small Si–O rings and edge-sharing Si tetrahedra. Oxygen vacancies, on the contrary, are found in relatively low abundance, suggesting that characterizations based on E′ centers do not adequately capture radiation-induced structural damage in quartz. Finally, we evaluate the dependence on the incident energy, of the amount of each type of the point defects formed, and quantify unambiguously the threshold displacement energies for both O and Si atoms. These results provide a comprehensive basis to assess the nature and extent of radiation damage in quartz.