This paper draws on data from a quantitative study of upper secondary students' general mathematical self-efficacy, anxiety towards mathematics, and their relationship to achievement in mathematics. The main objective of this article is to discuss the type of information that may be lost if potential problems of validity and extreme multicollinearity in exploratory factor analysis would be solved by only removing variables without doing a profound analysis. We also describe a method that treats Likert items in the questionnaire as ordinal variables that may represent the underlying continuous variable. Our study shows, for example, that removal of problematic variables without a profound analysis leads to a loss of significant information about test anxiety. Our qualitative analysis of problematic variables also led to an unexpected finding regarding the relationship between general mathematical self-efficacy and motivational values in mathematics.