Project Crossover

Measures of Sample Representativeness

The accuracy of survey data is largely dependant on the characteristics of survey respondents being similar to those of the population being studied. Over 4000 participants have completed the Project Crossover surveys, giving a total response rate of approximately 31%. Although the completed sample is large enough (i.e., greater than 1067) to allow for generalization of the chemistry and physics populations as defined by the National Science Foundation’s Characteristics of Doctoral Scientists and Engineers in the US: 2003 (2006), the issue of nonresponse error must still be addressed. Nonresponse error occurs when the characteristics of the completed sample (i.e., the population sample that completed the survey) differ from those of the population sample who did not complete the survey in such a way as to create an inaccurate data set (Dillman, 2000). One method to determine if the characteristics of the completed sample are representative of the sample population is to compare completed sample data to pre-collected data that is considered reflective of population characteristics.

Project Crossover samples were drawn from the membership lists of the American Chemical Society and American Physical Society, and it is assumed that the membership in these organizations are accurate reflections of the chemistry and physics populations, respectively. To validate Project Crossover data, race/ethnicity and gender data collected from Project Crossover (following 6 months of data collection) were compared to similar data collected by the NSF’s Characteristics of Doctoral Scientists and Engineers in the US: 2003. These comparisons include data from all individuals who had completed the Project Crossover “Scientist” survey as of December 3, 2007 and indicated that they hold a PhD degree. The first comparison between NSF and Crossover data is of the race/ethnicities of individuals in 5 sectors of employment: 1) College/University Faculty; 2) Non/Not-for-Profit Organizations; 3) For Profit Organizations/Industry; 4) Local, State, Federal Government Scientist; and 5) Other Employment (Figure 1):

Figure 1. Comparisons of National Science Foundation (NSF) Characteristics of Doctoral Scientists and Engineers in the United States: 2003 data versus Project Crossover “Scientist” survey data (as of December 3, 2007), by race/ethnicity and sector of employment.
¹ NSF data include physical science PhDs who hold faculty positions at 4-year postsecondary institutions; Crossover data include physical science PhDs who hold faculty positions at 2- and 4-year postsecondary institutions.
²NSF data include physical science PhDs who report working in private, for-profit organizations; Crossover data include individuals working in industry (for-profit).
³ NSF and Crossover data include physical science PhDs who report employment other than educational (all levels), non/not-for-profit, industry/for-profit, government-funded, and “self-employed”.

Gender comparisons between the NSF 2003 Survey of Doctorate Recipients data versus Project Crossover data are separated by broad field of employment (chemistry or physics) (Figure 2):
Validation by Gender and Field

Figure 2. Comparisons of National Science Foundation (NSF) 2003 Survey of Earned Doctorates data versus Project Crossover “Scientist” survey data (as of December 3, 2007), by gender and broad field of employment.
¹ Physicists in the NSF data include individuals identifying their employment as "Biophysics, Astronomy/Astrophys; Earth/Atmos/Ocean Sci; Physics; Aerospace/Astro Engineering; and Postsecondary Physics Teacher" in Characteristics of Doctoral Scientists and Engineers in the United States: 2003. Physicists in the Crossover data include participants from the American Physical Society membership list.
²Chemists in the NSF data include individuals identifying their employment as "Biochemistry; Chemistry; Chemical Engineering; or Postsecondary Chemistry Teacher" in Characteristics of Doctoral Scientists and Engineers in the United States: 2003. Chemists in the Crossover data include participants from the American Chemical Society membership list.

* Note: Percentage of Females in Crossover data for physics calculated after adjusting for oversampling females from the American Physical Society’s membership list.

Demographic similarities between NSF and Project Crossover data suggest that Project Crossover demographic data are representative of the physics and chemistry population characteristics, as defined by the NSF’s Characteristics of Doctoral Scientists and Engineers in the United States: 2003. While it is difficult to determine how generalizable demographic data representativeness is to all data collected by Project Crossover, the low nonresponse error for Project Crossover demographic data suggests that other data acquired by Project Crossover is representative of the chemistry and physics populations, as well.

References:

Dillman, D.A. (2000). Mail and Internet Surveys: The Tailored Design Method, Second Edition. New York: John Wiley & Sons.

National Science Foundation, Division of Science Resources Statistics (2006). Characteristics of Doctoral Scientists and Engineers in the United States: 2003, NSF 06-320, Project Officer, John Tsapogas (Arlington, VA 2006).