Is it reasonable or even possible to prioritize one method?
In research, a population represents the entirety of a group of items or subjects that are of interest to a researcher. Because populations can be too large for study or inaccessible as a whole to a researcher, a smaller subset of that population, known as a sample, is often used for observation and experimentation (Vogt & Johnson, 2015). The use of the data that a researcher collects from the sample is dependent upon the process used to select. Generally, the selection process is referred to as either probabilistic (e.g., random sampling) or non-probabilistic (e.g., convenience sampling). When data describe a population, they are known as a parameter; when they describe a sample, they are known as a statistic. Similarly, when the number of subjects or items reflects a population, the abbreviation used is 'N' (uppercase), and when the number reflects a sample, the abbreviation used is 'n' (lowercase).
A sample is considered both random and representative of a population when each item or subject has an equal probability of selection and when the probability for selecting any item or subject is independent. When sample selection meets these criteria, and the data are numerical with either interval or ratio strength, a researcher can validly and reliably apply parametric statistical procedures and generalize results to a population (Scott, 2006).
Random assignment, while similar to random selection, represents a subsequent process in a research design. While random selection is the method a researcher uses to create a sample in the first place, random assignment is the method of applying individual items or subjects from a sample to the experimental treatments of interest or control groups (Scott, 2006). Just as sample selection falls into the general categories of probabilistic and non-probabilistic, assignment also takes one of two forms, being either simple or matched (Vogt & Johnson, 2015). With simple assignment, the items or subjects of the sample are independently distributed among treatment groups; with matched assignment, they are paired on the basis of traits or attributes held in common (like gender or age). Matched assignment design assumes random selection, but it controls variables that might otherwise become confounders.
In sum, random selection creates a sample that represents a population and whose data can be analyzed statistically and generalized to a population. Random assignment distributes the items or subjects of that sample among the experimental treatments a researcher is studying. That said, the answer to the question of whether one is more important than the other, in my view, is dependent upon the needs of the research design. Moreover, both random selection and random assignment can be necessary preconditions of validity and reliability. In such a case, to prioritize one over the other would result in a research study of little or no value.
True experimental design requires that both the sample selection and assignment processes are random (Vogt & Johnson, 2015). If the goal is to generalize to a population, random selection and random assignment are arguably of equal importance. When one or the other is not random, the research design becomes quasi-experimental, and the researcher is no longer able to generalize results to a population (Scott, 2006). This is not to say that such research cannot contribute to a growing body of evidence in support of a theory; only that, from a statistical perspective, a true random sample and true random assignment are stringent assumptions of the generalizability of data from a sample to a population. Moreover, a completely randomized design is often not possible or even necessarily preferable. Variables that neither the independent (i.e., explanatory) nor dependent (i.e., response) variables capture may have a confounding effect on the study, and a researcher can control such lurking variables (often demographic in nature) through a matched approach to assignment. Peetsma et al.'s 2001 paper on inclusion in education is one such example. The study used a matched pairs design to increase internal validity and align their results to the factors of mainstream versus special education in psychological development. Similarly, in the sample selection process, constraints related to target population accessibility may mean that non-probabilistic, convenience sampling is required. Consider Wong-Ratcliff et al.'s 2010 paper on the effects of the federal Reading First program on the development of literacy as an example. Those authors, working with rural school populations in Louisiana, used a non-probabilistic convenience design to identify grade one students at five schools, three where Reading First programming was in place and two where it was not.
The question of whether random selection or random assignment is more important to the generalizability of results for a study may belie the complexity of research design more broadly. The accessibility of the target population and the need to generate a representative sample can be independent factors for consideration as a researcher undertakes to design a study. A quasi-experimental design, where one or both of the elements of sample selection and assignment may not be true-random, will still allow forms of statistical analysis and hypothesis testing. In situations where the researcher has prioritized generalizability, maintaining a true-random sample and true-random assignment should be of equal concern.
References:
Peetsma, T., Vergeer, M., Roeleveld, J. and Karsten, S., (2001) Inclusion in Education:
Comparing pupils' development in special and regular education,
Educational Review, 53:2, 125-135,
Scott, D. and Morrison, M. (2006). Key ideas in educational research. Continuum.
Vogt, W.P. & Johnson, R.B. (2015). The SAGE dictionary of statistics and
methodology (5th ed.). SAGE Publications, Inc.
Wong-Ratcliff, M., Powell, S., and Holland, G. (2010). Effects of the reading first
program on acquisition of early literacy skills. National Forum of Applied
Educational Research Journal, 23(3).