Research Methods, Statistics, and Measurement: Domain Guide

Two randomisations do two different jobs and neither substitutes for the other. Randomly selecting who takes part tells you the sample resembles the population, which is a claim about generalisation. Randomly allocating those participants to conditions tells you the groups did not differ beforehand, which is a claim about cause. A tightly controlled experiment on one university's undergraduates has the second without the first; a national survey has the first without the second. Almost every methodology item is built on that split.

Statistical significance is a statement about a hypothetical world, not about this one. A p-value answers: if there were truly no effect, how often would data this extreme arise? A small value makes the no-effect account uncomfortable. It does not report the probability that the null is true, it does not report the probability of replication, and because it depends on sample size, it does not report how large the effect is. Reading every 'p < .05' as 'the no-effect story fits badly' keeps all three misreadings out.

Reliability and validity stand in an asymmetric relationship worth stating out loud. A scale that reads three kilograms heavy is perfectly reliable and completely invalid. But a scale that returns a different number every time cannot be measuring anything consistently, so it cannot be valid either. Reliability is necessary for validity and nowhere near sufficient — which is why an item offering high reliability as evidence of validity is offering the wrong direction of the implication.

Designs form a ladder of entitlement. A case study describes one person or setting in depth and is invaluable for rare phenomena and hypothesis generation, but supports no general conclusion. Naturalistic observation records behaviour where it occurs and buys realism at the cost of control, with observer effects and observer bias as its characteristic threats. Surveys reach large samples cheaply and measure self-report, which is not the same as behaviour and is subject to social desirability. Correlational research measures two variables as they occur and supports prediction, but not causation, because of the directionality problem and the possibility of a third variable. Only an experiment — a manipulated independent variable, a controlled comparison condition, and random assignment — supports the conclusion that the manipulation produced the change. Quasi-experiments compare pre-existing groups such as sexes or diagnoses; because assignment was not random, they inherit the correlational limits no matter how experimental they look.

Within an experiment, the threats have names, and naming them is the skill. A confound is any variable that varies systematically with the independent variable. Demand characteristics are cues that let participants infer the hypothesis and adjust. Experimenter expectancy is the researcher's unintended influence on participants or on scoring, which is why double-blind procedures exist. Placebo effects require a placebo control to isolate. Order effects in within-subjects designs are handled by counterbalancing, and selection bias in between-subjects designs is what random assignment prevents. Between-subjects and within-subjects designs trade off against each other: within-subjects removes individual differences and needs fewer participants but risks order effects; between-subjects avoids order effects but requires more participants and relies on randomisation to equate the groups.

Descriptive statistics summarise a sample. The mean is the balance point and is pulled towards extreme values; the median is the middle score and is not; the mode is the most frequent. In a positively skewed distribution — a long right tail, such as income — the mean sits above the median, and in a negatively skewed distribution it sits below, which is why the median is reported for skewed data. Variability is summarised by the range, the interquartile range, and above all the standard deviation, the typical distance of a score from the mean. In a normal distribution roughly 68 per cent of scores fall within one standard deviation of the mean, 95 per cent within two, and 99.7 per cent within three, so a z-score — the number of standard deviations a score sits from the mean — converts any score into a position in that distribution.

Correlation is the domain's most abused statistic. The coefficient runs from −1 to +1, the sign gives direction and the absolute value gives strength, so −0.8 describes a stronger relationship than +0.5. Squaring it gives the proportion of variance shared, which is why an apparently respectable r of 0.30 accounts for only nine per cent of the variance. A correlation coefficient describes a linear relationship, so a strong curvilinear relationship can produce a coefficient near zero. Restriction of range attenuates correlations, and a single extreme outlier can create or destroy one. And it never licenses a causal claim: the relationship may run the other way, or both variables may depend on a third.

Inferential statistics ask whether a result would be surprising if nothing were going on. The null hypothesis states there is no effect; the p-value gives the probability of data at least this extreme under that assumption; and when it falls below the pre-set alpha, conventionally .05, the null is rejected. Two errors follow: rejecting a true null is a Type I error, whose long-run rate is alpha, and failing to reject a false null is a Type II error, with rate beta. Power, the probability of detecting an effect that is really there, is one minus beta, and it rises with sample size, with effect size, with a less stringent alpha, and with measurement precision. Lowering alpha therefore reduces Type I errors while increasing Type II errors — the trade-off is the point. Test selection is mechanical once the design is clear: a t-test compares two means, one-way analysis of variance compares three or more, chi-square handles frequencies in categories, and correlation or regression handles two measured continuous variables. Effect size measures such as Cohen's d, with the conventional landmarks near 0.2, 0.5, and 0.8, answer the question significance cannot.

Psychometrics applies the same rigour to instruments. Reliability comes in three flavours — stability across time, agreement across raters, and internal consistency across items — and internal consistency is usually reported as coefficient alpha. Validity asks whether the instrument measures the intended construct: content validity covers the domain adequately, criterion validity predicts a relevant outcome measured now (concurrent) or later (predictive), and construct validity assembles the whole pattern of evidence. Face validity, whether the test merely looks right, is the weakest form and is sometimes deliberately sacrificed. Standardisation means uniform administration and scoring plus norms from a representative sample, without which an individual score cannot be interpreted at all. The standard error of measurement expresses the uncertainty around an individual score and is why scores are best reported as bands.

Intelligence is the domain's extended case study. Spearman's factor analysis suggested a general factor underlying performance across tasks alongside task-specific factors; Thurstone argued for several primary mental abilities instead; Cattell distinguished fluid reasoning from crystallised knowledge; Gardner proposed multiple relatively independent intelligences; and Sternberg's triarchic account separated analytic, creative, and practical intelligence. Measurement began with Binet's practical task of identifying children needing additional schooling, produced the mental-age idea, and was converted into the ratio intelligence quotient of mental age over chronological age times one hundred — a formula that fails for adults, since chronological age keeps rising while performance does not. Modern tests therefore use a deviation score, placing an individual against the distribution of scores for their own age group, with the Wechsler scales set to a mean of 100 and a standard deviation of 15. Two further facts recur: the Flynn effect, the rise in raw test performance across generations that forces periodic renorming, and the correct reading of heritability as a statement about the proportion of variance within a population attributable to genetic differences, not about how much of an individual's ability is inherited.

Ethics constrains all of it. Human research requires informed consent given by a competent participant who understands what is involved, voluntary participation with the right to withdraw at any time without penalty, protection from harm beyond that of ordinary life, confidentiality of data, and debriefing afterwards. Deception is permitted only when the study cannot otherwise be done, when the risk is justified, and when full debriefing follows. Institutional review boards exist to make that judgement independently of the researcher, and animal research is governed by its own committees and standards of care. The historical cases that drove these requirements — medical research conducted without consent, and obedience and prison studies whose participants experienced real distress — are examinable both as history and as illustrations of which specific principle was breached.