Columbia Law Survey

When results from sub-groups of a CARAVAN sample appear in the detailed tabulations, an indicator of statistically significant differences is added to the tables run on our standard demographic banners. The test is performed on percentages as well as mean values. Each sub-sample is assigned a letter. When the percentage of one sub-sample is significantly different from the percentage of another sub-sample, the letter representing one of the two samples appears next to the percentage (or mean) of the other sample.

For instance the percentage of males answering yes to a particular question may be compared to the percentage of females answering yes to the same question. In the example below, the male sample is assigned the letter B, and the female sample is assigned the letter C. Here, respondents were asked whether a certain business practice is acceptable. 67% of women said that it was -- a proportion significantly greater than the 57% of males who believe that the practice is acceptable. To indicate that women are significantly more likely to find the practice acceptable than are men, the letter B -- the letter assigned to the male sub-sample -- appears next to the “67%” in the female column. Similarly, the 37% of men that find the practice unacceptable is significantly greater than the 29% of women who do so and, therefore, the letter C -- the letter assigned to the female sub-sample -- appears next to the “37%” in the male column.

			Sex
	Total	Male		Female
	(A)	(B)		(C)
Unweighted Total	977	488		489
Weighted Total	967	464		503

Acceptable	611	274		337
	63%	59%		67%B

Not Acceptable	319	171		148
	33%	37%C		29%

Don’t Know	37	18		19
	4%	4%		4%

Significance testing is done to the 95% confidence level. The columns compared are listed at the bottom of each table.

A number of factors need to be considered when determining which type of t-test should be applied, such as whether the samples being compared overlap, whether they are means or percentages, etc. Opinion Research Corporation's software has the capability to perform the appropriate test.

Note that any statistical test becomes less reliable when the sample sizes are small. Even though the test mathematically can be performed on samples as low as thirty, sixty respondents is the reasonable lower bound on the size of the sample.