Ggthjy

Is there a reliable way to say if two Pearson correlations from the same sample (do not) differ significantly? More concrete, I calculated the correlation between a total score on a questionnaire and an other variable, and a subscore of the same questionnaire and the variable. The correlations are respectively .239 and .234, so they look quite similar to me. (The other two subscales did not significantly correlate with the variable). Could I use a fisher Z to check if the two correlations indeed do not significantly differ, or is the fact that they are not independent a problem?

Firstly I would point out that these correlations are fairly low.

Second, have you plotted the data to investigate possible non-linear associations?

Third, I would say that common sense should dictate that correlations of 0.239 and 0.234 are essentially the same and searching for a test to confirm this, unless the sample size is absolutely enormous, is folly.

Fourth, you could calculate confidence intervals for both statistics, and if they do not overlap, then you can conclude that they are statistically significantly different. However, this would be invalid since the 2 samples are not independent. Moreover, as per my third point, even if you did have such an enormous sample and a test which validly concluded that a significant difference exists, I would find it hard to belive that the difference was practically significant.

Expanding on Robert Long's answer (+1 to Robert) I'd say that testing for a difference between these is folly, regardless of sample size. Look! Is 0.239 different from 0.234? Well, maybe it is. There are situations where a very small effect size is very important. If a plane crashes 1 in 1,000 flights, that's a big big problem. I can't think, offhand, of a situation where this tiny difference in correlations could be meaningful, but maybe there is one. Whether it is significant or not is not the point.

Also, the dependence will surely be a problem. If you really wanted to see something like this, I'd find a third correlation: The correlation between the test after removing the subtest. Then you can compare that to the correlation with the subtest.

Finally, it's unclear to me what you are trying to show, but I think you are trying to show that these are not different. In that case, the usual null hypothesis tests are inappropriate. You should be looking at tests of equivalence (if, in fact you want to look at significance at all).

Yes, it is possible to perform a significance test using the Fisher transform. This also depends on $N$, the number of samples used to compute the Pearson correlations. This blog post describes the method in more detail, and provides R code for it.