News

Jobs

Events

« February 2012 »
Mon	Tue	Wed	Thu	Fri	Sat	Sun
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29

p<0.001, CI 0.85 - 7.31 or "how can a confidence interval of a risk-ratio include 1 if the p-value is <0.001?"

Submitted by humbleservant on Mon, 03/23/2009 - 19:28

Hello all,
Hope you are well and enjoying the amazing weather.

I have a question with regard significance testing in a small cohort study.

If you type in Stata: csi 15 2 0 3 you get a RR of 2.5 with CI of 0.85-7.31 and a P-value of 0.001

But how can it be significant if the CI includes 1?

Clearly it has something to do with the small numbers, but how to proceed from here on? Is there another test that can be done for significance testing?

Input much appreciated.

Regards,

Maarten

Stats Questions

Comments

the answer

Submitted by humbleservant on Mon, 03/23/2009 - 19:45.

Transcribed from a mail-exchange and courtesy of Chris William's (cohort 11-12) efforts, the answer from stat's guru David Prieto (scroll down to 3rd post, emails in descending date). Important part is bold. Synopsis: if p-values are <0.05 and CIs overlap 1.0, then you most probably have used normal-approximations for calculating values with low numbers in your 2-by-2 table. The same is true for log-reg p-values below 0.05 including 1.0 (which are often calculated using normal-approximations). Solution: use "exact-methods" or "Fisher-method". Thanks, David!
===================================

Hi!
The P-val and CI using logistic regression are not exact ones. They are approximations using normal theory. Actually you can see it in the output where it prints "P>|z|". That "z" stands for a "normalized" value under a normal distribution N(0,1). So Fisher's and exact CI are better than logistic regression ones...
Regards

David

----- Mensaje original ----
De: "WilliamsC@rki.de"
Para: davidprieto56@yahoo.es
Enviado: miércoles, 18 de abril, 2007 10:13:04
Asunto: AW: AW: STATA, case-control odds ratio confidence intervals and p values

Dear David
many thanks, this makes sense
So I will use exact CI and Fisher exact p-values (two-tailed) and these should agree
However... what was the CI / p value calculated using logistic regression- is this like Fisher or another method?
best wishes
chris

--------------------------------------------------------------------------------
Von: David Prieto [mailto:davidprieto56@yahoo.es]
Gesendet: Mittwoch, 18. April 2007 11:04
An: Williams, Chris; mvalenciano@isciii.es
Cc: Bremer, Viviane; richard.pebody@hpa.org.uk; arnold.bosman@ecdc.eu.int
Betreff: Re: AW: STATA, case-control odds ratio confidence intervals and p values

Dear Chris,

I have been looking in Stata help to the different ways of calculating CI intervals in cc. The "exact" CI are the best ones always. The rest ("test based" "Cornfield" or "Wolf") are approximations to the "exact" ones. The reason for having so many approximated statistics is that the "exact" CI are very laborious to calculate by hand. In the pre-historical times (when there were no computers available) people had to figure out some approximations that could be calculated by hand. These are based in the normal distribution theory and are good approximations only when the sample size is large in all cells of the table. The same applies idea to the Chi-2 test. It gives an approximation to the exact P-value given by the Fisher's exact test.

Stata (and everybody) recomends to use always exact statistics (Fisher's test and exact CI). The only time when there could be an advantage in using approximated statistics is when you have LARGE samples. In that case all statistics tend to give very simmilar results but the approximations might take less computer time to calculate them compared to the exact statistics. However, this saving in time is minimal with modern computers and, anyway, given that you have small sample size, you should use the exact statistics (Fisher's test and exact CI)

Always use 2-sided tests unless under special conditions and always before the experiment you have specified a one side Hypothesis. By default exact confidence intervals are in accordance with 2 sided Fisher's exact tests.

Let me know if you have any other doubts.

Best wishes

David

----- Mensaje original ----

Enviado: miércoles, 18 de abril, 2007 8:35:04
Asunto: AW: STATA, case-control odds ratio confidence intervals and p values

Dear David and Marta,
Thanks for these suggestions.
With test-based CI both the CI and the p value (chi-squared) are significant- but I'm not sure what "test-based CI" are
(put code in fixed width font)
(1.010315 - 15.36038 test based, p (chi2) <0.0465)

With exact CI, non-significance appears to agree with the two-tailed Fisher exact p value.
(0.8497595 - 24.51525 exact CI, 1-sided Fisher's exact P = 0.0449 2-sided Fisher's exact P = 0.0629)

David, I tried LR which gives me the SAME OR but a different p value (again)-.
(OR = 3.939394 P>|z| = 0.057 95% CI: .9621557 - 16.12922)

STATA read-outs below

Still not sure we've got to the bottom of this!
best wishes
chris

. logistic case vorputen if vorputen<9

Logistic regression Number of obs = 61
LR chi2(1) = 4.16
Prob > chi2 = 0.0413
Log likelihood = -40.126858 Pseudo R2 = 0.0493

------------------------------------------------------------------------------
case | Odds Ratio Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
vorputen | 3.939394 2.833212 1.91 0.057 .9621557 16.12922
------------------------------------------------------------------------------

. cc case vorputen if vorputen<9, tb

--------------------------------------------------------------------------------
Von: David Prieto
Gesendet: Dienstag, 17. April 2007 20:00
An: Marta VALENCIANO; Williams, Chris
Betreff: Re: STATA, case-control odds ratio confidence intervals and p values

Dear all,

Sorry for answering so late. I am quite busy travelling around and not always have easy access to email. I remember we had the same question some time ago but I cannot find the emails with the discussions and the answers. I will have to re-think about it, although I vaguely rmember that we came up with some kind of satisfactory explanation of this "striking" phaenomenon of nature... ;-)

As a start... Have you tried to run a logistic regresion? You should get the same OR than with the cc command. What P-val and interval confidence you get with the log reg? I am not sure how this will help at all, but it might suggest some ideas... I´ll get on it when I have a spare minute...

regards to all and let me know if you find out something...
David

----- Mensaje original ----
De: Marta VALENCIANO
Enviado: domingo, 15 de abril, 2007 9:20:38
Asunto: Re: STATA, case-control odds ratio confidence intervals and p values

Chris

Don' t know if you have had an answer for this. I am currently on holidays in India and have no access to my computer. We had a similar question last year with EpiInfo and it depends on the method used to compute the 95% CI. There is a specific command in STATA to select the method to be used (but of course, I cannot remember it now...). I copy David Prieto who is the one who gave us the answer last year.

David, una ayudita? (y un beso de mi parte)
Greeting from India.

Von: Williams, Chris
Betreff: STATA, case-control odds ratio confidence intervals and p values

Dear colleagues
sorry for a round e-mail but this one is really very (a) specific and (b) epidemiological
Ahem
How is it that a case-control analysis can have a significant p value (i.e. <0.05) but the odds ratio CI overlaps 1 (see below for STATA example). If you use Cornfield CI, they no longer overlap 1.
There is probably a very simple answer and I am being dense
Many thanks
chris
(put code in fixed width font)
. cc case vvlow,exact
Proportion
| Exposed Unexposed | Total Exposed
-----------------+------------------------+------------------------
Cases | 12 27 | 39 0.3077
Controls | 4 33 | 37 0.1081
-----------------+------------------------+------------------------
Total | 16 60 | 76 0.2105
| |
| Point estimate | [95% Conf. Interval]
|------------------------+------------------------
Odds ratio | 3.666667 | .9514634 17.12941 (exact)
Attr. frac. ex. | .7272727 | -.0510126 .9416209 (exact)
Attr. frac. pop | .2237762 |
+-------------------------------------------------
1-sided Fisher's exact P = 0.0307
2-sided Fisher's exact P = 0.0482

Dr (Med) Chris J Williams