Discussion of Migraine Paper

Please read the paper e-mailed Sunday, focusing attention on statistical methods and results, and comment on strengths and weaknesses (one of each) of the paper.

weakness:

The paper said they used chi-squared test. However, the sample size is really small. Are they sure the cell counts are greater than 5? If not, the results are not accurate.

strength:

They offered confidence intervals, which are better than p-values.

Strength: patients for the study were chosen based on specific criteria in order to avoid confounding. For instance, patients with only migraine were included, whereas patients with other chronic pain syndrome were excluded.

Weakness: Due to the nature of the treatment the one given to the patients might not be representative of the whole acupuncture practice.

weakness: Data collection seems very basic. The diaries don't seem to provide accurate measures, although this might not be easy over such a long period.

strength: Well designed experiment for sham acupuncture

Weakness: The power for the study was only 20% which is a bit low when other problems in the study are taken into account.

Strength: The used Fisher's exact test in addition to the Chi-squared test which is helpful due to the small sample size.

weakness:

Although not necessarily a weakness per se, it seems strange to formally announce to patients that there would be both real and false treatments. While anyone participating in a study is likely aware that researchers will be studying differences through control groups, a formal announcement and anticipation of a questionnaire on patient's randomization arm seems to mitigate the placebo. For example, if patients knew that they were required to fill in a survey on their randomization arm impression they could easily research the differences in both sham and real treatments.

strength:

The researchers presented findings against the efficacy of semi-standardized methods instead of trying to make the data tell another story promoting these treatments. They also included and (recorded) detailed patient flows through the experiment (though not much commentary on whether these dropouts were systematic or random).

Weakness: use of diaries as a form of data collection could result in subjectivity and bias.

Strength: good use of randomization and double blinding to reduce certain bias such as by the evaluator.

strength: the procedure of this trial is very clear by showing the flow graph of participants through each stage of the trial. weakness: in the results of associated symptoms and the adverse effects, only p-values are show, no confidence interval.

Strength: Design for control group and treatment group is rigorous. To insure the accuracy of tests, the study puts lots of efforts on choosing randomization sample set. Weakness: The choice of treatment is related to Chinese medicine, which put limitation of sample selection. So it is possible that sample is not representative enough.

Weakness: From my point of view, the statement "There was only one statistically significant difference between groups, the age mean. The patients in the sham acupuncture group were older than in the real acupuncture group (P = 0.024)." is a very strange statement. Why would we have uncertainty (a p-value) in the difference in means of what are essentially completely observed populations? (the populations being the groups we assigned)? Could you please discuss the motivation behind this?

Strength: It appears that they put thought into their sample size calculations. The use of the results of the unpublished paper indicates they probably had decent estimates. As was mentioned above, the power seems small, however.