Sample size discussion for Thursday Jan. 8

To get the ball rolling, one area highlighted in Gerald van Belle's book is case when costs differ between samples. This could be a factor to consider on Thursday (section 2.11). Also since Neil mentioned we are dealing with a Binomial distribution, we could talk about the need for care when using certain methods to find the sample size. For example in the same book Equation 2.27 is considered appropriate only for the region of 10 < n < 100.

Extending on Daniel's point, an interesting point to considered and discuss, which is not fully fleshed out in the chapter, is what to do in situations where multiple constraints are influencing the sample size? For example, what to do in the situation where there exist constraints both in terms of costs (section 2.11), number of subjects available (section 2.10), and the requirement of a certain effect size dictated by a journal (section 2.5). Perhaps, we could delinate situations where ranking the importance of one constraint over another might be appropriate.

In this case mentioned by Andres, can we use different sample size calculations regarding each constraints and select the smallest number as sample size?

I believe it is important to consider whether the number of subjects required per group is available, and, if it is not, whether is useful to have unequal sample sizes (section 2.10). Moreover, as Andres mentioned in the previous comment we could discuss about the consequences of solely rely on the effect size (as the Rule of Thumb presented in most of the sections depends on it).

Yes, level of statistical significance, the value of the power desired, one-sided/two-sided test, costs difference, effect size, and unequal sample size are essential aspects to consider when we try to identify the sample size. Besides math calculation, in order to increase internal validity of the study design, do we need to consider about matching? Patients may have different characteristics, such as age and gender. For this example, suppose we only know whether the patients use activated carbon treatment or not, do we still have to go through all files to match two samples? Are there any better way or we do not care about matching? In addition, it is also important to consider the response rate in one of the groups (especially for surveys).

While it would be nice to match or block patients, I believe it would be too inefficient to implement this on an observational study such as the activated carbon case, unless there is a good number of patients available. And it would be rather unethical to create a randomized experiment, forcing subjects to consume poison and testing the use of activated carbon to other treatments.

As for the sample size, and in regards to Andres comment, it could be feasible to find the limiting factor and base it off that. But that factor should be a certain "mix" of other dependent factors such as costs and number of subjects available.

One point I'd like to ask is for a real experiment, how do we know some of the values that we need for sample size calculation (for example, standard deviation, probability of success, etc.). Should it be based on previous findings or a pilot study or something else?

A step toward answering Andres' question is to address the "number of subjects available" and the "cost" constraints as a single constraint. It would involve generalizing the concept of cost to allow for varying cost of additional observations as the sample size increases.

Increasing costs are often the case in reality; for example, suppose that the most recent charts are already digitized (very accessible, meaning cheap), slightly older charts are organized in a filing cabinet (still relatively cheap), but the pre- 2000 charts are disorganized in a cardboard box due to an office move or flood (much more expensive).

To control the number of subjects available, we could consider the cost of additional observations beyond that point as infinity. In this case, the simple method given by the book would no longer apply, but if costs were known it would not be too difficult to figure out the solution. I think we would be forced to use observations from the single remaining available set regardless of cost.

Another question, related to this one, is how to solve the cost problem when additional samples come in batches at a fixed cost. For example, sorting a box of charts gives 10 observations rather than just one.

Combining some of the ideas already mentioned could result in an interesting approach to choosing sample sizes. Consider regarding the currently available clinical results on Activated carbon as fixed with close to zero cost (for Neil suggests that these will likely be digitized and thus have zero incremental lookup cost). This also implicitly assumes this is a new treatment method that has not been conducted in the past. In comparison, the results of alternative treatments must be ascertained through manual searching (per Neil's suggestion). Hence, using section 2.10 on unequal sample sizes, as suggested by Chiara, will yield the number of past records to be manually searched. One intricacy not commented on is that the cost of retrieving one record of alternate treatment will not be uniform. Pre-digitized records are not organized by symptoms or treatment, therefore finding one positive record may actually be poisson distribution among all other causes bringing kids to the ER.

Alternatively, we can view this problem from a different perspective: What fixed cost are we willing to incur in the manual search process? With this cost we could estimate the number of applicable records found, consider this value fixed, and use section 2.10 to determine the number of cases where Activated carbon is used. If this sample size is greater than the number of records currently available simply wait and conduct the study when more records become available. [Here, the waiting time can be considered a generalized cost (likely modelled by an exponential random variable) to be compared against the costs from the search process. In this way the investigator could minimize his costs by balancing the relative sample sizes].

I agree with Sean's option. Also, if we only care about computing ability, an alternative way to deal with the case that sample size is greater than the number of records available may be to apply bootstrap. I think the cost in this case will be very small. However, it can be argued that this way does not increase the amount of information of original data, so it may not fit the requirement of our design.