WFN Randomization Test

Last Updated: 3 April 2012

A brief background to the randomization test can be found here.

WFN can be used to perform a randomization test by randomly permuting a covariate in the original data set. The bootstrap is run using nmrt. This command works exactly like nmgo but has two additional arguments, covname, which is the name of a data item in the $INPUT record of the NM-TRAN control stream, and NBOOT, the number of replications of the data set.

nmrt sex theopdsex 1000

This command randomizes the covariate SEX in the theopd.dat dataset with the theopdsex model and does 1000 NONMEM runs.

The randomization test can be restarted by specifying a range of bootstrap replicate values:

nmrt sex theopdsex 900 1000

will run the randomization test for samples 900 to 1000. The random number generator controlling the randomization test will start at the same place for run 900 as it would have done if the randomization test had completed 899 runs starting from run 1.

The randomization runs are performed in a temporary sub-directory named nmbs with the NMDIR extension. The randomization test results are saved in a randomization test sub-directory based on the runname with the extension ".rt_"COVNAME e.g. theopdb.rt_wt. Only the results of the last randomization test run are saved in the randomization test sub-directory.The numerical information in the WFN run summary is extracted and formatted in a tab delimited record which is added to a summary file called theopdbs.txt which is saved in the randomization test sub-directory. If the $COVARIANCE step runs successfully the standard errors of the parameters are added to that record in the final summary file. This tab delimited summary file can be easily loaded into other applications such as Excel for further analysis.

If theopdsex is run in the normal fashion with nmgo there is a fall in objective function to 5798.270 compared to the base model theopd value of 5802.086. The change in objective function is 3.816 units. If the chi-square distribution is used to predict the probability of falsely accepting the null hypothesis that sex did not affect Emax with alpha=0.05 and 1 degree of freedom then a change in objective function of greater than 3.84 would usually be accepted as sufficient evidence to conclude sex had an effect. The change in objective function in this example is close to the critical value if we assume the chi-square distribution but how close it the true P-value to the one based on this assumption?

The true P-value associated with rejection of the null hypothesis can be calculated by performing the following steps in Excel.
1. Open the thepdsex.rt_sex\theopdsex.txt file with Excel
2. Insert 3 columns between the "#rep" and "Obj" columns.
3. Name these columns "Delta", "Pvalue","Quantile".
4. Enter the following formulas into row 2 and copy to rows 3:1001.
    Delta:      = 5802.086 - E2    ; 5802.086 is the Obj value for the theopd base model. F2 is "Obj"
    Pvalue:    =chidist(B2,1)    ; B2 is "Delta". 1 degree of freedom.
    Quantile: =(row()-1)/count($E$2:$E$1001) ; Computes the quantile for each replication
5. Sort of column "Obj" ascending
6. Find the row with quantile closest to the desired alpha e.g. 0.05
7. The value of Delta in this row is the critical value of the difference in objective function that you should use with this data set to test the hypothesis that sex is a covariate that explains variability in Emax.
8. The value of Pvalue is the P value predicted by the chi-square distribution.