Visual Predictive Check

Last Updated: 22 January 2021

Visual Predictive Checks for Continuous Data

A visual predictive check (VPC) is used for non-linear mixed effect model evaluation and communication of results for continuous data (Nguyen, Mouksassi et al. 2017). The principle of a VPC is to compare the distribution of observations with the distribution of model predictions. This is done by simulating from the model and comparing percentiles of the observations and simulated predictions by visual inspection of graphs of percentiles of the observations and simulated predictions (Holford 2005, Karlsson and Holford 2008). VPC evaluation may be aided by scatterplots of the observations and plotting confidence intervals for the predicted percentiles.

WFN provides a mechanism for creating VPCs using NONMEM and R. The nmvpc.bat command is a Windows command file that is used to run NONMEM and R. It must be edited to specify the NM-TRAN control stream name for the model, observation types, independent variables, and plotting options.

Installation

The VPC run folder contains the NM-TRAN simulation control stream for simulating the model.

The files nmvpc2r.awk, vpc.R and nmvpc_functions.R files must be located in the run folder containing a Windows batch command file (e.g. named nmvpc.bat) to run the VPC simulation and plotting process. These files should be copied from the %WFNHOME%\bin\cont folder. An example batch command file, nmvpc.bat, can be found in %WFNHOME%\bin\tte\warfarin_PK.

WFN must be installed with NONMEM and a Fortran compiler. The R system must be installed with the path to the R.exe executable file in the file search path. This can be verified by typing ‘rgui’ at the command prompt in a WFN window. The R path may be set in wfn.bat near line 60 e.g.

set RPATH=C:\Apps\R\R-4.0.3\bin\x64

The NM-TRAN simulation control stream and the nmvpc.bat file will need to be customized before running nmvpc.bat.

No R packages need to be installed. VPCs are created using a pair of R scripts. These are called by nmvpc.bat using R in batch mode. The VPCs created by nmvpc.bat are created as pdf files.

When nmvpc.bat is run successfully a tmp_vpc.R script is available which can be sourced in R by users who want to explore how nmvpc is working.

The obsFile and simFile data frames created by sourcing tmp_vpc.R in R are compatible with the vpc() function found in the vpc package at https://rdrr.io/cran/vpc/. The vpc package will need to be installed and the vpc library loaded in order to use the vpc() function e.g.

vpc(sim = simFile, obs = obsFile, obs_cols=list("OBS"),show=list(pi=T,sim_median=T)) # vpc with obs and pred percentiles

It should be noted that the vpc package is not required to use the WFN VPC tools,

Customizing the NM-TRAN Simulation Control Stream

A VPC is constructed from the observed values used for estimation and simulated values predicted from the final estimates and model used for estimation.

1. Use nmctl.bat to put the final estimates from a model into the control stream used to run that model e.g.

nmctl mymodel.ctl

2. Copy the control stream containing the final estimates to another folder e.g. VPC folder in the estimation model run folder (the VPC folder). This allows the same model name to be used for estimation and VPC. Alternatively you may copy the control stream in the estimation model run folder and use a different name for the VPC control stream.

mkdir VPC

copy mymodel.ctl VPC\myvpc.ctl

3. Change to the VPC folder and edit the control stream

cd VPC

edit mymodel.ctl

4. The following changes should be made to convert the estimation control stream into one suitable for VPC simulation:

a. Copy the data file to the VPC folder or edit the $DATA record to locate the data file used for estimation e.g. the following change to the data file path could be used when the data file and VPC folder are in the estimation model run folder. Change $DATA from:

$DATA ka1_to_emax1_data.csv

$DATA ..\..\ka1_to_emax1_data.csv

Note the additional ..\ required because WFN creates a results folder below the VPC folder.

b. Remove all $ESTIMATION, $COVARIANCE and $TABLE records

c. Insert the following line immediately after $PK (or $PRED). This creates a copy of all DV data items so that they can be saved in the table file containing the simulated values. Because NONMEM uses the variable name DV for the simulated values the observed values must be copied with a different name i.e. OBS.

OBS=DV

d. Add the following lines to the end of $ERROR (or $PRED)

REP=IREP

$TABLE REP ID TIME DV PRED OBS

NOAPPEND ONEHEADER NOPRINT FILE=vpc.fit

$SIM (20201224) ONLYSIM NSUB=100

e. The NONMEM internal variable IREP is used to number each simulation replication. It is saved in the table file with the name REP.

f. The $TABLE record defines the NONMEM simulation output variables required to perform a VPC

g. The $SIM record has a random number generator seed which may be changed as required. The NSUB option indicates the number of sub-problems (i.e. replications) to be performed to create the VPC. NSUB=5 is useful for exploring VPC shapes. NSUB=100 is suggested as a minimum for reliable confidence intervals.

5. If there is a MDV data item in the data file then this should be added to the list of variables in the data file

$TABLE REP ID TIME DV PRED OBS MDV ; add MDV from data file

6. A DVID variable is required in the list of variables in the data file. The DVID value should be set to 1 in the NM-TRAN code if there is only one type of observation e.g. add this to the end of your $ERROR (or $PRED) NM-TRAN code.

DVID=1

7. If there is more than one type of observation e.g. a PKPD model with both concentration and effect observations then a DVID variable is used to distinguish the type of observation. A numerical value for each observation type should be defined in the data file to distinguish different observation types and a data item named DVID used in $INPUT.

8. A DVID variable based on another data item may be used e.g. the CMT data item might be used to distinguish parent and metabolite observations.

DVID=CMT ; use the CMT data item to distinguish observation types if DVID does not exist

$TABLE REP ID TIME DV PRED OBS MDV DVID ; add a DVID variable to the table file

9. VPCs may be created using categorical covariates e.g. SEX or RACE. Categories may be created during the simulation for continuous covariates e.g.

IF (WT.LT.70) THEN

SIZE=1

ELSE

SIZE=2

ENDIF

$TABLE REP ID TIME DV PRED OBS MDV DVID SEX SIZE ; add SEX and SIZE variables to the table file

Customizing the nmvpc.bat Command File

The nmvpc.bat file must be edited before running it to create a VPC. The changes involve setting environment variables with the set command. Take care not to include any blank characters immediately after the ‘=’ character and before the environment variable name. Blank characters should only be used as a list item separator when defining lists e.g. for models, observation types or covariates.

The essential changes are:

1. Specify if NONMEM should be run to create the simulation table file. This only has to be done once for each model. Different VPC options may be explored with the same simulation table file. The runNONMEM option should be set to ‘y’ the first time nmvpc.bat is run then it should be set to ‘n’ to skip the time consuming simulation step once a successful simulation has occurred.

:nonmem

rem set runNONMEM=y to execute nmgo for each model

set runNONMEM=y

2. List the names of simulation model control streams (without the file extension) in the models option. Usually only one simulation model is used but several models can be used to create VPCs if a list of models is provided. Model names must be separated by a blank character.

rem Set list of models to be simulated e.g. set models=mdl1 mdl2 mdl3

set models=ka1_to_emax1_simln

3. The xnames option identifies the independent variable (usually TIME) but any other suitable variable in the simulation table file may be used e.g. time after dose (TAD).

rem Names of variable in NONMEM table file to be used for the VPC x-axis

rem This can be used for evaluating continuous covariates e.g.

rem set xnames=TIME TAD WEIGHT for total time, time after dose and weight

set xnames=TIME

4. List the names of observation types to be used for the VPC. Observation names in the obsnames option must be separated by a blank character. At least one observation name must be specified.

rem Set list of names according to observation type e.g. set obsnames=CP PCA

set obsnames=CP PCA

5. Set the options for each observation type that determine the way the VPC plot is created.

1. The bintimes option is a list of times to be used as the centre of intervals for binning observations and predictions. The list of times should be separated by commas (no blanks). A R expression may be used to generate the list of times (see example below).

2. The timescale option may be useful for rescaling the independent variable e.g. a timescale variable set to 1/168 could be used to rescale the time variable from hours to weeks. The timescale applies to values in bintimes and to x axis variables such as xmax, xmin and xtick. The timescale applies to values in bintimes and to x axis variables such as xmin, xmax and xtick.

3. A yscale option may be useful for rescaling an observation variable e,g, yscale=1000 to rescale mg/L to mcg/L. The yscale applies to values to y axis variables such as ymin, ymax and ytick.

4. Either x or y axis or both axes of the VPC may be set to a logarithmic scale (base 10) by specifying ‘x’ or ‘y’ or ‘xy’ for the logaxis variable.

5. Each axis should have a label, a minimum, maximum and a tick value. Tick values should be a multiple of the minimum to maximum range. Because blanks may not be used in variable values the “#” character should be used in x and y axis labels to indicate where a blank should appear in the label.

6. A lower limit of quantitation value may be supplied. All predictions less than this lloq option value will be ignored when creating the VPC.

7. Each observation type has its own section identified by a label created from “:” prefixed to the observation name specified in the obsnames list. This allows specific options to be set according to the observation type. The user must create these labels and set the options for each observation type.

8. Some options may be the same for all observation types e.g. options related to the independent variable such as bintimes and x-axis options. These options may be set in the COMMON section and do not have to be set in the OBSERVATION SPECIFIC section.

rem *************************

rem OBSERVATION TYPES SECTION

rem *************************

:obstype

rem Each observation type may have its own properties

rem The dvid variable is required to distinguish types

rem Define R script variables for each observation type

rem No spaces are allowed in variable values.

rem Use '#' which will be replaced by a blank in xlabel and ylabel values

rem **** COMMON ******

rem Variables common to all observation types

rem Any of these variables may be observation (obsname) specific

set bintimes=c(seq(0,10,1),seq(12,144,12))

set logaxis=

set xlabel=Hour

set xmin=0

set xmax=144

set xtick=12

set lloq=0

rem obsname labels must correpond to names in the obsnames list

rem A obsname label must have a ":" before the obsname e.g. :CP for obsname=CP

goto %obsname%

rem **** OBSERVATION SPECIFIC ******

rem user defined obsname labels identify variables for each observation type

:CP

set dvid=1

set ylabel=%obsname%#mg/L

set ymin=0

set ymax=20

set ytick=5

goto select

:PCA

set dvid=2

set ylabel=%obsname%#%

set ymin=0

set ymax=120

set ytick=20

goto select

6. VPCs may be created by selecting observations and predictions according to covariate values. Use of covariate selection is optional. If covariate selection is not required then the covariates variable should be set to a null value (no blanks after the ‘=’):

set covariates=

7. If covariate selection is chosen then the covariates variable should be set to a list of one or more covariates using the name specified in the simulation control stream $TABLE record which determines the name in the simulation table file. For each covariate name there must be a covariate value list showing each of the covariate values to be used for VPC selection.

rem **********************

rem COVARIATES SECTION

rem **********************

:covariate

rem Covariate selection is optional. For VPC without covariates: set covariates=

rem Set list of covariates (upto 3) e.g. set covariates=SEX SIZE

rem Names in the covariates list must match exactly the names in the simulation table file

set covariates=SEX SIZE

rem Each covariate name must be matched with a list of numeric values for the covariate

rem which will be used to create VPCs for each value

rem select on covariate 1 e.g. sex values 0 1

set covlist1=0 1

rem select on covariate 2 e.g. size values 1 2

set covlist2=1 2

8. The MODELS SECTION contains some options that are not often changed. They apply to all the models and VPCs created by nmvpc.

1. The PIpercentile value of 0.9 will create a 90% interval (i.e. 0.05 and 0.95 percentiles) for observations and predictions.

2. The CIpercentile value of 0.95 will create a 95% confidence interval around each of the prediction percentiles.

3. The isstd option creates a standard VPC without modifying observations or prediction values.

4. The ispc option creates pred-corrected VPCs. Pred-correction modifies both the observations and predictions (Bergstrand et al. 2011).

5. The iscsv option writes comma separated value format files containing the numerical values used for the VPCs. These may be used by other programs to create VPC plots.

6. The isbig option may be set as y in order to re-read simulation file each time for each model to use less memory.

7. The hasmdv option is set as y to use the MDV data item to select valid observations otherwise all records are valid observations.

8. The mdvpname option is used to identify the name of a variable in the simulation table file that specifies the MDV status for each predicted value when it is different from the original MDV status. Some simulations may create simulated values at times when the original observation was missing (MDV=1) or sometimes simulated values should be ignored at times when the original observation was present (MDV=0). E.g. if a predicted value is less than the lower limit of quantitation then a variable MDVP in the simulation control stream could be set to 1 otherwise it is set to 0. The MDVP variable should be listed in the $TABLE record and the mdvpname variable in nmvpc.bat set to MDVP.

9. The dvpname option is used to identify the name of a variable in the simulation table file that specifies the equivalent of the DV data item when it is different from the original DV item. This allows VPCs to be created using other items in the $TABLE file.

10. The obspname option is used to identify the name of a variable in the simulation table file that specifies the equivalent of the OBS data item when it is different from the original OBS item. This allows VPCs to be created using other items in the $TABLE file.

11. Splotobsint may be set as y or n to show (y) or hide (n) the observation intervals on the scatter VPC

12. Splotpredint may be set as y or n to show (y) or hide (n) the prediction intervals on the scatter VPC

13. Splotsymbols may be set as y, id, or n to show observations as circles (y), id number (id) or hide (n) the observations on the scatter VPC

14. Splotspaghetti may set as y to show observations joined together for each subject (spaghetti plot) on the scatter VPC

15. PIplotobsint may be set as y or n to show (y) or hide (n) the observation intervals on the PI VPC

16. PIplotpredint may be set as y or n to show (y) or hide (n) the prediction intervals on the PI VPC

17. PIplotconfint may be set as y or n to show (y) or hide (n) the confidence bands on the PI VPC

18. PIplotsymbols may be set as y, id or n to show observations as circles (y), id number (id) or hide (n) the observations on the PI VPC

19. showbins may be set as y or n to show or hide the bintimes locations on the X-axis

20. plotObsNotch may be set as y or n to plot symbols on the observation intervals (to denote these lines as observation intervals).

21. showlegend may be set as a, b or n to plot the legend above (a; at Y=1.3*Ymax), below (b; at Y=-0.2*Ymax) or hide (n) the plot legend.

22. The user may wish to modify the vpc.R script. The name of the modified R script (without the .R extension) should be specified using the orgR variable.

23. It is possible to run NONMEM on a grid cluster using the nmgog command. The grid option should be set as g in order to use nmgog. It is also possible to set the args7 and cpus options for use with nmgog.

24. theme may be set as standard to use a standard theme or as user for a user defined theme of colours, line widths and symbols used for VPC plots. An example of user theme options is provided in nmvpc.bat. The standard theme options are defined in vpc.R.

Running nmvpc

After customizing the NM-TRAN simulation control stream and the nmvpc.bat command file the nmvpc command is used to create VPCs. nmvpc.bat uses the nmvpc2r awk script to create a temporary R script based on vpc.R. The variables specified in nmvpc.bat are written in the temporary R script then the R batch command processor is called to execute the temporary R script.

VPCs (and csv files if requested) are created in a vpc sub-folder identified according to the observation name (and covariate name if covariate selection is used).

Debugging

The vpc process is complex and it is easy to make errors. Errors in the NM-TRAN simulation will be displayed in the usual way when running nmgo.

A log of the R commands is created in a file called tmp_vpc.Rout. If there is an error in processing the R commands then this error will appear at the end of the log file.

The most common error is caused by incorrectly specifying a selection value. This causes the R script to create an inappropriate obsFile and/or simFile. If dvid is not correctly specified (e.g. a value of 3 is specified in nmvpc.bat but there are no records with a value of 3 (or all records are flagged as missing values)) or if a covariate name does not match the name in the simulation table file e.g. SOX is specified in covnames instead of SEX, then an error like this will occur:

Error in tapply(1L:0L, list(`cut(dat[, idvCol], br = c(0, binTimes), right = F, include.lowest = T)` = integer(0)), :

arguments must have same length

Calls: getObsPI ... do.call -> by -> by.data.frame -> eval -> eval -> tapply

Execution halted

When this error message occurs then check the dvid values and covnames to see if they appropriate. You may also need to check that there are non-missing values for both predictions and observations for the dvid and covnames that have been selected.

Examples

This example uses a model and data that were kindly provided by Dr Alison Thomson from a study of amikacin pharmacokinetics (Siebinga, Robb et al. 2020).

Model Used to Estimate Parameters

$PROB Model

$INPUT ID DAT1=DROP TIME TAD EVID AMT RATE DV SEX AGE WT AJBW CREA CRCA

$DATA data.csv

$ESTIMATION METHOD=COND INTERACTION MAX=9990 SIG=3 NOABORT

$COV

$THETA

0.05 ; POP_CL

0.5 ; POP_V

$OMEGA

0.1 ; PPV_CL

0.1 ; PPV_V

$SIGMA

0.01 ; RUV_PROP

1 ; RUV_ADD

$SUBROUTINE ADVAN1 TRANS2

$PK

GRP_CL=POP_CL*CRCA

GRP_V=POP_V*AJBW

CL=GRP_CL*EXP(PPV_CL)

V=GRP_V*EXP(PPV_V)

S1=V

$ERROR

Y=F*(1+RUV_PROP)+RUV_ADD

$TABLE ID TIME AMT RATE

TAD AGE SEX WT AJBW CRCA Y

NOPRINT ONEHEADER FILE=model.fit

Model Used to Simulate VPC

$PROB VPC

$INPUT ID DAT1=DROP TIME TAD EVID AMT RATE DV SEX AGE WT AJBW CREA CRCA

$DATA ..\..\Models\data.csv ; must specify path to data set from NONMEM run folder

;$ESTIMATION METHOD=COND INTERACTION MAX=9990 SIG=3 NOABORT

;$COV

$THETA

0.0464 ; POP_CL

0.344 ; POP_V

$OMEGA

0.108 ; PPV_CL

0.0501 ; PPV_V

$SIGMA

0.0273 ; RUV_PROP

2.49 ; RUV_ADD

$SUBROUTINE ADVAN1 TRANS2

$PK

OBS=DV ; for VPC

GRP_CL=POP_CL*CRCA

GRP_V=POP_V*AJBW

CL=GRP_CL*EXP(PPV_CL)

V=GRP_V*EXP(PPV_V)

S1=V

$ERROR

Y=F*(1+RUV_PROP)+RUV_ADD

;$TABLE ID TIME AMT RATE

;TAD AGE SEX WT AJBW CRCA Y

;NOPRINT ONEHEADER FILE=model.fit

; Simulation (this code must be in $ERROR)

DVID=1 ; must provide a DVID to identify the observation type for the VPC plot if DVID is not a data item or not defined elsewhere

REP=IREP ; simulation replication counter

; Simulation output

$TABLE REP ID TIME DV PRED OBS MDV DVID

TAD WT AGE SEX CRCA AJBW ; covariates for VPC

NOAPPEND ONEHEADER NOPRINT FILE=vpc.fit

$SIM (20201224) ONLYSIM NSUB=100

; Simulation end

Simple nmvpc (nmvpc1.bat)

Code changes shown for adapting the supplied nmvpc.bat for use with the amikacin example.

@echo off

if not '%1==' goto %1

:nonmem

rem set runNONMEM=y to execute nmgo for each model

set runNONMEM=y

rem Set list of models to be simulated e.g. set models=mdl1 mdl2 mdl3

set models=model_final

rem Names of variable in NONMEM table file to be used for the VPC x-axis

rem This can be used for evaluating continuous covariates e.g.

rem set xnames=TIME TAD WEIGHT for total time, time after dose and weight

set xnames=TIME

rem Set list of names according to observation type e.g. set obsnames=CP PCA

set obsnames=CP

goto models

rem *************************

rem OBSERVATION TYPES SECTION

rem *************************

:obstype

rem Each observation type may have its own properties

rem The dvid variable is required to distinguish types

rem Define R script variables for each observation type

rem No spaces are allowed in variable values.

rem Use '#' which will be replaced by a blank in xlabel and ylabel values

rem **** COMMON ******

rem Variables common to all observation types

rem Any of these variables may be observation (obsname) specific e.g.

rem if (%obsname%==CP) set logaxis=y

rem logaxis options: "x", "y", "xy", "n"

set logaxis=

rem List of times for binning observed and predicted values

set bintimes=c(seq(0,2400,100))

rem use timescale to scale TIME variable e.g.

rem set timescale=52 to scale years to weeks

rem set timescale=0.041666667 hours to days

rem set timescale=0.005952381 hours to weeks

set timescale=1

set xlabel=Hour

set xmin=0

set xmax=2000

set xtick=200

rem Use lloq1, lloq2, ... for covariate selection (see :covnum)

set lloq=0

rem obsname labels must correspond to names in the obsnames list

rem A obsname label must have a ":" before the obsname e.g. :CP for obsname=CP

goto %obsname%

rem **** OBSERVATION SPECIFIC ******

rem user defined obsname labels identify variables for each observation type

:CP

set dvid=1

rem use yscale to scale Observation variable e.g.

rem set yscale=1000 to scale mg/L to mcg/L)

set yscale=1

set ylabel=%obsname%#mg/L

set ymin=0

set ymax=100

set ytick=10

goto select

Advanced nmvpc (nmvpc2.bat)

Code changes shown using logarithmic scale on y-axis, time scale changed from hours to weeks, use of other independent variables.

rem Names of variable in NONMEM table file to be used for the VPC x-axis

rem This can be used for evaluating continuous covariates e.g.

rem set xnames=TIME TAD WEIGHT for total time, time after dose and weight

set xnames=TIME TAD WT AGE CRCA AJBW

rem use timescale to scale TIME variable e.g.

rem set timescale=52 to scale years to weeks

rem set timescale=0.041666667 hours to days

rem set timescale=0.005952381 hours to weeks

rem TIME (h) from 0 to 2000

if '%xname%=='TIME set timescale=0.005952381

if '%xname%=='TIME set bintimes=c(seq(0,14,1))

if '%xname%=='TIME (

set logaxis=y

set xlabel=Week

set xmin=0

set xmax=14

set xtick=2

)

rem WT from 36 to 148

if '%xname%=='WT set timescale=1

if '%xname%=='WT set bintimes=c(seq(30,150,10))

if '%xname%=='WT (

set logaxis=

set xlabel=Weight#kg

set xmin=30

set xmax=150

set xtick=30

)

rem AJBW from 36 to 92

if '%xname%=='AJBW set timescale=1

if '%xname%=='AJBW set bintimes=c(seq(0,100,5))

if '%xname%=='AJBW (

set logaxis=

set xlabel=AJBW

set xmin=30

set xmax=100

set xtick=10

)

rem AGE from 16 to 92

if '%xname%=='AGE set timescale=1

if '%xname%=='AGE set bintimes=c(seq(0,100,5))

if '%xname%=='AGE (

set logaxis=

set xlabel=Age#y

set xmin=10

set xmax=100

set xtick=10

)

rem CRCA from 17 to 185

if '%xname%=='CRCA set timescale=1

if '%xname%=='CRCA set bintimes=c(seq(10,200,10))

if '%xname%=='CRCA (

set logaxis=

set xlabel=CRCA

set xmin=10

set xmax=190

set xtick=20

)

rem TAD from 1 to 97

if '%xname%=='TAD set timescale=1

if '%xname%=='TAD set bintimes=c(seq(0,4,0.5),seq(6,24,2),seq(36,96,12))

if '%xname%=='TAD (

set logaxis=

set xlabel=TAD

set xmin=0

set xmax=24

set xtick=4

)

Advanced nmvpc (nmvpc3.bat)

Code changes shown to separate VPC by a categorical covariate.

:covariate

rem Covariate selection is optional. For VPC without covariates: set covariates=

rem Set list of categorical covariates (upto 3) e.g. set covariates=SEX SIZE

rem Names in the covariates list must match exactly the names in the simulation table file

set covariates=SEX

rem Each covariate name must be matched with a list of numeric values for the covariate

rem which will be used to create VPCs for each value

rem select on covariate 1 e.g. sex values 0 1

set covlist1=0 1

rem select on covariate 2 e.g. size values 1 2

set covlist2=

rem select on covariate 3 e.g. race values 1 2 3

set covlist3=

goto gotcov

VPC example from nmvpc.bat

1. Standard VPC

2. Prediction Corrected VPC

VPC examples from nmvpc2.bat

3. Logarithmic Y-axis; Independent variable scale transformed from hours to weeks

4. Independent variable AGE

5. Independent variable CRCA (creatinine clearance)

6. Independent variable AJBW (adjusted body weight)

7. Independent variable TAD (time after dose)

8. Independent variable WT (weight)

VPC examples from nmvpc3.bat

1. Covariate SEX=0 (female)

2. Covariate SEX=1 (male)

Holford, N. H. G. (2005). "The visual predictive check – superiority to standard diagnostic (Rorschach) plots www.page-meeting.org/?abstract=738. Last accessed 13 Feb 2019." PAGE 14.

Karlsson, M. O. and N. H. G. Holford (2008). "A Tutorial on Visual Predictive Checks." PAGE 17 (2008) Abstr 1434 [www.page-meeting.org/?abstract=1434] (last accessed 11 February 2012).

Nguyen, T. H., M. S. Mouksassi, N. Holford, N. Al-Huniti, I. Freedman, A. C. Hooker, J. John, M. O. Karlsson, D. R. Mould, J. J. Perez Ruixo, E. L. Plan, R. Savic, J. G. van Hasselt, B. Weber, C. Zhou, E. Comets, F. Mentre and C. Model Evaluation Group of the International Society of Pharmacometrics Best Practice (2017). "Model Evaluation of Continuous Data Pharmacometric Models: Metrics and Graphics." CPT Pharmacometrics Syst Pharmacol 6(2): 87-109.

Siebinga, H., F. Robb and A. H. Thomson (2020). "Population pharmacokinetic evaluation and optimization of amikacin dosage regimens for the management of mycobacterial infections." J Antimicrob Chemother 75(10): 2933-2940.