In this document, we show that the Wilcoxon-Mann-Whitney test is comparable or superior to alternative methods.
Two alternative methods could be compared with the Wilcoxon-Mann-Whitney (WMW) test proposed by BioQC: the Kolmogorov-Smirnov (KS) test, and the Student’s t-test, or more particularly, the Welch’s test which does not assume equal sample number or equal variance, which is appropriate in the setting of gene expression studies.
Based on these considerations, BioQC implements a computationally efficient version of the WMW test. In order not to confuse end-users, no alternative methods are implemented.
Nevertheless, in order to demonstrate the power of WMW test in comparison with the KS-test or the t-test, we performed the sensitivity benchmark described in the simulation studies, for the two alternative tests respectively.
Figure 1: Sensitivity benchmark. Expression levels of genes in the ovary signature are dedicately sampled randomly from normal distributions with different mean values. The lines show the enrichment score for the Wilcoxon-Mann-Whitney test, the t-test and the Kolmogorov-Smirnov test respectively. In the right panel, outliers were added by adding a random value to 1% of the simulated genes.
As expected, the results suggest, that both the KS-test and the WMW-test are robust to noise, while the performance of the t-test drops significantly on noisy data. Additionally, the WMW-test appears to be superior to the KS-test for low expression differences.
Since the KS-test is so slow, we did not replicate the sensitivity benchmark from the simulation studies using the enrichment score rank. While it takes BioQC about 3 seconds on a single thread to test all 155 signatures, it already takes the KS-test about 2 seconds to test a single signature.
## test replications elapsed relative
## 2 runKS() 5 11.510 1.000
## 1 runWMW() 5 17.175 1.492## R version 4.3.1 (2023-06-16)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 22.04.3 LTS
##
## Matrix products: default
## BLAS: /home/biocbuild/bbs-3.18-bioc/R/lib/libRblas.so
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_GB LC_COLLATE=C
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## time zone: America/New_York
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats4 stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] ggplot2_3.4.4 plyr_1.8.9 reshape2_1.4.4
## [4] hgu133plus2.db_3.13.0 rbenchmark_1.0.0 gplots_3.1.3
## [7] gridExtra_2.3 latticeExtra_0.6-30 lattice_0.22-5
## [10] org.Hs.eg.db_3.18.0 AnnotationDbi_1.64.0 IRanges_2.36.0
## [13] S4Vectors_0.40.0 testthat_3.2.0 limma_3.58.0
## [16] RColorBrewer_1.1-3 BioQC_1.30.0 Biobase_2.62.0
## [19] BiocGenerics_0.48.0 knitr_1.44
##
## loaded via a namespace (and not attached):
## [1] tidyselect_1.2.0 farver_2.1.1 dplyr_1.1.3
## [4] blob_1.2.4 Biostrings_2.70.0 bitops_1.0-7
## [7] fastmap_1.1.1 RCurl_1.98-1.12 digest_0.6.33
## [10] lifecycle_1.0.3 statmod_1.5.0 KEGGREST_1.42.0
## [13] RSQLite_2.3.1 magrittr_2.0.3 compiler_4.3.1
## [16] rlang_1.1.1 sass_0.4.7 tools_4.3.1
## [19] utf8_1.2.4 yaml_2.3.7 labeling_0.4.3
## [22] interp_1.1-4 bit_4.0.5 pkgload_1.3.3
## [25] KernSmooth_2.23-22 withr_2.5.1 desc_1.4.2
## [28] grid_4.3.1 fansi_1.0.5 caTools_1.18.2
## [31] colorspace_2.1-0 edgeR_4.0.0 scales_1.2.1
## [34] gtools_3.9.4 cli_3.6.1 rmarkdown_2.25
## [37] crayon_1.5.2 generics_0.1.3 httr_1.4.7
## [40] DBI_1.1.3 cachem_1.0.8 stringr_1.5.0
## [43] zlibbioc_1.48.0 XVector_0.42.0 vctrs_0.6.4
## [46] jsonlite_1.8.7 bit64_4.0.5 jpeg_0.1-10
## [49] locfit_1.5-9.8 jquerylib_0.1.4 glue_1.6.2
## [52] stringi_1.7.12 gtable_0.3.4 GenomeInfoDb_1.38.0
## [55] deldir_1.0-9 munsell_0.5.0 tibble_3.2.1
## [58] pillar_1.9.0 htmltools_0.5.6.1 brio_1.1.3
## [61] GenomeInfoDbData_1.2.11 R6_2.5.1 rprojroot_2.0.3
## [64] evaluate_0.22 png_0.1-8 memoise_2.0.1
## [67] bslib_0.5.1 Rcpp_1.0.11 xfun_0.40
## [70] pkgconfig_2.0.3