---
title: "Two Sample Testing Based on The 2-Wasserstein Distance"
output: rmarkdown::html_vignette
vignette: >
    %\VignetteIndexEntry{wasserstein_test}
    %\VignetteEngine{knitr::rmarkdown}
    %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
    collapse = TRUE,
    comment = "#>"
)
```


## Testing Procedures

The package `waddR` provides two testing procedures using the 2-Wasserstein
distance to test whether two distributions $F_A$ and $F_B$ given in the form of
samples are different by specifically testing the null hypothesis
$\mathcal{H}_0: F_A = F_B$ against the alternative
$\mathcal{H}_1: F_A \neq F_B$.


The first, semi-parametric (SP), procedure uses a test based on permutations
combined with a generalized Pareto distribution approximation to estimate
small p-values accurately.


The second procedure (ASY) uses a test based on asymptotic theory which is
valid only if the samples can be assumed to come from continuous distributions.

## Examples

To demonstrate the capabilities of the testing procedures, we consider models
based on normal distributions here.
We exemplarily construct three cases in which two distributions (samples)
differ with respect to location, size, and shape, respectively, and one case
without a difference.
For convenience, we focus on the p-value, the value of the (squared)
2-wasserstein distance, and the fractions of the location, size, and shape
terms (in \%) with respect to the (squared) 2-Wasserstein distance here, while
the functions also provide additional output.


```{r setup}
library(waddR)

spec.output <- c("pval", "d.wass^2", "perc.loc", "perc.size", "perc.shape")
```

We start with an example, in which the two distributions (samples) only differ
with respect to the location, and show the results for the two testing
procedures (semi-parametric (SP) and asymptotic theory-based (ASY)).

Please note that the method "SP" with the permnum 10000 is used by default in
calls to `wasserstein.test` if nothing different is specified.

```{r diff_loc}
set.seed(24)
ctrl <- rnorm(300 ,0 ,1)
set.seed(24)
dd1 <- rnorm(300, 1, 1)
wasserstein.test(ctrl, dd1, method="SP", permnum=10000)[spec.output]
wasserstein.test(ctrl, dd1, method="ASY")[spec.output]
```
We obtain a very low p-value, pointing at the existence of a difference, and
see that differences with respect to location make up by far the most part of
the 2-Wasserstein distance.


Analogously, we look at a case in which the two distributions (samples) only
differ with respect to the size.
```{r diff_size}
set.seed(24)
ctrl <- rnorm(300, 0, 1)
set.seed(24)
dd2 <- rnorm(300, 0, 2)
wasserstein.test(ctrl, dd2)
wasserstein.test(ctrl, dd2, method="ASY")
```


Similarly, we consider an example in which the two distributions (samples) only
differ with respect to the shape.
```{r diff_shape}
set.seed(24)
ctrl <- rnorm(300, 6.5, sqrt(13.25))
set.seed(24)
sam1 <- rnorm(300, 3, 1)
set.seed(24)
sam2 <- rnorm(300, 10, 1)
dd3 <- sapply(seq(1:300), 
              function(n) {
                sample(c(sam1[n], sam2[n]), 1, prob=c(0.5, 0.5))})
wasserstein.test(ctrl, dd3)[spec.output]
wasserstein.test(ctrl, dd3, method="ASY")[spec.output]
```


Finally, we show an example in which there is no difference between the
distributions. We obtain a high p-value, indicating that the null hypothesis
cannot be rejected.
```{r no_diff}
set.seed(24)
ctrl <- rnorm(300, 0, 1)
set.seed(24)
nodd <- rnorm(300, 0, 1)
wasserstein.test(ctrl, nodd)[spec.output]
wasserstein.test(ctrl, nodd, method="ASY")[spec.output]
```

## Reproducibility of Results

Using the method "SP" (default), the semi-parametric test is performed
to obtain p-values.
The permutation procedure is based on a random sampling of the input vectors
and for the results to be reproducible, a seed can be set from the user
environment.
```{r reproduce}
# set seed for reproducible p-values in re-runs
set.seed(123)
wasserstein.test(sam1, nodd)[spec.output]
# reproduce the result
set.seed(123)
wasserstein.test(sam1, nodd)[spec.output]
```

## See Also

* The [`waddR` package](waddR.html)

* Fast and accurate 
[calculation of the  Wasserstein distance](wasserstein_metric.html)

* Detect 
[differential gene expression distributions](wasserstein_singlecell.html)
in scRNAseq data

## Session Info

```{r session-info}
sessionInfo()
```