---
title: "Utilities for handling droplet-based single-cell RNA-seq data"
author: "Aaron Lun"
package: DropletUtils
output: 
  BiocStyle::html_document
vignette: >
  %\VignetteIndexEntry{Utilities for handling droplet-based single-cell RNA-seq data}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}  
---


```{r, echo=FALSE, results="hide", message=FALSE}
require(knitr)
opts_chunk$set(error=FALSE, message=FALSE, warning=FALSE)
```

```{r setup, echo=FALSE, message=FALSE}
library(DropletUtils)
```

# Introduction

Droplet-based single-cell RNA sequencing (scRNA-seq) technologies allow researchers to obtain transcriptome-wide expression profiles for thousands of cells at once.
Briefly, each cell is encapsulated in a droplet in a oil-water emulsion, along with a bead containing reverse transcription primers with a unique barcode sequence.
After reverse transcription inside the droplet, each cell's cDNA is labelled with that barcode (referred to a "cell barcode").
Bursting of the droplets yields a pool of cDNA for library preparation and sequencing.
Debarcoding of the sequences can then be performed to obtain the expression profile for each cell.

This package implements some general utilities for handling these data after quantification of expression.
In particular, we focus on the 10X Genomics platform, providing functions to load in the matrix of unique molecule identifier (UMI) counts as well as the raw molecule information.
Functions are also available for downsampling the UMI count matrix or the raw reads;
for distinguishing cells from empty droplets, based on the UMI counts;
and to eliminate the effects of barcode swapping on Illumina 4000 sequencing machine.

# Reading in 10X Genomics data

## From the UMI count matrix

The _CellRanger_ pipeline from 10X Genomics will process the raw sequencing data and produce a matrix of UMI counts.
Each row of this matrix corresponds to a gene, while each column corresponds to a cell barcode.
This is saved in a single directory for each sample, usually named like `<OUTPUT>/outs/filtered_gene_bc_matrices/<GENOME>`^[If you use the "filtered" matrix, each column corresponds to a putative cell. If you use the "raw" matrix, all barcodes are loaded, and no distinction is made between cells and empty droplets.].
We mock up an example directory below using some simulated data:

```{r}
# To generate the files.
example(write10xCounts, echo=FALSE) 
dir.name <- tmpdir
list.files(dir.name)
```

The `matrix.mtx` file contains the UMI counts, while the other two files contain the cell barcodes and the gene annotation.
We can load this into memory using the `read10xCounts` function, which returns a `SingleCellExperiment` object containing all of the relevant information.
This includes the barcode sequence for each cell (column), as well as the identifier and symbol for each gene (row).

```{r}
sce <- read10xCounts(dir.name)
sce
```

The counts themselves are loaded as a sparse matrix, specifically a `dgCMatrix` from the `r CRANpkg("Matrix")` package.
This reduces memory usage by only storing the non-zero counts, which is useful for sparse scRNA-seq data with lots of dropouts.

```{r}
class(counts(sce))
```

Users can also load multiple samples at once by supplying a character vector to `read10xCounts`.
This will return a single `SingleCellExperiment` where all of the individual matrices are combined by column.
Obviously, this only makes sense when the same set of genes is being used across samples.

## From the molecule information file

_CellRanger_ will also produce a molecule information file (`molecule_info.h5`) that contains... well, information about the transcript molecules.
This includes the UMI sequence^[For readers who are unfamiliar with UMIs, they allow reads from different PCR amplicons to be unambiguously assigned to the same original molecule.], the cell barcode sequence, the gene to which it was assigned, and the number of reads covering the molecule.
For demonstration purposes, we create an example molecule information file below:

```{r}
set.seed(1000)
mol.info.file <- DropletUtils:::sim10xMolInfo(tempfile())
mol.info.file
```

We can subsequently load this information into our R session using the `read10xMolInfo` function:

```{r}
mol.info <- read10xMolInfo(mol.info.file)
mol.info
```

This information can be useful for quality control purposes, especially when the underlying read counts are required, e.g., to investigate sequencing saturation.
Note that the function will automatically guess the length of the barcode sequence, as this is not formally defined in the molecule information file.
For most experiments, the guess is correct, but users can force the function to use a known barcode length with the `barcode.length` argument.

# Downsampling across batches

## From the count matrix

Given multiple batches of very different sequencing depths, it can be beneficial to downsample the deepest batches to match the coverage of the shallowest batches. 
This avoids differences in technical noise that can drive clustering by batch.
We can achieve this using the `downsampleMatrix` function on the count matrix:

```{r}
set.seed(100)
new.counts <- downsampleMatrix(counts(sce), prop=0.5)
library(Matrix)
colSums(counts(sce))
colSums(new.counts)
```

The above code will downsample the counts for each cell such that the total count is halved.
Calculation of an appropriate `prop` is the responsibility of the user, depending on the number of batches in their experiment and which ones have the lowest coverage.
You can also downsample with cell-specific proportions by supplying a vector to `prop`.

## From the reads

Technically, downsampling on the reads is more appropriate as it recapitulates the effect of differences in sequencing depth per cell.
This can be achieved by applying the `downsampleReads` function to the molecule information file containing the read counts:

```{r}
set.seed(100)
no.sampling <- downsampleReads(mol.info.file, prop=1)
sum(no.sampling)
with.sampling <- downsampleReads(mol.info.file, prop=0.5)
sum(with.sampling)
```

The above code will downsample the **reads** to 50% of the original coverage across the experiment.
However, the function will return a matrix of **UMI counts**, so the final total count may not actually decrease if sequencing saturation is high!
Users should use `downsampleMatrix` instead if they want to guarantee similar total counts after downsampling.

# Computing barcode ranks

A useful diagnostic for droplet-based data is the barcode rank plot, which shows the (log-)total UMI count for each barcode on the y-axis and the (log-)rank on the x-axis.
This is effectively a transposed empirical cumulative density plot with log-transformed axes.
It is useful as it allows users to examine the distribution of total counts across barcodes, focusing on those with the largest counts.
To demonstrate, let us mock up a count matrix:

```{r}
set.seed(0)
my.counts <- DropletUtils:::simCounts()
```

We compute the statistics using the `barcodeRanks` function, and then create the plot as shown below.

```{r}
br.out <- barcodeRanks(my.counts)

# Making a plot.
plot(br.out$rank, br.out$total, log="xy", xlab="Rank", ylab="Total")
o <- order(br.out$rank)
lines(br.out$rank[o], br.out$fitted[o], col="red")

abline(h=metadata(br.out)$knee, col="dodgerblue", lty=2)
abline(h=metadata(br.out)$inflection, col="forestgreen", lty=2)
legend("bottomleft", lty=2, col=c("dodgerblue", "forestgreen"), 
    legend=c("knee", "inflection"))
```

The knee and inflection points on the curve mark the transition between two components of the total count distribution.
This is assumed to represent the difference between empty droplets with little RNA and cell-containing droplets with much more RNA,
though a more rigorous method for distinguishing between these two possibilities is discussed below.

# Detecting empty droplets

Empty droplets often contain RNA from the ambient solution, resulting in non-zero counts after debarcoding.
The `emptyDrops` function is designed to distinguish between empty droplets and cells.
It does so by testing each barcode's expression profile for significant deviation from the ambient profile.
Given a matrix `my.counts` containing UMI counts for **all** barcodes, we call:

```{r}
set.seed(100)
e.out <- emptyDrops(my.counts)
e.out
```

Droplets with significant deviations from the ambient profile are detected at a specified FDR threshold, e.g., with `FDR` below 1%.
These can be considered to be cell-containing droplets, with a frequency of false positives (i.e., empty droplets) at the specified FDR.
Furthermore, droplets with very large counts are automatically retained by setting their _p_-values to zero.
This avoids discarding droplets containing cells that are very similar to the ambient profile.

```{r}
is.cell <- e.out$FDR <= 0.01
sum(is.cell, na.rm=TRUE)
```

The _p_-values are calculated by permutation testing, hence the need to set a seed.
The `Limited` field indicates whether a lower p-value could be obtained by increasing the number of permutations.
If there are any entries with `FDR` above the desired threshold and `Limited==TRUE`, it indicates that `npts` should be increased in the `emptyDrops` call.

```{r}
table(Limited=e.out$Limited, Significant=is.cell)
```

We recommend making some diagnostic plots such as the total count against the negative log-probability. 
Droplets detected as cells should show up with large negative log-probabilities _or_ very large total counts (based on the knee point reported by `barcodeRanks`).
Note that the example below is based on simulated data and is quite exaggerated.

```{r}
plot(e.out$Total, -e.out$LogProb, col=ifelse(is.cell, "red", "black"),
    xlab="Total UMI count", ylab="-Log Probability")
```

# Removing the effect of barcode swapping

Barcode swapping is a phenomenon that occurs upon multiplexing samples on the Illumina 4000 sequencer.
Molecules from one sample are incorrectly labelled with _sample_ barcodes from another sample, resulting in their misassignment upon demultiplexing.
Fortunately, droplet experiments provide a unique opportunity to eliminate this effect,
by assuming that it is effectively impossible to generate multiple molecules with the same combination of cell barcode, assigned gene and UMI sequence.
Thus, any molecules with the same combination across multiple samples are likely to arise from barcode swapping.

The `swappedDrops` function will identify overlapping combinations in the molecule information files of all multiplexed 10X samples sequenced on the same run.
It will remove these combinations and return "cleaned" UMI count matrices for all samples to use in downstream analyses.
To demonstrate, we mock up a set of molecule information files for three multiplexed 10X samples:

```{r}
set.seed(1000)
mult.mol.info <- DropletUtils:::sim10xMolInfo(tempfile(), nsamples=3)
mult.mol.info
```

We then apply `swappedDrops` to these files to remove the effect of swapping in our count matrices.

```{r}
s.out <- swappedDrops(mult.mol.info, min.frac=0.9)
length(s.out$cleaned)
class(s.out$cleaned[[1]])
```

For combinations where 90% of the reads belong to a single sample, the molecule is assigned to that sample rather than being removed.
This assumes that swapping is relatively rare, so that the read count should be highest in the sample of origin.
The exact percentage can be tuned by altering `min.frac` in the `swappedDrops` call.

# Session information

```{r}
sessionInfo()
```