---
title: "Netbenchmark"
author: "Pau Bellot, Catharina Olsen, Patrick Meyer"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Netbenchmark}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}
---

In the last decade, several methods have tackled the challenge of 
reconstructing gene regulatory networks from gene expression data. 
Several papers have compared and evaluated the different network 
inference methods relying on simulated data.

This is a new comparison that assesses different methods in a 
high-heterogeneity data scenario which could reveal the specialization 
of methods for the different network types and data.

This package allows repeating the comparison between different network
inference algorithms with only one line of code.

This package allows replication this comparison between the different 
networks inference algorithms with only one line of code. 

Toy example for main benchmark:
```{r  main_benchmark}
    library(netbenchmark)
    top20.aupr <- netbenchmark(methods="all",datasources.names = "Toy",
                               local.noise=20,global.noise=10,
                               noiseType=c("normal","lognormal"),
                               datasets.num = 2,experiments = 40,
                               seed=1422976420,verbose=FALSE)

```
The first element of the returned list is the $AUPR_{20}$:
```{r print}
   print(top20.aupr[[1]])
```

The package provides an easy way to compare new techniques with  
state-of-the-art ones and to make new different benchmarks in the future.

First, define the wrapper functions:
```{r wrapper_examples}
    Spearmancor <- function(data){
        cor(data,method="spearman")
    }

    Pearsoncor <- function(data){
        cor(data,method="pearson")
   }
```

Note that the wrapper function returns a matrix which is the weighted 
adjacency matrix of the network inferred by the algorithm and that the
columns and rows are named.

Evaluate five times these two simple inference methods with syntren300 
datasource:
```{r evaluate_grn}
    res <- netbenchmark(datasources.names="syntren300",
        methods=c("Spearmancor","Pearsoncor"),verbose=FALSE)
    aupr <- res[[1]][,-(1:2)]
```
Make a boxplot of the $AUPR_{20}$ results:
```{r fig.width=7, fig.height=6}
    boxplot(aupr, main="Syntren300",ylab=expression('AUPR'[20]))
```

Plot the mean Precision-Recall curves:
```{r fig.width=7, fig.height=6}
    PR <- res[[5]][[1]]
    col <- rainbow(3)
    plot(PR$rec[,1],PR$pre[,1],type="l",lwd=3,col=col[1],xlab="Recall",
        ylab="Precision",main="Syntren300",xlim=c(0,1),ylim=c(0,1))
    lines(PR$rec[,2],PR$pre[,2],type="l",lwd=3,col=col[2])
    lines(PR$rec[,3],PR$pre[,3],type="l",lwd=3,col=col[3])
    legend("topright", inset=.05,title="Method",colnames(PR$rec),fill=col)
```

We can also compare these two simple inference methods with the fast network 
inference algorithms using syntren300 datasource:
```{r compare_grn}
    comp <- netbenchmark(datasources.names="syntren300",
        methods=c("all.fast","Spearmancor","Pearsoncor"),verbose=FALSE)
    aupr <- comp[[1]][,-(1:2)]
```

Make a boxplot the $AUPR_{20}$ results:
```{r fig.width=7, fig.height=6}
    #make the name look prety
    library("tools")
    colnames(aupr) <- sapply(colnames(aupr),file_path_sans_ext)
    boxplot(aupr, main="Syntren300", ylab=expression('AUPR'[20]))
```