%\VignetteIndexEntry{2. Introduction to BatchtoolsParam}
%\VignetteKeywords{parallel, Infrastructure}
%\VignettePackage{BiocParallel}
%\VignetteEngine{knitr::knitr}

\documentclass{article}

<<style, eval=TRUE, echo=FALSE, results="asis">>=
BiocStyle::latex()
@

<<setup, echo=FALSE>>=
suppressPackageStartupMessages({
    library(BiocParallel)
})
@

\newcommand{\BiocParallel}{\Biocpkg{BiocParallel}}

\title{Introduction to \emph{BatchtoolsParam}}
\author{
  Nitesh Turaga\footnote{\url{Nitesh.Turaga@RoswellPark.org}}, 
  Martin Morgan\footnote{\url{Martin.Morgan@RoswellPark.org}}
}
\date{Edited: March 22, 2018; Compiled: \today}

\begin{document}

\maketitle

\tableofcontents

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The \Rcode{BatchtoolsParam} class is an interface to the
\CRANpkg{batchtools} package from within \BiocParallel{}. This aims to
replace BatchjobsParam as \BiocParallel{}'s class for computing on a
high performance cluster such as SGE, TORQUE, LSF, SLURM, OpenLava.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Quick start}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

This example demonstrates the easiest way to launch a 100000 jobs
using batchtools. The first step involves creating a
\Rcode{BatchtoolsParam} class. You can compute using 'bplapply' and
then the result is stored.

<<biocLite>>=
library(BiocParallel)

## Pi approximation
piApprox <- function(n) {
    nums <- matrix(runif(2 * n), ncol = 2)
    d <- sqrt(nums[, 1]^2 + nums[, 2]^2)
    4 * mean(d <= 1)
}

piApprox(1000)

## Apply piApprox over
param <- BatchtoolsParam()
result <- bplapply(rep(10e5, 10), piApprox, BPPARAM=param)
mean(unlist(result))
@

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{\emph{BatchtoolsParam} interface}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The \Rcode{BatchtoolsParam} interface allows you to replace the
BatchJobsParam interface, and allow more intuitive usage of your high
performance cluster with \BiocParallel{}.

The \Rcode{BatchtoolsParam} class allows the user to specify many
arguments to customize their jobs. Applicable to clusters with formal
schedulers.

\begin{itemize}

  \item{workers}{The number of workers used by the job.}

  \item{cluster}
    We currently support, SGE, SLURM, LSF, TORQUE and
    OpenLava. The 'cluster' argument is supported only if the R
    environment knows how to find the job scheduler. Each cluster type
    uses a template to pass the job to the scheduler. If the template is
    not given we use the default templates as given in the 'batchtools'
    package. The cluster can be accessed by 'bpbackend(param)'.

  \item{registryargs} 
    The 'registryargs' argument takes a list of arguments to create a
    new job registry for you \Rcode{BatchtoolsParam}. The job registry
    is a data.table which stores all the required information to
    process your jobs. The arguments we support for registryargs are,

    'file.dir': Path where all files of the registry are saved. Note that
    some templates do not handle relative paths well. If nothing is given,
    a temporary directory will be used in your current working directory.

    'work.dir': Working directory for R process for running jobs.

    'packages': Packages that will be loaded on each node.

    'namespaces': Namespaces that will be loaded on each node.

    'source': Files that are sourced before executing a job.

    'load': Files that are loaded before executing a job.

<<<>>=
registryargs <- batchtoolsRegistryargs(
    file.dir = "mytempreg",
    work.dir = getwd(),
    packages = character(0L),
    namespaces = character(0L),
    source = character(0L),
    load = character(0L)
)
param <- BatchtoolsParam(registryargs = registryargs)
param
@

\item{template}

  The template argument is unique to the \Rcode{BatchtoolsParam} class. It is
  required by the job scheduler. It defines how the jobs are submitted
  to the job scheduler. If the template is not given and the cluster
  is chosen, a default template is selected from the batchtools
  package. We recommend that the user define a template which works on
  their cluster, and supply a path to the template argument.

\item{log}

  The log option is logical, TRUE/FALSE. If it is set to TRUE, then
  the logs which are in the registry are copied to directory given by
  the user using the \Rcode{logdir} argument.

\item{logdir}

  Path to the logs. It is given only if \Rcode{log=TRUE}.

\item{resultdir}

  Path to the directory is given when the job has files to be saved in
  a directory.

\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Use cases}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

As an example for a BatchtoolParam job being run on an SGE cluster, we
use the same \Rcode{piApprox} function as defined earlier. The example
runs the function on 5 workers and submits 100 jobs to the SGE
cluster.

Example of SGE with minimal code:

<<simple_sge_example, eval=FALSE>>=
library(BiocParallel)

## Pi approximation
piApprox <- function(n) {
    nums <- matrix(runif(2 * n), ncol = 2)
    d <- sqrt(nums[, 1]^2 + nums[, 2]^2)
    4 * mean(d <= 1)
}

template <- system.file(
    package = "BiocParallel",
    "unitTests", "test_script", "test-sge-template.tmpl"
)
param <- BatchtoolsParam(workers=5, cluster="sge", template=template)

## Run parallel job
result <- bplapply(rep(10e5, 100), piApprox, BPPARAM=param)
@

Example of SGE demonstrating some of \Rcode{BatchtoolsParam} methods.

<<demo_sge, eval=FALSE>>=
library(BiocParallel)

## Pi approximation
piApprox <- function(n) {
    nums <- matrix(runif(2 * n), ncol = 2)
    d <- sqrt(nums[, 1]^2 + nums[, 2]^2)
    4 * mean(d <= 1)
}

template <- system.file(
    package = "BiocParallel",
    "unitTests", "test_script", "test-sge-template.tmpl"
)
param <- BatchtoolsParam(workers=5, cluster="sge", template=template)

## start param
bpstart(param)

## Display param
param

## To show the registered backend
bpbackend(param)

## Register the param
register(param)

## Check the registered param
registered()

## Run parallel job
result <- bplapply(rep(10e5, 100), piApprox)

bpstop(param)
@

\section{\Rcode{sessionInfo()}}

<<sessionInfo, results="asis">>=
toLatex(sessionInfo())
@

\end{document}