%\VignetteIndexEntry{parody: parametric and resistant outlier detection}
%\VignetteDepends{parody}
%\VignetteKeywords{outliers}
%\VignettePackage{parody}


%
% NOTE -- ONLY EDIT THE .Rnw FILE!!!  The .tex file is
% likely to be overwritten.
%
\documentclass[12pt]{article}

\usepackage{amsmath,pstricks}
\usepackage[authoryear,round]{natbib}
\usepackage{hyperref}


\textwidth=6.2in
\textheight=8.5in
%\parskip=.3cm
\oddsidemargin=.1in
\evensidemargin=.1in
\headheight=-.3in

\newcommand{\scscst}{\scriptscriptstyle}
\newcommand{\scst}{\scriptstyle}


\newcommand{\Rfunction}[1]{{\texttt{#1}}}
\newcommand{\Robject}[1]{{\texttt{#1}}}
\newcommand{\Rpackage}[1]{{\textit{#1}}}
\newcommand{\Rmethod}[1]{{\texttt{#1}}}
\newcommand{\Rfunarg}[1]{{\texttt{#1}}}
\newcommand{\Rclass}[1]{{\textit{#1}}}

\textwidth=6.2in

\bibliographystyle{plainnat} 
 
\begin{document}
%\setkeys{Gin}{width=0.55\textwidth}

\title{\texttt{parody}: a package for
\textbf{p}arametric \textbf{a}nd
\textbf{r}esistant \textbf{o}utlier
\textbf{dy}tection}
\author{VJ Carey}

\maketitle

\tableofcontents

\section{Introduction}
Outliers are data points that seem anomalous.
Precise quantitative measurement of anomalousness is
easiest when a parametric probability model
is adopted.  Significant statistical and
computational research has been devoted to devising and
rationalizing criteria for outlyingness that
do not require adoption of a parametric probability
model.  This package provides interfaces to
various methods presented in the literature
of statistical methods for measuring outlyingness
in univariate and multivariate samples.  It is offered
as a resource through bioconductor.org because formal
testing for outliers is a common concern of genome
scale data analysis, in many different contexts.

\section{Univariate samples}

Davies and Gather (1992) present a detailed
study of criteria for outlyingness in
univariate samples.  Inlier boundaries are defined,
with form $(\hat{m}-c(n)\hat{s},
\hat{m} + c(n) \hat{s})$,
where $\hat{m}$ is a location parameter estimate,
$\hat{s}$ is a scale parameter estimate,
and $c(n)$ are multipliers depending on sample size.
Candidates for $m$ are trimmed mean, median, midpoint of shorth;
candidates for $s$ include trimmed
standard deviation, median absolute deviation (MAD),
length of shorth.  The familiar boxplot
outlier labeling rules do not fall
immediately into this framework, as the left and
right inlier boundaries may not be equidistant from
the location parameter value.

This package provides
functions implementing components of
the various labeling rules; users may
mix components to define their own procedures.

<<rune,results=hide>>=
library(parody)
<<echo=FALSE>>=
options(width=56)
@
We define a dataset presented in Rosner (1983).
The data concern tapping frequencies achieved by
children with various exposures to lead.
<<run32>>=
lead <- c(83, 70, 62, 55, 56, 57, 57, 58, 59, 50, 51, 
        52, 52, 52, 54, 54, 45, 46, 48,
        48, 49, 40, 40, 41, 42, 42, 44, 
        44, 35, 37, 38, 38, 34, 13, 14)
@
We can use a boxplot to visualize the distribution.  This
includes an outlier labeling criterion, and the dots beyond
the whiskers are declared to be outlying.
\begin{center}
<<dobo,fig=TRUE>>=
boxplot(lead)
@
\end{center}

The first formal assessment uses the familiar boxplot rules.
For this, we need to supply a scaling function as described
in the manual page.
<<dob1>>=
calout.detect(lead,alpha=.05,method="boxplot", scaling=
function(n,alpha)1.5)
@
We can see that this agrees with the R visualization.

Theoretical work on the calibration of the boxplot
rule is deployed when \texttt{ftype} is set to \texttt{"ideal"}.
See the manual page for references.
<<dobi>>=
calout.detect(lead,alpha=.05,method="boxplot",ftype="ideal")
@

Another procedure that does not fit in the Davies and Gather
framework is the generalized extreme studentized deviate
procedure due to B Rosner.  In this procedure, we prespecify
the number of possibly outlying points $k$ (which may be any
number less than half the sample size), and the $k$ most
extreme studentized deviates are obtained and recorded
from the highest to lowest value.  Then repeated `outward testing'
is conducted using analytic critical values that bound the
rate of false outlier labeling for the entire testing procedure,
whether or not outliers are present.

Here we apply Rosner's procedure:
<<dor>>=
calout.detect(lead,alpha=.05,method="GESD",k=5)
@
It gives results that agree with the standard boxplot.

Davies and Gather characterize a rule based on median
and MAD using scaling functions $c(n)$ that they
associate with Hampel.  This procedure uses the
scaling function defined in their expression (3).
<<dha>>=
calout.detect(lead,alpha=.05,method="medmad",scaling=hamp.scale.3)
@
Finally we use the shorth-based detector.
<<dos>>=
calout.detect(lead,alpha=.05,method="shorth")
@

\section{Multivariate samples}

A sample dataset called \texttt{tcost} is supplied with
the package.  Before visualizing, we compute a default
multivariate outlier criterion, due to Caroni and Prescott,
that generalizes Rosner's GESD:
<<dot>>=
data(tcost)
ostr = mv.calout.detect(tcost)
ostr
@
We see that indices 9 and 21 are flagged.  Let's create
a pairs plot with coloring.
<<dop>>=
thecol = rep("black", nrow(tcost))
thecol[ostr$ind] = "red"
@

\begin{figure}
<<dopr,fig=TRUE>>=
pairs(tcost, col=thecol, pch=19)
@
\end{figure}

Another display of interest employs principal components:
<<dopc>>=
pc = prcomp(tcost)
@

\begin{center}
<<dobip,fig=TRUE>>=
biplot(pc)
@
\end{center}

@

To get a sense of the robustness (or lack thereof)
of the biplot, let us replot after removing the
datapoints that were marked as outliers by the
Caroni and Prescott procedure.
<<dor>>=
ftcost = tcost[-c(9,21),]
fpc = prcomp(ftcost)
@

\begin{center}
<<rfig,fig=TRUE>>=
biplot(fpc)
@
\end{center}


\end{document}