---
title: "Exploring Independence"
author: "R.W. Oldford"
date: "August 16, 2018"
bibliography: eikosograms.bib
header-includes:
- \usepackage{graphicx}
- \usepackage{epic}
- \usepackage{color}
- \usepackage{hyperref}
- \usepackage{multimedia}
- \PassOptionsToPackage{pdfmark}{hyperref}\RequirePackage{hyperref}
- \pgfdeclareimage[height=0.12\textheight]{university-logo}{../../UWlogo.png}
- \logo{\pgfuseimage{university-logo}}
- \newcommand{\code}[1]{\texttt{#1}}
- \newcommand{\ve}[1]{\mathbf{#1}}
- \newcommand{\pop}[1]{\mathcal{#1}}
- \newcommand{\samp}[1]{\mathcal{#1}}
- \newcommand{\subspace}[1]{\mathcal{#1}}
- \newcommand{\sv}[1]{\boldsymbol{#1}}
- \newcommand{\sm}[1]{\boldsymbol{#1}}
- \newcommand{\tr}[1]{{#1}^{\mkern-1.5mu\mathsf{T}}}
- \newcommand{\abs}[1]{\left\lvert ~{#1} ~\right\rvert}
- \newcommand{\size}[1]{\left\lvert {#1} \right\rvert}
- \newcommand{\norm}[1]{\left|\left|{#1}\right|\right|}
- \newcommand{\field}[1]{\mathbb{#1}}
- \newcommand{\Reals}{\field{R}}
- \newcommand{\Integers}{\field{Z}}
- \newcommand{\Naturals}{\field{N}}
- \newcommand{\Complex}{\field{C}}
- \newcommand{\Rationals}{\field{Q}}
- \newcommand{\widebar}[1]{\overline{#1}}
- \newcommand{\wig}[1]{\tilde{#1}}
- \newcommand{\bigwig}[1]{\widetilde{#1}}
- \newcommand{\leftgiven}{~\left\lvert~}
- \newcommand{\given}{~\vert~}
- \newcommand{\and}{~\&~}$
- \newcommand{\indep}{\bot\hspace{-.5em}\bot}
- \newcommand{\notindep}{\bot\hspace{-.5em}\bot\hspace{-0.85em} \times }
- \newcommand{\depend}{\Join}
- \newcommand{\notdepend}{\Join\hspace{-0.9 em}/\hspace{.4em}}
- \newcommand{\imply}{\Longrightarrow}
- \newcommand{\notimply}{\Longrightarrow \hspace{-1.5em}/ \hspace{0.8em}}
- \newcommand*{\intersect}{\cap}
- \newcommand*{\union}{\cup}
- \DeclareMathOperator*{\argmin}{arg\,min}
- \DeclareMathOperator*{\argmax}{arg\,max}
- \DeclareMathOperator*{\Ave}{Ave\,}
- \newcommand{\permpause}{\pause}
- \newcommand{\suchthat}{~:~}
- \newcommand{\st}{~:~} 
output:
  html_document:
    number_sections: no
    toc: yes
    toc_depth: 4
  html_notebook: default
  pdf_document:
    keep_tex: yes
    latex_engine: xelatex
    number_sections: no
    toc: yes
    toc_depth: 4
  html_vignette:
    number_sections: no
    toc: yes
    toc_depth: 4
  word_document: default
vignette: >
  %\VignetteIndexEntry{Exploring independence relations between two and three variates} 
  %\VignetteEncoding{UTF-8}
  %\VignetteDepends{gridExtra}
  %\VignetteEngine{knitr::rmarkdown} 
---
  
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(knitr)
set.seed(12314159)
```

---

$\renewcommand{\tr}[1]{{#1}^{\mkern-1.5mu\mathsf{T}}}$
$\renewcommand{\ve}[1]{\mathbf{#1}}$
$\renewcommand{\sv}[1]{\boldsymbol{#1}}$
$\renewcommand{\pop}[1]{\mathcal{#1}}$
$\renewcommand{\samp}[1]{\mathcal{#1}}$
$\renewcommand{\imply}{\Longrightarrow}$
$\renewcommand{\indep}{\bot\hspace{-.5em}\bot}$
$\renewcommand{\notindep}{\bot\hspace{-.5em}\bot\hspace{-0.85em} \times }$
$\renewcommand{\depend}{\Join}$
$\renewcommand{\notdepend}{\Join\hspace{-0.9 em}/\hspace{.4em}}$
$\renewcommand{\and}{~\&~}$
$\renewcommand{\given}{~\vert~}$
$\renewcommand{\suchthat}{~:~}$
$\renewcommand{\widebar}[1]{\overline{#1}}$
$\renewcommand{\wig}[1]{\tilde{#1}}$
$\renewcommand{\bigwig}[1]{\widetilde{#1}}$
$\renewcommand{\field}[1]{\mathbb{#1}}$
$\renewcommand{\Reals}{\field{R}}$
$\renewcommand{\abs}[1]{\left\lvert ~{#1} ~\right\rvert}$
$\renewcommand{\size}[1]{\left\lvert {#1} \right\rvert}$
$\renewcommand{\tr}[1]{{#1}^{\mkern-1.5mu\mathsf{T}}}$
$\renewcommand{\norm}[1]{\left|\left|{#1}\right|\right|}$
$\renewcommand{\intersect}{\cap}$
$\renewcommand{\union}{\cup}$

```{r libraries}
library(eikosograms)
library(gridExtra)
```


```{r set up conditional independence tables, eval = TRUE, echo = FALSE}
# This function will create all the joint probabilities
# and place them in the appropriate three-way table
create3WayBinaryTable <- function(widths, heights) {
    toprow_eikos <- widths * (1 - heights)
    bottomrow_eikos <- widths * heights
    probs <- array(c(bottomrow_eikos, toprow_eikos),
                  dim = c(2,2,2),
                  dimnames = list(Z = c("z1", "z2"),
                                  X = c("x1", "x2"),
                                  Y = c("y1", "y2")
                  ))
    as.table(probs)
}
# Marginal probabilities (widths) for 
# the two binary conditioning variates
# are given by the xz values,
# and the conditional probabilities (heights)
# for the response variate's first value
# are given by the b values.
# Figure numbers here refer to those figure numbers of
# the original Oldford (2003) article entitled
# "Understanding probabilistic independence and its 
#  modelling via Eikosograms and graphs."
xz1 <- c(2/7, 8/35, 1/7, 12/35)
b1 <- c(4/5, 1/2, 3/10, 7/10)
fig5a_data <-  create3WayBinaryTable(xz1, b1)
b2 <- c(4/5, 1/2, 3/5, 1/5)
fig5b_data <-  create3WayBinaryTable(xz1, b2)
b3 <- c(4/5, 1/2, 4/5, 1/2)
fig5c_data <-  create3WayBinaryTable(xz1, b3)
b4 <- c(7/10, 7/10, 7/10, 7/10)
fig6_data <- create3WayBinaryTable(xz1, b4)
xz2 <- c(10/33, 4/11, 5/33, 2/11)
fig7_data <- create3WayBinaryTable(xz2, b4)
xz3 <- c(1/3, 1/6, 1/3, 1/6)
b5 <- c(7/10, 7/10, 3/10, 3/10)
fig8_data <- create3WayBinaryTable(xz2, b5)
xz4 <- c(2/9, 1/9, 2/9, 4/9)
b6 <- c(2/3, 2/3, 1/6, 1/6)
fig9_data <- create3WayBinaryTable(xz4, b6)
xz5 <- c(1/7, 1/7, 3/7, 2/7)
b7 <- c(1/3, 2/3, 1/4, 1/6)
fig11_data <- create3WayBinaryTable(xz5, b7)
xz6 <- c(1/4, 1/4, 1/4, 1/4)
b8 <- c(3/4, 1/2, 1/4, 1/4)
fig12_data <- create3WayBinaryTable(xz6, b8)
xz7 <- c(1/3, 1/6, 1/3, 1/6)
b9 <- c(2/3, 1/2, 5/6, 1/6)
fig13_data <- create3WayBinaryTable(xz7, b9)
xz8 <- c(1/6, 1/3, 1/6, 1/3)
b10 <- c(1/6, 2/3, 5/6, 1/3)
fig14_data <- create3WayBinaryTable(xz8, b10)

# Create this now for early illustration
cond_indep <- fig5c_data
```

# Independence

In this vignette, we explore independence and conditional independence between two or more categorical variates.  It is recommended that the reader be familiar with the [introduction to eikosograms vignette](introduction.html) before continuing.  Independence, including conditional independence is discussed there as well.

## Of two variates
As discussed in the [introduction to eikosograms vignette](introduction.html), the probabilistic independence of two categorical random variates clearly appears as a **flat eikosogram**.
Two categorical random variates $X$ and $Y$  are distributed **independently** of one another if and only if their corresponding eikosgram is **flat**.

This is because independence is defined by the following relationship
\[Pr(Y \given X) = Pr(Y) \]
so that the conditional probabilities do not change as the value of the conditioning variate ($X$) change.  In fact, the conditional probabilities are identical to the marginal probabilities (as if $X$ were not there).

Equivalently, $X$ and $Y$ are independent if and only if
\[Pr(X \given Y) = \Pr(X). \]


These mathematical equivalences arise as flat eikosograms are flat:
```{r independence Y given X, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "Independence of variates X and Y"}
independenceExample <- as.table(array(c(10, 40, 15, 60),
                        dim = c(2,2),
                        dimnames = 
                        list(X =c("x_1", "x_2"), 
                             Y = c("y_1", "y_2"))))
```

```{r bivariate independence 1, eval = FALSE, echo = TRUE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%"}
eikos("Y", "X", data = independenceExample)
```

```{r png bivariate independence 1, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "Independence of variates Y and X"}
include_graphics("img/IndependenceExploration/indepYX.png")
```

```{r bivariate independence 2, eval = FALSE, echo = TRUE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%"}
eikos("X", "Y", data = independenceExample)
```

```{r png bivariate independence 2, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "Independence of variates X and Y"}
include_graphics("img/IndependenceExploration/indepXY.png")
```

Following @Dawid1979, we concisely denote the independence of $X$ and $Y$ by $X \indep Y$ or, being symmetric, equivalently $Y \indep X$. 
Similarly, when $X$ and $Y$ are known **not** to be independent we write $X \notindep Y$.


**Note**:  In the example both $X$ and $Y$ were chosen to be *binary* random variates.  This was done for two reasons.  

- For simplicity of presentation:  had there been more values of either $X$ or $Y$ there would have just been more columns in the eikosogram for the conditioning variate and more flat layers for the response variate.
- In some problems, we prefer to talk about events rather than random variates. The same diagram can be used to discuss events: e.g. we could define the event `A` as occurring whenever `X = x_1` and not occurring when `X = x_2`; similarly the event `B` would be defined as occurring or not whenever `Y` takes values `y_1` or `y_2` respectively.  

## Conditionally

When we have three variates, say $X$, $Y$, and $Z$, the pairwise independences  $Y \indep X$, $Y \indep Z$,  and $Z \indep X$ can be examined by looking at the corresponding eikosograms. 

But we can also construct eikosograms that have more than one conditioning variate and so examine, for example, the conditional probabilities of $Y$ **given the pair of variates** $X$ **and** $Y$.  Some interesting features may result.

For example, consider the following eikosogram
```{r conditional independence, eval = FALSE, echo = TRUE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%"}
eikos("Y", c("X",  "Z"), data = cond_indep, xaxs = FALSE, yaxs = FALSE)
```

```{r png conditional independence, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "Conditional independence of variates Y and X given Z"}
include_graphics("img/IndependenceExploration/indepYgivenXZ.png")
```
The eikosogram has two flat sections, one when $Z = z1$, the other when $Z=z2$.  

Looking only at the left hand flat first, we see from the eikosogram that
\[Pr(Y \given X = x1 \and Z = z1) = Pr(Y \given X = x2 \and Z = z1)\]
for all values of $Y$.  $Y$ is therefore  **conditionally independent** of $X$  when $Z = z1$, which we denote as
\[ Y \indep X \given (Z = z1). \]
It follows mathematically that
\[Pr(Y \given X \and (Z = z1)) = Pr(Y \given Z = z1)\]
for all values of of $Y$ and $X$.  


### Water container metaphor
This latter relation can also be observed directly from the eikosogram by applying the **water container metaphor** to the left hand flat.  

To do so, think of the eikosogram as a series of columnar containers with a varying amounts of water each, as shown by the bottom blue rectangles (if $Y$ had more values, think of the containers having liquids of various density to give the layers in each column). Were the boundary between two neighbouring containers to be perforated, then the liquid in the two containers would redistribute to find a new level across both containers.  

If, for example, the boundary between the two neighbouring containers at left (between $X = x1$ and $X = x2$ when $Z = z1$) were to be removed, then the resulting level would unchanged.  Removing this boundary removes the effect of changing $X$ when $Z = z1$; that is, we have marginalized over $X$ with the resulting level becoming $Pr(Y = y1 \given Z = z1)$.  The unchanging level for this particular eikosogram implies the previous result, namely
$Pr(Y \given X \and (Z = z1)) = Pr(Y \given Z = z1)$.

The same can be said about the right hand side of the eikosogram where $Z = z2$.  That is, it follows from the mathematics and/or eikosogram that we also have 
\[ Y \indep X \given (Z = z2). \]

Since the conditional independence holds whatever value $Z$ takes, we say more generally that $Y$ and $X$ are conditionally independent of one another given the value of $Z$ and more succinctly write
\[ Y \indep X \given Z \]
to assert that this holds no matter what the values of $Y$, $X$, or $Z$.

Clearly a number of independence relationships are possiblie for any collection of three variates which suggests a broader notation for probabilistic independence might be of some help.


# Independence when there are more than two variates 

This section is a little technical and can be skipped.  The next section focuses on only three variates and is illustrated via eikosograms.

When more than two variates are involved there are numerous ways in which the variates may exhibit some sort of independence, conditional and otherwise.   Moreover, the conditional independence graphs do not capture all of these possibilities  -- see @eikosIndependence for demonstration and alternative graph representations.  

However, the variants can be illustrated symbolically if we only slightly adjust the standard notation, switching from the standard "infix" use of $\indep$ to a "prefix" version.  

### A prefix notation for independence
$X \indep Y$ is the common notation to express the fact that random variates $X$ and $Y$ are probabilistically independent of one another.   

A slight variant of this notation can be used to convey more than pairwise independence.  The idea is to switch to using $\indep$ as a function of any number of variates.  For a pair of variates $\indep(X, Y)$ means, as before, that $X$ and $Y$ are probabilistically independent of one another.  The notation becomes useful when there it has more than two arguments (or variates); it then indicates that all variates are probabilistically independent in every way, a *complete independence* of one another.  This idea is made more precise in this section. 

Strictly speaking, the notation is not necessary to understand the rest of the vignette and readers may skip this subsection and return to it after observing eikosograms for the various possible cases of independence involving three variates.  The notation principally permits these cases to be symbolically described more succinctly.

As an operator on two or more symmetrically treated variables, the function $\indep(X, Y, Z, \ldots)$ is understood as follows:
\begin{eqnarray}
\indep (Y, X)
& \mbox{  means } &
Y \indep X \nonumber \\
\indep (Y, X)\given Z
& \mbox{  means } &
Y \indep X \given Z \nonumber \\
\indep (Y, X, Z)
 & \mbox{  means } & 
\indep (Y, X) \given Z, ~~  
\indep (Y, Z) \given X, ~~  
\indep (X, Z) \given Y,\nonumber\\
& &
\indep (Y, X), ~~  
\indep (Y, Z), ~  
\mbox{ and } ~
\indep (X, Z)    
\mbox{ all hold }
\nonumber \\
& &
\mbox{ or in the original (infix) notation, the following all hold: } ~~ \nonumber \\
 &  & 
Y \indep X  \given Z, ~~ 
Y \indep Z  \given X, ~~
X \indep Z  \given Y, ~~ 
Y \indep X, ~~ 
Y \indep Z, ~
\mbox{ and } ~
X \indep Z   
\nonumber
\end{eqnarray}
When only the binary operation is called
for, either the prefix or infix notation can be used depending on which seems clearer in the context.

As always, conditioning variables or events appear to the right of a vertical line
(if more than one variable is conditioned on, as in @Dawid1979, they can appear listed 
separately within parentheses
after the vertical bar). 

The prefix notation $\indep (Y, \cdots , Z)$ is intended 
to indicate the complete probabilistic independence
of its arguments which, as with pairwise independence, can occur for its
arguments either unconditionally or conditionally given other random variables 
or events. So,
\begin{eqnarray}
\indep (Y, X, Z) \given W
 & \mbox{  means } & 
\indep (Y, X) \given (Z, W),~~  
\indep (Y, Z) \given (X, W),~~  
\indep (X, Z) \given (Y, W),~~  \nonumber \\
& & 
\indep (Y, X) \given W, ~~  
\indep (Y, Z) \given W, ~  
\mbox{ and } ~
\indep (X, Z) \given W   \nonumber \\
& & 
\mbox{ all hold }
\nonumber
\end{eqnarray}
For four variables the recursive definition is
\begin{eqnarray}
\indep (Y, X, Z, W) & ~~\mbox{ means that }~~ &\nonumber \\
&
\indep (Y, X, Z) \given W,&
\indep (Y, X, W) \given Z, ~~  
\indep (Y, Z, W) \given X, ~~  
\indep (X, Z, W) \given Y, ~~ 
 \nonumber \\
& 
\indep (Y, X, Z), &
\indep (Y, X, W), ~~  
\indep (Y, Z, W), ~~ 
\mbox{ and } ~~
\indep (X, Z, W), ~~ 
 \nonumber \\
& 
\mbox{ all hold. }&   \nonumber 
\end{eqnarray}
The extension of the notation to more than four variables is entirely analogous. 
$\notindep (Y, \cdots , Z)$ means that at least one of the independencies on the right
hand side does not hold.

Given the number of independencies that are entailed, the phrase 
\textit{complete} independence seems more evocative of the strength of the
assertion than does the traditional \textit{mutual} independence.  @eikosIndependence studies these relationships in some detail (including linking them to log-linear and graphical models).  

The rest of the vignette explores eikosograms which show the various possible independencies that appear as part of the definition of $\indep(X,Y,Z)$.

# Three variates

In this section we explore the possible independence relations between three variates.  

@eikosIndependence provides some fake data constructed to illustrate the possible independence relationships between three categorical variates.  For each example, there will be a table of joint probabilities from which eikosograms may be determined.  

To simplify the presentation, each variate, $X$, $Y$, and $Z$, will be binary, taking values "x1" or "x2", "y1" or "y2", and  "z1" or "z2", respectively. That way the number of combinations of variate values is reduced to $8$, the number of cells in a complete $2 \times 2 \times 2$  table of the joint probabilities.

A function to create the tables with data in the form given in the Appendix of @eikosIndependence is the following:
```{r table creation function , echo = TRUE, eval = FALSE}
# This function will create all the joint probabilities
# and place them in the appropriate three-way table
create3WayBinaryTable <- function(widths, heights) {
    toprow_eikos <- widths * (1 - heights)
    bottomrow_eikos <- widths * heights
    probs <- array(c(bottomrow_eikos, toprow_eikos),
                  dim = c(2,2,2),
                  dimnames = list(Z = c("z1", "z2"),
                                  X = c("x1", "x2"),
                                  Y = c("y1", "y2")
                  ))
    as.table(probs)
}
```

Here the argument `widths` refers to the marginal joint probabilities of the values of the conditioning variate pairs $(X, Z)$ in the order $(x1, z1)$, $(x1, z2)$, $(x2, z1)$, and $(x2, z2)$.  The widths must sum to 1.  The argument `heights` gives the conditional probabilities of the response variate $Y$ taking value $y1$ for each of value of the conditioning pairs $(X, Y)$ in the same order as `widths`.  There is no restriction on the individual heights except that they be probabilities.

For example, 
```{r some table, echo = TRUE, eval = TRUE}
someTable <-   create3WayBinaryTable(widths =  c(10/35, 8/35, 5/35, 12/35),
                                     heights = c(7/10, 7/10, 7/10, 7/10))
```

produces the table of joint probabilities
```{r cond_indep table, echo = FALSE}
knitr::kable(cond_indep)
```

from which, for example, the following eikosogram can be produced 
(note that left to right in the eikosogram, the values of the conditioning variates change from most to least frequently in the call to `eikos()` according to the order in Which they appear in its argument `x`).

```{r eikos of someTable, eval = FALSE, echo = TRUE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%"}
eikos(y="Y", x = c("Z", "X"), data = someTable)
```

```{r png eikos of someTable, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "What is the independence structure?"}
include_graphics("img/IndependenceExploration/someTableYgivenZX.png")
```
This eikosogram suggests some sort of independence.  Certainly, applying the water container metaphor makes it easy to see the "flat water" in the diagram implies that $Y \indep Z \given X$ and also that $Y \indep X \given Z$.  That all of the levels are identical also allows us to see that any perforation of vertical barriers would have no effect on any of the heights and hence we can conclude that pairwise independencies $\indep(Y, X)$ and $\indep(Y, Z)$ also hold.  

One might be tempted to jump to the conclusion that the flat structure across all values of $X$ and $Z$ implies that $\indep(X,Y,Z)$ also holds (i.e. that $X$, $Y$, and $Z$ are **completely independent**).  This would be a mistake.  What this eikosogram fails to reveal immediately is the **pairwise relationship** between $X$ and $Z$ and what effect it might have, if any, on the remaining conditional relations $X \given (Y, Z)$ and $Z \given (X, Z)$.

It turns our that for this set of probabilities,  the three variables $X$, $Y$, and $Z$  are **not completely independent**. 


```{r conditional independence tables display, echo = FALSE, eval = FALSE}
# Marginal probabilities (widths) for 
# the two binary conditioning variates
# are given by the xz values,
# and the conditional probabilities (heights)
# for the response variate's first value
# are given by the b values.
# Figure numbers here refer to those figure numbers of
# the original Oldford (2003) article entitled
# "Understanding probabilistic independence and its 
#  modelling via Eikosograms and graphs."
xz1 <- c(2/7, 8/35, 1/7, 12/35)
b1 <- c(4/5, 1/2, 3/10, 7/10)
fig5a_data <-  create3WayBinaryTable(xz1, b1)
b2 <- c(4/5, 1/2, 3/5, 1/5)
fig5b_data <-  create3WayBinaryTable(xz1, b2)
b3 <- c(4/5, 1/2, 4/5, 1/2)
fig5c_data <-  create3WayBinaryTable(xz1, b3)
b4 <- c(7/10, 7/10, 7/10, 7/10)
fig6_data <- create3WayBinaryTable(xz1, b4)
xz2 <- c(10/33, 4/11, 5/33, 2/11)
fig7_data <- create3WayBinaryTable(xz2, b4)
xz3 <- c(1/3, 1/6, 1/3, 1/6)
b5 <- c(7/10, 7/10, 3/10, 3/10)
fig8_data <- create3WayBinaryTable(xz2, b5)
xz4 <- c(2/9, 1/9, 2/9, 4/9)
b6 <- c(2/3, 2/3, 1/6, 1/6)
fig9_data <- create3WayBinaryTable(xz4, b6)
xz5 <- c(1/7, 1/7, 3/7, 2/7)
b7 <- c(1/3, 2/3, 1/4, 1/6)
fig11_data <- create3WayBinaryTable(xz5, b7)
xz6 <- c(1/4, 1/4, 1/4, 1/4)
b8 <- c(3/4, 1/2, 1/4, 1/4)
fig12_data <- create3WayBinary Table(xz6, b8)
xz7 <- c(1/3, 1/6, 1/3, 1/6)
b9 <- c(2/3, 1/2, 5/6, 1/6)
fig13_data <- create3WayBinaryTable(xz7, b9)
xz8 <- c(1/6, 1/3, 1/6, 1/3)
b10 <- c(1/6, 2/3, 5/6, 1/3)
fig14_data <- create3WayBinaryTable(xz8, b10)

```

## The set of possibilities

For three variates, say $X$, $Y$, and $Z$, a great many  eikosograms are possibly of interest.

To begin with any one of the three variates can be chosen as the response variate (or vertical axis), with the combinations of the remaining two appearing as conditioning variates.  The order of the conditioning variates (and possibly their values) could be rearranged to produce further variations in the display.  

We also need to consider the eikosograms of every pair of variates **unconditionally**. That is, we need to consider the joint distribution of each pair without any consideration of the third variate (i.e. marginalizing over the third). 

It turns out that **there are only eight substantively different types of independence relationships between three categorical variates**. When all three are binary variates, this is reduced to only seven possibilities.  See @eikosIndependence for details and proofs.  These seven possibilities will be illustrated in turn below.  

### Case 1: All three 3-way diagrams are flat

As was suggested earlier, it is not enough to consider only one of the variates as the response.  While it may be possible to infer some independence structure from one three-way eikosogram, all three must be considered.  If all three 3-way diagrams are flat, then 
 $\indep(X, Y, Z)$ holds and we have **complete independence**.  
 
An example where this occurs is very nearly the same as `someTable` above and is given by the following table of joint probabilities:

```{r complete independence table, echo = TRUE}
complete_indep <- create3WayBinaryTable(widths = c(10/33, 12/33, 5/33, 6/33),
                                        heights = c(7/10, 7/10, 7/10, 7/10))
```

The joint distribution can be shown in three different eikosograms depending on which variate is the response (for simplicity, none of the axis probabilities will be shown)

```{r complete independence code, eval = TRUE, fig.height = 0.1}
eikosY <- eikos(y="Y", x = c("Z", "X"), 
                xaxs = FALSE, yaxs= FALSE,
                data = complete_indep, draw = FALSE)
eikosX <- eikos(y="X", x = c("Z", "Y"), 
                xaxs = FALSE, yaxs= FALSE,
                data = complete_indep, draw = FALSE)
eikosZ <- eikos(y="Z", x = c("X", "Y"), 
                xaxs = FALSE, yaxs= FALSE,
                data = complete_indep, draw = FALSE)
```

These can be displayed in a single figure using the function `grid.arrange()` from the package `gridExtra`.

```{r complete independence arrange 3, eval = FALSE, echo = TRUE, fig.width=11, fig.height=3.5, fig.align="center", out.width="100%"}
grid.arrange(eikosY, eikosX, eikosZ, nrow=1)
```

```{r png complete independence arrange 3, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "Complete independence"}
include_graphics("img/IndependenceExploration/completeIndep.png")
```
Note that each of the three eikosograms is completely flat.  This is the hallmark of complete independence.  In fact, it can be proved that $\indep(X,Y,Z)$ occurs if, and only if, all three of these 3-way eikosograms is flat.

The conditional independencies are obvious from the diagrams, as are the marginal pairwise independencies.  To check the latter we could construct the corresponding histograms.

```{r conditional on one, echo = TRUE, fig.width=9, fig.height=0.1, fig.align="center",  out.width="80%"}
eikosYX <- eikos(y = "Y", x = "X",
                 data = complete_indep, main = "Y | X",
                 xaxs = FALSE, yaxs= FALSE, draw = FALSE)
eikosYZ <- eikos(y = "Y", x = "Z",
                 data = complete_indep, main = "Y | Z",
                 xaxs = FALSE, yaxs= FALSE, draw = FALSE)
eikosXZ <- eikos(y = "X", x = "Z", 
                 data = complete_indep, main = "X | Z",
                 xaxs = FALSE, yaxs= FALSE, draw = FALSE)
```


```{r conditional on one arranged, eval = FALSE, echo = TRUE, fig.width=10, fig.height=3, fig.align="center", out.width="80%"}
grid.arrange(eikosYX, eikosYZ, eikosXZ, nrow = 1) 
```


```{r png conditional on one arranged, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "Pairwise independence"}
include_graphics("img/IndependenceExploration/pairwise.png")
```
For the other cases, it will be more convenient to produce all of these eikosograms at once.
To that end, the following function will put simplified versions of all 3-way and all 2-way eikosograms together in one display.
```{r grid code, eval = TRUE, fig.height = 0.1}
layout3and2way <- function(table) {
    if(length(dimnames(table))!=3) stop("Must be a three-way table")
    varNames <- names(dimnames(table))
    zVar <- varNames[1]
    xVar <- varNames[2]
    yVar <- varNames[3]
    eikosY <- eikos(y = yVar, x = c(zVar, xVar),
                    data = table, main = paste0(yVar, " | ", zVar, "&", xVar),
                    xlabs = FALSE, ylabs = FALSE, xaxs = FALSE, yaxs= FALSE, 
                    draw = FALSE)
    eikosX <- eikos(y = xVar, x = c(zVar, yVar), 
                    data = table, main = paste0(xVar, " | ", zVar, "&", yVar),
                    xlabs = FALSE, ylabs = FALSE, xaxs = FALSE, yaxs= FALSE, 
                    draw = FALSE)
    eikosZ <- eikos(y = zVar, x = c(xVar, yVar), 
                    data = table, main = paste0(zVar, " | ", xVar, "&", yVar),
                    xlabs = FALSE, ylabs = FALSE, xaxs = FALSE, yaxs= FALSE, 
                    draw = FALSE)
    eikosYX <- eikos(y = yVar, x = xVar,
                     data = table, main = paste(yVar, xVar, sep =" | "),
                     xlabs = FALSE, ylabs = FALSE, xaxs = FALSE, yaxs= FALSE, 
                     draw = FALSE)
    eikosYZ <- eikos(y = yVar, x = zVar,
                     data = table, main = paste(yVar, zVar, sep =" | "),
                     xlabs = FALSE, ylabs = FALSE, xaxs = FALSE, yaxs= FALSE, 
                     draw = FALSE)
    eikosXY <- eikos(y = xVar, x = yVar, 
                     data = table, main = paste(xVar, yVar, sep =" | "),
                     xlabs = FALSE, ylabs = FALSE, xaxs = FALSE, yaxs= FALSE, 
                     draw = FALSE)
    eikosXZ <- eikos(y = xVar, x = zVar, 
                     data = table, main = paste(xVar, zVar, sep =" | "),
                     xlabs = FALSE, ylabs = FALSE, xaxs = FALSE, yaxs= FALSE, draw = FALSE)
    eikosZY <- eikos(y = zVar, x = yVar, 
                     data = table, main = paste(zVar, yVar, sep =" | "),
                     xlabs = FALSE, ylabs = FALSE, xaxs = FALSE, yaxs= FALSE, draw = FALSE)
    eikosZX <- eikos(y = zVar, x = xVar,  
                     data = table, main = paste(zVar, xVar, sep =" | "),
                     xlabs = FALSE, ylabs = FALSE, xaxs = FALSE, yaxs= FALSE, 
                     draw = FALSE)
    layout <- rbind(c(1,1, NA, 2, 2, NA,  3, 3),
                    rep(NA, 8),
                    c(4, 5, NA, 6, 7, NA, 8, 9))
    grid.arrange(eikosY, eikosX, eikosZ,
                 eikosYX, eikosYZ, 
                 eikosXY, eikosXZ, 
                 eikosZY, eikosZX,
                 layout_matrix = layout,
                 widths = c(2,2,1,2,2,1,2,2), 
                 heights = c(2, 0.5, 1.1)
    )
}
```

Calling this function on the `complete_indep` table, It is clear that

```{r complete independence layout, eval = FALSE, echo = TRUE, fig.width=11, fig.height=5.5, fig.align="center", out.width="80%"}
layout3and2way(complete_indep)
```

```{r png complete independence layout, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "Complete independence"}
include_graphics("img/IndependenceExploration/complete_indep.png")
```
All eikosograms are flat, indicating the complete independence of all three variates from one another $\indep(X, Y, Z)$

### Case 2: one 4-flat, two 2 by 2-flats

As with the first case, it will be simplest to describe this by the shape of the 3-way eiksograms.  In this case, one has the flar crossing all four conditions, the remaining two have flats crossing the values of one of the conditioning variates and not the others.

An example, and its simplified display is the following

```{r Case 2, eval = FALSE, echo = TRUE, fig.width=11, fig.height=5.5, fig.align="center", out.width="80%"}
table <- create3WayBinaryTable(widths = c(10/33, 12/33, 5/33, 6/33),
                               heights = c(7/10, 7/10, 3/10, 3/10))

layout3and2way(table)
```

```{r png Case2, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "One 4-flat; two 2 by 2 flats"}
include_graphics("img/IndependenceExploration/case2.png")
```
Because not all eikosograms are flat, we can conclude that $\notindep(X, Y, Z)$; there are at least some dependencies amongst the three variates.

Recall that the value of the first conditioning variate mentioned changes fastest left to right across the vertical bars of any one eikosogram; the value of the second named conditioning variate changes more slowly.

Reading the top row of eikosograms from left to right, we see 

- from the first top row eikosogram that 
    - $Y \indep Z \given X$, but that
    - $Y \notindep X \given Z$;
    
- from the second eikosogram in the top row that 
    - $X \indep Z \given Y$, but that
    - $X \notindep Y \given Z$; and
    
- from the third eikosogram in the top row that 
    - $Z \indep Y \given X$ **and** $Z \indep X \given Y$.
    
The second row reveals the joint dependence structure between two variates **un**conditionally, that is each eikosograms shows the joint distribution of the named pair of variates after marginalizing over the remaining third variate. 

Reading the pairs of eikosograms in the second row from left to right we see 

- from the first pair of eikosograms in the second row that 
    - $Y \notindep X$, but
    - $Y \indep Z$;
- from the second pair of eikosograms in the second row that 
    - $X \notindep Y$, but
    - $X \indep Z$;
- from the third pair of eikosograms in the second row that 
    - $Z \indep Y$, and
    - $Z \indep X$.
    
In summary, the only independencies presented are  

- $Z \indep X \given Y$, 
- $Z \indep Y \given X$, 
- $Z \indep X$, and 
- $Z \indep Y$.  

For the remaining cases, only the summary independencies will be provided.

### Case 3: two 2 by 2-flats, one no-flat
This happens when there is only one conditional-independence.
```{r Case 3, eval = FALSE, echo = TRUE, fig.width=11, fig.height=5.5, fig.align="center", out.width="80%"}
table <- create3WayBinaryTable(widths = c(2/9, 1/9, 2/9, 4/9),
                               heights = c(2/3, 2/3, 1/6, 1/6))

layout3and2way(table)
```

```{r png Case 3, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "One no-flat; two 2 by 2 flats"}
include_graphics("img/IndependenceExploration/case3.png")
```
Reading across the eikosograms, there is only a single independence relation of any kind, namely $Y \indep Z \given X$.

### Case 4: three no-flats

Whenever no flat areas appear in the 3-way eikosograms, there are no **conditional** independencies.


#### Case 4.1: No flats; no marginal independence
```{r Case 4.1, eval = FALSE, echo = TRUE, fig.width=11, fig.height=5.5, fig.align="center", out.width="80%"}
table <- create3WayBinaryTable(widths = c(1/7, 1/7, 3/7, 2/7),
                               heights = c(1/3, 2/3, 1/4, 1/6))

layout3and2way(table)
```

```{r png Case 4.1, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "No flats; no marginal independence"}
include_graphics("img/IndependenceExploration/case41.png")
```
There are **no** independencies present.

#### Case 4.2:  No flats; one marginal independence
```{r Case 4.2, eval = FALSE, echo = TRUE, fig.width=11, fig.height=5.5, fig.align="center", out.width="80%"}
table <- create3WayBinaryTable(widths = c(1/4, 1/4, 1/4, 1/4),
                               heights = c(3/4, 1/2, 1/4, 1/4))

layout3and2way(table)
```
```{r png Case 4.2, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "No flats; one marginal independence"}
include_graphics("img/IndependenceExploration/case42.png")
```

Only independence is present is the *marginal* independence $Z \indep X$.

Note however that there is also a conditional indepenence but **only for a single value of the conditioning variate**, namely $Y \indep Z \given (X = x2)$, which can be seen in the first eikosogram of the top row.

#### Case 4.3:  No flats; two marginal independences
```{r Case 4.3, eval = FALSE, echo = TRUE, fig.width=11, fig.height=5.5, fig.align="center", out.width="80%"}
table <- create3WayBinaryTable(widths = c(2/6, 1/6, 2/6, 1/6),
                               heights = c(2/3, 1/2, 5/6, 1/6))

layout3and2way(table)
```

```{r png Case 4.3, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "No flats; two marginal independences"}
include_graphics("img/IndependenceExploration/case43.png")
```
Two marginal independences exist, $X \indep Y$, and $X \indep Z$.

#### Case 4.4:  No flats; three marginal independences
```{r Case 4.4, eval = FALSE, echo = TRUE, fig.width=11, fig.height=5.5, fig.align="center", out.width="80%"}
table <- create3WayBinaryTable(widths = c(1/6, 2/6, 1/6, 2/6),
                               heights = c(1/6, 2/3, 5/6, 1/3))

layout3and2way(table)
```

```{r png Case 4.4, eval = TRUE, echo = FALSE, fig.width=4, fig.height=3.5, fig.align="center", out.width="50%", fig.cap = "No flats; three marginal independences"}
include_graphics("img/IndependenceExploration/case44.png")
```
All variates are pairwise independent but are not mutually independent.  That is,
$X \indep Y$, $Y \indep Z$, and $X \indep Z$ but $\notindep (X, Y, Z)$.  No conditional independence exists.


# References