---
title: "cytoviewer - Interactive multi-channel image visualization in R"
date: "`r BiocStyle::doc_date()`"
package: "`r BiocStyle::pkg_ver('cytoviewer')`"
author:
- name: Lasse Meyer
  affiliation: 
  - Department for Quantitative Biomedicine, University of Zurich
  - Institute for Molecular Health Sciences, ETH Zurich
  email: lasse.meyer@dqbm.uzh.ch
- name: Nils Eling
  affiliation: 
  - Department for Quantitative Biomedicine, University of Zurich
  - Institute for Molecular Health Sciences, ETH Zurich
  email: nils.eling@dqbm.uzh.ch
output:
    BiocStyle::html_document:
        toc_float: yes
bibliography: library.bib
abstract: |
    This R/Bioconductor package supports interactive visualization of multi-channel 
    images and segmentation masks generated by imaging mass cytometry and other 
    highly multiplexed imaging techniques using shiny. The cytoviewer interface 
    is divided into image-level (Composite and Channels) and cell-level 
    visualization (Masks). It allows users to overlay individual images with 
    segmentation masks, integrates well with SingleCellExperiment and 
    SpatialExperiment objects for metadata visualization and 
    supports image downloads.
vignette: |
    %\VignetteIndexEntry{"Interactive multi-channel image visualization in R"}
    %\VignetteEncoding{UTF-8}
    %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  markdown: 
    wrap: 72
---

```{r, echo=FALSE, results="hide"}
knitr::opts_chunk$set(error=FALSE, warning=FALSE, message=FALSE,
                        fig.retina = 0.75, crop = NULL)
library(BiocStyle)
```

`r fontawesome::fa(name = "github", fill = "#333")` <a href="https://github.com/lassedochreden">\@lassedochreden</a>  
`r fontawesome::fa(name = "github", fill = "#333")` <a href="https://github.com/nilseling">\@nilseling</a>

![](imgs/cytoviewer_sticker.png)

```{r library, echo=FALSE}
library(cytoviewer)
```

# Introduction

This vignette introduces the `cytoviewer` package for interactive
multi-channel image visualization. Images as well as corresponding
segmentation masks generated by imaging mass cytometry (IMC) and other
highly multiplexed imaging techniques can be interactively visualized
and explored.

The `cytoviewer` package builds on top of the `r Biocpkg("cytomapper")`
Bioconductor package [@Eling2020] and extends the static visualization strategies
provided by `cytomapper` via an **interactive Shiny application**. The
`cytoviewer` package leverages the image handling, analysis and
visualization strategies provided by the `r Biocpkg("EBImage")`
Bioconductor package and offers interactive image visualization
strategies in a similar fashion as `r Biocpkg("iSEE")` for single-cell
data. In addition, building up on `r Biocpkg("SingleCellExperiment")`,
`r Biocpkg("SpatialExperiment")` and `cytomapper::CytoImageList`
classes, the `cytoviewer` package integrates into the Bioconductor
framework for single-cell and image analysis.

Read the **pre-print** [here](https://doi.org/10.1101/2023.05.24.542115).

## Highly multiplexed imaging

Highly multiplexed imaging allows simultaneous spatially and single-cell
resolved detection of dozens of biological molecules (e.g. proteins) in
their native tissue context. As a result, these technologies allow an
in-depth analysis of complex systems and diseases such as the tumor
microenvironment [@Jackson2020] and type 1 diabetes progression
[@Damond2019].

Imaging-based spatial proteomics methods [@Moffitt2022] can be broadly
divided into fluorescent cyclic approaches such as tissue-based cyclic
immunofluorescence (t-CyCIF) [@Lin2018] and one-step mass-tag based
approaches such as multiplexed ion beam imaging (MIBI) [@Angelo2014] and
IMC [@Giesen2014].

Of note, the instructions below will focus on the visualization and
exploration of **IMC data** as an example. However, data from other
technologies such as t-CyCIF or MIBI, which produce pixel-level
intensities and (optionally) segmentation masks, can be interactively
visualized with `cytoviewer` as long as they have the appropriate input
format (see Section [Data input format](#DataInputFormat)).

### Imaging mass cytometry

IMC, an advancement of CyTOF, combines antibodies tagged with
isotopically pure rare earth metals with laser ablation and
mass-spectrometry-based detection to produce high-dimensional images
[@Giesen2014]. It captures the spatial expression of over 40 proteins in
parallel at a sub-cellular resolution of 1 μm. Thus, IMC is able to
detect cytoplasmic and nuclear localization of proteins.

## Highly multiplexed image analysis

To fully leverage the information contained in IMC and multiplexed
imaging data in general, computational tools are of key importance.

The main analysis steps, irrespective of the biological question,
include 1) Visual inspection of images for quality control, 2) Image
pre-processing and segmentation and 3) Single-cell and spatial analysis
[@Windhager2021].

A comprehensive end-to-end workflow for multiplexed image processing and
analysis with detailed information for every analysis step can be found
[here](https://bodenmillergroup.github.io/IMCDataAnalysis/).

Importantly, the `cytoviewer` package can support, simplify and improve
any of these analysis steps with its easy-to-use interactive
visualization interface in R.

Below we will showcase an [example workflow](#ExampleWorkflow) that
highlights the different functionality and potential application fields
of `cytoviewer`.

## Application overview

The `cytoviewer` interface is broadly divided into
[image-level](#ImageLevel) (Composite and Channels) and
[cell-level](#CellLevel) visualization (Masks). It allows users to
overlay individual images with segmentation masks, integrates well with
`SingleCellExperiment` and `SpatialExperiment` objects for metadata
visualization and supports [image downloads](#Download) **(Figure 2B)**.

![**Figure 1: cytoviewer interface and functionality.** **(A)** The
supported functionality (right) of *cytoviewer* depends on the data
inputs (left). To match information between the objects, cell (cell_id)
and image (img_id) identifiers can be provided. SCE/SPE =
*SingleCellExperiment*/*SpatialExperiment*. **(B)** The graphical user
interface of *cytoviewer* is divided into a body, header, and sidebar.
The body of *cytoviewer* includes the image viewer, which has three
tabs: Composite (Image-level), Channels (Image-level), and Mask
(Cell-level). Zooming is supported for Composite and Mask tabs. The
package version, R session information, help page, and a drop-down menu
for image downloads are located in the header. The sidebar menu has
controls for sample selection, image visualization, mask visualization,
and general settings. Scale bar: 150 µm **(C)** *cytoviewer* supports
different viewing modes. Top: The "channels" tab of image-level
visualization displays individual channels. Shown are Ecad (magenta),
CD8a (cyan), and CD68 (yellow) marking tumor cells, CD8+ T cells, and
myeloid cells, respectively. Center: The "composite" tab of image-level
visualization visualizes image composites combining multiple channels.
These composite images can be overlayed with cell outlines, which can be
colored by cell-specific metadata. Shown here are cell outlines colored
by cell area (continous value) and cell type (categorical value; tumor
cells in white). Channel color settings are as follows for all markers:
Contrast: 2,5; Brightness: 1; Gamma: 1.2. Bottom: The "mask" tab can be
used to visualize segmentation masks that can be colored by
cell-specific metadata. Shown here are segmentation masks colored by
cell area (continuous) and cell type (categorical; tumor cells in
magenta). Scale bars: 150 µm. **(D)** "Image appearance" controls can be
used to add legends or titles and to change the scale bar length for
image-level (top) and cell level (bottom) visualization. The cell-level
mask plot depicts tumor (magenta), myeloid (yellow), and CD8+ T cells
(cyan). Scale bars: 100 µm. Adapted from
[@Meyer2023]](imgs/cytoviewer_overview.png)

### Data input format {#DataInputFormat}

The `cytoviewer` package combines objects of
`r Biocpkg("SingleCellExperiment")`, `r Biocpkg("SpatialExperiment")`
and `cytomapper::CytoImageList` classes (from `r Biocpkg("cytomapper")`)
to visualize image- and cell-level information.

The `cytoviewer` function takes up to **five arguments**.

Firstly, `image` refers to a `CytoImageList` object containing one or
multiple multi-channel images where each channel represents the
pixel-intensities of one marker (proteins in IMC).

Secondly, `mask` refers to a `CytoImageList` object containing one or
multiple segmentation masks. Segmentation masks are defined as
one-channel images containing integer values, which represent the cell
ids or background.

Thirdly, the `object` entry refers to a `SingleCellExperiment` or
`SpatialExperiment` class object that contains cell-specific metadata in
the `colData` slot.

Lastly, to match information between the `CytoImageList` objects and the
`SingleCellExperiment`/`SpatialExperiment` object, two additional spots
can be specified:

-   `img_id`: a single character indicating the `colData` (of the
    `SingleCellExperiment`/`SpatialExperiment` object) and
    `elementMetadata` (of the `CytoImageList` object) entry that
    contains the image identifiers. These image ids have to match
    between the `SingleCellExperiment`/ `SpatialExperiment` object and
    the `CytoImageList` objects.

-   `cell_id`: a single character indicating the `colData` entry that
    contains the cell identifiers. These should be integer values
    corresponding to pixel-values in the segmentation masks.

For more detailed information on the input objects,
please refer to the respective documentation (e.g. the vignettes of
the `r Biocpkg("cytomapper")` or `r Biocpkg("SingleCellExperiment")`/
`r Biocpkg("SpatialExperiment")` packages).

In the [Read in data](#ReadData) section, we provide 
example code to directly read in images and masks (e.g. in .tiff format) 
into a `CytoImageList` object and create a `SingleCellExperiment` object 
from them, which we can then visualize with `cytoviewer`.

### Data input variations {#DataInputVariation}

The functionality of `cytoviewer` depends on which input objects are
user-provided. Below we describe the **four use cases** in respect to
input objects and functionality.

**1. Usage of cytoviewer with images, masks and object**

The *full* functionality of cytoviewer can be leveraged when `image`,
`mask` and `object` are provided, which is the main intended use case.

This allows [image-level](#ImageLevel) visualization (Composite and
Channels), [cell-level](#CellLevel) visualization, overlaying images
with segmentation masks as well as metadata visualization.

**2. Usage of cytoviewer with images only**

If only the `image` object is specified, [image-level](#ImageLevel)
visualization (Composite and Channels) is possible.

**3. Usage of cytoviewer with images and masks**

[Image-level](#ImageLevel) visualization (Composite and Channels),
overlaying of images with segmentation masks and
[cell-level](#CellLevel) visualization is feasible when `image` and
`mask` objects are provided.

**4. Usage of cytoviewer with masks and object**

If `mask` and `object` are specified, [cell-level](#CellLevel)
visualization as well as metadata visualization is possible.

# Example workflow {#ExampleWorkflow}

## Installation

The `cytoviewer` package can be installed from `Bioconductor` via:

```{r bioc-install, eval=FALSE}
if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")

BiocManager::install("cytoviewer")
```

The development version of `cytoviewer` can be installed from Github
via:

```{r devel-install, eval=FALSE}
if (!requireNamespace("remotes", quietly = TRUE))
    install.packages("remotes")

remotes::install_github("BodenmillerGroup/cytoviewer")
```

To load the package in your R session, type the following:

```{r load}
library(cytoviewer)
```

## Example dataset

For visualization purposes, we will use a toy dataset provided by the
`r Biocpkg("cytomapper")` package.

The dataset contains 3 images of $100\mu{m}$ x $100\mu{m}$ dimensions
with 362 segmented cells and pixel-intensities for 5 proteins: H3, CD99,
PIN, CD8a, and CDH. It is a small subset from a Type 1 Diabetes dataset
[@Damond2019].

Pixel-level intensities for all 5 markers (5 channels) are stored in the
`pancreasImages` object.

The corresponding segmentation masks are stored in the `pancreasMasks`
object.

All cell-specific metadata are stored in the `colData` slot of the
corresponding `SingleCellExperiment` object: `pancreasSCE`.

For more detailed information on the dataset, please refer to the
respective documentation (e.g. via `?pancreasImages` or the vignette of
the `r Biocpkg("cytomapper")` package).

We also provide example code to directly read in images and masks 
(e.g. in .tiff format) into a `CytoImageList` object and create a 
`SingleCellExperiment` object from them in the 
[Read in data](#ReadData) section. 

```{r dataset}
# Load example datasets 
library(cytomapper)
data("pancreasImages")
data("pancreasMasks")
data("pancreasSCE")

pancreasImages
pancreasMasks 
pancreasSCE
```

## Function call

Here as an example, we call `cytoviewer` with `image`, `mask` and
`object` data to leverage all provided functionality.

This setting allows [image-level](#ImageLevel) visualization (Composite
and Channels), [cell-level](#CellLevel) visualization, overlaying images
with segmentation masks as well as metadata visualization.

For further details, please refer to the `?cytoviewer` manual or the
`Help page` within the shiny application.

```{r cytoviewer}
# Use cytoviewer with images, masks and object
app <- cytoviewer(image = pancreasImages, 
                  mask = pancreasMasks, 
                  object = pancreasSCE, 
                  img_id = "ImageNb", 
                  cell_id = "CellNb")

if (interactive()) {
  
  shiny::runApp(app, launch.browser = TRUE)

  }
```

## Interface {#Interface}

The `cytoviewer` interface is divided into a **Header**, **Sidebar** and
**Body** section (see Figure below).

The **Header** includes package version information, access to session
information and the help page as well as a dropdown-menu for [image
downloads](#Download).

The **Body** features a Tabset-Panel layout allowing the user to switch
between three image modes: [Image-level](#ImageLevel) (Composite and
Channels) and [Cell-level](#CellLevel) (Mask). Furthermore, the
Composite and Mask tabs have zoom controls.

The **Sidebar** panel is subdivided into four sections: *Sample
selection*, *Image-level*, *Cell-level* and [*General*](#General)
controls.

## Image-level visualization {#ImageLevel}

Image visualization control is split into *basic* and *advanced
controls*.

*Basic controls* supports the selection of up to six markers/channels
for `image` display. Each marker has color control settings that allow
the user to set contrast, brightness, gamma and select a channel color.

![**Figure 2: Image-level visualization - Basic controls**. The graphical user 
interface of cytoviewer for image-level-composite with basic controls. For 
image-level visualization, Ecad (magenta), CD8a (cyan) and CD68 (yellow) 
marking tumor cells, CD8+ T cells and myeloid cells, respectively, are shown 
and channel color settings are as follows for all markers: Contrast: 2,5; 
Brightness: 1; Gamma: 1.2. Note that the Composite tab is zoomable. 
Scale bars: 150 µm. Adapted from [@Meyer2023]](imgs/cytoviewer_image_basic.png)

In the *advanced controls* part, the user can choose to overlay the
displayed images with provided segmentation `masks`. Outline color and
mask thickness can be adjusted by the user. Moreover, the masks can be
outlined by cell-specific metadata provided in `colData` slot of the
`object`.

Of note, for categorical and continuous metadata entries the user can
choose between discrete colors and continuous color palettes (viridis,
inferno, plasma), respectively.

![**Figure 3: Image-level visualization - Advanced controls**. The graphical user 
interface of cytoviewer for image-level-composite with advanced controls. For 
image-level visualization, Ecad (magenta), CD8a (cyan) and CD68 (yellow) 
marking tumor cells, CD8+ T cells and myeloid cells, respectively, are shown 
and channel color settings are as follows for all markers: Contrast: 2,5; 
Brightness: 1; Gamma: 1.2. Note that the Composite tab is zoomable. 
Scale bars: 150 µm. Adapted from [@Meyer2023]](imgs/cytoviewer_image_advanced.png)

## Cell-level visualization {#CellLevel}

Cell visualization has *basic controls*.

Here, the user can choose to display the provided segmentation `masks`.
If an `object` is provided, the masks can be colored by cell-specific
metadata.

Please note again that for categorical and continuous metadata entries
the user can choose between discrete colors and continuous color
palettes (viridis, inferno, plasma), respectively.

![**Figure 4: Cell-level visualization - Basic controls**. The graphical user 
interface of cytoviewer for cell-level-mask with basic controls. For cell-level
visualization, tumor cells (magenta) are highlighted. Note that the Mask tab
is zoomable. Adapted from [@Meyer2023]](imgs/cytoviewer_cells.png)

## General controls {#General}

General controls is subdivided into an *Image appearance* and *Image
filters* part.

In the *Image appearance* section, the user can adjust the scale bar
length and include legend/image titles, while the *Image filters*
section allows to control pixel-wise interpolation (default) and apply a
Gaussian filter.

## Image download {#Download}

The `cytoviewer` package supports fast and uncomplicated image
downloads.

Download controls are part of the **Header** (see Section
[Interface](#Interface)).

The user can specify a file name, select the image of interest
(Composite, Channels, Mask) and the file format (pdf, png). Upon
clicking the download button, a pop-window should appear where the user
can specify the download location.

# Additional Information

## Read in data {#ReadData}

To conveniently read in images and segmentation masks into a `CytoImageList` 
object and then visualize these using `cytoviewer`, 
the `cytomapper` package provides the `loadImages` function. 

The `loadImages` function returns a `CytoImageList` object containing the
multi-channel images or segmentation masks. Refer to the `?loadImages` function
to see the full functionality.

As an example, we will read in multi-channel images and segmentation masks
provided by the `cytomapper` package.  

To correctly scale pixel values of the segmentation masks when reading them in, 
we will need to set `as.is = TRUE`. Users needs to take care that pixel values 
are scaled correctly in more complex cases.

```{r read-in-images}
library(cytomapper)

# Data directory that stores images and masks in tiff format
data_path <- system.file("extdata", package = "cytomapper")

# Read in images
cur_images <- loadImages(data_path, pattern = "_imc.tiff")
cur_images

# Read in masks
cur_masks <- loadImages(data_path, pattern = "_mask.tiff", as.is = TRUE)
cur_masks
```

### Add metadata

To link images between the two `CytoImageList` objects and the corresponding
`SingleCellExperiment` object, the image ids need to be added to the
`elementMetadata` slot of the `CytoImageList` objects.

```{r add-metadata}
names(cur_images)
names(cur_masks)
mcols(cur_masks)$ImageNb <- mcols(cur_images)$ImageNb <- c("E34", "G01", "J02")
```

### Add channel names

To access the correct images in the multi-channel `CytoImageList` object, the
user needs to set the correct channel names. For this, the `cytomapper` package
provides the `?channelNames` getter and setter function:

```{r channelNames-example}
channelNames(cur_images) <- c("H3", "CD99", "PIN", "CD8a", "CDH")
```

### Generating the object

Based on the processed segmentation masks and multi-channel images, 
`cytomapper` can be used to measure cell-specific intensities and morphological features.
Here, these features are stored in form of a `SingleCellExperiment` object.

```{r measureObjects}
cur_sce <- measureObjects(image = cur_images, 
                          mask = cur_masks, 
                          img_id = "ImageNb")
cur_sce
```

### Run cytoviewer 

Next, we can again call `cytoviewer` with the generated `image`, `mask` and
`object` data and leverage all provided functionality.

```{r cytoviewer-manual}
# Use cytoviewer with images, masks and object
app_1 <- cytoviewer(image = cur_images, 
                  mask = cur_masks, 
                  object = cur_sce, 
                  img_id = "ImageNb", 
                  cell_id = "object_id")

if (interactive()) {
  
  shiny::runApp(app_1, launch.browser = TRUE)

  }
```

For more detailed information on the input objects,
please refer to the respective documentation (the vignettes of
the `r Biocpkg("cytomapper")` or `r Biocpkg("SingleCellExperiment")`/
`r Biocpkg("SpatialExperiment")` packages).

# Session info {.unnumbered}

```{r sessionInfo, echo=FALSE}
sessionInfo()
```

# References