--- title: "Onassis: Ontology Annotation and Semantic Similarity software" author: "Eugenia Galeota" date: "`r Sys.Date()`" output: BiocStyle::html_document bibliography: bibliography.bib vignette: > %\VignetteIndexEntry{Onassis: Ontology Annotation and Semantic Similarity software} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} %\VignettePackage{Onassis} %\VignetteDepends{Onassis} --- ```{r imports, echo=FALSE,eval=TRUE, message=FALSE, warning=FALSE} library(Onassis) library(DT) library(gplots) library(org.Hs.eg.db) library(kableExtra) ``` # Introduction to OnASSis Public repositories contain thousands of experiments and samples that are difficult to mine. Annotating the description of this data with controlled vocabularies or ontology terms could improve the retrieval of data of interest both programmatically or manually [@galeota2016ontology]. OnASSiS (Ontology Annotations and Semantic Similarity software) is a package aimed at matching metadata associated with biological experiments with concepts from ontologies, thus aiming at obtaining semantically coherent omics datasets, possibly representing various data types as derived from independent studies. The recognition of domain specific entities not only allows users to retrieve samples related to a given cell type or experimental condition, but also to discover different and not immediately obvious relationships between experiments. Onassis applies Natural Language Processing tools to annotate sample's and experiments' descriptions, recognizing concepts from a multitude of biomedical ontologies and quantifying the similarities/divergences between pairs or groups of query studies. In particular the software includes modules to assist on: * the retrieval of samples’ metadata from repositories of large scale biologial data * the annotation of these data with concepts belonging to OBO biomedical ontologies * the organization of available samples in comparable and coherent groups based on semantic similarity measures * the comparison of biological signal stored in a matrix of scores (e.g. gene expression in different samples) based on the annotated entities associated to each sample Onassis uses Conceptmapper, an Apache UIMA (Unstructured Information Management Architecture) dictionary lookup tool to retrieve dictionary terms in a given text. https://uima.apache.org/downloads/sandbox/ConceptMapperAnnotatorUserGuide/ConceptMapperAnnotatorUserGuide.html In particular, the ccp-nlp Conceptmapper wrapper, specific for the biomedical domain, implements a pipeline through which it is possible to retrieve concepts from OBO ontologies in any given text with different adjustable options [@Verspoor:2009b]. Onassis features can be easily accessed through a main class named Onassis, having as slots 'dictionary', 'entities', 'similarity' and 'scores'. In the following sections we first show details on the usage of the classes and methods that constitute the building blocks of typical metadata integration workflows and than we show how the Onassis class wraps all these functions for a simplified access and usage. Onassis can handle any type of text as input, but is particularly well suited for the analysis of the metadata from Gene Expression Omnibus (GEO).Indeed it provides the possibility to associate concepts from any OBO ontology to GEO metadata retrieved using `r BiocStyle::Biocpkg("GEOmetadb")`. This represents a fundamental first step in the integrative analysis of the data from large scale repositories [@galeota2016ontology]. In general, any table or database (Sequence Read Archive (SRA) [@Zhu2013] or Cistrome [@Shengling2017] ) containing textual descriptions that can be easily imported in R as a data frame can be used as input for Onassis. In addition to the ontological concepts, the recognition of gene/protein symbols or epigenetic modifications can be highly relevant, especially for experiments directed to those specific factors or marks (such as ChIP-seq experiments). The semantic similarity module uses different semantic similarity measures to determine the semantic similarity of concepts in a given ontology. This module has been developed on the basis of the Java slib http://www.semantic-measures-library.org/sml. The score module applies simple statistical tests to determine if biological signals from samples annotated with different concepts are significantly different. # Installing additional libraries to run the examples in the vignette To run Onassis Java (>= 1.8) is needed. Some of the optional functions, which will be described in the followign parts of the vignette, require additional libraries. For the analyses in this vignette please install following libraries: ```{r echo=TRUE, eval=FALSE} source("https://bioconductor.org/biocLite.R") biocLite('org.Hs.eg.db') biocLite("GenomicRanges") install.packages('data.table') install.packages('DT') install.packages('gplots') ``` # Retrieving public repositories metadata One of the most straightforward ways to retrieve metadata of samples provided in GEO is through `r BiocStyle::Biocpkg("GEOmetadb")` package. In order to use GEOmetadb through Onassis, users should download the corresponding SQLite database file, by following the instructions provided in the package vignette. Onassis provides functions to help GEO metadata retrieval without the need of explicitly making SQL queries to the database. While GEOmetadb can be accessed on any platform, another important database `r BiocStyle::Biocpkg("SRAdb")`, is not available for Windows users. In the following sections we show how to query GEOmetadb through Onassis, but we also provide an example accessing SRAdb metadata. ## Handling GEO (Gene Expression Omnibus) Metadata Firstly, it is necessary to obtain a connection to the SQLite database. `connectToGEODB` returns a connection to the database given the path of the SQLite database file. If the latter is missing, it will be automatically downloaded into the current working directory. Because of the size of this file (0.5 GB), the results of the queries illustrated below are available into Onassis for the subsequent analyses illustrated in this document. Then, the `getGEOmetadata` function can be used to retrieve the metadata related to specific GEO samples, taking as minimal parameters the connection to the database and one of the experiment types available. Optionally it is possible to specify the organism and the platform. ```{r connectTodb, echo=TRUE,eval=FALSE} ## Running this function might take long time if the database has to be downloaded. geo_con <- connectToGEODB(download=TRUE) #Showing the experiment types available in GEO experiments <- experiment_types(geo_con) #Showing the organism types available in GEO species <- organism_types(geo_con) #Retrieving the metadata associated to experiment type "Methylation profiling by high througput sequencing" meth_metadata <- getGEOMetadata(geo_con, experiment_type='Methylation profiling by high throughput sequencing', organism = 'Homo sapiens') #Retrieving Human gene expression metadata, knowing the GEO platform identifier, e.g. the Affymetrix Human Genome U133 Plus 2.0 Array expression <- getGEOMetadata(geo_con, experiment_type='Expression profiling by array', gpl='GPL570') ``` Some of the experiment types available are the following: ```{r experimentTypesshow, echo=FALSE, eval=TRUE} experiments <- readRDS(system.file('extdata', 'vignette_data', 'experiment_types.rds', package='Onassis')) knitr::kable(as.data.frame(experiments[1:10]), col.names = c('Experiment')) %>% kable_styling(bootstrap_options = c("striped"), position="center") %>% scroll_box(width = "300px", height = "200px") ``` Some of the organisms available are the following: ```{r speciesShow, echo=FALSE,eval=TRUE} species <- readRDS(system.file('extdata', 'vignette_data', 'organisms.rds', package='Onassis')) knitr::kable(as.data.frame(species[1:10]), col.names=c('Species')) %>% kable_styling(bootstrap_options = c("striped"), position="center") %>% scroll_box(width = "300px", height = "200px") ``` As specified before in this document, to correctly query GEOmetadb, it is necessary to download the corresponding database SQLite file, which occupies sever GB of disk space. Only for this vignette, meth_metadata was previously saved from the getGEOmetadata function and can be loaded from the Onassis package external data: ```{r loadgeoMetadata, echo=TRUE, eval=TRUE} meth_metadata <- readRDS(system.file('extdata', 'vignette_data', 'GEOmethylation.rds', package='Onassis')) ``` ```{r printmeta, echo=FALSE,eval=TRUE} methylation_tmp <- meth_metadata methylation_tmp$experiment_summary <- sapply(methylation_tmp$experiment_summary, function(x) substr(x, 1, 50)) knitr::kable(methylation_tmp[1:10,], caption = 'Methylation profiling by high througput sequencing metadata from GEOmetadb.') %>% kable_styling(bootstrap_options = c("striped"), position="center") %>% scroll_box(width = "80%", height = "300px") ``` ## Handling SRA (Sequence Read Archive) Metadata In this section we provide an example showing how it is possible to retrieve data from other sources such as SRA. In this case we only show an example on how to query the database and store the metadata in a data frame table. The following code requires the file SRAmetadb.sqlite, containing SRA metadata. Since also this database occupies disk space (~4GB), we provide the results of the query as external data and running the following part of code is optional. The database file and the queries can be obtained and executed in R through the Bioconductor package `SRAdb`. The following code shows how to obtain SRA metadata of ChIP-Seq human samples and Bisulfite sequencing samples: ```{r connectSRA, echo=TRUE,eval=FALSE} # Connection to the SRAmetadb and potential download of the sqlite file sqliteFileName <- './data/SRAdb.sqlite' sra_con <- dbConnect(SQLite(), sqliteFileName) # Query for the ChIP-Seq experiments contained in GEO for human samples library_strategy <- 'ChIP-Seq' #ChIP-Seq data library_source='GENOMIC' taxon_id=9606 #Human samples center_name='GEO' #Data from GEO # Query to the sample table samples_query <- paste0("select sample_accession, description, sample_attribute, sample_url_link from sample where taxon_id='", taxon_id, "' and sample_accession IS NOT NULL", " and center_name='", center_name, "'", ) samples_df <- dbGetQuery(sra_con, samples_query) samples <- unique(as.character(as.vector(samples_df[, 1]))) # Query to the experiment table experiment_query <- paste0("select experiment_accession, center_name, title, sample_accession, sample_name, experiment_alias, library_strategy, library_layout, experiment_url_link, experiment_attribute from experiment where library_strategy='", library_strategy, "'" , " and library_source ='", library_source, "' " ) experiment_df <- dbGetQuery(sra_con, experiment_query) #Merging the columns from the sample and the experiment table experiment_df <- merge(experiment_df, samples_df, by = "sample_accession") # Replacing the field separators with white spaces experiment_df$experiment_attribute <- sapply(experiment_df$experiment_attribute, function(value) { gsub("||", " ", value) }) experiment_df$sample_attribute <- sapply(experiment_df$sample_attribute, function(value) { gsub("||", " ", value) }) # Replacing the '_' character with white spaces experiment_df$sample_name <- sapply(experiment_df$sample_name, function(value) { gsub("_", " ", value) }) experiment_df$experiment_alias <- sapply(experiment_df$experiment_alias, function(value) { gsub("_", " ", value) }) sra_chip_seq <- experiment_df ``` The previously saved sra_chip_seq can be retrieved in Onassis: ```{r readCHIP, echo=TRUE, eval=TRUE} sra_chip_seq <- readRDS(system.file('extdata', 'vignette_data', 'GEO_human_chip.rds', package='Onassis')) ``` ```{r printchromatinIP, echo=FALSE,eval=TRUE} knitr::kable(head(sra_chip_seq, 10), rownames=FALSE, caption = 'ChIP-Seq metadata obtained from SRAdb') %>% kable_styling(bootstrap_options = c("striped"), position="center") %>% scroll_box(width = "80%", height = "300px") ``` # Annotating text with Ontology Concepts The Onassis `EntityFinder` class has methods for annotating any text with dictionary terms. More specifically, Onassis can take advantage of the OBO dictionaries (http://www.obofoundry.org/), as shown in the next section where we will define relationships between different samples annotated with different ontology concepts thanks to the structure of the ontology. ## Data preparation Input text can be provided as: * The path of a directory containing named documents (findEntities method). * The path of a single file containing multiple documents. In this case each row contains the name/identifier of the document followed by a '|' separator and the text to annotate (findEntities method). * A data frame. In this case each row represents a document, first column has to be the document identifier, and the remaining columns will be combined and contain the text to analyze (annotateDF method). This option can be conveniently used with the metadata retrieved from `r BiocStyle::Biocpkg("GEOmetadb")` and `r BiocStyle::Biocpkg("SRAdb")`, possibly selecting a subset of the available columns. ## Creation of a Conceptmapper Dictionary Conceptmapper dictionaries are XML files with a set of entries specified by the xml tag `````` with a canonical name (the name of the entry) and one or more variants (synonyms). Additional properties are allowed. The following code represents a sample of the Conceptmapper dictionary obtained from the Brenda tissue ontology. ```xml ``` The constructor `CMdictionary` creates an instance of the class `CMdictionary`. * If an XML file containing the Conceptmapper dictionary is already available, it can be uploaded into Onassis indicating its path and setting the `dictType` option to "CMDICT". * If the dictionary has to be built from an OBO ontology (as a file in the OBO or OWL format), its path has to be provided and dictType has to be set to "OBO". The synonymType argument can be set to EXACT_ONLY or ALL to consider only canonical concept names or also to include any synonym. The resulting XML file is written in the indicated outputdir. Alternatively, to automatically download the ontology, the URL where the OBO file is located can be provided. * To build a dictionary containing only gene/protein names, dictType has to be set to either TARGET or ENTREZ, to include histone types and marks or not, respetively. If a specific Org.xx.eg.db Bioconductor library is indicated in the inputFileOrDb parameter as a character string, gene names will be derived from it. Instead, if inputFileOrDb is empty and a specific species is indicated in the taxID parameter, the gene_info.gz file hosted at NCBI will be downloaded and used to find gene names. If available, this file can be located with the inputFile parameter. Otherwise, it will be automatically downloaded (300MB). ```{r createSampleAndTargetDict, echo=TRUE,eval=TRUE, message=FALSE} # If a Conceptmapper dictionary is already available the dictType CMDICT can be specified and the corresponding file loaded sample_dict <- CMdictionary(inputFileOrDb=system.file('extdata', 'cmDict-sample.cs.xml', package = 'Onassis'), dictType = 'CMDICT') #Creation of a dictionary from the file sample.cs.obo available in OnassisJavaLibs obo <- system.file('extdata', 'sample.cs.obo', package='OnassisJavaLibs') sample_dict <- CMdictionary(inputFileOrDb=obo, outputDir=getwd(), synonymType='ALL') # Creation of a dictionary for human genes/proteins require(org.Hs.eg.db) targets <- CMdictionary(dictType='TARGET', inputFileOrDb = 'org.Hs.eg.db') ``` ## Setting the options for the annotator Conceptmapper includes 7 different options controlling the annotation step. These are documented in detail in the documentation of the CMoptions function. They can be listed through the `listCMOptions` function. The `CMoptions` constructor instantiates an object of class CMoptions with the different parameters that will be required for the subsequent step of annotation. We also provided getter and setter methods for each of the 7 parameters. ```{r settingOptions, echo=TRUE,eval=TRUE} #Creating a CMoptions object and showing hte default parameters opts <- CMoptions() show(opts) ``` To list the possible combinations: ```{r listCombinations, echo=TRUE, eval=TRUE} combinations <- listCMOptions() ``` To create a CMoptions object having has SynonymType 'EXACT_ONLY' ```{r setsynonymtype, echo=TRUE, eval=TRUE} myopts <- CMoptions(SynonymType = 'EXACT_ONLY') myopts ``` To change a given parameter ```{r changeparameter, echo=TRUE, eval=TRUE} #Changing the SearchStrategy parameter SearchStrategy(myopts) <- 'SKIP_ANY_MATCH_ALLOW_OVERLAP' myopts ``` ## Running the entity finder The class `EntityFinder` defines a type system and runs the Conceptmapper pipeline. It can find concepts of any OBO ontology in a given text. The `findEntities` and `annotateDF` methods accept text within files or data.frame, respectively, as described in Section 4.1. The function `EntityFinder` automatically adapts to the provided input type, creates an instance of the `EntityFinder` class to initialize the type system and runs the pipeline with the provided options and dictionary. For example, to annotate the metadata derived from ChIP-seq experiments obtained from SRA with tissue and cell type concepts belonging to BRENDA ontology the following code can be used: ```{r EntityFinder, echo=TRUE, eval=TRUE, results='hide', message=FALSE, warning=FALSE } chipseq_dict_annot <- EntityFinder(sra_chip_seq[1:20,c('sample_accession', 'title', 'experiment_attribute', 'sample_attribute', 'description')], dictionary=sample_dict, options=myopts) ``` The resulting data.frame contains, for each row, a match to the provided dictionary for a specific document/sample (indicated in the first column). The annotation is reported with the id of the concept (term_id), its canonical name (term name), its URL in the obo format, and the matching sentence of the document. ```{r showchipresults, echo=FALSE, eval=TRUE, message=FALSE} #methylation_brenda_annot <- readRDS(system.file('extdata', 'vignette_data', 'methylation_brenda_annot.rds', package='Onassis')) #UPDATE con ChIP-seq knitr::kable(head(chipseq_dict_annot, 20), rownames=FALSE, caption = 'Annotations of the methylation profiling by high througput sequencing metadata obtained from GEO with BRENDA ontology concepts') %>% kable_styling() %>% scroll_box(width = "80%", height = "400px") ``` \r \r The function `EntityFinder` can also be used to identify the targeted entity of each ChIP-seq experiment, by retrieving gene names and histone types or modifications in the ChIP-seq metadata. ```{r annotateGenes, echo=TRUE, eval=TRUE, results='hide', message=FALSE, warning=FALSE} #Finding the TARGET entities target_entities <- EntityFinder(input=sra_chip_seq[1:20,c('sample_accession', 'title', 'experiment_attribute', 'sample_attribute', 'description')], options = myopts, dictionary=targets) ``` ```{r printKable, echo=FALSE, eval=TRUE} knitr::kable(target_entities, caption = 'Annotations of ChIP-seq test metadata obtained from SRAdb and stored into files with the TARGETs (genes and histone variants)') %>% kable_styling(bootstrap_options = c("striped"), position="center") %>% scroll_box(width = "80%", height = "400px") ``` # Semantic similarity With Onassis it is possible to quantify the semantic similarity between the concepts of a given ontology using different semantic similarity measures. `Similarity` is an Onassis class applying methods of the Java library slib \url{http://www.semantic-measures-library.org/sml/} [@harispe2014semantic], which builds a semantic graph starting from OBO ontology concepts and their hierarchical relationships. The following methods are available and are automatically chosen depending on the settings of the `Similarity` function. The `sim` and `groupsim` methods allow the computation of semantic similarity between single terms (pairwise measures) and between group of terms (groupwise measures), respectively. Pairwise measures can be edge based, if they rely only on the structure of the ontology, or information-content based if they also consider the information that each term in the ontology carries. Rather, groupwise measures can be indirect, if they compute the pairwise similarity between each couple of terms, or direct if they consider each set of concepts as a whole. The `samplesim` method allows to determine the semantic similarity between two documents, each possibly associated to multiple concepts, using groupwise measures. Finally, the `multisim` method allows to determine the semantic similarity between documents annotated with two or more ontologies: first `samplesim` is run for each ontology, then a user defined function can be used to aggregate the resulting semantic similarities for each pair of documents. The function `listSimilarities` shows all the measures supported by Onassis. For details about the measures run `{?Similarity}`. ```{r similarity, echo=TRUE, eval=TRUE, message=FALSE} #Instantiating the Similarity similarities <- listSimilarities() ``` The following example shows the pairwise similarities between sample cell concepts obtained annotating the ChIP-seq metadata. The lin similarity measure is used by default, which relies on a ratio between the Information content (IC) of the Most Specific Common Ancestor of the compared concepts and the sum of their IC based on the information content of the most informative common ancestor of the considered concepts. In particular, the seco information content is used by default, which determines the specificity of each concept based on the number of concepts it subsumes. ```{r computing measures, echo=TRUE, eval=TRUE, message=FALSE} found_terms <- unique(chipseq_dict_annot$term_url) n <- length(found_terms) ontologyfile <- obo pairwise_results <- data.frame(term1 = character(0), term2= character(0), value = double(0L)) for(i in 1:(n-1)){ term1 <- as.character(found_terms[i]) j = i + 1 for(k in j:n){ term2 <- as.character(found_terms[k]) two_term_similarity <- Similarity(ontologyfile, term1, term2 ) new_row <- cbind(term1, term2, two_term_similarity) pairwise_results <- rbind(pairwise_results, new_row ) } } pairwise_results <- unique(pairwise_results) pairwise_results <- merge(pairwise_results, chipseq_dict_annot[, c('term_url', 'term_name')], by.x='term2', by.y='term_url', all.x=TRUE) colnames(pairwise_results)[length(colnames(pairwise_results))] <- 'term2_name' pairwise_results <- merge(pairwise_results, chipseq_dict_annot[, c('term_url', 'term_name')], by.x='term1', by.y='term_url', all.x=TRUE) colnames(pairwise_results)[length(colnames(pairwise_results))] <- 'term1_name' pairwise_results <- unique(pairwise_results) ``` ```{r showSim, echo=FALSE, eval=TRUE} knitr::kable(pairwise_results, caption = 'Pairwise similarities of cell line terms annotating the ChIP-seq metadata') %>% kable_styling(bootstrap_options = c("striped"), position="center") %>% scroll_box(width = "80%", height = "400px") ``` In the following code the semantic similarity between two groups of terms is computed using the ui measure, a groupwise direct measure combining the intersection and the union of the set of ancestors of the two groups of concepts. ```{r groupwise_measures, echo=TRUE, eval=TRUE, message=FALSE} Similarity(obo, found_terms[1:2], found_terms[3]) ``` Lastly, the pariwise semantic similarity between ChIP-seq samples is illustrated. ```{r samples_similarity, echo=TRUE, eval=TRUE, message=FALSE} annotated_samples <- as.character(as.vector(unique(chipseq_dict_annot$sample_id))) n <- length(annotated_samples) samples_results <- data.frame(sample1 = character(0), sample2= character(0), value = double(0L)) samples_results <- matrix(0, nrow=n, ncol=n) rownames(samples_results) <- colnames(samples_results) <- annotated_samples for(i in 1:(n-1)){ sample1 <- as.character(annotated_samples[i]) j = i + 1 for(k in j:n){ sample2 <- as.character(annotated_samples[k]) two_samples_similarity <- Similarity(ontologyfile, sample1, sample2, chipseq_dict_annot) samples_results[i, k] <- samples_results[k, i] <- two_samples_similarity } } diag(samples_results) <- 1 heatmap.2(samples_results, density.info = "none", trace="none", main='Semantic similarity of annotated samples', margins=c(5,5)) ``` # Onassis class The class Onassis was built to wrap the main functionalities of the package in a single class. It consists of 4 slots: * dictionary: stores the source dictionary used to find entities. * entities: a table containing the annotations of documents (samples). The list of unique concepts belonging to the dictionary and found in the metadata representing a given sample can be defined as a semantic set * similarity: a matrix of the similarities between the unique semantic sets identified in the entities table * scores: a dataset of quantitative measurements (e.g. gene expression) associated to the samples annotated in entities and separated in the different semantic sets identified in the annotation process. In this section we illustrate the use of the Onassis class to annotate the previously retrieved metadata. The method `annotate` takes as input a data frame of metadata to annotate, the type of dictionary and the path of an ontology file and returns an instance of class Onassis. ```{r onassis_class_usage, echo=TRUE, eval=TRUE, results='hide', message=FALSE, warning=FALSE } onassis_annotations <- annotate(sra_chip_seq, 'OBO',obo ) ``` To retrieve the annotations in an object of class Onassis we provided the accessor method `entities` ```{r show_onassis_annotations, echo=TRUE, eval=TRUE} onassis_entities <- entities(onassis_annotations) ``` ```{r showing_entities, echo=FALSE, eval=TRUE} knitr::kable( onassis_entities[sample(nrow(onassis_entities), 10),], caption = 'Entities in Onassis object') %>% kable_styling(bootstrap_options = c("striped"), position="center") %>% scroll_box(width = "80%", height = "400px") ``` The `filterconcepts` method can be used to filter out unwanted annotations. It takes the Onassis object and removes from its entities the undesired concepts. ```{r term_filtering, echo=TRUE, eval=TRUE} filtered_onassis <- filterconcepts(onassis_annotations, c('cell')) ``` ```{r showing_filt_entities, echo=FALSE, eval=TRUE} knitr::kable(entities(filtered_onassis), caption = 'Entities in filtered Onassis object') %>% kable_styling(bootstrap_options = c("striped"), position="center") %>% scroll_box(width = "80%", height = "400px") ``` The method `sim` cretes a matrix of the semantic similarities between the annotations of each couple of samples annotated in the entities slot of an Onassis object. ```{r similarity_of_samples, echo=TRUE, eval=TRUE} filtered_onassis <- sim(filtered_onassis) ``` Annotations with semantic similarities above a given threshold can be unified using the method `collapse`. This method unifies the similar annotations by concatenating their unique concepts. Entities are replaced with the new concatenated annotations. For each concept in the concatenated annotations the number of samples associated is also reported, together with the total number of samples annotated with the new annotations. The similarity slot will be consequently updated. ```{r collapsing_similarities, echo=TRUE, eval=TRUE, message=FALSE, results='hide', fig.width=6, fig.height=6} collapsed_onassis <- Onassis::collapse(filtered_onassis, 0.8) head(entities(collapsed_onassis)) heatmap.2(simil(collapsed_onassis), margins=c(15,15), cexRow = 1, cexCol = 1) ``` # Session Info Here is the output of sessionInfo() on the system on which this document was compiled through kintr: ```{r sessionInfo(), echo=FALSE, eval=TRUE} sessionInfo() ``` # References