Package {rdomains}

rdomains: Classify Domains by their Content

Description

Want to know what kind of content is carried on a domain? Get the results quickly using rdomains. The package provides access to virustotal API, shalla, aws, OpenAI GPT models, Anthropic Claude models, and validated ML model based off shallalist data to predict content of a domain.

Details

To learn how to use rdomains, see this vignette: ../doc/rdomains.html.

Author(s)

Gaurav Sood

Probability that Domain Hosts Adult Content Based on features of Domain Name and Suffix alone.

Description

Uses a validated ML model that uses keywords in the domain name and suffix to predict probability that the domain hosts adult content. For more information see https://github.com/themains/keyword_porn

Usage

adult_ml1_cat(domains = NULL)

Arguments

Value

data.frame with original list and content category of the domains

Examples

## Not run: 
adult_ml1_cat("http://www.google.com")

## End(Not run)

Get Category from Anthropic Claude

Description

Fetches category of content hosted by a domain using Anthropic's Claude API. The function uses Claude models to classify domains into specified categories.

Usage

claude_cat(
  domains = NULL,
  api_key = NULL,
  categories = NULL,
  model = "claude-3-haiku-20240307",
  rate_limit = 0.5
)

Arguments

Value

data.frame with original list and content category of the domain

Examples

## Not run: 
claude_cat("google.com")
claude_cat(c("google.com", "facebook.com"))
claude_cat("google.com", categories = c("search", "social", "ecommerce", "news", "other"))

## End(Not run)

Get Category from DMOZ

Description

Fetches category (or categories) of content hosted by a domain according to DMOZ. The function checks if path to the DMOZ file is provided by the user. If not, it looks for dmoz_domain_cateory.csv in the working directory. It also returns results for prominent subdomains.

Usage

dmoz_cat(domains = NULL, use_file = NULL)

Arguments

Value

data.frame with original list and content category of the domain

Examples

## Not run: 
dmoz_cat(domains = "http://www.google.com")
dmoz_cat(domains = c("http://www.google.com", "http://plus.google.com"))

## End(Not run)

Get DMOZ Data

Description

Downloads archived DMOZ (Open Directory Project) data. DMOZ was discontinued in March 2017. This function downloads our preserved copy of the final DMOZ dataset. For more details, check: https://github.com/themains/rdomains/tree/master/data-raw/dmoz/

Usage

get_dmoz_data(outdir = ".", overwrite = FALSE)

Arguments

References

https://archive.org/details/dmoz-rdf-20150327

Examples

## Not run: 
get_dmoz_data()

## End(Not run)

Get Shalla Data

Description

Shallalist service was discontinued in January 2022. This function downloads the last archived copy (from 1/14/22) that we have preserved on GitHub. The original service at shallalist.de is no longer available. Downloads, unzips and saves the final version of shallalist data. By default, saves shalla data as shalla_domain_category.csv.

Usage

get_shalla_data(outdir = "./", overwrite = FALSE)

Arguments

References

https://web.archive.org/web/20210502020725/http://www.shallalist.de/

Examples

## Not run: 
get_shalla_data()

## End(Not run)

Get Steven Black's Host List Data

Description

Downloads the latest version of Steven Black's unified hosts file. The hosts file contains domains known for serving ads, malware, and tracking.

Usage

get_stevenblack_data(outdir = "./", variant = "base", overwrite = FALSE)

Arguments

References

https://github.com/StevenBlack/hosts

Examples

## Not run: 
get_stevenblack_data()
get_stevenblack_data(variant = "all")

## End(Not run)

ML Model

Description

ML Model

Usage

glm_shalla

Format

A list

Author(s)

Gaurav Sood

Source

ML model based on shallalist using keywords and domain suffixes,

Classify News and Non-News Based on keywords in the URL

Description

Based on a slightly amended version of the regular expression used to classify news, and non-news in: “Exposure to ideologically diverse news and opinion on Facebook” by Bakshy, Messing, and Adamic. Science. 2015.

Usage

not_news(url_list = NULL)

Arguments

Details

Amendment: sport rather than sports

URL containing any of the following words is classified as soft news: "sport|entertainment|arts|fashion|style|lifestyle|leisure|celeb|movie|music|gossip|food|travel|horoscope|weather|gadget"

URL containing any of following words is classified as hard news: "politi|usnews|world|national|state|elect|vote|govern|campaign|war|polic|econ|unemploy|racis|energy|abortion|educa|healthcare|immigration"

Note that it is based on patterns existing in a small set of domains. See paper for details.

Value

data.frame with 3 columns: url, not_news, news

References

https://www.science.org/doi/10.1126/science.aaa1160

Examples

## Not run: 
not_news("http://www.bbc.com/sport")
not_news(c("http://www.bbc.com/sport", "http://www.washingtontimes.com/news/politics/"))

## End(Not run)

Get Category from OpenAI

Description

Fetches category of content hosted by a domain using OpenAI's chat completion API. The function uses GPT models to classify domains into specified categories.

Usage

openai_cat(
  domains = NULL,
  api_key = NULL,
  categories = NULL,
  model = "gpt-4o-mini",
  rate_limit = 0.5
)

Arguments

Value

data.frame with original list and content category of the domain

Examples

## Not run: 
openai_cat("google.com")
openai_cat(c("google.com", "facebook.com"))
openai_cat("google.com", categories = c("search", "social", "ecommerce", "news", "other"))

## End(Not run)

Get Category from Shallalist

Description

Fetches category of content hosted by a domain according to Shalla. The function checks if path to the shalla file is provided by the user. If not, it looks for shalla_domain_category.csv in the working directory.

Usage

shalla_cat(domains = NULL, use_file = NULL)

Arguments

Value

data.frame with original list and content category of the domain

Examples

## Not run: 
shalla_cat(domains = "http://www.google.com")

## End(Not run)

Get Category from Steven Black's Host List

Description

Classifies domains based on Steven Black's unified host list which blocks ads, malware, and tracking domains. The function checks if a domain appears in the blocklist and categorizes it accordingly.

Usage

stevenblack_cat(domain = NULL, use_file = NULL)

Arguments

Details

Steven Black's host list is a consolidated list from multiple sources including adaway.org, mvps.org, malwaredomainlist.com, and someonewhocares.org.

Value

data.frame with original domain name and category

References

https://github.com/StevenBlack/hosts

Examples

## Not run: 
stevenblack_cat("doubleclick.net")
stevenblack_cat(c("google.com", "googleadservices.com", "malware-example.com"))

## End(Not run)

Get Category from University Domain List

Description

Fetches university domain json from: https://raw.githubusercontent.com/Hipo/university-domains-list/master/world_universities_and_domains.json

Usage

uni_cat(domains = NULL)

Arguments

Value

data.frame with original list and all the other columns from the university json

Examples

## Not run: 
uni_cat(domains = "http://www.google.com")

## End(Not run)

Get Category from VirusTotal

Description

Returns category of content from multiple security vendors using the VirusTotal API v3. The function retrieves domain analysis results including categories from various security services. Not all services will have categories for all domains.

Usage

virustotal_cat(domains = NULL, apikey = NULL)

Arguments

Details

Get the API Access Key from https://www.virustotal.com/. Either pass the API Key to the function or set the environmental variable: VirustotalToken. Environment variables persist within a R session.

Value

data.frame with domain and VirusTotal analysis results

References

https://docs.virustotal.com/reference/domains

Examples

## Not run: 
virustotal_cat("http://www.google.com")
virustotal_cat(c("google.com", "facebook.com"))

## End(Not run)