NAME
    Catmandu::Importer::getJSON - load JSON-encoded data from a server using
    a GET HTTP request

SYNOPSIS
    The following three examples are equivalent:

        Catmandu::Importer::getJSON->new(
            file => \"http://example.org/alice.json\nhttp://example.org/bob.json"
        )->each(sub { my ($record) = @_; ... );

        Catmandu::Importer::getJSON->new(
            url  => "http://example.org",
            file => \"/alice.json\n/bob.json"
        )->each(sub { my ($record) = @_; ... );

        Catmandu::Importer::getJSON->new(
            url  => "http://example.org/{name}.json",
            file => \"{\"name\":\"alice\"}\n{\"name\":\"bob\"}"
        )->each(sub { my ($record) = @_; ... );

    For more convenience the catmandu command line client can be used:

        echo http://example.org/alice.json | catmandu convert getJSON to YAML
        catmandu convert getJSON --from http://example.org/alice.json to YAML
        catmandu convert getJSON --dry 1 --url http://{domain}/robots.txt < domains

DESCRIPTION
    This Catmandu::Importer performs a HTTP GET request to load JSON-encoded
    data from a server. The importer expects a line-separated input. Each
    line corresponds to a HTTP request that is mapped to a JSON-record on
    success. The following input formats are accepted:

    plain URL
        A line that starts with ""http://"" or ""https://"" is used as plain
        URL.

    URL path
        A line that starts with ""/"" is appended to the configured url
        parameter.

    variables
        A JSON object with variables to be used with an URL template or as
        HTTP query parameters. For instance the input line "{"name":"Karl
        Marx"}" with URL "http://api.lobid.org/person" or the input line
        "{"entity":"person","name":"Karl Marx"}" with URL template
        "http://api.lobid.org/{entity}{?id}{?name}{?q}" are both expanded to
        <http://api.lobid.org/person?name=Karl+Marx>.

    If the JSON data returned in a HTTP response is a JSON array, its
    elements are imported as multiple items. If a JSON object is returned,
    it is imported as one item.

CONFIGURATION
    url An URI or an URI templates (URI::Template) as defined by RFC 6570
        <http://tools.ietf.org/html/rfc6570> to load JSON from. If no url is
        configured, plain URLs must be provided as input or option "from"
        must be used instead.

    from
        A plain URL to load JSON without reading any input lines.

    timeout / agent / proxy / headers
        Optional HTTP client settings.

    client
        Instance of a Furl HTTP client to perform requests with.

    dry Don't do any HTTP requests but return URLs that data would be
        queried from.

    file / fh
        Input to read lines from (see Catmandu::Importer). Defaults to
        STDIN.

    fix An optional fix to be applied on every item (see Catmandu::Fix).

    wait
        Number of seconds to wait between requests.

    cache
        Cache JSON response of URLs to not request the same URL twice. HTTP
        error codes in the 4xx range (e.g. 404) are also cached but 5xx
        errors are not.

        The value of this option can be any objects that implements method
        "get" and "set" (e.g. "CHI"), an existing directory for file
        caching, a true value to enable global in-memory-caching, or a false
        value to disable caching (default).

        File caching uses file names based on MD5 of an URL so for instance
        "http://example.org/" is cached as
        "4389382917e51695b759543fdfd5f690.json".

    warn
        Show error messages on the standard error.

METHODS
  time
    Returns the UNIX timestamp right before the last request. This can be
    used for instance to add timestamps or the measure how fast requests
    were responded.

  construct_url( [ $base_url, ] $vars_url_or_path )
    Returns an URL given a hash reference with variables, a plain URL or an
    URL path. The optional first argument can be used to override option
    "url".

        $importer->construct_url( %query_vars )
        $importer->construct_url( $importer->url, %query_vars ) # equivalent

  request($url)
    Perform a HTTP GET request of a given URL including logging, caching,
    request hook etc. Returns a hash/array reference or "undef".

EXTENDING
    This importer provides two methods to filter requests and responses,
    respectively. See Catmandu::Importer::Wikidata for an example.

  request_hook
    Gets a whitespace-trimmed input line and is expected to return an
    unblessed hash reference, an URL, or undef. Errors are catched and
    treated equal to undef.

  response_hook
    Gets the queried response object and is expected to return an object.

LOGGING
    URLs are emitted before each request on DEBUG log level.

LIMITATIONS
    Future versions of this module may also support asynchronous HTTP
    fetching modules such as HTTP::Async, for retrieving multiple URLs at
    the same time.

SEE ALSO
    Catmandu::Fix::get_json provides this importer as fix function.

COPYRIGHT AND LICENSE
    Copyright Jakob Voß, 2014-

    This library is free software; you can redistribute it and/or modify it
    under the same terms as Perl itself.