# The curl package: a modern R interface to libcurl

#### 2016-02-17

The curl package provides bindings to the libcurl C library for R. The package supports retrieving data in-memory, downloading to disk, or streaming using the R “connection” interface. Some knowledge of curl is recommended to use this package. For a more user-friendly HTTP client, have a look at the httr package which builds on curl with HTTP specific tools and logic.

### Download in memory

The curl package implements three ways to retrieve data from a URL. The `curl_fetch_memory` function is a synchronous interface which returns a list with contents of the server response.

    req <- curl_fetch_memory("https://httpbin.org/get")
    str(req)

    List of 6
     $ url        : chr "https://httpbin.org/get"
     $ status_code: int 200
     $ headers    : raw [1:220] 48 54 54 50 ...
     $ modified   : POSIXct[1:1], format: NA
     $ times      : Named num [1:6] 0 0.000923 0.11623 0.52658 0.640177 ...
      ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect" "pretransfer" ...
     $ content    : raw [1:230] 7b 0a 20 20 ...

    parse_headers(req$headers)

    [1] "HTTP/1.1 200 OK"
    [2] "Server: nginx"
    [3] "Date: Wed, 17 Feb 2016 20:23:37 GMT"
    [4] "Content-Type: application/json"
    [5] "Content-Length: 230"
    [6] "Connection: keep-alive"
    [7] "Access-Control-Allow-Origin: *"
    [8] "Access-Control-Allow-Credentials: true"

    cat(rawToChar(req$content))

    {
      "args": {},
      "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "r/curl/jeroen"
      },
      "origin": "84.26.114.237",
      "url": "https://httpbin.org/get"
    }

The `curl_fetch_memory` interface is the easiest interface and most powerful for buidling API clients. However because it is fully in-memory, it is not suitable for downloading really large files. If you are expecting 100G of data, you probably need one of the other interfaces.

### Download to disk

The second method is `curl_download`, which has been designed as a drop-in replacement for `download.file` in r-base. It writes the response straight to disk, which is useful for downloading (large) files.

    tmp <- tempfile()
    curl_download("https://httpbin.org/get", tmp)
    cat(readLines(tmp), sep = "\n")

    {
      "args": {},
      "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "r/curl/jeroen"
      },
      "origin": "84.26.114.237",
      "url": "https://httpbin.org/get"
    }

### Streaming

The most flexible interface is the `curl` function, which has been designed as a drop-in replacement for base `url`. It will create a so-called connection object, which allows for incremental (asynchronous) reading of the response.

    con <- curl("https://httpbin.org/get")
    open(con)

    # Get 3 lines
    out <- readLines(con, n = 3)
    cat(out, sep = "\n")

    {
      "args": {},
      "headers": {

    # Get 3 more lines
    out <- readLines(con, n = 3)
    cat(out, sep = "\n")

        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",

    # Get remaining lines
    out <- readLines(con)
    close(con)
    cat(out, sep = "\n")

        "User-Agent": "r/curl/jeroen"
      },
      "origin": "84.26.114.237",
      "url": "https://httpbin.org/get"
    }

The example shows how to use `readLines` on an opened connection to read `n` lines at a time. Similarly `readBin` is used to read `n` bytes at a time for stream parsing binary data.

### Status codes

It is important to note that `curl_fetch_memory` will not automatically raise an error if the request was completed but returned a non-200 status code. When using `curl_fetch_memory` you need to implement the application logic yourself.

    req <- curl_fetch_memory("https://httpbin.org/status/418")
    print(req$status_code)

    [1] 418

The `curl` and `curl_download` functions on the other hand will automatically raise an error if the HTTP response was non successful, as would the base functions `url` and `download.file` do.

    curl_download("https://httpbin.org/status/418", tempfile())

    Error in curl_download("https://httpbin.org/status/418", tempfile()): HTTP error 418.

    con <- curl("https://httpbin.org/status/418")
    open(con)

    Error in open.connection(con): HTTP error 418.

## Handles

By default libcurl uses HTTP GET to issue a request to an HTTP url. To send a customized request, we first need to create and configure a curl handle object that is passed to the specific download interface.

### Configuring a handle

Creating a new handle is done using `new_handle`. After creating a handle object, we can set the libcurl options and http request headers.

    h <- new_handle()
    handle_setopt(h, copypostfields = "moo=moomooo");
    handle_setheaders(h,
      "Content-Type" = "text/moo",
      "Cache-Control" = "no-cache",
      "User-Agent" = "A cow"
    )

Use the `curl_options()` function to get a list of the options supported by your version of libcurl. The libcurl documentation explains what each option does. Option names are not case sensitive.

After the handle has been configured, it can be used with any of the download interfaces to perform the request. For example `curl_fetch_memory` will load store the output of the request in memory:

    req <- curl_fetch_memory("http://httpbin.org/post", handle = h)
    cat(rawToChar(req$content))

    {
      "args": {},
      "data": "moo=moomooo",
      "files": {},
      "form": {},
      "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Cache-Control": "no-cache",
        "Content-Length": "11",
        "Content-Type": "text/moo",
        "Host": "httpbin.org",
        "User-Agent": "A cow"
      },
      "json": null,
      "origin": "84.26.114.237",
      "url": "http://httpbin.org/post"
    }

Alternatively we can use `curl()` to read the data of via a connetion interface:

    con <- curl("http://httpbin.org/post", handle = h)
    cat(readLines(con), sep = "\n")

    {
      "args": {},
      "data": "moo=moomooo",
      "files": {},
      "form": {},
      "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Cache-Control": "no-cache",
        "Content-Length": "11",
        "Content-Type": "text/moo",
        "Host": "httpbin.org",
        "User-Agent": "A cow"
      },
      "json": null,
      "origin": "84.26.114.237",
      "url": "http://httpbin.org/post"
    }

Or we can use `curl_download` to write the response to disk:

    tmp <- tempfile()
    curl_download("http://httpbin.org/post", destfile = tmp, handle = h)
    cat(readLines(tmp), sep = "\n")

    {
      "args": {},
      "data": "moo=moomooo",
      "files": {},
      "form": {},
      "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Cache-Control": "no-cache",
        "Content-Length": "11",
        "Content-Type": "text/moo",
        "Host": "httpbin.org",
        "User-Agent": "A cow"
      },
      "json": null,
      "origin": "84.26.114.237",
      "url": "http://httpbin.org/post"
    }

### Cookies

Curl handles automatically keep track of cookies set by the server. At any given point we can use `handle_cookies` to see a list of current cookies in the handle.

    # Start with a fresh handle
    h <- new_handle()

    # Ask server to set some cookies
    req <- curl_fetch_memory("http://httpbin.org/cookies/set?foo=123&bar=ftw", handle = h)
    req <- curl_fetch_memory("http://httpbin.org/cookies/set?baz=moooo", handle = h)
    handle_cookies(h)

           domain  flag path secure expiration name value
    1 httpbin.org FALSE    /  FALSE       <NA>  foo   123
    2 httpbin.org FALSE    /  FALSE       <NA>  bar   ftw
    3 httpbin.org FALSE    /  FALSE       <NA>  baz moooo

    # Unset a cookie
    req <- curl_fetch_memory("http://httpbin.org/cookies/delete?foo", handle = h)
    handle_cookies(h)

           domain  flag path secure          expiration name value
    1 httpbin.org FALSE    /  FALSE 2016-02-17 21:23:41  foo  <NA>
    2 httpbin.org FALSE    /  FALSE                <NA>  bar   ftw
    3 httpbin.org FALSE    /  FALSE                <NA>  baz moooo

The `handle_cookies` function returns a data frame with 7 columns as specified in the netscape cookie file format.

### On re-using handles

As we have already seen, curl allows for reusing a single handle for multiple requests. However it is not always a good idea to do so. The performance overhead of creating and configuring a new handle object is usually negligible. The safest way to issue mutiple requests, either to a single server or multiple servers is by using a separate handle for each request.

    req1 <- curl_fetch_memory("https://httpbin.org/get", handle = new_handle())
    req2 <- curl_fetch_memory("http://www.r-project.org", handle = new_handle())

There are two reasons why you might want to reuse a handle for multiple requests. The first one is that it will automatically keep track of cookies set by the server. This might be useful if your host requires use of a session cookies.

The other reason is to take advantage of http Keep-Alive. Curl automatically maintains a pool of open http connections within each handle. When using a single handle to issue many requests to the same server, curl uses existing connections when possible. This eliminites a little bit of connection overhead, although on a decent network this might not be very significant.

    h <- new_handle()
    system.time(curl_fetch_memory("https://api.github.com/users/ropensci", handle = h))

       user  system elapsed
      0.047   0.010   0.632

    system.time(curl_fetch_memory("https://api.github.com/users/rstudio", handle = h))

       user  system elapsed
      0.009   0.002   0.479

The argument against reusing handles is that curl does not cleanup the handle after each request. All of the options and internal fields will linger around for all future request until explicitly reset or overwritten. This can sometimes leads to unexpected behavior.

    handle_reset(h)

The `handle_reset` function will reset all curl options and request headers to the default values. It will not erease cookies and it will still keep alive the connections. Therefore it is good practice to call `handle_reset` after performing a request if you want to reuse the handle for a subsequent request. Still it is always safer to create a new fresh handle when possible, rather than recycling old ones.

### Forms

The `handle_setform` function is used to perform a `multipart/form-data` HTTP POST request (a.k.a. posting a form). Values can be either strings, raw vectors (for binary data) or files.

    # Posting multipart
    h <- new_handle()
    handle_setform(h,
      foo = "blabla",
      bar = charToRaw("boeboe"),
      description = form_file(system.file("DESCRIPTION")),
      logo = form_file(file.path(Sys.getenv("R_DOC_DIR"), "html/logo.jpg"), "image/jpeg")
    )
    req <- curl_fetch_memory("http://httpbin.org/post", handle = h)

The `form_file` function is used to upload files with the form post. It has two arguments: a file path, and optionally a content-type value. If no content-type is set, curl will guess the content type of the file based on the file extention.

### Using pipes

All of the `handle_xxx` functions return the handle object so that function calls can be chained using the popular pipe operators:

    library(magrittr)

    new_handle() %>%
      handle_setopt(copypostfields = "moo=moomooo") %>%
      handle_setheaders("Content-Type" = "text/moo", "Cache-Control" = "no-cache", "User-Agent" = "A cow") %>%
      curl_fetch_memory(url = "http://httpbin.org/post") %$% content %>% rawToChar %>% cat

    {
      "args": {},
      "data": "moo=moomooo",
      "files": {},
      "form": {},
      "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Cache-Control": "no-cache",
        "Content-Length": "11",
        "Content-Type": "text/moo",
        "Host": "httpbin.org",
        "User-Agent": "A cow"
      },
      "json": null,
      "origin": "84.26.114.237",
      "url": "http://httpbin.org/post"
    }