Head’s Up! Roll Your Own HTTP Headers Investigations with the ‘hdrs’ Package

This article is originally published at https://rud.is/b

I blathered alot about HTTP headers in the last post.

In the event you wanted to dig deeper I threw together a small package that will let you grab HTTP headers from a given URL and take a look at them. The README has examples for most things but we’ll go through a bit of them here as well.

For those that just want to play, you can do:

install.packages("hdrs", repos = "https://cinc.rud.is/")

hdrs::explore_app()

and use the diminutive Shiny app to explore a site’s security headers or look at all the headers they return. (Oh, yeah…if you read the previous post then looked at the above screenshot you’ll notice how completely useless IP blocking is to determined attackers individuals.)

NOTE: There are binaries for macOS and Windows at my CINC repo for hdrs so you’ll be getting those if you use the above method. Use type='source' on that call or use various remotes package functions to install the source package (after reading it b/c you really shouldn’t trust any package, ever) from:

• SourceHut
• GitLab
• if you must GitHub

Moving Ahead

Let’s use the command-line to poke at my newfound most favorite site to use in security-related examples:

library(hdrs)

assess_security_headers("https://cran.r-project.org/") %>% 
  dplyr::select(-url)
## # A tibble: 13 x 4
##    header                            value                  status_code message            
##  * <chr>                             <chr>                  <chr>       <chr>              
##  1 access-control-allow-origin       NA                     WARN        Header not set     
##  2 content-security-policy           NA                     WARN        Header not set     
##  3 expect-ct                         NA                     WARN        Header not set     
##  4 feature-policy                    NA                     WARN        Header not set     
##  5 public-key-pins                   NA                     WARN        Header not set     
##  6 referrer-policy                   NA                     WARN        Header not set     
##  7 server                            Apache/2.4.10 (Debian) NOTE        Server header found
##  8 strict-transport-security         NA                     WARN        Header not set     
##  9 x-content-type-options            NA                     WARN        Header not set     
## 10 x-frame-options                   NA                     WARN        Header not set     
## 11 x-permitted-cross-domain-policies NA                     WARN        Header not set     
## 12 x-powered-by                      NA                     WARN        Header not set     
## 13 x-xss-protection                  NA                     WARN        Header not set     

Ouch. Not exactly a great result (so, perhaps it matters little how poorly maintained the downstream mirrors are after all, or maybe it’s perfectly fine to run a five year old web server with some fun vulns).

Anyway…

The assess_security_headers() function looks at 13 modern “security-oriented” HTTP headers, performs a very light efficacy assessment and returns the results.

• access-control-allow-origin
• content-security-policy
• expect-ct
• feature-policy
• server
• public-key-pins
• referrer-policy
• strict-transport-security
• x-content-type-options
• x-frame-options
• x-permitted-cross-domain-policies
• x-powered-by
• x-xss-protection

Since you likely do not have every HTTP header’s name, potential values, suggested values, and overall purpose memorized, you can also pass in include_ref = TRUE to the function to get more information with decent textual descriptions like you saw in the screenshot (the Shiny app omits many fields).

The full reference is available in a data element:

data("http_headers")

dplyr::glimpse(http_headers)
## Observations: 184
## Variables: 14
## $ header_field_name    <chr> "A-IM", "Accept", "Accept-Additions", "Accept-Charset", "Accept-Datetime", "Accept-Encoding…
## $ type_1               <chr> "Permanent", "Permanent", "Permanent", "Permanent", "Permanent", "Permanent", "Permanent", …
## $ protocol             <chr> "http", "http", "http", "http", "http", "http", "http", "http", "http", "http", "http", "ht…
## $ status               <chr> "", "standard", "", "standard", "informational", "standard", "", "standard", "", "standard"…
## $ reference            <chr> "https://tools.ietf.org/html/rfc3229#section-10.5.3", "https://tools.ietf.org/html/rfc7231#…
## $ type_2               <chr> "Request", "Request", "Request", "Request", "Request", "Request", "Request", "Request", "Re…
## $ enable               <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, TRUE, FALSE, FAL…
## $ required             <lgl> NA, NA, NA, NA, NA, NA, NA, NA, TRUE, TRUE, NA, TRUE, NA, NA, NA, TRUE, NA, NA, NA, NA, NA,…
## $ https                <lgl> NA, NA, NA, NA, NA, NA, NA, NA, TRUE, TRUE, NA, TRUE, NA, NA, NA, TRUE, NA, NA, NA, NA, NA,…
## $ security_description <chr> "", "", "", "", "", "", "", "", "Sometimes an HTTP intermediary might try to detect viruses…
## $ security_reference   <chr> "", "", "", "", "", "", "", "", "https://tools.ietf.org/html/rfc5789#section-5", "https://t…
## $ recommendations      <chr> "", "", "", "", "", "", "", "", "Antivirus software scans for viruses or worms.", "Servers …
## $ cwe                  <chr> "", "", "", "", "", "", "", "", "CWE-509: Replicating Malicious Code (Virus or Worm)", "CWE…
## $ cwe_url              <chr> "\r", "\r", "\r", "\r", "\r", "\r", "\r", "\r", "https://cwe.mitre.org/data/definitions/509…

There will eventually be a lovely vignette with well-formatted sections that include the above information so you can reference it at your leisure (it’s great bedtime reading).

The http_headers object is fully documented but here’s what those fields mean:

• header_field_name: header field
• type_1: Permanent (in a standard); Provisional (experimental); Personal (unofficial)
• protocol: should always be http for now but may be different (e.g. quic)
• status: blank == unknown; otherwise the value describes the status well
• reference: where to look for more info
• type_2: Request (should only be found in requests); Response (should only be found in responses); Request/Response found in either; Reserved (not in use yet)
• enable: should you have this enabled
• required: Is this header required
• https: HTTPS-specific header?
• security_description: Information on the header
• security_reference: Extra external reference information on the header
• recommendations: Recommended setting(s)
• cwe: Associated Common Weakness Enumeration (CWE) identifier
• cwe_url: Associated CWE URL

Even Moar Headers

HTTP servers can spit out tons of headers and we can catch’em all with hdrs::explain_headers(). That function grabs the headers, merges in the full metadata from http_headers and returns a big ol’ data frame. We’ll only pull out the security reference URL for this last example:

explain_headers("https://community.rstudio.com/") %>% 
  dplyr::select(header, value, security_reference)
## # A tibble: 18 x 3
##    header               value                                                         security_reference                   
##    <chr>                <chr>                                                         <chr>                                
##  1 cache-control        no-cache, no-store                                            https://tools.ietf.org/html/rfc7234#…
##  2 connection           keep-alive                                                    ""                                   
##  3 content-encoding     gzip                                                          https://en.wikipedia.org/wiki/BREACH…
##  4 content-security-po… base-uri 'none'; object-src 'none'; script-src 'unsafe-eval'… https://www.owasp.org/index.php/List…
##  5 content-type         text/html; charset=utf-8                                      https://tools.ietf.org/html/rfc7231#…
##  6 date                 Tue, 05 Mar 2019 20:40:31 GMT                                 ""                                   
##  7 referrer-policy      strict-origin-when-cross-origin                               NA                                   
##  8 server               nginx                                                         https://tools.ietf.org/html/rfc7231#…
##  9 strict-transport-se… max-age=31536000                                              https://tools.ietf.org/html/rfc6797  
## 10 vary                 Accept-Encoding                                               ""                                   
## 11 x-content-type-opti… nosniff                                                       https://www.owasp.org/index.php/List…
## 12 x-discourse-route    list/latest                                                   NA                                   
## 13 x-download-options   noopen                                                        NA                                   
## 14 x-frame-options      SAMEORIGIN                                                    https://tools.ietf.org/html/rfc7034  
## 15 x-permitted-cross-d… none                                                          NA                                   
## 16 x-request-id         12322c6e-b47e-4960-b384-32138097886c                          NA                                   
## 17 x-runtime            0.106664                                                      NA                                   
## 18 x-xss-protection     1; mode=block                                                 https://www.owasp.org/index.php/List…

FIN

Have some fun and poke at some headers. Perhaps even use this to do a survey of key web sites in your field of work/study and see how well they rate. As usual, post PRs & issues at your fav social coding site.

Thanks for visiting r-craft.org
This article is originally published at https://rud.is/b
Please visit source website for post related comments.