readr 1.3.1

This article is originally published at https://www.tidyverse.org/articles/

readr 1.3.1 is now available on CRAN! Learn more about readr at https://readr.tidyverse.org. Detailed notes are always in the change log.

The readr package makes it easy to get rectangular data out of comma separated (csv), tab separated (tsv) or fixed width files (fwf) and into R. It is designed to flexibly parse many types of data found in the wild, while still cleanly failing when data unexpectedly changes. If you are new to readr, the best place to start is the data import chapter in R for data science.

The easiest way to install the latest version from CRAN is to install the whole tidyverse.

install.packages("tidyverse")

Alternatively, install just readr from CRAN:

install.packages("readr")

readr is part of the core tidyverse, so load it with:

library(tidyverse)
#> ── Attaching packages ────────────────────────────────── tidyverse 1.2.1 ──
#> ✔ ggplot2 3.1.0     ✔ purrr   0.2.5
#> ✔ tibble  1.4.2     ✔ dplyr   0.7.8
#> ✔ tidyr   0.8.2     ✔ stringr 1.3.1
#> ✔ readr   1.3.1     ✔ forcats 0.3.0
#> ── Conflicts ───────────────────────────────────── tidyverse_conflicts() ──
#> ✖ dplyr::filter() masks stats::filter()
#> ✖ dplyr::lag()    masks stats::lag()

There have been three releases of readr recently, this post will highlight the most significant changes across all three releases.

Breaking Changes

Blank line skipping

The logic for blank line skipping has been overhauled, which should make it easier to predict and more robust.

Now both commented lines and completely blank lines are skipped by default.

file <-
"# A file with comments

first_name,last_name
Mary,Sue
John,Doe
"

read_csv(file, comment = "#")
#> # A tibble: 2 x 2
#>   first_name last_name
#>   <chr>      <chr>    
#> 1 Mary       Sue      
#> 2 John       Doe

Some useful properties about skip.

• It is a lower bound on the number of lines that will actually be skipped. e.g. both of the following examples result in identical output.

read_csv(file, skip = 1)
#> # A tibble: 2 x 2
#>   first_name last_name
#>   <chr>      <chr>    
#> 1 Mary       Sue      
#> 2 John       Doe
read_csv(file, skip = 2)
#> # A tibble: 2 x 2
#>   first_name last_name
#>   <chr>      <chr>    
#> 1 Mary       Sue      
#> 2 John       Doe

• skip is enforced without looking at the line contents, so both commented and blank lines are counted.

So if you wanted to skip all the way including the first_name,last_name line you would need to use skip = 3. N.B. the results are identical with and without comment = "#".

read_csv(file, skip = 3, col_names = FALSE)
#> # A tibble: 2 x 2
#>   X1    X2   
#>   <chr> <chr>
#> 1 Mary  Sue  
#> 2 John  Doe

Integer column guessing

readr functions no longer guess columns are of type integer, instead these columns are guessed as numeric. Because R uses 32 bit integers and 64 bit doubles all integers can be stored in doubles, guaranteeing no loss of information. This change was made to remove errors when numeric columns were incorrectly guessed as integers. If you know a certain column is an integer and would like to read it as such you can do so by specifying the column type explicitly with the col_types argument.

file <-
"a,b
1,3
2,4
"

read_csv(file, col_type = c(b = "i"))
#> # A tibble: 2 x 2
#>       a     b
#>   <dbl> <int>
#> 1     1     3
#> 2     2     4

tibble subclass

readr now returns results with a spec_tbl_df subclass. This differs from a regular tibble only in that the spec attribute (which holds the column specification) is lost as soon as the object is subset and a normal tbl_df object is returned.

Historically tbl_df’s lost their attributes once they were subset. However recent versions of tibble retain the attributes when subsetting, so the spec_tbl_df subclass is needed to retain the previous behavior.

This should only break compatibility if you are explicitly checking the class of the returned object. A way to get backwards compatible behavior is to call subset with no arguments on your object, e.g. x[].

data <- read_csv(file)
class(data)
#> [1] "spec_tbl_df" "tbl_df"      "tbl"         "data.frame"

class(data[])
#> [1] "tbl_df"     "tbl"        "data.frame"

New features

Colored specifications

The most user visible change is coloration of the column specifications. The column types are now colored based on 4 broad classes

• Red - Character data (characters, factors)
• Green - Numeric data (doubles, integers)
• Yellow - Logical data (logicals)
• Blue - Temporal data (dates, times, datetimes)

By coloring the specification, we hope to make it easier to spot when a column differs from the rest or when guessing leads to import with an unexpected type.

The coloring can be disabled by setting options(crayon.enabled = FALSE).

Melt functions

There is now a family of melt_*() functions in readr. These functions store data in ‘long’ or ‘melted’ form, where each row corresponds to a single value in the dataset. This form is useful when your data is ragged and not rectangular.

file <-
"a,b,c
1,2
w,x,y,z"

readr::melt_csv(file)
#> # A tibble: 9 x 4
#>     row   col data_type value
#>   <dbl> <dbl> <chr>     <chr>
#> 1     1     1 character a    
#> 2     1     2 character b    
#> 3     1     3 character c    
#> 4     2     1 integer   1    
#> 5     2     2 integer   2    
#> 6     3     1 character w    
#> 7     3     2 character x    
#> 8     3     3 character y    
#> 9     3     4 character z

Thanks to Duncan Garmonsway (@nacnudus) for great work on the idea and implementation of the melt_*() functions!

Connection improvements

In previous versions of readr the connections were read into an in-memory raw vector, then passed to the readr functions. This made reading connections for small to medium datasets fast, but also meant that the dataset had to fit into memory at least twice (once for the raw data, once for the parsed data). It also meant that reading could not begin until the full vector was read through the connection.

We now write the connection to a temporary file (in the R temporary directory), than parse that temporary file. This means connections may take a little longer to be read, but also means they will no longer need to fit into memory. It also allows the use of the chunked readers to process the data in parts.

Future improvements to readr could allow it to parse data from connections in a streaming fashion, which would avoid many of the drawbacks of either approach.

Acknowledgements

Thank you to the 188 contributors who made this release possible by opening issues or submitting pull requests: @aaronschiff, @abelew, @Abhijitsj, @ADraginda, @adrtod, @ajdamico, @alessandro-gentilini, @arendsee, @batpigandme, @bbrewington, @benmarwick, @benwhalley, @benzipperer, @BerwinTurlach, @bheavner, @billdenney, @biostu24, @blablablerg, @blakeboswell, @brainfood, @brianstamper, @brisk022, @Brunohnp, @bschneidr, @cameron-faerber, @Carenser, @cboettig, @cderv, @ChiWPak, @christellacaze, @cimentadaj, @conchoecia, @cuttlefish44, @dan-reznik, @danielsjf, @dapperjapper, @DarioS, @DataStrategist, @DDWetzel, @DevGri, @dfjenkins3, @dhbrand, @dhimmel, @digital-dharma, @dioh, @djbirke, @dkulp2, @dongzhuoer, @dpprdan, @Dulani, @eddelbuettel, @ElToro1966, @EmilHvitfeldt, @Enchufa2, @epetrovski, @eric-pedersen, @EricHe98, @EruIluvatar, @evelynmitchell, @EvgenyPetrovsky, @gandalf-randolph, @GegznaV, @gergness, @ghost, @gksmyth, @gregrs-uk, @gwarnes-mdsol, @ha0ye, @hadley, @hbeale, @Henri-Lo, @hiltonmbr, @hmorzaria, @holstius, @hstojic, @huftis, @ilyasustun, @IndrajeetPatil, @isomorphisms, @isteves, @j-bartholome, @J535D165, @jaearick, @jasonserviss, @javierluraschi, @jdeboer, @jennybc, @jhpoelen, @jiho, @jimhester, @jimmo, @johncassil, @jomuller, @jonovik, @josiekre, @jpiggod, @jschelbert, @jstaf, @jtelleria, @jtelleriar, @jtr13, @juyeongkim, @jzadra, @kadekillary, @kendonB, @kerry-ja, @kevinushey, @kforner, @kk22boy, @KKulma, @kos59125, @kwstat, @laresbernardo, @lbartnik, @ldecicco-USGS, @Leo-Lee15, @lsusatyo-gcmlp, @martinjhnhadley, @martj42, @matthewarbo, @MatthieuStigler, @mawds, @md0u80c9, @mgirlich, @mikmart, @mirkhosro, @mohkar123, @momeara, @msberends, @msmall318, @Naboum, @nacnudus, @namelessjon, @nan1949, @nbenn, @neveldo, @nevrome, @NicolasImberty, @normandev, @olgamie, @osorensen, @pachevalier, @pakuipers, @pgensler, @PMassicotte, @prebours, @prosoitos, @QuLogic, @rafazaya, @ramiromagno, @randomgambit, @raw937, @rdisalv2, @rensa, @richierocks, @RickPack, @rmcd1024, @RomeroBarata, @roshnamohan, @ryanatanner, @s-fleck, @saghirb, @sambrady3, @seandavi, @seanpor, @sellisd, @sibojan, @SimonGoring, @sindile, @statsccpr, @stelsemeyer, @StevenMMortimer, @stianlagstad, @tarmenic, @tdsmith, @thanosgatos, @tigertoes, @tomsing1, @TPreijers, @tres-pitt, @tungmilan, @wgrundlingh, @wibeasley, @willemvdb42, @xhudik, @yutannihilation, @Zack-83, and @zeehio

Thanks for visiting r-craft.org
This article is originally published at https://www.tidyverse.org/articles/
Please visit source website for post related comments.