collapse version 2.0.17

• In GRP.default(), the "group.starts" attribute is always returned, even if there is only one group or every observation is its own group. Thanks @JamesThompsonC (#631).

• Fixed a bug in pivot() if na.rm = TRUE and how = "wider"|"recast" and there are multiple value columns with different missingness patterns. In this case na_omit(values) was applied with default settings to the original (long) value columns, implying potential loss of information. The fix applies na_omit(values, prop = 1), i.e., only removes completely missing rows.

• qDF()/qDT()/qTBL() now allow a length-2 vector of names to row.names.col if X is a named atomic vector, e.g., qDF(fmean(mtcars), c("cars", "mean")) gives the same as pivot(fmean(mtcars, drop = FALSE), names = list("car", "mean")).

• Added a subsection on using internal (ad-hoc) grouping to the collapse for tidyverse users vignette.

• qsu() now adds a WeightSum column giving the sum of (non-zero or missing) weights if the w argument is used. Thanks @mayer79 for suggesting (#650). For panel data (pid) the 'Between' sum of weights is also simply the number of groups, and the 'Within' sum of weights is the 'Overall' sum of weights divided by the number of groups.

• Fixed an inaccuracy in fquantile()/fnth() with weights: As per documentation the target sum is sumwp = (sum(w) - min(w)) * p, however, in practice, the weight of the minimum element of x was used instead of the minimum weight. Since the smallest element in the sample usually has a small weight this was unnoticed for a long while, but thanks to @Jahnic-kb now reported and fixed (#659).

• Fixed a bug in recode_char() when regex = TRUE and the default argument was used. Thanks @alinacherkas for both reporing and fixing (#654).

collapse version 2.0.16

• Fixes an installation bug on some Linux systems (conflicting types) (#613).

• collapse now enforces string encoding in fmatch() / join(), which caused problems if strings being matched had different encodings (#566, #579, and #618). To avoid noticeable performance implications, checks are done heuristically, i.e., the first, middle and last string of a character vector are checked, and if not UTF8, the entire vector is coerced to UTF8 strings before the matching process. In general, character vectors in R can contain strings of different encodings, but this is not the case with most regular data. For performance reasons, collapse assumes that character vectors are uniform in terms of string encoding.

• Fixes a bug using qualified names for fast statistical functions inside across() (#621, thanks @alinacherkas).

• collapse now depends on R >= 3.4.0 due to the enforcement of STRICT_R_HEADERS = 1 from R v4.5.0. In particular R API functions were renamed Calloc -> R_Calloc and Free -> R_Free.

collapse version 2.0.15

• Some changes on the C-side to move the package closer to C API compliance (demanded by R-Core). One notable change is that gsplit() no longer supports S4 objects (because SET_S4_OBJECT is not part of the API and asS4() is too expensive for tight loops). I cannot think of a single example where it would be necessary to split an S4 object, but if you do have applications please file an issue.

• pivot() has new arguments FUN = "last" and FUN.args = NULL, allowing wide and recast pivots with aggregation (default last value as before). FUN currently supports a single function returning a scalar value. Fast Statistical Functions receive vectorized execution. FUN.args can be used to supply a list of function arguments, including data-length arguments such as weights. There are also a couple of internal functions callable using function strings: "first", "last", "count", "sum", "mean", "min", or "max". These are built into the reshaping C-code and thus extremely fast. Thanks @AdrianAntico for the request (#582).

• join() now provides enhanced verbosity, indicating the average order of the join between the two tables, e.g.

  join(data.frame(id = c(1, 2, 2, 4)), data.frame(id = c(rep(1,4), 2:3)))
  #> left join: x[id] 3/4 (75%) <1.5:1st> y[id] 2/6 (33.3%)
  #>   id
  #> 1  1
  #> 2  2
  #> 3  2
  #> 4  4
  join(data.frame(id = c(1, 2, 2, 4)), data.frame(id = c(rep(1,4), 2:3)), multiple = TRUE)
  #> left join: x[id] 3/4 (75%) <1.5:2.5> y[id] 5/6 (83.3%)
  #>   id
  #> 1  1
  #> 2  1
  #> 3  1
  #> 4  1
  #> 5  2
  #> 6  2
  #> 7  4

• In collap(), with multiple functions passed to FUN or catFUN and return = "long", the "Function" column is now generated as a factor variable instead of character (which is more efficient).

collapse version 2.0.14

• Updated 'collapse and sf' vignette to reflect the recent support for units objects, and added a few more examples.

• Fixed a bug in join() where a full join silently became a left join if there are no matches between the tables (#574). Thanks @D3SL for reporting.

• Added function group_by_vars(): A standard evaluation version of fgroup_by() that is slimmer and safer for programming, e.g. data |> group_by_vars(ind1) |> collapg(custom = list(fmean = ind2, fsum = ind3)). Or, using magrittr:

library(magrittr)
set_collapse(mask = "manip") # for fgroup_vars -> group_vars

data %>% 
  group_by_vars(ind1) %>% {
  add_vars(
    group_vars(., "unique"),
    get_vars(., ind2) %>% fmean(keep.g = FALSE) %>% add_stub("mean_"),
    get_vars(., ind3) %>% fsum(keep.g = FALSE) %>% add_stub("sum_")
  ) 
}

• Added function as_integer_factor() to turn factors/factor columns into integer vectors. as_numeric_factor() already exists, but is memory inefficient for most factors where levels can be integers.

• join() now internally checks if the rows of the joined datasets match exactly. This check, using identical(m, seq_row(y)), is inexpensive, but, if TRUE, saves a full subset and deep copy of y. Thus join() now inherits the intelligence already present in functions like fsubset(), roworder() and funique() - a key for efficient data manipulation is simply doing less.

• In join(), if attr = TRUE, the count option to fmatch() is always invoked, so that the attribute attached always has the same form, regardless of verbose or validate settings.

• roworder[v]() has optional setting verbose = 2L to indicate if x is already sorted, making the call to roworder[v]() redundant.

collapse version 2.0.13

• collapse now explicitly supports xts/zoo and units objects and concurrently removes an additional check in the .default method of statistical functions that called the matrix method if is.matrix(x) && !inherits(x, "matrix"). This was a smart solution to account for the fact that xts objects are matrix-based but don't inherit the "matrix" class, thus wrongly calling the default method. The same is the case for units, but here, my recent more intensive engagement with spatial data convinced me that this should be changed. For one, under the previous heuristic solution, it was not possible to call the default method on a units matrix, e.g., fmean.default(st_distance(points_sf)) called fmean.matrix() and yielded a vector. This should not be the case. Secondly, aggregation e.g. fmean(st_distance(points_sf)) or fmean(st_distance(points_sf), g = group_vec) yielded a plain numeric object that lost the units class (in line with the general attribute handling principles). Therefore, I have now decided to remove the heuristic check within the default methods, and explicitly support zoo and units objects. For Fast Statistical Functions, the methods are FUN.zoo <- function(x, ...) if(is.matrix(x)) FUN.matrix(x, ...) else FUN.default(x, ...) and FUN.units <- function(x, ...) if(is.matrix(x)) copyMostAttrib(FUN.matrix(x, ...), x) else FUN.default(x, ...). While the behavior for xts/zoo remains the same, the behavior for units is enhanced, as now the class is preserved in aggregations (the .default method preserves attributes except for ts), and it is possible to manually invoke the .default method on a units matrix and obtain an aggregate statistic. This change may impact computations on other matrix based classes which don't inherit from "matrix" (mts does inherit from "matrix", and I am not aware of any other affected classes, but user code like m <- matrix(rnorm(25), 5); class(m) <- "bla"; fmean(m) will now yield a scalar instead of a vector. Such code must be adjusted to either class(m) <- c("bla", "matrix") or fmean.matrix(m)). Overall, the change makes collapse behave in a more standard and predictable way, and enhances its support for units objects central in the sf ecosystem.

• fquantile() now also preserves the attributes of the input, in line with quantile().

collapse version 2.0.12

• Fixes some issues with signed int overflows inside hash functions and possible protect bugs flagged by RCHK. With few exceptions these fixes are cosmetic to appease the C/C++ code checks on CRAN.

collapse version 2.0.11

• An article on collapse has been submitted to the Journal of Statistical Software. The preprint is available through arXiv.

• Removed magrittr from most documentation examples (using base pipe).

• Improved plot.GRP a little bit - on request of JSS editors.

collapse version 2.0.10

• Fixed a bug in fmatch() when matching integer vectors to factors. This also affected join().

• Improved cross-platform compatibility of OpenMP flags. Thanks @kalibera.

• Added stub = TRUE argument to the grouped_df methods of Fast Statistical Functions supporting weights, to be able to remove or alter prefixes given to aggregated weights columns if keep.w = TRUE. Globally, users can set st_collapse(stub = FALSE) to disable this prefixing in all statistical functions and operators.

collapse version 2.0.9

• Added functions na_locf() and na_focb() for fast basic C implementations of these procedures (optionally by reference). replace_na() now also has a type argument which supports options "locf" and "focb" (default "const"), similar to data.table::nafill. The implementation also supports character data and list-columns (NULL/empty elements). Thanks @BenoitLondon for suggesting (#489). I note that na_locf() exists in some other packages (such as imputeTS) where it is implemented in R and has additional options. Users should utilize the flexible namespace i.e. set_collapse(remove = "na_locf") to deal with this.

• Fixed a bug in weighted quantile estimation (fquantile()) that could lead to wrong/out-of-range estimates in some cases. Thanks @zander-prinsloo for reporting (#523).

• Improved right join such that join column names of x instead of y are preserved. This is more consistent with the other joins when join columns in x and y have different names.

• More fluent and safe interplay of 'mask' and 'remove' options in set_collapse(): it is now seamlessly possible to switch from any combination of 'mask' and 'remove' to any other combination without the need of setting them to NULL first.

collapse version 2.0.8

• In pivot(..., values = [multiple columns], labels = "new_labels_column", how = "wieder"), if the columns selected through values already have variable labels, they are concatenated with the new labels provided through "new_labels_col" using " - " as a separator (similar to names where the separator is "_").

• whichv() and operators %==%, %!=% now properly account for missing double values, e.g. c(NA_real_, 1) %==% c(NA_real_, 1) yields c(1, 2) rather than 2. Thanks @eutwt for flagging this (#518).

• In setv(X, v, R), if the type of R is greater than X e.g. setv(1:10, 1:3, 9.5), then a warning is issued that conversion of R to the lower type (real to integer in this case) may incur loss of information. Thanks @tony-aw for suggesting (#498).

• frange() has an option finite = FALSE, like base::range. Thanks @MLopez-Ibanez for suggesting (#511).

• varying.pdata.frame(..., any_group = FALSE) now unindexes the result (as should be the case).