Collapse infrequent categorical levels — step

step_other2() creates a specification of a recipe step that will potentially pool infrequently occurring values into an "other_ml" category.

Usage

step_other2(
  recipe,
  ...,
  role = NA,
  trained = FALSE,
  threshold = 0.05,
  other = "other_ml",
  single_low_level = c("as-is", "rename"),
  objects = NULL,
  skip = FALSE,
  id = recipes::rand_id("other2")
)

Arguments

recipe: A recipe object. The step will be added to the sequence of operations for this recipe.
...: One or more selector functions to choose variables for this step. See recipes::selections() for more details.
role: Not used by this step since no new variables are created.
trained: A logical to indicate if the quantities for preprocessing have been estimated.
threshold: A numeric value between 0 and 1, or an integer greater or equal to one. If less than one, then factor levels with a rate of occurrence in the training set below threshold will be pooled to other. If greater or equal to one, then this value is treated as a frequency and factor levels that occur less than threshold times will be pooled to other.
other: A single character value for the other category, defaults to "other_ml".
single_low_level: character controlling handling of a single low rate/frequency class. Defaults to 'as-is', where data is unmodified if only a single level meets the criterion for pooling. This is different from recipes::step_other()'s behavior ('rename'), where the low rate/frequency class would not be pooled with other classes but renamed to other
objects: A list of objects that contain the information to pool infrequent levels that is determined by recipes::prep().
skip: A logical. Should the step be skipped when the recipe is baked by recipes::bake()? While all operations are baked when recipes::prep() is run, some operations may not be able to be conducted on new data (e.g. processing the outcome variable(s)). Care should be taken when using skip = TRUE as it may affect the computations for subsequent operations.
id: A character string that is unique to this step to identify it.

Value

An updated version of recipe with the new step added to the sequence of any existing operations.

Details

The overall proportion (or total counts) of the categories are computed. The other category is used to pool any two or more categorical levels whose individual proportion (or frequency) in the training set is less than threshold.

If no pooling is done the data are unmodified (although character data may be changed to factors based on the value of strings_as_factors in recipes::prep()/recipes::recipe()). Otherwise, a factor is always returned with different factor levels.

If threshold is less than the largest category proportion, all levels except for the most frequent are collapsed to the other level.

If other_ml is in the list of discarded levels, no error occurs.

If no pooling is done, novel factor levels are converted to missing. If pooling is needed, they will be placed into the other_ml category.

When data to be processed contains novel levels (i.e., not contained in the training set), the other category is assigned.

Differences to step_other()

a single class subject to lumping is kept as-is as opposed to renamed to other (during prep)
If the level defined in other is an original class level that was not subject to pooling, the user is informed, but no error is raised.

TODO check message in test case

novel factor levels are not pooled with an existing lumped category (during bake)

Tidying

When you recipes::tidy() this step, a tibble is returned with columns terms, retained , and id:

terms: character, the selectors or variables selected
retained: character, factor levels not pulled into other
id: character, id of this step

Case weights

This step performs an unsupervised operation that can utilize case weights. As a result, case weights are only used with frequency weights. For more information, see the documentation in recipes::case_weights and the examples on tidymodels.org.

The underlying operation does not allow for case weights.

Author

This step is based on recipes::step_other() with only minor modifications.