Abess
abess is a machine learning method designed to address the problem of best subset selection. It aims to determine which features or variables are crucial for optimal model performance when provided with a dataset and a prediction task. abess was introduced by Zhu in 2020 and it dynamically selects the appropriate model size adaptively, eliminating the need for selecting regularization parameters.
abess is applicable in various statistical and machine learning tasks, including linear regression, the Single-index model, and other common predictive models.
abess can also be applied in biostatistics.
Basic Form
The basic form of abess is employed to address the optimal subset selection problem in general linear regression. abess is an method, it is characterized by its polynomial time complexity and the property of providing both unbiased and consistent estimates.In the context of linear regression, assuming we have knowledge of independent samples, where and, we define and. The following equation represents the general linear regression model:
To obtain appropriate parameters, one can consider the loss function for linear regression:
In abess, the initial focus is on optimizing the loss function under the constraint. That is, we consider the following problem:
where represents the desired size of the support set, and is the norm of the vector.
To address the optimization problem described above, abess iteratively exchanges an equal number of variables between the active set and the inactive set. In each iteration, the concept of sacrifice is introduced as follows:
- For j in the active set :
- For j in the inactive set :
- : This represents the estimate of obtained in the previous iteration.
- : It denotes the estimated active set from the previous iteration.
- : This is a vector where the j-th element is set to 0, while the other elements are the same as.
- : Here, represents a vector where all elements are 0 except the j-th element.
- : This is calculated based on the equation mentioned.
In abess, select an appropriate and optimize the above problem for active sets size using the information criterion to adaptively choose the appropriate active set size and obtain its corresponding abess estimator.
Generalizations
The splicing algorithm in abess can be employed for subset selection in other models.Distribution-Free Location-Scale Regression
In 2023, Siegfried extends abess to the case of Distribution-Free and Location-Scale. Specifically, it considers the optimization problemsubject to where is a loss function, is a parameter vector, and are vectors, and is a data vector.
This approach, demonstrated across various applications, enables parsimonious regression modeling for arbitrary outcomes while maintaining interpretability through innovative subset selection procedures.
Groups Selection
In 2023, Zhang applied the splicing algorithm to group selection, optimizing the following model:Here are the symbols involved:
- : Total number of feature groups, representing the existence of non-overlapping feature groups in the dataset.
- : Index set for the -th feature group, where ranges from 1 to, representing the feature grouping structure in the data.
- : Model size, a positive integer determined from the data, limiting the number of selected feature groups.
Regression with Corrupted Data
Single Index Models
In 2023, Tang applied the splicing algorithm to optimal subset selection in the Single-index model.The form of the Single Index Model is given by
where is the parameter vector,
is the error term.
The corresponding loss function is defined as
where is the rank vector, is the rank of in.
The Estimation Problem addressed by this algorithm is
Eographically Weighted Regression Model
In 2023, Wu applied the splicing algorithm to geographically weighted regression. GWR is a spatial analysis method, and Wu's research focuses on improving GWR performance in handling geographical data regression modeling. This is achieved through the application of an l0-norm variable adaptive selection method, which simultaneously performs model selection and coefficient optimization, enhancing the accuracy of regression modeling for geographic spatial data.Distributed Systems
In 2023, Chen introduced an innovative method addressing challenges in high-dimensional distributed systems, proposing an efficient algorithm for abess.A distributed system is a computational model that distributes computing tasks across multiple independent nodes to achieve more efficient, reliable, and scalable data processing. In a distributed system, individual computing nodes can work simultaneously, collaboratively completing the overall tasks, thereby enhancing system performance and processing capabilities.
However, within distributed systems, there is a lack of efficient algorithms for optimal subset selection. To address this gap, Chen introduces a novel communication-efficient approach for handling optimal subset selection in distributed systems.