Filter Based Feature Selection or Permutation Feature Importance

Ivan Casana-Gallen 26 Reputation points


I am aware that Feature Based Selection kind of measure the prediction power of each variable before any model has been built.

I know that Permutation Feature importance measures, once the model has been built, how relevant each variable is. Thanks to this we can prune our model and strike a good balance between accuracy and simplicity. Fair enough.

However, most of the time I encounter contradictory messages. There is a variable with 0 importance on the Feature Based Selection but it ends up becoming one of the most relevant variables of my model according to its ranking on Permutation Feature importance.

So I guess I cannot rely on what Feature Based Selection says in order to do a preliminary assessment. I guess I should see it as some theoretical exercise.

Thank you

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,711 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. romungi-MSFT 43,686 Reputation points Microsoft Employee

    @Ivan Casana-Gallen Thanks for adding details about your observations. As documented under Permutation feature importance module the rankings the component provides are often different from the ones we get from Filter Based Feature Selection as the Filter Based Feature Selection calculates scores before a model is created. Thanks!!

    0 comments No comments