Auto Tune requires the least amount of input from the user. It intelligently searches a large grid of experiments and only creates model experiments when the probability of successfully increasing the metric of interest is high. Sometimes it is able to find an optimal combination of hyperparameters by creating and evaluating only one third of all combinations. This efficiency will vary by dataset.
Auto Tune uses the default xAI Workbench grid specification. The required inputs are:
Class Type: Binomial (two different values in the class column) or Multinomial (more than two different values in the class column)
Metric to Optimize: This selection depends upon the Class Type
Experiment Mode: NFold cross validation or date split into Training, Test, and Validation
The metrics that can be optimized for Binomial class types are
- AUC (Area Under the ROC Curve),
- Log_Loss, and
- MCC (Matthews Correlation Coefficient).
For Multinomial class types, the metrics that can be optimized are:
- F1 Score
The first three are optimized for a specific class and are listed as Multi_Recall, Multi_Precision, and Multi_F1_Score.
Experiments Mode can be either NFold or Date Split. These are explained later in this manual.
You can edit the parameters used to create the grid that will be searched. If you slide the Edit Initial Parameters switch to on, the grid parameters will be shown and you can modify them. The default parameters cover many types of models and is a good choice for an initial run.
When the Execute button is clicked, the Auto Tune Process will start and you will be navigated to the model details page. From there, the Grid Results menu item in the Model Actions menu will show you the grid models as they are created and evaluated. Clicking on the Grid Results & Model Creation tab will show you the best model found so far.
You can select the All Results radio button to see all the grid models, if you want. The blue Create Model button will take you to a new Create Model page with the parameters from the corresponding grid experiment filled in. You need to specify a name and the data folder, but clicking the Execute button in this new Create Model page will create the corresponding model.
Please sign in to leave a comment.