(^i%ddlmZddlmZddlZddlZddlZddlm Z ddl m Z ddl m Z ddlmZer ddlmZddlmZndd lmZed Ze d Gd d e Zy)) annotations) TYPE_CHECKINGN)experimental_class) BasePruner)StudyDirection) FrozenTrial)Literal) _LazyImportz scipy.statsz3.6.0c2eZdZdZddd ddZd dZy) WilcoxonPruneraPruner based on the `Wilcoxon signed-rank test `__. This pruner performs the Wilcoxon signed-rank test between the current trial and the current best trial, and stops whenever the pruner is sure up to a given p-value that the current trial is worse than the best one. This pruner is effective for optimizing the mean/median of some (costly-to-evaluate) performance scores over a set of problem instances. Example applications include the optimization of: * the mean performance of a heuristic method (simulated annealing, genetic algorithm, SAT solver, etc.) on a set of problem instances, * the k-fold cross-validation score of a machine learning model, and * the accuracy of outputs of a large language model (LLM) on a set of questions. There can be "easy" or "hard" instances (the pruner handles correspondence of the instances between different trials). In each trial, it is recommended to shuffle the evaluation order, so that the optimization doesn't overfit to the instances in the beginning. When you use this pruner, you must call ``Trial.report(value, step)`` method for each step (instance id) with the evaluated value. The instance id may not be in ascending order. This is different from other pruners in that the reported value need not converge to the real value. To use pruners such as :class:`~optuna.pruners.SuccessiveHalvingPruner` in the same setting, you must provide e.g., the historical average of the evaluated values. .. seealso:: Please refer to :meth:`~optuna.trial.Trial.report`. Example: .. testcode:: import optuna import numpy as np # We minimize the mean evaluation loss over all the problem instances. def evaluate(param, instance): # A toy loss function for demonstrative purpose. return (param - instance) ** 2 problem_instances = np.linspace(-1, 1, 100) def objective(trial): # Sample a parameter. param = trial.suggest_float("param", -1, 1) # Evaluate performance of the parameter. results = [] # For best results, shuffle the evaluation order in each trial. instance_ids = np.random.permutation(len(problem_instances)) for instance_id in instance_ids: loss = evaluate(param, problem_instances[instance_id]) results.append(loss) # Report loss together with the instance id. # CAVEAT: You need to pass the same id for the same instance, # otherwise WilcoxonPruner cannot correctly pair the losses across trials and # the pruning performance will degrade. trial.report(loss, instance_id) if trial.should_prune(): # Return the current predicted value instead of raising `TrialPruned`. # This is a workaround to tell the Optuna about the evaluation # results in pruned trials. (See the note below.) return sum(results) / len(results) return sum(results) / len(results) study = optuna.create_study(pruner=optuna.pruners.WilcoxonPruner(p_threshold=0.1)) study.optimize(objective, n_trials=100) .. note:: This pruner cannot handle ``infinity`` or ``nan`` values. Trials containing those values are never pruned. .. note:: If :func:`~optuna.trial.FrozenTrial.should_prune` returns :obj:`True`, you can return an estimation of the final value (e.g., the average of all evaluated values) instead of ``raise optuna.TrialPruned()``. This is a workaround for the problem that currently there is no way to tell Optuna the predicted objective value for trials raising :class:`optuna.TrialPruned`. Args: p_threshold: The p-value threshold for pruning. This value should be between 0 and 1. A trial will be pruned whenever the pruner is sure up to the given p-value that the current trial is worse than the best trial. The larger this value is, the more aggressive pruning will be performed. Defaults to 0.1. .. note:: This pruner repeatedly performs statistical tests between the current trial and the current best trial with increasing samples. The false-positive rate of such a sequential test is different from performing the test only once. To get the nominal false-positive rate, please specify the Pocock-corrected p-value. n_startup_steps: The number of steps before which no trials are pruned. Pruning starts only after you have ``n_startup_steps`` steps of available observations for comparison between the current trial and the best trial. Defaults to 2. Note that the trial is not pruned at the first and second steps even if the `n_startup_steps` is set to 0 or 1 due to the lack of enough data for comparison. g?) p_thresholdn_startup_stepsc|dkrtd|dd|cxkrdksntd|d||_||_y)Nrz,n_startup_steps must be nonnegative but got .gg?z,p_threshold must be between 0 and 1 but got ) ValueError_n_startup_steps _p_threshold)selfrrs Y/var/www/html/hubwallet-dev/venv/lib/python3.12/site-packages/optuna/pruners/_wilcoxon.py__init__zWilcoxonPruner.__init__sV Q KOK\\]^_ _k(S(KK=XYZ[ [ /'ct|jdk(rytjt |jj j \}}tjtj|r$tjd|jdy |j}t|jdk(rtjdytjt |jj j \}}tjtj|r$tjd|jdytj||d \}} } t| t|krtjd || || z } t| td |j kry|j"t$j&k(r2d } t)|t|z t)|t|z k} n1d } t)|t|z t)|t|z k\} t+j,| | dj.}||j0kr| ryt3||j0kS#t$rYywxYw)NrFz4The intermediate values of the current trial (trial zB) contain infinity/NaNs. WilcoxonPruner will not prune this trial.zThe best trial has no intermediate values so WilcoxonPruner cannot prune trials. If you have added the best trial with Study.add_trial, please consider setting intermediate_values argument.z1The intermediate values of the best trial (trial zI) contain infinity/NaNs. WilcoxonPruner will not prune the current trial.T)return_indiceszxWilcoxonPruner finds steps existing in the current trial but does not exist in the best trial. Those values are ignored.r lessgreaterzsplit) alternative zero_method)lenintermediate_valuesnparraylistitemsTanyisfinitewarningswarnnumber best_trialr intersect1dmaxr directionrMAXIMIZEsumsswilcoxonpvaluerbool)rstudytrialsteps step_valuesr, best_stepsbest_step_values_idx1idx2 diff_valuesaltaverage_is_bestps rprunezWilcoxonPruner.prunes u(( )Q .XXd5+D+D+J+J+L&MNPP{ 662;;{++ , MMFu||nUST  ))J z-- .! 3 MM0  ')xxZ5S5S5Y5Y5[0\']'_'_$ $ 662;;/00 1 MMCJDUDUCVWZ[ ujN 4 t9s;' ' MM,  "$'*:4*@@ { c!T%:%:; ; ??n55 5C!"23c:J6KKsPK P!!OC!"23c:J6KKsPK P!!O KK ( K R R t  _ A)))**w  s+ J77 KKN)rfloatrintreturnNone)r6z'optuna.study.Study'r7rrFr5)__name__ __module__ __qualname____doc__rrCrrr r s:lb! ( ( (  (J+rr ) __future__rtypingrr)numpyr"optunaoptuna._experimentalroptuna.prunersroptuna.study._study_directionr optuna.trialrr scipy.statsstatsr2optuna._importsr r rLrrrXs["  3%8$+ ] #BGG+ZG+G+r