(^i7ddlmZddlZddlmZddlZddlZddlmZddlm Z ddl m Z ddl m Z ejeZGdd e Zy) ) annotationsN) Container)logging) BasePruner)SuccessiveHalvingPruner) TrialStatecteZdZdZ d d dZd dZd dZd dZ ddZ ddZ y)HyperbandPruneruPruner using Hyperband. As SuccessiveHalving (SHA) requires the number of configurations :math:`n` as its hyperparameter. For a given finite budget :math:`B`, all the configurations have the resources of :math:`B \over n` on average. As you can see, there will be a trade-off of :math:`B` and :math:`B \over n`. `Hyperband `__ attacks this trade-off by trying different :math:`n` values for a fixed budget. .. note:: * In the Hyperband paper, the counterpart of :class:`~optuna.samplers.RandomSampler` is used. * Optuna uses :class:`~optuna.samplers.TPESampler` by default. * `The benchmark result `__ shows that :class:`optuna.pruners.HyperbandPruner` supports both samplers. .. note:: If you use ``HyperbandPruner`` with :class:`~optuna.samplers.TPESampler`, it's recommended to consider setting larger ``n_trials`` or ``timeout`` to make full use of the characteristics of :class:`~optuna.samplers.TPESampler` because :class:`~optuna.samplers.TPESampler` uses some (by default, :math:`10`) :class:`~optuna.trial.Trial`\ s for its startup. As Hyperband runs multiple :class:`~optuna.pruners.SuccessiveHalvingPruner` and collects trials based on the current :class:`~optuna.trial.Trial`\ 's bracket ID, each bracket needs to observe more than :math:`10` :class:`~optuna.trial.Trial`\ s for :class:`~optuna.samplers.TPESampler` to adapt its search space. Thus, for example, if ``HyperbandPruner`` has :math:`4` pruners in it, at least :math:`4 \times 10` trials are consumed for startup. .. note:: Hyperband has several :class:`~optuna.pruners.SuccessiveHalvingPruner`\ s. Each :class:`~optuna.pruners.SuccessiveHalvingPruner` is referred to as "bracket" in the original paper. The number of brackets is an important factor to control the early stopping behavior of Hyperband and is automatically determined by ``min_resource``, ``max_resource`` and ``reduction_factor`` as :math:`\mathrm{The\ number\ of\ brackets} = \mathrm{floor}(\log_{\texttt{reduction}\_\texttt{factor}} (\frac{\texttt{max}\_\texttt{resource}}{\texttt{min}\_\texttt{resource}})) + 1`. Please set ``reduction_factor`` so that the number of brackets is not too large (about 4 – 6 in most use cases). Please see Section 3.6 of the `original paper `__ for the detail. .. note:: ``HyperbandPruner`` computes bracket ID for each trial with a function taking ``study_name`` of :class:`~optuna.study.Study` and :attr:`~optuna.trial.Trial.number`. Please specify ``study_name`` to make the pruning algorithm reproducible. Example: We minimize an objective function with Hyperband pruning algorithm. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) n_train_iter = 100 def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.HyperbandPruner( min_resource=1, max_resource=n_train_iter, reduction_factor=3 ), ) study.optimize(objective, n_trials=20) Args: min_resource: A parameter for specifying the minimum resource allocated to a trial noted as :math:`r` in the paper. A smaller :math:`r` will give a result faster, but a larger :math:`r` will give a better guarantee of successful judging between configurations. See the details for :class:`~optuna.pruners.SuccessiveHalvingPruner`. max_resource: A parameter for specifying the maximum resource allocated to a trial. :math:`R` in the paper corresponds to ``max_resource / min_resource``. This value represents and should match the maximum iteration steps (e.g., the number of epochs for neural networks). When this argument is "auto", the maximum resource is estimated according to the completed trials. The default value of this argument is "auto". .. note:: With "auto", the maximum resource will be the largest step reported by :meth:`~optuna.trial.Trial.report` in the first, or one of the first if trained in parallel, completed trial. No trials will be pruned until the maximum resource is determined. .. note:: If the step of the last intermediate value may change with each trial, please manually specify the maximum possible step to ``max_resource``. reduction_factor: A parameter for specifying reduction factor of promotable trials noted as :math:`\eta` in the paper. See the details for :class:`~optuna.pruners.SuccessiveHalvingPruner`. bootstrap_count: Parameter specifying the number of trials required in a rung before any trial can be promoted. Incompatible with ``max_resource`` is ``"auto"``. See the details for :class:`~optuna.pruners.SuccessiveHalvingPruner`. c||_||_||_g|_||_d|_g|_d|_t|jts3|jdk7r$tdj|j|jdkDr4|jdk(r$tdj|jyy)NrautozEThe 'max_resource' should be integer or 'auto'. But max_resource = {}z_bootstrap_count > 0 and max_resource == 'auto' are mutually incompatible, bootstrap_count is {}) _min_resource _max_resource_reduction_factor_pruners_bootstrap_count_total_trial_allocation_budget_trial_allocation_budgets _n_brackets isinstanceint ValueErrorformat)self min_resource max_resourcereduction_factorbootstrap_counts Z/var/www/html/hubwallet-dev/venv/lib/python3.12/site-packages/optuna/pruners/_hyperband.py__init__zHyperbandPruner.__init__s*)!179 /./+46&'+$,,c2t7I7IV7S((.t/A/A(B   1 $););v)ECCI6$J_J_C` *F $cTt|jdk(r*|j|t|jdk(ry|j||}tj dj ||j||}|j|j||S)NrFz{}th bracket is selected) lenr_try_initialization_get_bracket_id_loggerdebugr_create_bracket_studyprune)rstudytrial bracket_id bracket_studys rr(zHyperbandPruner.prunes t}}  "  $ $U +4==!Q&))%7  077 CD225*E }}Z(..}eDDr c~|jdk(r^|jdtjf}|Dcgc]}|j|j}}|syt |dz|_t |jtsJ|jRtjtj|j|jz |jdz|_tjdj!|jt#|jD]}|j%|}|xj&|z c_|j(j+|t-|j|j||j.}|j0j+|ycc}w)Nr FdeepcopystateszHyperband has {} brackets)rrmin_early_stopping_rater)r get_trialsrCOMPLETE last_stepmaxrrrmathfloorlogr rr%r&rrange"_calculate_trial_allocation_budgetrrappendrrr)rr)trialstn_stepsr+trial_allocation_budgetpruners rr#z#HyperbandPruner._try_initializations    '%%uj>Q>Q=S%TF,2Nqakk6Mq{{NGN!$W!1D $,,c222    # HHT//$2D2DDdF\F\]    1889I9IJK 0 01 )J&*&M&Mj&Y #  / /3J J /  * * 1 12I J,!//!%!7!7(2 $ 5 5 F MM  ( )5Os F:F:c|jJ|jdz |z }tj|j|j|zz|dzz S)aCompute the trial allocated budget for a bracket of ``bracket_id``. In the `original paper `, the number of trials per one bracket is referred as ``n`` in Algorithm 1. Since we do not know the total number of trials in the leaning scheme of Optuna, we calculate the ratio of the number of trials here instead. r1)rr7ceilr)rr+ss rr;z2HyperbandPruner._calculate_trial_allocation_budgetsY+++   q : -yy))T-C-CQ-FG1q5QRRr cpt|jdk(ry|jJtjdj |j |jj|jz}t|jD]}||j|z}|dks|cSJd)zCompute the index of bracket for a trial of ``trial_number``. The index of a bracket is noted as :math:`s` in `Hyperband paper `__. rz{}_{}z This line should be unreachable.) r"rrbinasciicrc32r study_namenumberencoderr:r)rr)r*nr+s rr$zHyperbandPruner._get_bracket_ids t}}  "+++ NN7>>%*:*:ELLIPPR S11 2   0 01 "J // ; ;A1u!! " 988ur cbGfddtjj|||S)NcdeZdZdZ dfd Z d dfd Zd dZfdZxZS) ._BracketStudy)r3 _get_trials directions direction _directions_storage _study_idrArH _bracket_idsamplerr=_is_multi_objectivestop_study _thread_localct||j|j|j|||_||_y)N)rHstoragerVrA)superrrHrSrVrYrU)rr)rAr+ __class__s rrzEHyperbandPruner._create_bracket_study.._BracketStudy.__init__!sB $//!NN!MM! ! $ #- r ct|||}|j}t|tsJ|Dcgc]$}|j |||j k(s#|&c}Scc}w)Nr.)r]rOrArr r$rU)rr/r0r=rAr>r^s rr3zGHyperbandPruner._create_bracket_study.._BracketStudy.get_trials-sa ,hv,N!&/:::#)aaV-C-CD!-LPTP`P`-`aaas $A!A!c8|jjy)N)rYrX)rs rrXzAHyperbandPruner._create_bracket_study.._BracketStudy.stop7s   "r c|jvrtdj|tj ||S)Nz-_BracketStudy does not have attribute of '{}') _VALID_ATTRSAttributeErrorrobject__getattribute__)r attr_name _BracketStudys rrezMHyperbandPruner._create_bracket_study.._BracketStudy.__getattribute__<sAM$>$>>(GNNyY"224CCr )r)'optuna.study.Study'rAr r+rreturnNone)TN)r/boolr0zContainer[TrialState] | Noneriz list['optuna.trial.FrozenTrial'])rirj) __name__ __module__ __qualname__rbrr3rXre __classcell__)r^rgs@rrgrN smL& .1 .;J .X[ . ."&7; b b5 b2  b #  D Dr rg)optunar)Study)rr)r+rgs @rr'z%HyperbandPruner._create_bracket_studys-5 DFLL..5 DnUD*55r N)r1r r) rrrz str | intrrrrrirj)r)rhr*'optuna.trial.FrozenTrial'rirk)r)rhrirj)r+rrir)r)rhr*rsrir)r)rhr+rrirh) rlrmrn__doc__rr(r#r;r$r'r rr r s|@"( !       8 E()T S9)92L9 90>6)>67:>6 >6r r ) __future__rrFcollections.abcrr7rproptuna.pruners._baser"optuna.pruners._successive_halvingroptuna.trial._stater get_loggerrlr%r rur rr|sB"% +F* '  X &s6js6r