(^iR8ddlmZddlmZddlmZddlZddlZddlm Z ddlm Z ddl m Z ddl m Z dd l mZdd l mZdd lmZdd lmZdd lmZddlmZddlmZerJddlmZddlmZddlZddlmcmZ ddl!mcm"Z"ddl#mcm$Z$ddl%mcm&Z&ddl'mcm(Z)ddl*m+Z+ddlm,Z,n6ddl-m.Z.e.dZe.dZ)e.dZ"e.dZ$e.dZ e.dZ&ddl/Z/e/j`e1Z2dZ3d!dZ4e dGddeZ5 d"d Z6y)#) annotations)Any) TYPE_CHECKINGN)experimental_class)warn_experimental_argument)_CONSTRAINTS_KEY)&_INDEPENDENT_SAMPLING_WARNING_TEMPLATE) _process_constraints_after_trial) BaseSampler)LazyRandomState)StudyDirection)_is_pareto_front) FrozenTrial) TrialState)Callable)Sequence)BaseDistribution)Study) _LazyImporttorchzoptuna._gp.search_spacez optuna._gp.gpzoptuna._gp.optim_mixedzoptuna._gp.acqfzoptuna._gp.priorg|=ctj|}tj|d}tj|d}||z tj t |z }|||fS)Nraxis)gpwarn_and_convert_infnpmeanstdmaximumEPS)valuesclipped_valuesmeansstdsstandardized_valuess \/var/www/html/hubwallet-dev/venv/lib/python3.12/site-packages/optuna/samplers/_gp/sampler.py_standardize_valuesr'5sZ,,V4N GGN +E 66.q )D)E1RZZT5JJ t ++z3.6.0ceZdZdZddddddd ddZddZdd Z dd Z dd Z dd Z dd Z ddZ ddZ ddZ ddZy) GPSamplerauSampler using Gaussian process-based Bayesian optimization. This sampler fits a Gaussian process (GP) to the objective function and optimizes the acquisition function to suggest the next parameters. The current implementation uses Matern kernel with nu=2.5 (twice differentiable) with automatic relevance determination (ARD) for the length scale of each parameter. The hyperparameters of the kernel are obtained by maximizing the marginal log-likelihood of the hyperparameters given the past trials. To prevent overfitting, Gamma prior is introduced for kernel scale and noise variance and a hand-crafted prior is introduced for inverse squared lengthscales. As an acquisition function, we use: - log expected improvement (logEI) for single-objective optimization, - log expected hypervolume improvement (logEHVI) for Multi-objective optimization, and - the summation of logEI and the logarithm of the feasible probability with the independent assumption of each constraint for (black-box inequality) constrained optimization. For further information about these acquisition functions, please refer to the following papers: - `Unexpected Improvements to Expected Improvement for Bayesian Optimization `__ - `Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization `__ - `Bayesian Optimization with Inequality Constraints `__ Please also check our articles: - `[Optuna v4.5] Gaussian Process-Based Sampler (GPSampler) Can Now Perform Constrained Multi-Objective Optimization `__ - `[Optuna v4.2] Gaussian Process-Based Sampler (GPSampler) Can Now Handle Inequality Constraints `__ - `Introducing Optuna's Native GPSampler `__ The optimization of the acquisition function is performed via: 1. Collect the best param from the past trials, 2. Collect ``n_preliminary_samples`` points using Quasi-Monte Carlo (QMC) sampling, 3. Choose the best point from the collected points, 4. Choose ``n_local_search - 2`` points from the collected points using the roulette selection, 5. Perform a local search for each chosen point as an initial point, and 6. Return the point with the best acquisition function value as the next parameter. Note that the procedures for non single-objective optimization setups are slightly different from the single-objective version described above, but we omit the descriptions for the others for brevity. The local search iteratively optimizes the acquisition function by repeating: 1. Gradient ascent using l-BFGS-B for continuous parameters, and 2. Line search or exhaustive search for each discrete parameter independently. The local search is terminated if the routine stops updating the best parameter set or the maximum number of iterations is reached. We use line search instead of rounding the results from the continuous optimization since EI typically yields a high value between one grid and its adjacent grid. .. note:: This sampler requires ``scipy`` and ``torch``. You can install these dependencies with ``pip install scipy torch``. Args: seed: Random seed to initialize internal random number generator. Defaults to :obj:`None` (a seed is picked randomly). independent_sampler: Sampler used for initial sampling (for the first ``n_startup_trials`` trials) and for conditional parameters. Defaults to :obj:`None` (a random sampler with the same ``seed`` is used). n_startup_trials: Number of initial trials. Defaults to 10. deterministic_objective: Whether the objective function is deterministic or not. If :obj:`True`, the sampler will fix the noise variance of the surrogate model to the minimum value (slightly above 0 to ensure numerical stability). Defaults to :obj:`False`. Currently, all the objectives will be assume to be deterministic if :obj:`True`. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraints is violated. A value equal to or smaller than 0 is considered feasible. If ``constraints_func`` returns more than one value for a trial, that trial is considered feasible if and only if all values are equal to 0 or smaller. The ``constraints_func`` will be evaluated after each successful trial. The function won't be called when trials fail or are pruned, but this behavior is subject to change in future releases. warn_independent_sampling: If this is :obj:`True`, a warning message is emitted when the value of a parameter is sampled by using an independent sampler, meaning that no GP model is used in the sampling. Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case. N FT)seedindependent_samplern_startup_trialsdeterministic_objectiveconstraints_funcwarn_independent_samplingct||_|xs tjj ||_tj j|_||_ tj|_ tj|_d|_d|_||_||_||_| t)dd|_d|_d|_y)N)r,r0ir+g-C6?)r _rngoptunasamplers RandomSampler_independent_sampler search_spaceIntersectionSearchSpace_intersection_search_space_n_startup_trialspriordefault_log_prior _log_priorDEFAULT_MINIMUM_NOISE_VAR_minimum_noise_gprs_cache_list_constraints_gprs_cache_list_deterministic_constraints_func_warn_independent_samplingr_n_preliminary_samples_n_local_search_tol)selfr,r-r.r/r0r1s r&__init__zGPSampler.__init__s$D) $7$c6??;X;X^b;X;c!*0*=*=*U*U*W'!1DID[D[%*%D%D>BIM)5!1*C'  ' &'9 :,0#! r(ctj||j|jjj |jj d}t j|y)Nz2dynamic search space is not supported by GPSampler) param_name trial_numberindependent_sampler_name sampler_namefallback_reason)r formatnumberr7 __class____name___loggerwarning)rItrialrLmsgs r&_log_independent_samplingz#GPSampler._log_independent_samplingsO4;;!%)%>%>%H%H%Q%Q00P   r(c|jjj|jj yN)r3rngr,r7 reseed_rng)rIs r&r]zGPSampler.reseed_rngs(  !!,,.r(ci}|jj|jD]\}}|jr|||<|Sr[)r: calculateitemssingle)rIstudyrWr8name distributions r&infer_relative_search_spacez%GPSampler.infer_relative_search_spacesX "&"A"A"K"KE"R"X"X"Z . D,""$!-L  . r(c|t|jdk(sJtj|||j|j |j |jj\}}|S)N)!warmstart_normalized_params_arrayn_preliminary_samplesn_local_searchtolr\) lenshape optim_mixedoptimize_acqf_mixedrFrGrHr3r\)rIacqf best_paramsnormalized_params _acqf_vals r&_optimize_acqfzGPSampler._optimize_acqfsj "c+*;*;&<&AAA'2'F'F .9"&"="=//  ( $9! r(c >t| \}}}|j6t|jdj|jk7rd|_|j }g}g} | tjt|z j} t|jD]m\} } |j |j | nd} tj|| ||j|j | |j"} |j%| o||_|| fS)NrXYis_categorical log_prior minimum_noise gpr_cacher/)r'rArlinverse_squared_lengthscalesdimrBryrrr tolist enumerateTrfit_kernel_paramsr>r@rCappend)rIconstraint_valsinternal_search_spacerrstandardized_constraint_valsr#r$ryconstraints_gprsconstraints_threshold_listivalscachegprs r&_get_constraints_acqf_argsz$GPSampler._get_constraints_acqf_argss15HHX4Y1$eT  ! ! -D))!,IIJ$(()15D -.==%'"',frzz#t/D&D%L%L%N" !=!?!?@ )GAt44@11!4  &&#-//"11(,(;(;C  # #C ( )"-=)!;;;r(c|t| d}t|}t|jdz|}|jj j ||d}||S)NF)assume_unique_lexsortedrg)sizereplace)rrlminrGr3r\choice)rIrrstandardized_score_vals pareto_params n_pareto_solsrchosen_indicess r&$_get_best_params_for_multi_objectivez.GPSampler._get_best_params_for_multi_objectivesj * 55u U M* 4''1,m<--m$PU-V^,,r(c |ik(riStjf}|jd|d}t||jkriSt j |}|j|}tj|jDcgc]}|tjk(rdndc}} t| tj|Dcgc]}|jc}z\} } } |j6t|jdj |j"k7rd|_g} | j$d} |j&}t)| D]o}|j|j|nd}| j+t-j.|| dd|f||j0|j2||j4q| |_|j6| d k(rjt| d k(sJt9j:| d|| dddfj= }|tj>| tj@f}nt9jB| |tEjF| d |jHjJjMd  }|jO|| }n| d k(rt| d k(sJtQ||\}}tjR|| dddftjT }|jW|||\}}tj>|}||}t9jX| d||||}|j$dd|j$k(sJtjZ|rdn||tj@f}ntQ||\}}|jW|||\}}t]| }t9j^| ||stEjF| |ndd |jHjJjMd ||}|s|jO||| |nd}|ja||}|jc|Scc}wcc}w)NFTdeepcopystates use_cachegg?rrv)rr8 thresholdi@)gpr_listr8Y_train n_qmc_samplesqmc_seed)rr8rconstraints_gpr_listr)rr8 Y_feasiblerrrr)2rCOMPLETE _get_trialsrlr;gp_search_space SearchSpaceget_normalized_paramsrarray directionsr MINIMIZEr'r!rAr}r~rmryrangerrrr>r@rCrD acqf_moduleLogEImaxargmaxnewaxisLogEHVIr from_numpyr3r\randintr$_get_constraint_vals_and_feasibilitywhereinfrConstrainedLogEIisneginfanyConstrainedLogEHVIrtget_unnormalized_param)rIrbrWr8rtrialsrrrd_signr_ gprs_list n_objectivesryrrrprqr is_feasible y_with_neginfconstr_gpr_listconstr_threshold_listi_optbest_feasible_yis_all_infeasiblenormalized_params r&sample_relativezGPSampler.sample_relative's~ 2 I%%'""E&D"Q v;// /I / ; ;L I1GGOQVQaQabA!~'>'>">$CGbc(; BHH?uell?@ @) %A  ! ! -D))!,IIJ$(()%)D ! .44R8 .==|$ A040E0E0QD))!,W[E   $$'-ad3#1"oo"&"5"5#,0,?,?  !*  ! ! )q 9~***"((!! !65ad;??A 0 :Q0RTVT^T^0^_ "**&!6!,,-DE"%!YY]]227; #GG%'> q 9~***/STY[a/b, "6MaQRd6SVXV\V\U\ ] :>9X9X#%:OPUWYWaWaPa>b0TTY[a/b,9=9X9X#%:z7_get_constraint_vals_and_feasibility..s% H!3" #s1v - Hs$'z:The number of constraints must be the same for all trials.rgz#constraint_vals must be a 2d array.rrrz#MyPy Redefinition for NumPy v2.2.0.)_storageget_trial_system_attrs _trial_idgetrr ValueErrorrrrlrmall isinstancebool_)rbrrWrrrs @r&rrs   --eoo>BBCSUWX H7G HHUVVhh/0O $$ % *Q,QQ *&&A-A6K+rxx0W2WW 0 K ''sAC)r!rrz)tuple[np.ndarray, np.ndarray, np.ndarray])rbrrzlist[FrozenTrial]rztuple[np.ndarray, np.ndarray])7 __future__rtypingrrnumpyrr4optuna._experimentalrroptuna.samplers._baserr r r "optuna.samplers._lazy_random_stater optuna.studyr optuna.study._multi_objectiver optuna.trialrrcollections.abcrrroptuna._gp.acqf_gprpr optuna._gp.gproptuna._gp.optim_mixedrnoptuna._gp.priorr<optuna._gp.search_spacer8roptuna.distributionsrroptuna._importsrlogging getLoggerrTrUr r'r*rrr(r&rs"  3;2HB->':$#(())00$$555"+  E!";