h$,fm dZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZmZmZmZdd lmZdd lmZmZmZmZmZmZmZdd lmZddl m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9ddl:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCdZDGddZEGddeZFGddeFZGedeHeIdgiddZJGddZKdZLdZMeeIgdeNeOehd gd!gd!eed"hgd!eed"hgd#dddd"d"d$d%ZPePe&ZQePe4ZRePe+d&'ZSePe0d&'ZTePe1d&'ZUePe,d&'ZVePe-d&'ZWePe2d&'ZXePe7d&'ZYePe8d&'ZZePe/d&'Z[ePe.d&'Z\ePe!Z]ePe#Z^ePe*Z_d(Z`d)ZaePe`ZbePead&'ZcePe9dd*+ZdePe6dd*+ZeePe"d*,ZfePe6d-d./ZgePe6d-d.d01ZhePe6d-d2/ZiePe6d-d2d01ZjePe)d&d-+ZkePe$d&d-+ZlePe$d&d-+ZmePe<ZnePeBZoePe?ZpePe=ZqePeCZrePe@ZsePe;ZtePeAZuePe>Zvewddid3eQd4eRd5eSd6e_d7eXd8eVd9eWd:eTd;eUde[d?e\d@e]dAeddBeedCeidDegdEejdFehdGe^dHefdIekdJeldKebdLecdMendNeodOepdPeqdQerdResdSetdTeudUevZxdVZydWe3fdXe5fdYe'fdZe(ffD]9\ZzZ{ePe{d[\exez<d]D]"Z|d^jeze|Z~ePe{de|_exe~<$;eed`geeeyeIdgd!gdadded&dbdcZy)faH The :mod:`sklearn.metrics.scorer` submodule implements a flexible interface for model selection and evaluation using arbitrary score functions. A scorer object is a callable that can be passed to :class:`~sklearn.model_selection.GridSearchCV` or :func:`sklearn.model_selection.cross_val_score` as the ``scoring`` parameter, to specify how a model should be evaluated. The signature of the call is ``(estimator, X, y)`` where ``estimator`` is the model to be evaluated, ``X`` is the test data and ``y`` is the ground truth labeling (or ``None`` in the case of unsupervised models). N)Counter)partial) signature) format_exc) is_regressor)Bunch) HasMethodsHidden StrOptionsvalidate_params_get_response_values)MetadataRequestMetadataRouter_MetadataRequester_raise_for_params_routing_enabledget_routing_for_objectprocess_routing)_check_response_method)accuracy_scoreaverage_precision_scorebalanced_accuracy_scorebrier_score_lossclass_likelihood_ratiosexplained_variance_scoref1_score jaccard_scorelog_lossmatthews_corrcoef max_errormean_absolute_errormean_absolute_percentage_errormean_gamma_deviancemean_poisson_deviancemean_squared_errormean_squared_log_errormedian_absolute_errorprecision_scorer2_score recall_score roc_auc_scoreroot_mean_squared_errorroot_mean_squared_log_errortop_k_accuracy_score) adjusted_mutual_info_scoreadjusted_rand_scorecompleteness_scorefowlkes_mallows_scorehomogeneity_scoremutual_info_scorenormalized_mutual_info_score rand_scorev_measure_scorecR| ||vr||St|g|d|i|\}}||||<|S)z/Call estimator with method and args and kwargs.response_methodr)cache estimatorr<argskwargsresult_s 7lib/python3.12/site-packages/sklearn/metrics/_scorer.py _cached_callrDRsZ _5_%%$*9=CIFA !'o Mc.eZdZdZdddZdZdZdZy) _MultimetricScoreraCallable for multimetric scoring used to avoid repeated calls to `predict_proba`, `predict`, and `decision_function`. `_MultimetricScorer` will return a dictionary of scores corresponding to the scorers in the dictionary. Note that `_MultimetricScorer` can be created with a dictionary with one key (i.e. only one actual scorer). Parameters ---------- scorers : dict Dictionary mapping names to callable scorers. raise_exc : bool, default=True Whether to raise the exception in `__call__` or not. If set to `False` a formatted string of the exception details is passed as result of the failing scorer. T) raise_excc ||_||_yN)_scorers _raise_exc)selfscorersrHs rC__init__z_MultimetricScorer.__init__ts #rEc hi}|j|rind}tt|}trt |dfi|}n.t di|j Dcic]}|t |c}}|j jD]q\}} t| tr1| j||g|i|j|j} n%| |g|i|j|j} | ||<s|Scc}w#t$r%} |jr| t||<Yd} ~ d} ~ wwxYw)z!Evaluate predicted target values.NscorerQ) _use_cacherrDrrr rKitems isinstance _BaseScorer_scoregetrQ ExceptionrLr) rMr>r?r@scoresr= cached_call routed_paramsnamescorerrQes rC__call__z_MultimetricScorer.__call__xs6ooi0dlE2  +D'DVDM"9=G4V,,GM!MM//1 0LD& 0fk2)FMM#Y159F9J9J49P9V9VE#9UtU}7H7H7N7T7TUE$t  0 #H 0??G#- A+D D1 D,,D1c @t|jdk(ryt|jjDcgc]2}t |t r t ||jj4c}}td|jDryycc}w)zuReturn True if using a cache is beneficial, thus when a response method will be called several time. rFc3&K|] }|dkD yw)rNrS).0vals rC z0_MultimetricScorer._use_cache..s33sQw3sT) lenrKrvaluesrVrWr_response_method__name__any)rMr>r_counters rCrTz_MultimetricScorer._use_caches t}}  "#mm224 fk2'y&2I2IJSS   3'.."23 3 s7Bc~t|jjjdi|jddiS)abGet metadata routing of this object. Please check :ref:`User Guide ` on how the routing mechanism works. .. versionadded:: 1.3 Returns ------- routing : MetadataRouter A :class:`~utils.metadata_routing.MetadataRouter` encapsulating routing information. ownermethod_mappingrQrS)r __class__rjaddrKrMs rCget_metadata_routingz'_MultimetricScorer.get_metadata_routings=A~DNN$;$;<@@ mm ,3  rEN)rj __module__ __qualname____doc__rOrarTrtrSrErCrGrGas $.2$<* rErGc4eZdZddZdZdZd dZdZdZy) rWc<||_||_||_||_yrJ) _score_func_sign_kwargsri)rM score_funcsignr@r<s rCrOz_BaseScorer.__init__s %  /rEcd|jvr|jdSt|jj}d|vr|djSy)N pos_label)r|rrz parametersdefault)rMscore_func_paramss rC_get_pos_labelz_BaseScorer._get_pos_labelsQ $,, &<< , ,%d&6&67BB + +$[199 9rEc |jdkDrdnd}d|j}dj|jj Dcgc] \}}d|d|c}}}d|j j |||dScc}}w) Nrz, greater_is_better=Falsez, response_method=z, =z make_scorer())r{rijoinr|rUrzrj)rM sign_stringresponse_method_stringkv kwargs_strings rC__repr__z_BaseScorer.__repr__s JJNb0K #5d6K6K5N!O$,,:L:L:N O$!Q2aS! OP 4++445k]%&}oQ 8 !Ps B Nc t||dtj|}|||d<|jt t d|||fi|S)aEvaluate predicted target values for X relative to y_true. Parameters ---------- estimator : object Trained estimator to use for scoring. Must have a predict_proba method; the output of that is used to compute the score. X : {array-like, sparse matrix} Test data that will be fed to estimator.predict. y_true : array-like Gold standard target values for X. sample_weight : array-like of shape (n_samples,), default=None Sample weights. **kwargs : dict Other parameters passed to the scorer. Refer to :func:`set_score_request` for more details. Only available if `enable_metadata_routing=True`. See the :ref:`User Guide `. .. versionadded:: 1.3 Returns ------- score : float Score function applied to prediction of estimator on X. N sample_weight)rcopydeepcopyrXrrD)rMr>Xy_truerr@r|s rCraz_BaseScorer.__call__sS@ &$---'  $'4GO $t{{7<6 1fXPWXXrEc|j tn"t|jj}|j|j}|r t j |d|t yy)zWarn if there is any overlap between ``self._kwargs`` and ``kwargs``. This method is intended to be used to check for overlap between ``self._kwargs`` and ``kwargs`` passed as metadata. Nz Overlapping parameters are: )r|setkeys intersectionwarningswarn UserWarning)rMmessager@r|overlaps rC _warn_overlapz_BaseScorer._warn_overlapsc  <</#%S9J9J9L5M&&v{{}5  MM)8 BK  rEc ts td|jd|t|jj |_|jD],\}}|j jj||.|S)awSet requested parameters by the scorer. Please see :ref:`User Guide ` on how the routing mechanism works. .. versionadded:: 1.3 Parameters ---------- kwargs : dict Arguments should be of the form ``param_name=alias``, and `alias` can be one of ``{True, False, None, str}``. zThis method is only available when metadata routing is enabled. You can enable it using sklearn.set_config(enable_metadata_routing=True).zYou are setting metadata request for parameters which are already set as kwargs for this metric. These set values will be overridden by passed metadata if provided. Please pass them either as metadata or kwargs to `make_scorer`.rr@rn)paramalias) r RuntimeErrorrrrqrj_metadata_requestrUrQ add_request)rMr@rrs rCset_score_requestz_BaseScorer.set_score_request s !E  :   "1t~~7N7N!O"LLN OLE5  " " ( ( 4 45 4 N O rE)predictrJ) rjrurvrOrrrarrrSrErCrWrWs"0  &YP !rErWceZdZdZy)_Scorerc |jd|t|rdn|j}t||j}|||j ||}i|j |} |j|j||fi| zS)aEvaluate the response method of `estimator` on `X` and `y_true`. Parameters ---------- method_caller : callable Returns predictions given an estimator, method name, and other arguments, potentially caching results. estimator : object Trained estimator to use for scoring. X : {array-like, sparse matrix} Test data that will be fed to clf.decision_function or clf.predict_proba. y_true : array-like Gold standard target values for X. These must be class labels, not decision function values. **kwargs : dict Other parameters passed to the scorer. Refer to :func:`set_score_request` for more details. Returns ------- score : float Score function applied to prediction of estimator on X. zThere is an overlap between set kwargs of this scorer instance and passed metadata. Please pass them either as kwargs to `make_scorer` or metadata, but not both.rN)r) rrrrrirjr|r{rz) rM method_callerr>rrr@rr<y_predscoring_kwargss rCrXz_Scorer._score1s: .  )3D9L9L9N 0D`. :func:`~sklearn.metrics.get_scorer_names` can be used to retrieve the names of all available scorers. Parameters ---------- scoring : str, callable or None Scoring method as string. If callable it is returned as is. If None, returns None. Returns ------- scorer : callable The scorer. Notes ----- When passed a string, this function always returns a copy of the scorer object. Calling `get_scorer` twice for the same scorer results in two separate scorer objects. Examples -------- >>> import numpy as np >>> from sklearn.dummy import DummyClassifier >>> from sklearn.metrics import get_scorer >>> X = np.reshape([0, 1, -1, -0.5, 2], (-1, 1)) >>> y = np.array([0, 1, 1, 0, 1]) >>> classifier = DummyClassifier(strategy="constant", constant=0).fit(X, y) >>> accuracy = get_scorer("accuracy") >>> accuracy(classifier, X, y) 0.4 z]%r is not a valid scoring value. Use sklearn.metrics.get_scorer_names() to get valid options.)rVstrrr_SCORERSKeyError ValueError)rr_s rC get_scorerraseT'3 ]]8G#45F M M (*12  s 4A ceZdZdZdZdZy)_PassthroughScorerc||_yrJ) _estimator)rMr>s rCrOz_PassthroughScorer.__init__s #rEc&|j|i|S)z!Method that wraps estimator.scorerR)rMr>r?r@s rCraz_PassthroughScorer.__call__sy///rEc,t|jS)a\Get requested data properties. Please check :ref:`User Guide ` on how the routing mechanism works. .. versionadded:: 1.3 Returns ------- routing : MetadataRouter A :class:`~utils.metadata_routing.MetadataRouter` encapsulating routing information. )rrrss rCrtz'_PassthroughScorer.get_metadata_routings(&doo66rEN)rjrurvrOrartrSrErCrrs$07rErc  d|d}t|tttfrd} t|}t |t |k7rt |d|t |dkDratd|Ds2td|Drt |d |t |d ||Dcic]}|t|| }}|St |d |t|trwt|}td |Dst d|t |dk(rt d||jDcic]\}}|t|| }}}|St |#t$r}t ||d}~wwxYwcc}wcc}}w)a!Check the scoring parameter in cases when multiple metrics are allowed. In addition, multimetric scoring leverages a caching mechanism to not call the same estimator response method multiple times. Hence, the scorer is modified to only use a single response method given a list of response methods and the estimator. Parameters ---------- estimator : sklearn estimator instance The estimator for which the scoring will be applied. scoring : list, tuple or dict Strategy to evaluate the performance of the cross-validated model on the test set. The possibilities are: - a list or tuple of unique strings; - a callable returning a dictionary where they keys are the metric names and the values are the metric scores; - a dictionary with metric names as keys and callables a values. See :ref:`multimetric_grid_search` for an example. Returns ------- scorers_dict : dict A dict mapping each scorer name to its validated scorer. zscoring is invalid (got zh). Refer to the scoring glossary for details: https://scikit-learn.org/stable/glossary.html#term-scoringzFThe list/tuple elements must be unique strings of predefined scorers. Nz2 Duplicate elements were found in the given list. rc3<K|]}t|tywrJrVrrdrs rCrfz-_check_multimetric_scoring..s8az!S)8c32K|]}t|ywrJ)callablers rCrfz-_check_multimetric_scoring..s1qx{1szi One or more of the elements were callables. Use a dict of score name mapped to the scorer callable. Got z4 Non-string types were found in the given list. Got )rz Empty list was given. c3<K|]}t|tywrJrrs rCrfz-_check_multimetric_scoring..s4!:a%4rzCNon-string types were found in the keys of the given dict. scoring=zAn empty dict was passed. ) rVlisttupler TypeErrorrrgallrk check_scoringdictrU) r>rerr_msg_genericerr_msgrr`r_rNkeys rC_check_multimetric_scoringrs> #7+.E E 'D%-. T  -w PWEK i@@G, N%y(?{KL L GT "7|4t44++2+7  t9>9'EF F '}} V y&9 9   N))U -W%1 , -, s# E(-FF ( F1 E==Fc|dk7}|dk7}|du}|dk(rdn|}|dk(rdn|}|r|s|r td|s|rtjdt|r|S|dur|dur td|durd}|S|durd }|Sd }|S) zmHandles deprecation of `needs_threshold` and `needs_proba` parameters in favor of `response_method`. deprecatedNFzYou cannot set both `response_method` and `needs_proba` or `needs_threshold` at the same time. Only use `response_method` since the other two are deprecated in version 1.4 and will be removed in 1.6.zThe `needs_threshold` and `needs_proba` parameter are deprecated in version 1.4 and will be removed in 1.6. You can either let `response_method` be `None` or set it to `predict` to preserve the same behaviour.TzYou cannot set both `needs_proba` and `needs_threshold` at the same time. Use `response_method` instead since the other two are deprecated in version 1.4 and will be removed in 1.6. predict_probadecision_functionrr)rrr FutureWarning)r<needs_threshold needs_probaneeds_threshold_providedneeds_proba_providedresponse_method_provideds rC_get_response_methodrs /,>&,6.d:.,>eOO&,6%KK%9=U V  7 "   d$6 9  d)  D @ $ rE>rrrbooleanr)r}r<greater_is_betterrr)r<rrrc Dt|||}|rdnd}t||||S)aMake a scorer from a performance metric or loss function. A scorer is a wrapper around an arbitrary metric or loss function that is called with the signature `scorer(estimator, X, y_true, **kwargs)`. It is accepted in all scikit-learn estimators or functions allowing a `scoring` parameter. The parameter `response_method` allows to specify which method of the estimator should be used to feed the scoring/loss function. Read more in the :ref:`User Guide `. Parameters ---------- score_func : callable Score function (or loss function) with signature ``score_func(y, y_pred, **kwargs)``. response_method : {"predict_proba", "decision_function", "predict"} or list/tuple of such str, default=None Specifies the response method to use get prediction from an estimator (i.e. :term:`predict_proba`, :term:`decision_function` or :term:`predict`). Possible choices are: - if `str`, it corresponds to the name to the method to return; - if a list or tuple of `str`, it provides the method names in order of preference. The method returned corresponds to the first method in the list and which is implemented by `estimator`. - if `None`, it is equivalent to `"predict"`. .. versionadded:: 1.4 greater_is_better : bool, default=True Whether `score_func` is a score function (default), meaning high is good, or a loss function, meaning low is good. In the latter case, the scorer object will sign-flip the outcome of the `score_func`. needs_proba : bool, default=False Whether `score_func` requires `predict_proba` to get probability estimates out of a classifier. If True, for binary `y_true`, the score function is supposed to accept a 1D `y_pred` (i.e., probability of the positive class, shape `(n_samples,)`). .. deprecated:: 1.4 `needs_proba` is deprecated in version 1.4 and will be removed in 1.6. Use `response_method="predict_proba"` instead. needs_threshold : bool, default=False Whether `score_func` takes a continuous decision certainty. This only works for binary classification using estimators that have either a `decision_function` or `predict_proba` method. If True, for binary `y_true`, the score function is supposed to accept a 1D `y_pred` (i.e., probability of the positive class or the decision function, shape `(n_samples,)`). For example `average_precision` or the area under the roc curve can not be computed using discrete predictions alone. .. deprecated:: 1.4 `needs_threshold` is deprecated in version 1.4 and will be removed in 1.6. Use `response_method=("decision_function", "predict_proba")` instead to preserve the same behaviour. **kwargs : additional arguments Additional parameters to be passed to `score_func`. Returns ------- scorer : callable Callable object that returns a scalar score; greater is better. Examples -------- >>> from sklearn.metrics import fbeta_score, make_scorer >>> ftwo_scorer = make_scorer(fbeta_score, beta=2) >>> ftwo_scorer make_scorer(fbeta_score, response_method='predict', beta=2) >>> from sklearn.model_selection import GridSearchCV >>> from sklearn.svm import LinearSVC >>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, ... scoring=ftwo_scorer) r)rr)r}r<rrrr@r~s rC make_scorerrBs2^++O"1rD :tV_ ==rEF)rc t||dS)Nrrrrs rCpositive_likelihood_ratior "66 21 55rEc t||dS)Nrrrs rCnegative_likelihood_ratiorrrEr)rr<)r<rovo)r< multi_classweighted)r<raverageovrexplained_variancer2r#r"neg_median_absolute_errorneg_mean_absolute_error"neg_mean_absolute_percentage_errorneg_mean_squared_errorneg_mean_squared_log_errorneg_root_mean_squared_errorneg_root_mean_squared_log_errorneg_mean_poisson_devianceneg_mean_gamma_devianceaccuracytop_k_accuracyroc_auc roc_auc_ovr roc_auc_ovoroc_auc_ovr_weightedroc_auc_ovo_weightedbalanced_accuracyaverage_precision neg_log_lossneg_brier_scorerneg_negative_likelihood_ratior3r9r6r4r:r7r2r8r5c<ttjS)aGet the names of all available scorers. These names can be passed to :func:`~sklearn.metrics.get_scorer` to retrieve the scorer object. Returns ------- list of str Names of all available scorers. Examples -------- >>> from sklearn.metrics import get_scorer_names >>> all_scorers = get_scorer_names() >>> type(all_scorers) >>> all_scorers[:3] ['accuracy', 'adjusted_mutual_info_score', 'adjusted_rand_score'] >>> "roc_auc" in all_scorers True )sortedrrrSrErCget_scorer_namesrLs, (--/ ""rE precisionrecallf1jaccardbinary)r)macromicrosamplesrz{0}_{1})rrfit)r>r allow_none)rcnt|tr t|St|ret |dd}t |drA|j dr0|j ds|j dstd|zt|S|(t |dr t|S|rytd |zy) aDetermine scorer from user options. A TypeError will be thrown if the estimator cannot be scored. Parameters ---------- estimator : estimator object implementing 'fit' The object to use to fit the data. scoring : str or callable, default=None A string (see model evaluation documentation) or a scorer callable object / function with signature ``scorer(estimator, X, y)``. If None, the provided estimator object's `score` method is used. allow_none : bool, default=False If no scoring is specified and the estimator has no score function, we can either return None or raise an exception. Returns ------- scoring : callable A scorer callable object / function with signature ``scorer(estimator, X, y)``. Examples -------- >>> from sklearn.datasets import load_iris >>> from sklearn.metrics import check_scoring >>> from sklearn.tree import DecisionTreeClassifier >>> X, y = load_iris(return_X_y=True) >>> classifier = DecisionTreeClassifier(max_depth=2).fit(X, y) >>> scorer = check_scoring(classifier, scoring='accuracy') >>> scorer(classifier, X, y) 0.96... ruN startswithzsklearn.metrics.zsklearn.metrics._scorerzsklearn.metrics.tests.zscoring value %r looks like it is a metric function rather than a scorer. A scorer should require an estimator as its first parameter. Please use `make_scorer` to convert a metric to a scorer.rQziIf no scoring is specified, the estimator passed should have a 'score' method. The estimator %r does not.) rVrrrgetattrhasattrrrrr)r>rrmodules rCrrqsZ'3'"",5 FL )!!"45%%&?@%%&>?") ) '"" 9g &%i0 0 DFOP  rErSrJ)rwrr collectionsr functoolsrinspectr tracebackrbaserutilsr utils._param_validationr r r r utils._responserutils.metadata_routingrrrrrrrutils.validationrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1clusterr2r3r4r5r6r7r8r9r:rDrGrWrrrrrrrrrrexplained_variance_scorer r2_scorermax_error_scorerneg_mean_squared_error_scorer!neg_mean_squared_log_error_scorerneg_mean_absolute_error_scorer)neg_mean_absolute_percentage_error_scorer neg_median_absolute_error_scorer"neg_root_mean_squared_error_scorer&neg_root_mean_squared_log_error_scorer neg_mean_poisson_deviance_scorerneg_mean_gamma_deviance_scoreraccuracy_scorerbalanced_accuracy_scorermatthews_corrcoef_scorerrr positive_likelihood_ratio_scorer$neg_negative_likelihood_ratio_scorertop_k_accuracy_scorerroc_auc_scoreraverage_precision_scorerroc_auc_ovo_scorerroc_auc_ovo_weighted_scorerroc_auc_ovr_scorerroc_auc_ovr_weighted_scorerneg_log_loss_scorerneg_brier_score_scorerbrier_score_loss_scoreradjusted_rand_scorer rand_scorerhomogeneity_scorercompleteness_scorerv_measure_scorermutual_info_scoreradjusted_mutual_info_scorernormalized_mutual_info_scorerfowlkes_mallows_scorerrrrr^metricrformatqualified_namerrrSrErCrGs (  UU266    Z Z zo$od.Ok.ObC4(#'  /  /d77>Vr.bj    H I  ([!6*l^*D#EF%vj,.H'IJ #'$ d>d>P((@A  ! yEB +,>RW X$/e%!"-5"-8"e-)$/U$ &1u&"*55*&$/U$ "-5" n-&'>?&'8966$//H#I '2($ $: : ':!?*#   !?*#  " %& ##67*%  !23!"45/ !23)*DE +,H I$%:; % 0% % % / %  ? %  ; % (Q% 9%  A% !C% %K% ?% ;% % )%  !% "##% $#%% &5'% (5)% */+% ,/-% .%/% 0+1% 2?3% 4#G5% 8-9% :;% <)=% >+?% @%A% B)C% D ;E% F"?G% H1I% P#4/" | 8   XLD& !:HTN<X"))$8#.vw#W X X '(s#3#5674H k #' AAArE