h$,f ` ddlZddlZddlmZddlmZmZddlm Z m Z m Z m Z m Z mZddlmZmZmZddlmZddlmZmZddlmZdd lmZdd lmZdd lmZdd l m!Z!m"Z"dd l#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0ddl1m2Z2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;mZ>ddl?m@Z@ddlAmBZBdZCejddZEejjdeBejjdddgejjd d!d"gd#ZHd$ZIejjd d!d"gd%ZJejjdddgejjd d!d"gd&ZKejjdddgejjd d!d"gd'ZLejjdddgejjd d!d"gejjd(eMd)d*ZNd+ZOejjdeBd,ZPejjdddgd-ZQd.ZRd/ZSejjd d!d"gd0ZTejjd d!d"gd1ZUejjd d!d"gd2ZVejjd3ejjd4jd5d6d)ejjd4jd5d6d)d7gd8ZZejd9Z[ejd:Z\d;Z]ejjd<eje5d=d>?d)eje5d=d>?d@gdAZ_dBZ`dCZaejddDZbejddEZcejjdFd6dGgejjdHdIdJgdKZddLZeejjdMdNdOgdPZfdQZgejjdRdSdTgdUZhdVZiejjdWejekgdXZlejjdWejekgdYZmejjdZgd[d\Znejjdddgejjd d!d"gd]Zoejjd^d_d`gdaZpejjdbdcgeCzejeCgddZrdeZsejjdddgejjd d!d"gdfZtdgZudhZvdiZwdjZxy)kN)assert_allclose) BaseEstimatorclone)CalibratedClassifierCVCalibrationDisplay_CalibratedClassifier_sigmoid_calibration_SigmoidCalibrationcalibration_curve) load_iris make_blobsmake_classification)DummyClassifier)RandomForestClassifierVotingClassifier)NotFittedError)DictVectorizer) SimpleImputer)IsotonicRegression)LogisticRegression SGDClassifier)brier_score_loss)KFold LeaveOneOutcheck_cvcross_val_predictcross_val_scoretrain_test_split) MultinomialNB)Pipeline make_pipeline) LabelEncoderStandardScaler) LinearSVC)DecisionTreeClassifier)CheckingClassifier)_convert_containerassert_almost_equalassert_array_almost_equalassert_array_equal)softmax)CSR_CONTAINERSmodule)scopec4ttdd\}}||fS)N* n_samples n_features random_state)r N_SAMPLES)Xys >lib/python3.12/site-packages/sklearn/tests/test_calibration.pydatar;7s qr RDAq a4K csr_containermethodsigmoidisotonicensembleTFctdz}|\}}tjjdj |j }||j z}|d||d||d|} } }||d||d} } tj|| | } | j| dddf}t| |j dz|}tjt5|j||ddd|| f|||| ffD]>\}}t| |d| }|j|| | |j|dddf}t| |t| |kDsJ|j|| dz| |j|dddf}t|||j|d| zdz | |j|dddf}t|||j|| dzdz| |j|dddf}|d k(rt|d|z t| |t| dzdz|kDr?Jy#1swYaxYw) Nr2seedsize sample_weightcvrAr>rLrAr?)r7nprandom RandomStateuniformrGminrfit predict_probarpytestraises ValueErrorrr))r;r>r=rAr4r8r9rIX_trainy_trainsw_trainX_testy_testclf prob_pos_clfcal_clf this_X_train this_X_testprob_pos_cal_clfprob_pos_cal_clf_relabeleds r:test_calibrationre=s QI DAqII))r)2:::GMLA"#:I*9 }Zi?XhWGyz]AijMFF /  gwh  GC$$V,QT2L$SQVVaZ(KG z " Aq & w v!67&#! k)VHU  L' B"00=adC  58H $9     L'A+X F%,%:%:;%G1%M"!"24NO  L!g+/ J%,%:%:;%G1%M"!"24NO  L7Q;!"38 L%,%:%:;%G1%M" Y  %&6rArH皙?) r7rOrPrQrRrpr$rrTrUlinalgnorm)r;r>rAr4r8r9rIrYrZr[r\ricalibrated_clf probs_with_swprobs_without_swdiffs r:test_sample_weightrsQI DAqII))r)2::A:GM!":I*9 }Zi?XhWG yz]FvB7I+IfxXNwx@"008Mw(%33F; 99>>-*:: ;D #::r<cR|\}}t||d\}}}}tttdd} t | |d|} | j ||| j |} t | |d|} | j ||| j |} t| | y) zTest parallel calibrationr2)r6rvrwrC)r>n_jobsrArJN)rr!r#r$rrTrUr)r;r>rAr8r9rYr\rZr]rical_clf_parallelprobs_parallelcal_clf_sequentialprobs_sequentials r:test_parallel_executionrs DAq'712'N$GVWfn. vTV0WXI-&X'*%33F;N/&X7G,)77?N$45r<rErCcd}tdd}tdd|dd \}}d ||d kD<tj|jd }|ddd |ddd } }|d dd |d dd } } |j || t ||d |} | j || | j| } ttj| d tjt| d|j| | cxkrdksJJ| j| | d|j| | zkDsJ|| t|j| |}|| | |}|d|zksJtdd}|j || |j| }|| ||}t ||d |} | j || | j| }|| ||}|d|zksJy)Nctj||}tj||z dz|jdz S)NrCr)rOeyesumshape)y_true proba_pred n_classesY_onehots r:multiclass_brierz5test_calibration_multiclass..multiclass_briers<66)$V,vvx*,23hnnQ6GGGr<rvrwid .@r4r5r6centers cluster_stdrCrrJrMrNaxis?gffffff?)rg?r2) n_estimatorsr6)r$r rOuniquerrTrrUrronesrpscorer+decision_functionr)r>rArErr^r8r9rrYrZr\r]r`probasuncalibrated_briercalibrated_brier clf_probs cal_clf_probss r:test_calibration_multiclassrs H a 0C #D"RV DAqAa!eH ! ""1%I1vq1vWGqt!tWa1gFFGGGW$SAQG KK!  " "6 *FBFF6*BGGCK,@A #))FF+ 2d 22 22 2 == (4#))FF2K+K KK K *--f56)()L c$66 66 6 !br BCGGGW!!&)I)&)yQ$SAQG KK!))&1M' S c$66 66 6r<cGdd}tddddd\}}tj||}|}t||g|j}|j |}t |d |jz y) NceZdZdZy)9test_calibration_zero_probability..ZeroCalibratorcFtj|jdS)Nr)rOzerosrselfr8s r:predictzAtest_calibration_zero_probability..ZeroCalibrator.predicts88AGGAJ' 'r<N)__name__ __module__ __qualname__rr<r:ZeroCalibratorr s (r<r2rrrr)ri calibratorsclasses?)r rrTrclasses_rUr n_classes_)rr8r9r^ calibratorr`rs r:!test_calibration_zero_probabilityrs (( !RT DAq    1 %C!J#J<G " "1 %FFC#..01r<c d}td|zdd\}}tjjdj |j }||j z}|d||d||d|}}}||d |z||d |z||d |z} } }|d |zd|d |zd} } t} t| d }tjt5|j|| ddd| j|||| j| ddd f}|| f|||| ffD]\}}d D]}t| |d }| dfD]}|j|| ||j|}|j|}|ddd f}t|tj dd gtj"|d t%| |t%| |kDrJy#1swYxYw)z*Test calibration for prefitted classifiersrr1r2r3rDrFNrCprefitrgrJ)r@r?)r>rLrHrr)rrOrPrQrRrGrSrrrVrWrrTrUrr*arrayargmaxr)r=r4r8r9rIrYrZr[X_caliby_calibsw_calibr\r]r^ unfit_clfr_ this_X_calibrbr>r`swy_proby_predrcs r:test_calibration_prefitr s0I Y1SU VDAqII))r)2:::GMLA"#:I*9 }Zi?XhWG )a)m$ )a)m$i!i-0WG q9}'1y=?);FF /C&sx8I ~ &( gw'(GGGWh'$$V,QT2L & w v!67&! k. F,SHMG&  L' D ..{; 5#)!Q$< "6288QF+;BIIfSTr@clip) out_of_boundsrJ) r$rrTrUrrr rrr) r;r>r8r9r^r` cal_probasunbiased_predsrclf_df manual_probass r:test_calibration_ensemble_falserPs DAq a 0C$SANG KK1&&q)J'sAqQ?RSN 'f= (* NN>1%GGAqM  " "1 %F&&v.MJq!t$m4r<c0tjgd}tjgd}tjddg}t|t||dddtj|d|z|dzzz }t j ||j|}t||d tjt5t j tj||f|d d d y #1swYy xYw) z0Test calibration values with Platt sigmoid model)rMr)rJrgj=ɿgY90(?rrrrJr1N) rOrr)r expr rTrrVrWrXvstack)exFexY AB_lin_libsvmlin_probsk_probs r:test_sigmoid_calibrationris ((< C ((; CHH24GHIMm-A#s-KQOcBFF=#3c#9M!Gh3 z ">!!"))S#J"7=>>>s 0D  Dctjgd}tjgd}t||d\}}t|t|k(sJt|dk(sJt |ddgt |ddgt j t5tdgd gd d d tjgd }tjgd }t||dd \}}t|t|k(sJt|dk(sJt |ddgt |ddgt j t5t||dd d d y #1swYxYw#1swYy xYw)z Check calibration_curve function)rrrrJrJrJ)ry皙?皙??rrCn_binsrrJryrgN)rrrrrJrJ)rryr?rrquantilerstrategygUUUUUU?r percentile)r)rOrr rpr(rVrWrX)rr prob_true prob_predy_true2y_pred2prob_true_quantileprob_pred_quantiles r:test_calibration_curverzsi XX( )F XX4 5F,VVAFIy y>S^ ++ + y>Q   Aq6* C:. z "'1#v&'hh)*Ghh56G->Z.** ! "c*<&= == = ! "a '' '*QJ7*S#J7 z "C'7\BCC!'' CCsE+E7+E47Fctddddd\}}tj|d<tdt fdt d fg}t |dd | }|j|||j|y )z$Test that calibration can accept nanrrCrr2)r4r5 n_informative n_redundantr6rrimputerrfrJ)rr@)rLr>rAN) rrOnanr rrrrTr)rAr8r9r^clf_cs r:test_calibration_nan_imputerrs} !QS DAqffAdG  ]_ %.DRS.T'UV C #31Z( SE IIaO MM!r<c2d}tdd|\}}tddd}t|d t| }|j |||j |}t |jd tj|jd y)NrCrrM)r4r5rrvrrrxCr6r?rNrJrr) rr$rrrTrUr)rrOrr)rA num_classesr8r9r^clf_probprobss r:test_calibration_prob_sumrsK [ QDAq 3Q 7C% I+-(H LLA  " "1 %EeiiQi/Q1HIr<c ~tjjdd}tjd}t ddd}t |dt |}|j||t|jD]\}}|j|}|ryt|dd|ftjt|tj|ddd|fd kDsJtj|dd|d zdfd kDrJtj|d |j d z rJy) NrrMrvrrrr?rNrrJ)rOrPrandnaranger$rrrT enumeraterhrUr*rrpallallcloser)rAr8r9r^r`icalibrated_classifierprobas r:test_calibration_less_classesr s AA " A 3Q 7C$ I+-(G KK1$-g.M.M$N :  %33A6  uQT{BHHSV,< =66%2A2,*+ ++66%1q57 +a/0 00;;ua%++a.&89 99 :r<r8r2rMr1cngd}Gddt}t|}|j||y)z;Test that calibration accepts n-dimensional arrays as input)rJrrrJrJrrJrJrrrJrrrJrc eZdZdZdZdZdZy)>test_calibration_accepts_ndarray..MockTensorClassifierz*A toy estimator that accepts tensor inputs classifierc:tj||_|SN)rOrr)rr8r9s r:rTzBtest_calibration_accepts_ndarray..MockTensorClassifier.fitsIIaLDMKr<c`|j|jddjdS)NrrrJr)reshaperrrs r:rzPtest_calibration_accepts_ndarray..MockTensorClassifier.decision_functions)99QWWQZ,00a08 8r<N)rrr__doc___estimator_typerTrrr<r:MockTensorClassifierrs8&  9r<rN)rrrT)r8r9rr|s r: test_calibration_accepts_ndarrayrs5 6A 9} 9,,@,BCNq!r<c.dddddddddg}gd}||fS)NNYadult)stateageTXVTchild)rJrrJr) dict_data text_labelss r:r r s5w'w'w'I K k !!r<cr|\}}tdtfdtfg}|j||S)N vectorizerr^)r rrrT)r r8r9pipeline_prefits r:dict_data_pipeliner%sC DAq ( )E3I3K+LMO   q! $$r<c |\}}|}t|d}|j||t|j|jt |drJt |drJ|j ||j |y)aRTest that calibration works in prefit pipeline with transformer `X` is not array-like, sparse matrix or dataframe at the start. See https://github.com/scikit-learn/scikit-learn/issues/8710 Also test it can predict without running into validation errors. See https://github.com/scikit-learn/scikit-learn/issues/19637 rrgn_features_in_N)rrTr*rhasattrrrU)r r%r8r9r^rks r:test_calibration_dict_pipeliner)s DAq C&sx8I MM!Qy))3<<8s,-- -y"233 3a Ar<zclf, cvrvrJrxrrctdddd\}}|dk(r|j||}t||}|j|||dk(r ?CGGAqM  + +CA 6C!002HIGG&&(224M } %;!< << <; $::::;r<zname, expected_label)N_line1)my_estractjgd}tjgd}tjg}t||||}|j|gn|g}|j d|j j j}t|t|k(sJ|D]} | j|vrJy)NrrJrJrrrr皙?rFrD) rOrrplotappendrLrTrUrprV) rWnameexpected_labelrrrrXrZr[r\s r:'test_calibration_display_default_labelsrks&I-.I XXb\F Y 6$ OCHHJ#'?K,0@@c1a[q&kF v -H -C    )) ) LLHHJ&(>?GG&&(224M } %;!< << <; $::::; LLHHH(H } %;!< << <; $::::;r<cP|\}}tj||}tj||}tj|||}tj||||j }|j j d}|jddk(sJy)N)axrJrD)rrTr%rrErLget_legend_handles_labelscount) rWrAr8r9r4dtrXviz2r\s r:!test_calibration_display_ref_liners DAq   ! !!Q 'B  ! % %a +B  + +B1 5C  , ,RA#'' BD XX / / 1! 4F <<. /1 44 4r< dtype_y_strc>tjjd}tjdgdzdgdzz|}|j dd|j }d }t jt| 5t||d d d y #1swYy xYw) zKCheck error message when a `pos_label` is not specified with `str` targets.r2spamreggsrCdtyperrFzy_true takes value in {'eggs', 'spam'} and pos_label is not specified: either make y_true take value in {0, 1} or {-1, 1} or pass pos_label explicitlyr/N) rOrPrQrrandintrGrVrWrXr )rrngy1y2err_msgs r:*test_calibration_curve_pos_label_error_strrs ))   #C 6(Q,&A-[ AB Q (B $ z 1""b!"""s = BBctjgd}tjddg|}||}tjgd}t||d\}}t|gdt||dd \}}t|gdt|d |z dd \}}t|gd t|d |z dd \}}t|gd y )z8Check the behaviour when passing explicitly `pos_label`.) rrrrJrJrJrJrJrJreggr) ryrg333333?rergffffff?rrrr)rrrJrJ)r pos_labelrJr)rrrrJN)rOrr r)rrr y_true_strrr_s r: test_calibration_curve_pos_labelrsXX1 2Fhhk:GJ XXD EF%VVA>LIqI~.$ZUSLIqI~.$VQZQOLIqI~.$ZVAQWXLIqI~.r<zpos_label, expected_pos_label))NrJr)rJrJc|\}}tj||}tj||||}|j |dd|f}t |||\} } t |j| t |j| t |j||jjd|dk(sJ|jjd|dk(sJ|jjdg} |jjj!} t#| t#| k(sJ| D]} | j%| vrJy)z?Check the behaviour of `pos_label` in the `CalibrationDisplay`.)rNz,Mean predicted probability (Positive class: )z'Fraction of positives (Positive class: rD)rrTrrErUr rrrrrLrRrS __class__rrTrUrprV)rWrArexpected_pos_labelr8r9r4rXrrrrZr[r\s r:"test_calibration_display_pos_labelrsc DAq   ! !!Q 'B  + +B1 JC   a $6!6 7F,Q)LIyCMM9-CMM9-CJJ'  9:L9MQ O P P  45G4H J K K!ll335KLGG&&(224M } %;!< << <; $::::;r<crtd\}}tj|}|dd|dd}}tj|dz}tj |j ddz|j df|j}||dddddf<||dddddf<tj |j ddz|j}||ddd<||ddd<t}t|||d }t|} | j||| |j||t| j|jD]9\} } t| jj | jj ;| j#|} |j#|} t| | y) zrCheck that passing repeating twice the dataset `X` is equivalent to passing a `sample_weight` with a factor 2.Tr=NrrCrrJrr>rArLrH)r r# fit_transformrO ones_likerrrrrrrTziprhrricoef_rU)r>rAr8r9rIX_twicey_twicericalibrated_clf_without_weightscalibrated_clf_with_weightsest_with_weightsest_without_weightsy_pred_with_weightsy_pred_without_weightss r:?test_calibrated_classifier_cv_double_sample_weights_equivalencer3s  %DAq&&q)A Tc7AdsGqALLOa'Mhh Q 3177CGGCaCFOGADqD!GhhqwwqzA~QWW5GGCaCLGADqDM"$I%; &" #((F"G##Aq #F"&&w825#;;&>>2 --   & & , ,  ) ) / /  6CCAF;II!L')?@r<fit_params_typelistrc|\}}t||t||d}tddg}t|}|j||fi|y)zTests that fit_params are passed to the underlying base estimator. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/12384 )abrr)expected_fit_paramsN)r'r&rrT)rr;r8r9 fit_paramsr^pc_clfs r: test_calibration_with_fit_paramsresW DAq ? 3 ? 3J #s .ClfWithoutSampleWeightc2d|vsJt|||fi|S)NrIsuperrT)rr8r9rrs r:rTzTtest_calibration_without_sample_weight_estimator..ClfWithoutSampleWeight.fits'"*4 447;q!2z2 2r<rrrrT __classcell__rs@r:ClfWithoutSampleWeightrs  3 3r<rrHN)rOrr&rrVwarns UserWarningrT)r;r8r9rIrr^rs r:0test_calibration_without_sample_weight_estimatorrsn DAqLLOM3!33 ! "C #C (F k "6 1a} 5666s A33A<ctd\}}tj|}tj|dd|ddf}tj |dd|ddf}tj |}d|ddd<t}t|||d }t|}|j||| |j|ddd|dddt|j|jD]9\}} t|jj| jj;|j!|} |j!|} t| | y) z|Check that passing removing some sample from the dataset `X` is equivalent to passing a `sample_weight` with a factor 0.Tr=N(rZrJrCrrH)r r#rrOrhstack zeros_likerrrrTrrhrrirrU) r>rAr8r9rIrirrrrrrs r:>test_calibrated_classifier_cv_zeros_sample_weights_equivalencersj  %DAq&&q)A 1Sb61R8$%A 1Sb61R8$%AMM!$MM#A#"$I%; &" #((F"G##Aq #F"&&q1vq1v625#;;&>>2 --   & & , ,  ) ) / /  6CCAF;II!L')?@r<c Gddt}t|j|dtjt |ddziy)z[Check that CalibratedClassifierCV does not enforce sample alignment for fit parameters.c eZdZdfd ZxZS)Jtest_calibration_with_non_sample_aligned_fit_param..TestClassifierc0|Jt||||S)NrHr)rr8r9rI fit_paramrs r:rTzNtest_calibration_with_non_sample_aligned_fit_param..TestClassifier.fits$( ((7;q!=;A Ar<)NNrrs@r:TestClassifierrs B Br<r)rirrJN)rrrTrOrrp)r;rs r:2test_calibration_with_non_sample_aligned_fit_paramrsMB+B ;^%56:: T!W!12r<c d}d}tjj|j|}tjdgt ||zzdg|t ||zz zz}d|j dz|z}td|d }|j||}|D]Y\}} ||||} } || } td | } | j| | | j| }|d kDjrYJttd | d}t|||d}ttd | d}t|||d}t||y)zTest that :class:`CalibratedClassifierCV` works with large confidence scores when using the `sigmoid` method, particularly with the :class:`SGDClassifier`. Non-regression test for issue #26766. gq= ףp?irFrJrgj@)rrJNT)rLr9r squared_hinge)lossr6g@r?)r>roc_auc)scoringr@)rOrP default_rngnormalrintrrsplitrrTranyrrr)global_random_seedprobn random_noiser9r8rLindicestraintestrYrZr\sgd_clf predictions clf_sigmoid score_sigmoid clf_isotonicscore_isotonics r:@test_calibrated_classifier_cv_works_with_large_confidence_scoresrsq D A99(();<CCCKL !s1t8}$sa#a$h-.?'@@AA aii  </A TQ4 0Bhhq!nG) tU8QuX4_CUV GW%//7 c!&&((( ))?9KLK$KAyIM*?9KLL%\1aKNM>2r<cxtjj|}d}|jdd|}|j ddd}d}t ||| \}}d }t ||| \} } t || \} } d } t || | t | | | t || | t | | | y)NrDrrrCrF)lowhighrGry)rr9max_abs_prediction_thresholdr)rr9gư>)atol)rOrPrQrrRr r)rr6rr9predictions_small threshold_1a1b1 threshold_2a2b2a3b3rs r:5test_sigmoid_calibration_max_abs_prediction_thresholdr s99((.@(AL AQ*A%,,!#,FK !% %0FBK !% %0FB"% FB DB&B&B&B&r<cdGddt}|}t|}|j|y)zoCheck that CalibratedClassifierCV works with float32 predict proba. Non-regression test for gh-28245. ceZdZfdZxZS)4test_float32_predict_proba..DummyClassifer32c\t||jtjSr)rrUastyperOfloat32)rr8rs r:rUzBtest_float32_predict_proba..DummyClassifer32.predict_proba<s"7(+222::> >r<)rrrrUrrs@r:DummyClassifer32r;s  ? ?r<rN)rrrT)r;rmodelrs r:test_float32_predict_probar5s0 ???  E'.JJNNDr<)ynumpyrOrV numpy.testingr sklearn.baserrsklearn.calibrationrrrr r r sklearn.datasetsr r r sklearn.dummyrsklearn.ensemblerrsklearn.exceptionsrsklearn.feature_extractionrsklearn.imputersklearn.isotonicrsklearn.linear_modelrrsklearn.metricsrsklearn.model_selectionrrrrrrsklearn.naive_bayesrsklearn.pipeliner r!sklearn.preprocessingr"r# sklearn.svmr$ sklearn.treer%sklearn.utils._mockingr&sklearn.utils._testingr'r(r)r*sklearn.utils.extmathr+sklearn.utils.fixesr,r7fixturer;mark parametrizerermrtrrr8rrrrrrrrr rPrQrrr r%r)paramr-r2r;r?rAr]r_rkrnrxrr9objectrrrrrrrrrrrrrrr<r:rs )-HG).5(/B,.4>!/5 *.  h  .9Iz#:;dE]384<:8v+dE]3 D4 DIz#:;dE]34<,Iz#:;dE]364<6,Iz#:;dE]3q*:7+4< :7z22.9,:,^Iz#:;5<50>"C>dE]3 4 dE]3 J4 JdE]3:4:4 b!''Aq1 b!''Aq!4,""%%4  YFa0!4 YFa0(;66" #  h& &h  Ar7+i%<=&;>,&;R ;-/CD ; ; ;(+.>@R-ST;U;D 5f 6 "7 "f 6/7/(8:UV;W;@Iz#:;dE]3-A4<-A`*VW,=>#?#$   226,Iz#:;dE]3'A4<'AT /3d&'R r<