:'aE=ddlmZmZddlmZmZmZmZmZ ddl m Z ddl mZmZmZddlmZddlmZmZddlmZdd lZdd lmZmZmZmZmZmZdd l m!Z!m"Z"m#Z#m$Z$ejJd Z&e&jOejPejRZ*e*jWejXd e&j[e* dde.de/de/de0dee dee de1de1defdZ2 ddede/de/de0dee dee de1de1defdZ3 dde de/de/de0dee dee de1de1defdZ4dde de/de/de0dee dee de1defdZ5y #e $r ee dfZ YEwxYw))splitextbasename)ListBinaryIOOptionalSetUnion)PathLikezos.PathLike[str])TOO_SMALL_SEQUENCETOO_BIG_SEQUENCEIANA_SUPPORTED) mess_ratio)CharsetMatches CharsetMatch)warnN)any_specified_encodingis_multi_byte_encodingidentify_sig_or_bomshould_strip_sig_or_bom is_cp_similar iana_name)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratioscharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s sequencessteps chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainreturnc |s$tjtjn#tjtjt |}|dk(r/tj dtt|dddgdgS|?tj dd j||D cgc]} t| d}} ng}|?tj d d j||D cgc]} t| d}} ng}|||zkrtj d |||d }|}|d kDr||z |krt||z }t |tk} t |tk\} | rtd j|g} |dur t!|nd} | '| j#| tj%d| t'}g}g}d}d}d}d}d}t}t)|\}}|1| j#|tj%dt ||| j#dd| vr| j#d| t*zD]}|r||vr |r||vr||vr|j-|d}||k(}|xr t/|}|dvr|durtj%d|] t1|} | r9|dur5t9|dur|dtdn|t |td|nt9|dur|n |t |d|}d}|D]} t?|| sd}n|rtj d| tA|durdn t ||t||z }!|xr|duxrt ||k}"|"rtj%d|tt |!dz }#|#dkrd}#d}$g}%g}&|!D]o}'||'|'|z}(|r |dur||(z}(|(jC|d})|%j#|)|&j#tE|)||&d|k\r|$d z }$|$|#k\s|sj|duson|&rtG|&t |&z }*nd}*|*|k\s|$|#k\ri|j#||s|d z }tj d||$tI|*d zd!"|dd| fvr"t|||dg|}+|| k(r|+}n |dk(r|+}n|+}vtj%d#|tI|*d zd!"|s tK|},n tM|},|,r.tj%d$j|t9|,g}-|%D]3})tO|)d%|,rd&j|,nd}.|-j#|.5tQ|-}/|/r%tj%d'j|/||j#t|||*||/||| ddfvr,|*d%kr'tj%d(|t||gcS||k(r'tj%d)|t||gcS|djRstj%d*|||jTt |dk(r|s|s|rtj d+|r3tj d,|jV|j#||S|r||rA|jX|jXk7r(tj d-|j#||S|r&tj d.|j#||Scc} wcc} w#t2t4f$rtj7d|YwxYw#t:$rC}tj d|t9||j#||s|d z }Yd}~Rd}~wt<$r|j#||s|d z }YywxYw)/aD Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. rzXGiven content is empty, stopping the process very early, returning empty utf_8 str matchutf_8gFNz`cp_isolation is set. use this flag for debugging purpose. limited list of encoding allowed : %s.z, zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.z>Trying to detect encoding from a tiny portion of ({}) byte(s).Tz@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_16utf_32z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecodergA)encodingz9Code page %s does not fit given bytes sequence at ALL. %szW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.ignore)errorszc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}g?,z We detected language {} using {}z0%s is most likely the one. Stopping the process.z[%s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.z:Using %s code page we detected the following languages: %szONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z#%s will be used as a fallback matchz&utf_8 will be used as a fallback matchz&ascii will be used as a fallback match)-loggersetLevelloggingCRITICALINFOlenwarningrrjoinrintr r rformatrappendinfosetrr addrrModuleNotFoundError ImportErrordebugstrUnicodeDecodeError LookupErrorrrangedecodersumroundrrrr languages _languagesr- fingerprint)0rrrr r!r"r#r$lengthcpis_too_small_sequenceis_too_large_sequenceprioritized_encodingsspecified_encodingtestedtested_but_hard_failuretested_but_soft_failurefallback_ascii fallback_u8fallback_specifiedsingle_byte_hard_failure_countsingle_byte_soft_failure_countresults sig_encoding sig_payload encoding_ianadecoded_payloadbom_or_sig_availablestrip_sig_or_bomis_multi_byte_decoderesimilar_soft_failure_testencoding_soft_failedr_multi_byte_bonusmax_chunk_gave_upearly_stop_count md_chunks md_ratiosi cut_sequencechunkmean_mess_ratiofallback_entrytarget_languages cd_ratioschunk_languagescd_ratios_mergeds0 6lib/python3.12/site-packages/charset_normalizer/api.py from_bytesr{s2 (() % ^F {qr    @yy. 08DD "e,D D  6 IIl # %8DD "e,D D  *u$% l :v '  qyVe^j0%(  N-?? N.>> M T TU[ \]>RVZ>Z/ :`d%$$%78 VXjk UF  NK%&"%&"G 3I >L+$$\2 _adepaqs A  )++$$W-.~=D M=  M\9  F "  =!+}</Z4KM4Z 0 05IU5R KKuxE F  $:=$I !  $)>%)G-=-FIjs4y)IVYZeVfgjkogpLq* #&!1U!:I #kJZJ[@\*# %*!$;  ],@A,0)  % NNtwDFZ [  %.AC 4D     1r_D5PrUXYhUilrUr  KKKMZ [B! , q !    A$Qq:~6L#(8E(A*<7  '' h'GE   U #    } ) A%  $55;OTdhmTm+ . !)ns9~=O O i '+;?P+P # * *= 9(.!3. NN;(+ 3!6B  D '3E FF!-!# "!$66)7&"g-%3N"0K  K  /C' 3 %1-@ 4]C   KK@GG WZ[kWlm n  E-eSXh#((CS:TnrsO     2)<  KK:AABRTab c $     /'B BY\G\ KKJM Z!'(  L ( KKm "'(  2; KKL &11 ADL 7|q .,> NNl m  NN@BTB]B] ^ NN- . Nn4++JaJaeseeJ NNC D NN; ' N  NNC D NN> * NcEEV$[1  LLM} ]  "  NNVXegjklgm n # * *= 9(.!3.   # * *= 9(.!3.   s=4\17\6: \;A]';%]$#]$' _08^..$__fpc Bt|j|||||||S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )r{read)r|rrr r!r"r#r$s rzfrom_fprbs/      pathc jt|d5}t||||||||cdddS#1swYyxYw)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openr) rrrr r!r"r#r$r|s rz from_pathr|sA dD tRr5*i|Uikrsttts)2c t|||||||}t|}tt|} t |dk(rt dj ||j} | dxxd| jzz cc<tdj |j|dj| d5} | j| jddd| S#1swY| SxYw)zi Take a (text-based) file path and try to create another file next to it, this time using UTF-8. rz;Unable to normalize "{}", no encoding charset seems to fit.-z{}r(wbN)rrlistrr<IOErrorr@bestr-rreplacer>writeoutput) rrrr r!r"r#r`filenametarget_extensionsresultr|s rz normalizers  G~HXh/0 7|qSZZ[cdee \\^FaC&//11 dkk$,,x9J1KLMt T XZ  MMO  M  Ms = C''C1)皙?NNTF)rrrNNT)6os.pathrrtypingrrrrr osr rFrHcharset_normalizer.constantr r r charset_normalizer.mdrcharset_normalizer.modelsrrwarningsrr9charset_normalizer.utilsrrrrrrcharset_normalizer.cdrrrr getLoggerr7r8DEBUG StreamHandlerhandler setFormatter Formatter addHandlerbytesr?floatboolr{rrrrrzrs&77.]\,B66tt   / 0  '   !&W&&'RST' "&"&%)DDDD D 3i D 3i D#DDDR "&"&%)   3i  3i #8"&"&%)tttt t 3i t 3i t#ttt$HS#PUlpqtluMQRUMVuyEQQ .S,,-H.sE E$#E$