a BiX @sRddlmZddlZddlmZddlmZddlmZm Z m Z m Z ddl m Z mZmZmZddlmZdd lmZmZdd lmZmZmZmZmZmZmZed Ze Z!e!"e#d d(dddddddddddd ddZ$d)dddddddddddd ddZ%d*d ddddddddddd! d"d#Z&d+d$ddddddddddd% d&d'Z'dS),) annotationsN)PathLike)BinaryIO)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encodingcut_sequence_chunks iana_nameidentify_sig_or_bom is_cp_similaris_multi_byte_encodingshould_strip_sig_or_bomZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s皙?TF皙?zbytes | bytearrayintfloatzlist[str] | Noneboolr) sequencessteps chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainlanguage_thresholdenable_fallbackreturnc 2 Cs t|ttfs tdt||r>tj} tt t t t |} | dkrt d|rvtt t | prtjtt|dddgdgS|durtt d d |d d |D}ng}|durtt d d |dd |D}ng}| ||krtt d||| d}| }|dkr:| ||kr:t| |}t |tk} t |tk} | rltt d| n| rtt d| g}|rt|nd}|dur||tt d|t}g}g}d}d}d}t}t}t|\}}|dur||tt dt |||dd|vr4|d|tD]}|rV||vrVq<|rj||vrjq<||vrxq<||d}||k}|ot|}|dvr|stt d|q<|dvr|stt d|q[zfrom_bytes..zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.cSsg|]}t|dqSr,r-r.r1r1r2r3fr4z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.rz>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_16utf_32z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.>utf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.z2Encoding %s does not provide an IncrementalDecodergA)encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sgj@strict)errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsr6r7)Zpreemptive_declarationz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {},z We detected language {} using {}rz.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4 isinstance bytearraybytes TypeErrorformattypeloggerlevel addHandlerexplain_handlersetLevelr lendebug removeHandlerloggingWARNINGrrlogjoinrr r rappendsetrr addrrModuleNotFoundError ImportErrorstrUnicodeDecodeError LookupErrorrrangemaxrrdecodesumroundrrrr r9best fingerprint)2rr r!r"r#r$r%r&r'r(Zprevious_logger_levellengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodingsZspecified_encodingZtestedZtested_but_hard_failureZtested_but_soft_failureZfallback_asciiZ fallback_u8Zfallback_specifiedresultsZearly_stop_resultsZ sig_encodingZ sig_payloadZ encoding_ianaZdecoded_payloadZbom_or_sig_availableZstrip_sig_or_bomZis_multi_byte_decodereZsimilar_soft_failure_testZencoding_soft_failedZr_Zmulti_byte_bonusZmax_chunk_gave_upZearly_stop_countZlazy_str_hard_failureZ md_chunksZ md_ratioschunkZmean_mess_ratioZfallback_entryZtarget_languagesZ cd_ratiosZchunk_languagesZcd_ratios_mergedZ current_matchZprobable_resultr1r1r2 from_bytes!s(                       $      &   $                             rhr) fpr r!r"r#r$r%r&r'r(r)c Cst|||||||||| S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )rhread) rir r!r"r#r$r%r&r'r(r1r1r2from_fp!srkzstr | bytes | PathLike) pathr r!r"r#r$r%r&r'r(r)c CsHt|d*} t| ||||||||| WdS1s:0YdS)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openrk) rlr r!r"r#r$r%r&r'r(rir1r1r2 from_path?s roz!PathLike | str | BinaryIO | bytes) fp_or_path_or_payloadr r!r"r#r$r%r&r'r(r)c Cszt|ttfr,t|||||||||| d } nHt|ttfrXt|||||||||| d } nt|||||||||| d } | S)a) Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string. Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match are disabled to be stricter around ASCII-compatible but unlikely to be a string. ) r r!r"r#r$r%r&r'r()rCrZrrorErDrhrk) rpr r!r"r#r$r%r&r'r(Zguessesr1r1r2 is_binary^sX   rq) rrrNNTFrT) rrrNNTFrT) rrrNNTFrT) rrrNNTFrF)( __future__rrQosrtypingrcdrrrr Zconstantr r r r mdrmodelsrrutilsrrrrrrr getLoggerrI StreamHandlerrL setFormatter Formatterrhrkrorqr1r1r1r2sr    $ $$ $!