eTh 9SSKrSSKrSSKrSSKJr SSKJrJrJrJ r J r J r J r SSK JrJr \R R#\5r\R R'\S5r\"\5r\R/5rSSS5 "SS5r\ \r\ \ \r"SS \ 5r"S S 5r"S S 5r"SS5r \4S\ \!S\!4Sjjr"g!,(df  N^=f)N)Template)AnyCallableDictList NamedTupleOptionalTuple)Encoding Tokenizerzvisualizer-styles.csscJ\rSrSr%\\S'\\S'\\S'S\S\S\4SjrSrg) Annotationstartendlabelc(XlX lX0lgN)rrr)selfrrrs S/var/www/auris/envauris/lib/python3.13/site-packages/tokenizers/tools/visualizer.py__init__Annotation.__init__s  )rrrN) __name__ __module__ __qualname____firstlineno__int__annotations__strr__static_attributes__rrrrs+ J H JcCrrc6\rSrSr%\\\S'\\\S'Srg) CharStateKeytoken_ixanno_ixr"N)rrrrr rrr!r"rrr$r$ssm c]rr$cZ\rSrSr%\\\S'Sr\S5r \S5r S\ 4Sjr Sr g ) CharState$char_ixc,XlSUl/Ulgr)r+r'tokens)rr+s rrCharState.__init__'s &* !# rcV[UR5S:aURS$S$)Nrlenr-rs rr&CharState.token_ix-s%!$T[[!1A!5t{{1~?4?rc2[UR5S:$)z: BPE tokenizers can output more than one token for a char r0r2s r is_multitokenCharState.is_multitoken1s 4;;!##rreturnc>[URURS9$)N)r&r')r$r&r'r2s r partition_keyCharState.partition_key8s]]LL  r)r'r+r-N)rrrrr rrrpropertyr&r6r$r:r!r"rrr)r)$sG c]$ @@$$  | rr)c\rSrSrSrg)Aligned?r"N)rrrrr!r"rrr>r>?srr>c f\rSrSrSr\R "S\RS9rSS\ S\ S\ \ \ /\44S jjr/S4S \S \S\ \ S \ \4S jjr\S \S \\\44Sj5r\S\\S \S\4Sj5r\S \S\S \S \4Sj5r\S \S \S \4Sj5r\S \S\S \S \\4Sj5rSrg)EncodingVisualizerCa Build an EncodingVisualizer Args: tokenizer (:class:`~tokenizers.Tokenizer`): A tokenizer instance default_to_notebook (:obj:`bool`): Whether to render html output in a notebook by default annotation_converter (:obj:`Callable`, `optional`): An optional (lambda) function that takes an annotation in any format and returns an Annotation object z(.{1})?(unk|oov)(.{1})?)flagsN tokenizerdefault_to_notebookannotation_converterczU(a SSKJnJn XlX lX0lg![a [ S5ef=f)NrHTMLdisplayzWe couldn't import IPython utils for html display. Are you running in a notebook? You can also pass `default_to_notebook=False` to get back raw HTML )IPython.core.displayrIrJ ImportError ExceptionrDrEannotation_coverter)rrDrErFrIrJs rrEncodingVisualizer.__init__VsH  >##6 #7    s$:text annotationsr8c`URnUbUnU(a SSKJnJn UR b[[UR U55nURRU5n[RXU5nU(aW"W"U55 gU$![a [ S5ef=f)aw Build a visualization of the given text Args: text (:obj:`str`): The text to tokenize annotations (:obj:`List[Annotation]`, `optional`): An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function default_to_notebook (:obj:`bool`, `optional`, defaults to `False`): If True, will render the html in a notebook. Otherwise returns an html string. Returns: The HTML string if default_to_notebook is False, otherwise (default) returns None and renders the HTML in the notebook NrrHzeWe couldn't import IPython utils for html display. Are you running in a notebook?) rErKrIrJrLrMrNlistmaprDencoderA_EncodingVisualizer__make_html) rrPrQrEfinal_default_to_notebookrIrJencodinghtmls r__call__EncodingVisualizer.__call__ls2%)$<$<!  *(; % $ >  # # /s4#;#;[IJK>>((.!--dkJ $ DJ K 6 s BB-c[U5S:Xa0$[[SU55n[U5n[SU- 5nUS:aSnSnSnSn0n[ U5HnSUS US US 3Xx'Xc- nM U$) z Generates a color palette for all the labels in a given set of annotations Args: annotations (:obj:`Annotation`): A list of annotations Returns: :obj:`dict`: A dictionary mapping labels to colors in HSL format rcUR$r)r)xs r;EncodingVisualizer.calculate_label_colors..s177r @ zhsl(,z%,%)r1setrTrsorted) rQlabels num_labelsh_stepslhcolorsrs rcalculate_label_colors)EncodingVisualizer.calculate_label_colorss { q IS*K89[ S:%& B;F   F^E"1#QqcA3a0FM KA$ rconsecutive_chars_listrXcUSnURcURURnSUS3$USnURnURS-nXUn/n 0n URbU RS5 UR(aU RS5 URS-(aU RS 5 OU RS 5 [ R RURUR5b-U RS 5 URURU S 'OU RS 5 SSRU 5S3n Sn U R5HupU SU SUS3- n M SU SU SUS3$)as Converts a list of "consecutive chars" into a single HTML element. Chars are consecutive if they fall under the same word, token and annotation. The CharState class is a named tuple with a "partition_key" method that makes it easy to compare if two chars are consecutive. Args: consecutive_chars_list (:obj:`List[CharState]`): A list of CharStates that have been grouped together text (:obj:`str`): The original text being processed encoding (:class:`~tokenizers.Encoding`): The encoding returned from the tokenizer Returns: :obj:`str`: The HTML span for a set of consecutive chars rz(r5tokenz multi-tokenz odd-tokenz even-tokenz special-tokenstokz non-tokenzclass=" "z data-z="z) r+r-r&appendr6rAunk_token_regexsearchjoinitems)rsrPrXfirststokenlastrr span_text css_classes data_itemscssdatakeyvals rconsecutive_chars_to_html,EncodingVisualizer.consecutive_chars_to_htmls2'q) == __U^^4F>fXXN N%b) llQsO   >> %   w '""""=1~~! "";/""<0!11889XYe""?3%-__U^^%D 6"   { +#((;/04"((*HC fSEC5* *D+uAdV2i[88rc 4[RXU5nUS/nUSRn/n[RU5nUSRnUb,X(n U Rn Xzn UR SU SU S35 USSHn U RnX:wajUR [R UUUS95 U /nUbUR S5 Ub,X(n U Rn Xzn UR SU SU S35 UnU R5USR5:XaUR U 5 MUR [R UUUS95 U /nM UR [R UUUS95 [U5n U $)Nrz&r5)rPrXr|) rA%_EncodingVisualizer__make_char_statesr'rqrr}rr:HTMLBody)rPrXrQ char_statescurrent_consecutive_chars prev_anno_ixspanslabel_colors_dict cur_anno_ixannorcolorcsress r __make_htmlEncodingVisualizer.__make_htmls(;;DKX %0^$4!"1~-- .EEkR!!n,,  "+DJJE%,E LLA%W\V]]_` aab/B**K* &@@1!!)A.0D)+LL+*&3D JJE-4ELL#I%P^_d^eeg!hi&L!%>q%A%O%O%QQ)004 &@@1!!)A.0D)M"R   8 8)! 9  uo rcS/[U5-n[U5H0up4[URUR5HnX2U'M M2 U$)a Args: text (:obj:`str`): The raw text we want to align to annotations (:obj:`AnnotationList`): A (possibly empty) list of annotations Returns: A list of length len(text) whose entry at index i is None if there is no annotation on character i or k, the index of the annotation that covers index i where k is with respect to the list of annotations N)r1 enumeraterangerr)rPrQannotation_mapr'ais r__make_anno_map"EncodingVisualizer.__make_anno_map<sN#d)+#K0JG177AEE*$+q!+1rc[RX5n[[U55Vs/sHn[ U5PM nn[ UR 5HNupgURU5nUcMUup[X5H n X[R RU5 M" MP [ U5HupLXUl M U$s snf)aJ For each character in the original text, we emit a tuple representing it's "state": * which token_ix it corresponds to * which word_ix it corresponds to * which annotation_ix it corresponds to Args: text (:obj:`str`): The raw text we want to align to annotations (:obj:`List[Annotation]`): A (possibly empty) list of annotations encoding: (:class:`~tokenizers.Encoding`): The encoding returned from the tokenizer Returns: :obj:`List[CharState]`: A list of CharStates, indicating for each char in the text what it's state is ) rA"_EncodingVisualizer__make_anno_maprr1r)rr-token_to_charsr}r') rPrXrQrr+rr&rvoffsetsrrrr's r__make_char_states%EncodingVisualizer.__make_char_statesQs.,;;DNJOPSTXPYJZ'[JZw '(:JZ '[(9OH--h7G"$ u*AN))00:+ : !*. 9 G+2 (!:(\sC )rNrErD)TN)rrrr__doc__recompile IGNORECASEr~r boolr rrrrr AnnotationListrZ staticmethodrrqrr)r rrVPartialIntListrrr!r"rrrArACs jj!>bmmTO %)FJ   " 'xz0A'BC  2').2 ++$+&d^ + # +ZNtCH~8A9 $YA9A9A9A9F?#???SV??Bc>("""~"Z^_hZi""rrAchildrenr8c6SRU5nSUSUS3$)a7 Generates the full html with css from a list of html spans Args: children (:obj:`List[str]`): A list of strings, assumed to be html elements css_styles (:obj:`str`, `optional`): Optional alternative implementation of the css Returns: :obj:`str`: An HTML string with style markup r{z?
z4
)r)r css_styles children_texts rrrws9GGH%M  O  r)# itertoolsosrstringrtypingrrrrrr r tokenizersr r pathdirname__file__r css_filenameopenfreadrrrrrr$r)r>rAr rr"rrrs III* ''//( #ww||G%<=  ,1 &&(Cj!hsm$:   6  qqh .1tCySW s ,C C#