o Zh1@sndZddlZddlmZmZddlmZmZmZer%ddl Z ddl m Z e e Z GdddeZdgZdS) z' Feature extractor class for MarkupLM. N) BatchFeatureFeatureExtractionMixin)is_bs4_availableloggingrequires_backends) BeautifulSoupcsFeZdZdZfddZddZddZdd Zd efd d Z Z S) MarkupLMFeatureExtractorao Constructs a MarkupLM feature extractor. This can be used to get a list of nodes and corresponding xpaths from HTML strings. This feature extractor inherits from [`~feature_extraction_utils.PreTrainedFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. c s"t|dgtjdi|dS)Nbs4)rsuper__init__)selfkwargs __class__r g/var/www/auris/lib/python3.10/site-packages/transformers/models/markuplm/feature_extraction_markuplm.pyr +s z!MarkupLMFeatureExtractor.__init__csg}g}|jr |n|jjD]+}|jjdd}|j|dt|kr)dn tfddt|dD|q||||fS)NF) recursiverc3s |] \}}|ur|VqdS)Nr ).0ischildr r 7sz6MarkupLMFeatureExtractor.xpath_soup..) nameparentparentsZfind_allappendlennext enumeratereverse)relement xpath_tagsxpath_subscriptsrZsiblingsr rr xpath_soup/s  *z#MarkupLMFeatureExtractor.xpath_soupc Cst|d}g}g}g}|jD]3}t|tjjrAt|jtjjur!qt | }|s+q| || |\}} | || | qt|t|krNtdt|t|krZtd|||fS)Nz html.parserz3Number of doc strings and xtags does not correspondz3Number of doc strings and xsubs does not correspond)rZ descendants isinstancer r#ZNavigableStringtyperTaghtmlunescapestriprr&r ValueError) r html_stringZ html_codeall_doc_stringsstring2xtag_seqstring2xsubs_seqr#Ztext_in_this_tagr$r%r r rget_three_from_single>s*      z.MarkupLMFeatureExtractor.get_three_from_singlecCsBd}t||D]\}}|d|7}|dkr|d|d7}q|S)N/r[])zip)rr$r%Zxpathtagnamesubsr r rconstruct_xpath[sz(MarkupLMFeatureExtractor.construct_xpathreturncCsd}t|tr d}nt|ttfr t|dkst|dtr d}|s,tdt|dtt|ttfo:t|dt}|sA|g}g}g}|D]-}||\}}} | |g} t ||| D]\} } } | | | }| |q^| | qG||d}t |dd}|S) a\ Main method to prepare for the model one or several HTML strings. Args: html_strings (`str`, `List[str]`): The HTML string or batch of HTML strings from which to extract nodes and corresponding xpaths. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **nodes** -- Nodes. - **xpaths** -- Corresponding xpaths. Examples: ```python >>> from transformers import MarkupLMFeatureExtractor >>> page_name_1 = "page1.html" >>> page_name_2 = "page2.html" >>> page_name_3 = "page3.html" >>> with open(page_name_1) as f: ... single_html_string = f.read() >>> feature_extractor = MarkupLMFeatureExtractor() >>> # single example >>> encoding = feature_extractor(single_html_string) >>> print(encoding.keys()) >>> # dict_keys(['nodes', 'xpaths']) >>> # batched example >>> multi_html_strings = [] >>> with open(page_name_2) as f: ... multi_html_strings.append(f.read()) >>> with open(page_name_3) as f: ... multi_html_strings.append(f.read()) >>> encoding = feature_extractor(multi_html_strings) >>> print(encoding.keys()) >>> # dict_keys(['nodes', 'xpaths']) ```FTrzQHTML strings must of type `str`, `List[str]` (batch of examples), but is of type .)nodesxpathsN)dataZ tensor_type) r'strlisttuplerr-r(boolr2rr7r:r)rZ html_stringsZ valid_stringsZ is_batchedr=r>r.r/r0r1Z xpath_stringsnodeZtag_listZsub_listZ xpath_stringr?Zencoded_inputsr r r__call__cs80        z!MarkupLMFeatureExtractor.__call__) __name__ __module__ __qualname____doc__r r&r2r:rrE __classcell__r r rrr !s r )rIr*Zfeature_extraction_utilsrrutilsrrrr rZ get_loggerrFloggerr __all__r r r rs