o rZŽh>ã@stdZddlZddlmZddlmZddlTddlmZddl m Z ddl m Z Gd d „d eƒZ Gd d „d eƒZdS) z‚ Corpus reader for corpora whose documents are xml files. (note -- not named 'xml' to avoid conflicting w/ standard xml package) éN)Ú ElementTree)Ú CorpusReader)Ú*)ÚSeekableUnicodeStreamReader)ÚElementWrapper)ÚWordPunctTokenizerc@s.eZdZdZd dd„Zd dd„Zd dd „ZdS) ÚXMLCorpusReadera Corpus reader for corpora whose documents are xml files. Note that the ``XMLCorpusReader`` constructor does not take an ``encoding`` argument, because the unicode encoding is specified by the XML files themselves. See the XML specs for more info. FcCs||_t |||¡dS©N)Ú _wrap_etreerÚ__init__)ÚselfÚrootZfileidsZ wrap_etree©rúI/var/www/auris/lib/python3.10/site-packages/nltk/corpus/reader/xmldocs.pyr !szXMLCorpusReader.__init__NcCs€|durt|jƒdkr|jd}t|tƒstdƒ‚| |¡ ¡}t |¡  ¡}Wdƒn1s2wY|j r>t |ƒ}|S)Nérz(Expected a single file identifier string) ÚlenZ_fileidsÚ isinstanceÚstrÚ TypeErrorÚabspathÚopenrÚparseZgetrootr r)r ÚfileidÚfpÚeltrrrÚxml%s  ÿzXMLCorpusReader.xmlc Cs€| |¡}| |¡}tƒ}z| ¡}Wn| ¡}Yg}|D]}|j}|dur=t|tƒr3| |¡}|  |¡} |  | ¡q |S)aE Returns all of the words and punctuation symbols in the specified file that were in text nodes -- ie, tags are ignored. Like the xml() method, fileid can only specify one file. :return: the given file's text nodes as a list of words and punctuation symbols :rtype: list(str) N) rÚencodingrZ getiteratorÚiterÚtextrÚbytesÚdecodeÚtokenizeÚextend) r rrrZword_tokenizerÚiteratorÚoutÚnoderÚtoksrrrÚwords4s"       €zXMLCorpusReader.words)Fr )Ú__name__Ú __module__Ú __qualname__Ú__doc__r rr'rrrrrs   rc@sveZdZdZdZdZddd„Zdd„Zd d „Ze   d e j e j B¡Z e   d ¡Ze   d e j e j B¡Zdd„Zddd„ZdS)Ú XMLCorpusViewam A corpus view that selects out specified elements from an XML file, and provides a flat list-like interface for accessing them. (Note: ``XMLCorpusView`` is not used by ``XMLCorpusReader`` itself, but may be used by subclasses of ``XMLCorpusReader``.) Every XML corpus view has a "tag specification", indicating what XML elements should be included in the view; and each (non-nested) element that matches this specification corresponds to one item in the view. Tag specifications are regular expressions over tag paths, where a tag path is a list of element tag names, separated by '/', indicating the ancestry of the element. Some examples: - ``'foo'``: A top-level element whose tag is ``foo``. - ``'foo/bar'``: An element whose tag is ``bar`` and whose parent is a top-level element whose tag is ``foo``. - ``'.*/foo'``: An element whose tag is ``foo``, appearing anywhere in the xml tree. - ``'.*/(foo|bar)'``: An wlement whose tag is ``foo`` or ``bar``, appearing anywhere in the xml tree. The view items are generated from the selected XML elements via the method ``handle_elt()``. By default, this method returns the element as-is (i.e., as an ElementTree object); but it can be overridden, either via subclassing or via the ``elt_handler`` constructor parameter. FiNcCsF|r||_t |d¡|_ ddi|_ | |¡}tj|||ddS)aW Create a new corpus view based on a specified XML file. Note that the ``XMLCorpusView`` constructor does not take an ``encoding`` argument, because the unicode encoding is specified by the XML files themselves. :type tagspec: str :param tagspec: A tag specification, indicating what XML elements should be included in the view. Each non-nested element that matches this specification corresponds to one item in the view. :param elt_handler: A function used to transform each element to a value for the view. If no handler is specified, then ``self.handle_elt()`` is called, which returns the element as an ElementTree object. The signature of elt_handler is:: elt_handler(elt, tagspec) -> value z\Zrr)rN)Ú handle_eltÚreÚcompileÚ_tagspecÚ _tag_contextÚ_detect_encodingÚStreamBackedCorpusViewr )r rÚtagspecÚ elt_handlerrrrrr us  zXMLCorpusView.__init__cCsôt|tƒrz| ¡}| ¡}W| ¡n| ¡wt|dƒ }| ¡}Wdƒn1s-wY| tj¡r:dS| tj¡rBdS| tj ¡rJdS| tj ¡rRdS| tj ¡rZdSt   d|¡}|ri| d¡ ¡St   d |¡}|rx| d¡ ¡SdS) NÚrbz utf-16-bez utf-16-lez utf-32-bez utf-32-lezutf-8s!\s*<\?xml\b.*\bencoding="([^"]+)"rs!\s*<\?xml\b.*\bencoding='([^']+)')rZ PathPointerrÚreadlineÚcloseÚ startswithÚcodecsÚ BOM_UTF16_BEÚ BOM_UTF16_LEÚ BOM_UTF32_BEÚ BOM_UTF32_LEÚBOM_UTF8r.ÚmatchÚgroupr )r rÚinfileÚsÚmrrrr2™s2    ÿ       zXMLCorpusView._detect_encodingcCs|S)a Convert an element into an appropriate value for inclusion in the view. Unless overridden by a subclass or by the ``elt_handler`` constructor argument, this method simply returns ``elt``. :return: The view value corresponding to ``elt``. :type elt: ElementTree :param elt: The element that should be converted. :type context: str :param context: A string composed of element tags separated by forward slashes, indicating the XML context of the given element. For example, the string ``'foo/bar/baz'`` indicates that the element is a ``baz`` element whose parent is a ``bar`` element and whose grandparent is a top-level ``foo`` element. r)r rÚcontextrrrr-¶szXMLCorpusView.handle_elta; [^<]* ( (() | # comment () | # doctype decl (<[^!>][^>]*>)) # tag or PI [^<]*)* \Zz<\s*(?:/\s*)?([^\s>]+)a6 # Include these so we can skip them: (?P )| (?P )| (?P <\?.*?\?> )| (?P ]*(\[[^\]]*])?\s*>)| # These are the ones we actually care about: (?P <\s*[^>/\?!\s][^>]*/\s*> )| (?P <\s*[^>/\?!\s][^>]*> )| (?P <\s*/[^>/\?!\s][^>]*> )cCsôd}t|tƒr | ¡} | |j¡}||7}|j |¡r|St d|¡  d¡dkr?| ¡t |ƒt d|¡  ¡}t d|ƒ‚|sEt dƒ‚|  d¡}|dkry|j |d |…¡ryt|tƒrh| |¡| |¡n | t |ƒ| d ¡|d |…Sq ) a{ Read a string from the given stream that does not contain any un-closed tags. In particular, this function first reads a block from the stream of size ``self._BLOCK_SIZE``. It then checks if that block contains an un-closed tag. If it does, then this function either backtracks to the last '<', or reads another block. ÚTz[<>]rú>zUnexpected ">" near char %sz&Unexpected end of file: tag not closedú25} {}ú/iìÿÿÿZ START_TAGrZEND_TAGzUnmatched tag éÿÿÿÿzUnmatched tag ...‡süþÿz,XMLCorpusView.read_block..)r0r-Úlistr1ÚgetrIrrrWrOÚ _XML_PIECEÚfinditerÚ_DEBUGÚprintÚformatÚjoinrAÚ _XML_TAG_NAMEr@Úappendr.ÚstartrrNÚpoprQrRÚtuple)r rSr4r5rEÚeltsZ elt_startZ elt_depthZelt_textrUZ xml_fragmentZpieceÚnamerVrr^rÚ read_block"sˆ   "  €       €     ¹J  ûzXMLCorpusView.read_blockr )NN)r(r)r*r+rdrKr r2r-r.r/ÚDOTALLÚVERBOSErLrhrbrWrorrrrr,Qs$ $ ö  õ1r,)r+r:Z xml.etreerZnltk.corpus.reader.apirZnltk.corpus.reader.utilZ nltk.datarZnltk.internalsrZ nltk.tokenizerrr3r,rrrrÚs     9