o rZh6@sdZddlZddlZddlmZddlmZmZddlm Z ddZ Gdd d e Z Gd d d Z Gd d d Z GdddeZdS)z9 A reader for corpora whose documents are in MTE format. N)reduce)TaggedCorpusReaderconcat) XMLCorpusViewcCs |||SN)findall)rootpathnsr E/var/www/auris/lib/python3.10/site-packages/nltk/corpus/reader/mte.pyxpath s r c@s$eZdZdZdddZdddZdS) MTECorpusViewz0 Class for lazy viewing the MTE Corpus. NcCst||||dSr)r__init__)selfZfileidtagspec elt_handlerr r r rszMTECorpusView.__init__c Csttddt||||S)NcS|duSrr xr r r z*MTECorpusView.read_block..)listfilterr read_block)rstreamrrr r r rs zMTECorpusView.read_blockr)NN)__name__ __module__ __qualname____doc__rrr r r r rs rc@seZdZdZdddZdZdZdZdZd Z d d Z e d d Z e ddZ e ddZe ddZe ddZe ddZe ddZe ddZe ddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0S)1 MTEFileReaderz Class for loading the content of the multext-east corpus. It parses the xml files and does some tag-filtering depending on the given method parameters. zhttps://www.tei-c.org/ns/1.0z%https://www.w3.org/XML/1998/namespace)Zteixmlz{https://www.tei-c.org/ns/1.0}z'{https://www.w3.org/XML/1998/namespace}zTEI/text/body/div/div/p/s/(w|c)zTEI/text/body/div/div/p/szTEI/text/body/div/div/pcCs ||_dSr)_MTEFileReader__file_path)r file_pathr r r r3s zMTEFileReader.__init__cCs|jSr)textclseltcontextr r r _word_elt6szMTEFileReader._word_eltcfddt|djDS)Ncg|]}|dqSr)r).0wr&r r <z+MTEFileReader._sent_elt..*r r r%r r/r _sent_elt:zMTEFileReader._sent_eltcr*)Ncr+r)r4r-sr/r r r0@r1z+MTEFileReader._para_elt..r2r3r%r r/r _para_elt>r5zMTEFileReader._para_eltcCsd|jvr |jdfS|jdkr|jdkr|j|jdfS|jdkr1|jdkr1|jt|jdfStdtdd|jd}| |jdr`|jdkrU|j|jdfS|jt|jdfSdS) NZanamsd universal^-.z.*$) attribr$_MTEFileReader__tags_MTEFileReader__tagsetMTETagConvertermsd_to_universalrecompilesubmatch)r&r'r(tagsr r r _tagged_word_eltBs   zMTEFileReader._tagged_word_eltc*ttddfddt|djDS)NcSrrr rr r r r\rz0MTEFileReader._tagged_sent_elt..cr+r)rIr,r/r r r0]r1z2MTEFileReader._tagged_sent_elt..r2rrr r r%r r/r _tagged_sent_eltX zMTEFileReader._tagged_sent_eltcrJ)NcSrrr rr r r rerz0MTEFileReader._tagged_para_elt..cr+r)rLr6r/r r r0fr1z2MTEFileReader._tagged_para_elt..r2rKr%r r/r _tagged_para_eltarMzMTEFileReader._tagged_para_eltcCs$d|jvr |jdfS|j|jdfS)NZlemmar9)r?r$r%r r r _lemma_word_eltjs  zMTEFileReader._lemma_word_eltcr*)Ncr+r)rOr,r/r r r0sr1z1MTEFileReader._lemma_sent_elt..r2r3r%r r/r _lemma_sent_eltqr5zMTEFileReader._lemma_sent_eltcr*)Ncr+r)rPr6r/r r r0wr1z1MTEFileReader._lemma_para_elt..r2r3r%r r/r _lemma_para_eltur5zMTEFileReader._lemma_para_eltcCt|jtjtjSr)rr"r word_pathr)rr r r wordsy zMTEFileReader.wordscCrRr)rr"r sent_pathr4rTr r r sents~rVzMTEFileReader.sentscCrRr)rr"r para_pathr8rTr r r parasrVzMTEFileReader.parascCrRr)rr"r rSrOrTr r r lemma_wordsrVzMTEFileReader.lemma_wordscC|t_|t_t|jtjtjSr)r rAr@rr"rSrIrtagsetrHr r r tagged_words  zMTEFileReader.tagged_wordscCrRr)rr"r rWrPrTr r r lemma_sentsrVzMTEFileReader.lemma_sentscCr\r)r rAr@rr"rWrLr]r r r tagged_sentsr`zMTEFileReader.tagged_sentscCrRr)rr"r rYrQrTr r r lemma_parasrVzMTEFileReader.lemma_parascCr\r)r rAr@rr"rYrNr]r r r tagged_parasr`zMTEFileReader.tagged_parasN)rrrrr Ztag_nsZxml_nsrSrWrYr classmethodr)r4r8rIrLrNrOrPrQrUrXrZr[r_rarbrcrdr r r r r "sL          r c @s:eZdZdZdddddddd d d d d d ZeddZdS)rBzu Class for converting msd tags to universal tags, more conversion options are currently not implemented. ZADJZADPZADVZCONJZDETZNOUNZNUMZPRTZPRONZVERBr>X) ASRCDNMQPVr>r=cCs4|ddks |dn|d}|tjvrd}tj|S)z This function converts the annotation from the Multex-East to the universal tagset as described in Chapter 5 of the NLTK-Book Unknown Tags will be mapped to X. Punctuation marks are not supported in MSD tags, so r#r=)rBmapping_msd_universal)tagZ indicatorr r r rCs  z MTETagConverter.msd_to_universalN)rrrrrs staticmethodrCr r r r rBs"rBc@s|eZdZdZdddZddZddd Zdd d Zdd d ZdddZ dddZ dddZ dddZ dddZ dddZdS)MTECorpusReaderz Reader for corpora following the TEI-p5 xml scheme, such as MULTEXT-East. MULTEXT-East contains part-of-speech-tagged words with a quite precise tagging scheme. These tags can be converted to the Universal tagset Nutf8cCst||||d|_dS)a. Construct a new MTECorpusreader for a set of documents located at the given root directory. Example usage: >>> root = '/...path to corpus.../' >>> reader = MTECorpusReader(root, 'oana-*.xml', 'utf8') # doctest: +SKIP :param root: The root directory for this corpus. (default points to location in multext config file) :param fileids: A list or regexp specifying the fileids in this corpus. (default is oana-en.xml) :param encoding: The encoding of the given files (default is utf8) z 00README.txtN)rrZ_readme)rrfileidsencodingr r r rs zMTECorpusReader.__init__csP|durj}nt|tr|g}tfdd|}tdd|}|s&td|S)Ncs |jvSr)_fileidsrrTr r rs z+MTECorpusReader.__fileids..cSs|dvS)N)z oana-bg.xmlz oana-mk.xmlr rr r r rrz$No valid multext-east file specified)rz isinstancestrrprintrrxr rTr Z __fileidss zMTECorpusReader.__fileidsctfdd|DS)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of words and punctuation symbols. :rtype: list(str) c$g|]}ttjj|qSr )r osr join_rootrUr-frTr r r0z)MTECorpusReader.words..r_MTECorpusReader__fileidsr~r rTr rUs  zMTECorpusReader.wordscr)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of sentences or utterances, each encoded as a list of word strings :rtype: list(list(str)) crr )r rr rrrXrrTr r r0 rz)MTECorpusReader.sents..rr~r rTr rX  zMTECorpusReader.sentscr)a :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as lists of word string :rtype: list(list(list(str))) crr )r rr rrrZrrTr r r0rz)MTECorpusReader.paras..rr~r rTr rZrzMTECorpusReader.parascr)a :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of words, the corresponding lemmas and punctuation symbols, encoded as tuples (word, lemma) :rtype: list(tuple(str,str)) crr )r rr rrr[rrTr r r0&rz/MTECorpusReader.lemma_words..rr~r rTr r[rzMTECorpusReader.lemma_wordsr:r9c<dksdkrtfdd|DStddS)a; :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of tagged words and punctuation symbols encoded as tuples (word, tag) :rtype: list(tuple(str, str)) r;r:c(g|]}ttjj|qSr )r rr rrr_rrrHr^r r r09 z0MTECorpusReader.tagged_words..Unknown tagset specified.Nrrr}rrxr^rHr rr r_,  zMTECorpusReader.tagged_wordscr)aB :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of sentences or utterances, each encoded as a list of tuples of the word and the corresponding lemma (word, lemma) :rtype: list(list(tuple(str, str))) crr )r rr rrrarrTr r r0Lrz/MTECorpusReader.lemma_sents..rr~r rTr raC  zMTECorpusReader.lemma_sentscr)aH :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of sentences or utterances, each each encoded as a list of (word,tag) tuples :rtype: list(list(tuple(str, str))) r;r:crr )r rr rrrbrrr r r0_rz0MTECorpusReader.tagged_sents..rNrrr rr rbRrzMTECorpusReader.tagged_sentscr)am :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a list of tuples of the word and the corresponding lemma (word, lemma) :rtype: list(List(List(tuple(str, str)))) crr )r rr rrrcrrTr r r0rrz/MTECorpusReader.lemma_paras..rr~r rTr rcirzMTECorpusReader.lemma_parascr)a :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a list of (word,tag) tuples :rtype: list(list(list(tuple(str, str)))) r;r:crr )r rr rrrdrrr r r0rz0MTECorpusReader.tagged_paras..rNrrr rr rdxs  zMTECorpusReader.tagged_paras)NNrwr)Nr:r9)rrrrrrrUrXrZr[r_rarbrcrdr r r r rvs       rv)rrrD functoolsrZnltk.corpus.readerrrZnltk.corpus.reader.xmldocsrr rr rBrvr r r r s   %