o rZh @sdZddlmZddlmZmZmZddlmZm Z ddl m Z m Z ddl mZ ddd Zd d Zd d ZGdddZdddZdS)z Utility functions for parsers. )load)CFGPCFGFeatureGrammar)Chart ChartParser) FeatureChartFeatureChartParser)InsideChartParserNcKst|fi|}t|tstdt|tr#|durt}||||dSt|tr;|dur.t}|dur4t}||||dS|durAt }|durGt }||||dS)a Load a grammar from a file, and build a parser based on that grammar. The parser depends on the grammar format, and might also depend on properties of the grammar itself. The following grammar formats are currently supported: - ``'cfg'`` (CFGs: ``CFG``) - ``'pcfg'`` (probabilistic CFGs: ``PCFG``) - ``'fcfg'`` (feature-based CFGs: ``FeatureGrammar``) :type grammar_url: str :param grammar_url: A URL specifying where the grammar is located. The default protocol is ``"nltk:"``, which searches for the file in the the NLTK data package. :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; and higher numbers will produce more verbose tracing output. :param parser: The class used for parsing; should be ``ChartParser`` or a subclass. If None, the class depends on the grammar format. :param chart_class: The class used for storing the chart; should be ``Chart`` or a subclass. Only used for CFGs and feature CFGs. If None, the chart class depends on the grammar format. :type beam_size: int :param beam_size: The maximum length for the parser's edge queue. Only used for probabilistic CFGs. :param load_args: Keyword parameters used when loading the grammar. See ``data.load`` for more information. z1The grammar must be a CFG, or a subclass thereof.N)trace beam_size)r chart_class) r isinstancer ValueErrorrr rr rrr)Z grammar_urlr parserr r Z load_argsgrammarr>/var/www/auris/lib/python3.10/site-packages/nltk/parse/util.py load_parsers$"   rc csPt|ddD]\}\}}t||d||dddddg }d|d}|VqdS) a A module to convert a single POS tagged sentence into CONLL format. >>> from nltk import word_tokenize, pos_tag >>> text = "This is a foobar sentence." >>> for line in taggedsent_to_conll(pos_tag(word_tokenize(text))): # doctest: +NORMALIZE_WHITESPACE ... print(line, end="") 1 This _ DT DT _ 0 a _ _ 2 is _ VBZ VBZ _ 0 a _ _ 3 a _ DT DT _ 0 a _ _ 4 foobar _ JJ JJ _ 0 a _ _ 5 sentence _ NN NN _ 0 a _ _ 6 . _ . . _ 0 a _ _ :param sentence: A single input sentence to parse :type sentence: list(tuple(str, str)) :rtype: iter(str) :return: a generator yielding a single sentence in CONLL format. )start_0a  N) enumeratestrjoin)sentenceiwordtagZ input_strrrrtaggedsent_to_conllOs r#ccs$|D] }t|EdHdVqdS)aV A module to convert the a POS tagged document stream (i.e. list of list of tuples, a list of sentences) and yield lines in CONLL format. This module yields one line per word and two newlines for end of sentence. >>> from nltk import word_tokenize, sent_tokenize, pos_tag >>> text = "This is a foobar sentence. Is that right?" >>> sentences = [pos_tag(word_tokenize(sent)) for sent in sent_tokenize(text)] >>> for line in taggedsents_to_conll(sentences): # doctest: +NORMALIZE_WHITESPACE ... if line: ... print(line, end="") 1 This _ DT DT _ 0 a _ _ 2 is _ VBZ VBZ _ 0 a _ _ 3 a _ DT DT _ 0 a _ _ 4 foobar _ JJ JJ _ 0 a _ _ 5 sentence _ NN NN _ 0 a _ _ 6 . _ . . _ 0 a _ _ 1 Is _ VBZ VBZ _ 0 a _ _ 2 that _ IN IN _ 0 a _ _ 3 right _ NN NN _ 0 a _ _ 4 ? _ . . _ 0 a _ _ :param sentences: Input sentences to parse :type sentence: list(list(tuple(str, str))) :rtype: iter(str) :return: a generator yielding sentences in CONLL format. Nz )r#) sentencesrrrrtaggedsents_to_conllis !r%c@s$eZdZdZdddZd ddZdS) TestGrammarz Unit tests for CFG. NcCs*||_t|dd|_||_||_||_dS)Nr)r )Z test_grammarrcpsuite_acceptZ_reject)selfrr(acceptrejectrrr__init__s  zTestGrammar.__init__Fc Cs|jD][}t|dddddD]D}||D]=}|}t|j|}|r9|r9tt||D]}t|q2|dkrJ|gkrGtd|d}q|rRtd |d} qq|r^| r^td qd S) a} Sentences in the test suite are divided into two classes: - grammatical (``accept``) and - ungrammatical (``reject``). If a sentence should parse according to the grammar, the value of ``trees`` will be a non-empty list. If a sentence should be rejected according to the grammar, then the value of ``trees`` will be None. doc: )end)r+r,r+zSentence '%s' failed to parse'TzSentence '%s' received a parse'zAll tests passed!N)r(printsplitlistr'parser) r*Z show_treestestkeysenttokensZtreestreeacceptedZrejectedrrrruns.     zTestGrammar.run)NN)F)__name__ __module__ __qualname____doc__r-r<rrrrr&s r&#%;cCs|dur ||}g}|dD]F}|dks|d|vrq|dd}d}t|dkrF|ddvr<|dd v}|d}n t|d}|d}|}|gkrOq|||fg7}q|S) a Parses a string with one test sentence per line. Lines can optionally begin with: - a bool, saying if the sentence is grammatical or not, or - an int, giving the number of parse trees is should have, The result information is followed by a colon, and then the sentence. Empty lines and lines beginning with a comment char are ignored. :return: a list of tuple of sentences and expected results, where a sentence is a list of str, and a result is None, or bool, or int :param comment_chars: ``str`` of possible comment characters. :param encoding: the encoding of the string, if it is binary Nrrr/r)TruetrueFalsefalse)rDrE)decoder3lenint)stringZ comment_charsencodingr$rZ split_inforesultr9rrrextract_test_sentencess&       rN)rNNr)rAN)r@Z nltk.datarZ nltk.grammarrrrZnltk.parse.chartrrZnltk.parse.featurechartrr Znltk.parse.pchartr rr#r%r&rNrrrrs   9+1