o
rZh @ sZ d dl mZ d dlmZ d dlmZmZ ejZeedddddZ de _
d d
Z d d Z d
S ) )partial)chain)
everygrams pad_sequenceTzz)pad_leftZleft_pad_symbol pad_rightZright_pad_symbolzPads both ends of a sentence to length specified by ngram order.
Following convention pads the start of sentence pads its end.
c C s t tt|| d| dS )zpHelper with some useful defaults.
Applies pad_both_ends to sentence and follows it up with everygrams.
nmax_len)r list
pad_both_ends)orderZsentence r D/var/www/auris/lib/python3.10/site-packages/nltk/lm/preprocessing.pypadded_everygrams s r c s. t t d fdd|D tt|fS )a Default preprocessing for a sequence of sentences.
Creates two iterators:
- sentences padded and turned into sequences of `nltk.util.everygrams`
- sentences padded as above and chained together for a flat stream of words
:param order: Largest ngram length produced by `everygrams`.
:param text: Text to iterate over. Expected to be an iterable of sentences.
:type text: Iterable[Iterable[str]]
:return: iterator over text as ngrams, iterator over text as vocabulary data
r c 3 s$ | ]
}t t| d V qdS )r
N)r r ).0sentr Z
padding_fnr r 1 s " z,padded_everygram_pipeline..)r r
flattenmap)r textr r r padded_everygram_pipeline" s
r N)
functoolsr itertoolsr Z nltk.utilr r
from_iterabler r
__doc__r r r r r r s