o rZh!@sdZddlZzddlmZddlmZWney)ddZddZd d ZYnwe d Z Gd d d Z ddZ ddZ e fddZe fddZddZddZdS)z A port of the Gale-Church Aligner. Gale & Church (1993), A Program for Aligning Sentences in Bilingual Corpora. https://aclweb.org/anthology/J93-1004.pdf N)logsf)normcCst|}ddd|}|t| |d|d|d|d|d|d|d |d |d |d }|d krB|Sd|S)zComplementary error function.?gś??g5?g`yg?gƸ?gꪂIǿg#v?g9)gS?gޅ1Ogv(?gg@)absmathexp)xztrr I/var/www/auris/lib/python3.10/site-packages/nltk/translate/gale_church.pyerfccsLrcCsddt|tdS)u>Return the area under the normal distribution from M{-∞..x}.rr)rrsqrtr r r rnorm_cdf@srcCs0z tdt|WStytdYSw)Nr-inf)rlogr ValueErrorfloatrr r r norm_logsfDs   rrc@s&eZdZdddddddZdZdZdS) LanguageIndependentgׁsF?g{Gz?gbX9ȶ?gI +?))rr)rr)rr)rr)rr)rrrg333333@N)__name__ __module__ __qualname__PRIORSAVERAGE_CHARACTERSVARIANCE_CHARACTERSr r r rrNs rc Csg}t|t|f}|dkrotdd|Droz||\}}Wnty4|dd|ddf}Yq wt|D]}t|D]}||d|d|d|dfq?q9|d||d|f}|dkrotdd|Ds|dddS)a Traverse the alignment cost from the tracebacks and retrieves appropriate sentence pairs. :param backlinks: A dictionary where the key is the alignment points and value is the cost (referencing the LanguageIndependent.PRIORS) :type backlinks: dict :param source_sents_lens: A list of target sentences' lengths :type source_sents_lens: list(int) :param target_sents_lens: A list of target sentences' lengths :type target_sents_lens: list(int) )rrcss|]}|dkVqdS)rNr ).0pr r r nztrace..rrN)lenall TypeErrorrangeappend) backlinkssource_sents_lenstarget_sents_lenslinkspositionsr ijr r rtrace`s     ( r2c stfddt|dD}tfddt|dD}z|||jd}||j|t||j} Wn tyGtdYSwtt t | t |j | S)aPReturns the log probability of the two sentences C{source_sents[i]}, C{target_sents[j]} being aligned with a specific C{alignment}. @param i: The offset of the source sentence. @param j: The offset of the target sentence. @param source_sents: The list of source sentence lengths. @param target_sents: The list of target sentence lengths. @param alignment: The alignment type, a tuple of two integers. @param params: The sentence alignment parameters. @returns: The log probability of a specific alignment between the two sentences, given the parameters. c3 |] }|dVqdSrNr r offset)r0 source_sentsr rr"z!align_log_prob..rc3r3r4r r5)r1 target_sentsr rr"r8rrr) sumr(rrrrZeroDivisionErrorrLOG2rrrr) r0r1r7r9Z alignmentparamsZl_sZl_tmdeltar )r0r1r7r9ralign_log_prob|s   "r@c Cst|j}gg}i}tt|dD]l}tt|dD]Q}td}d} |D]1} d| d} || d} | t| ksA| dkrBq(|| | t||||| |} | |krY| }| } q(|tdkrbd}| |||f<|d|qt|dkr{|d|gqt |||S)aReturn the sentence alignment of two text blocks (usually paragraphs). >>> align_blocks([5,5,5], [7,7,7]) [(0, 0), (1, 1), (2, 2)] >>> align_blocks([10,5,5], [12,20]) [(0, 0), (1, 1), (2, 1)] >>> align_blocks([12,20], [10,5,5]) [(0, 0), (1, 1), (1, 2)] >>> align_blocks([10,2,10,10,2,10], [12,3,20,3,12]) [(0, 0), (1, 1), (2, 2), (3, 2), (4, 3), (5, 4)] @param source_sents_lens: The list of source sentence lengths. @param target_sents_lens: The list of target sentence lengths. @param params: the sentence alignment parameters. @return: The sentence alignments, a list of index pairs. rinfNr$rr) listrkeysr(r%rr@r)popr2)r+r,r=Zalignment_typesDr*r0r1Zmin_distZ min_alignaZprev_iZprev_jr!r r r align_blockss6          rGcs0t|t|kr tdfddt||DS)aCreates the sentence alignment of two texts. Texts can consist of several blocks. Block boundaries cannot be crossed by sentence alignment links. Each block consists of a list that contains the lengths (in characters) of the sentences in this block. @param source_blocks: The list of blocks in the source text. @param target_blocks: The list of blocks in the target text. @param params: the sentence alignment parameters. @returns: A list of sentence alignment lists z>Source and target texts do not have the same number of blocks.csg|] \}}t||qSr )rG)r Z source_blockZ target_blockr=r r s zalign_texts..)r%rzip)Z source_blocksZ target_blocksr=r rHr align_textss rKc#s"fdd} |Vq )zSplits an iterator C{it} at values of C{split_value}. Each instance of C{split_value} is swallowed. The iterator produces subiterators which need to be consumed fully before the next subiterator can be used. c3s,|}|kr|V}|ksdSdSNnext)firstvit split_valuer r_chunk_iterators z!split_at.._chunk_iteratorrM)rRrSrTr rQrsplit_ats rUcsfddt||DS)zParses a stream of tokens and splits it into sentences (using C{soft_delimiter} tokens) and blocks (using C{hard_delimiter} tokens) for use with the L{align_texts} function. cs g|] }ddt|DqS)cSsg|] }tdd|DqS)css|]}t|VqdSrL)r%)r tokenr r rr"r#z;parse_token_stream....)r:)r Z sentence_itr r rrIsz1parse_token_stream...rU)r Zblock_itsoft_delimiterr rrIs z&parse_token_stream..rW)streamrYZhard_delimiterr rXrparse_token_streams r[)__doc__rrrrZ scipy.stats ImportErrorrrrr<rr2r@rGrKrUr[r r r rs$   '  4  6