/h/lSrSSKrSSKJr SSKJrJrJrJr SSK J r "SS\5r "SS \ 5r g) a Lexical translation model that considers word order. IBM Model 2 improves on Model 1 by accounting for word order. An alignment probability is introduced, a(i | j,l,m), which predicts a source word position, given its aligned target word's position. The EM algorithm used in Model 2 is: :E step: In the training data, collect counts, weighted by prior probabilities. - (a) count how many times a source language word is translated into a target language word - (b) count how many times a particular position in the source sentence is aligned to a particular position in the target sentence :M step: Estimate new probabilities based on the counts from the E step Notations --------- :i: Position in the source sentence Valid values are 0 (for NULL), 1, 2, ..., length of source sentence :j: Position in the target sentence Valid values are 1, 2, ..., length of target sentence :l: Number of words in the source sentence, excluding NULL :m: Number of words in the target sentence :s: A word in the source language :t: A word in the target language References ---------- Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York. Peter E Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19 (2), 263-311. N defaultdict) AlignedSent AlignmentIBMModel IBMModel1)Countsc`^\rSrSrSrS U4SjjrSrSrSrSr Sr S r S r S r S rU=r$) IBMModel2;u Lexical translation model that considers word order >>> bitext = [] >>> bitext.append(AlignedSent(['klein', 'ist', 'das', 'haus'], ['the', 'house', 'is', 'small'])) >>> bitext.append(AlignedSent(['das', 'haus', 'ist', 'ja', 'groß'], ['the', 'house', 'is', 'big'])) >>> bitext.append(AlignedSent(['das', 'buch', 'ist', 'ja', 'klein'], ['the', 'book', 'is', 'small'])) >>> bitext.append(AlignedSent(['das', 'haus'], ['the', 'house'])) >>> bitext.append(AlignedSent(['das', 'buch'], ['the', 'book'])) >>> bitext.append(AlignedSent(['ein', 'buch'], ['a', 'book'])) >>> ibm2 = IBMModel2(bitext, 5) >>> print(round(ibm2.translation_table['buch']['book'], 3)) 1.0 >>> print(round(ibm2.translation_table['das']['book'], 3)) 0.0 >>> print(round(ibm2.translation_table['buch'][None], 3)) 0.0 >>> print(round(ibm2.translation_table['ja'][None], 3)) 0.0 >>> print(round(ibm2.alignment_table[1][1][2][2], 3)) 0.939 >>> print(round(ibm2.alignment_table[1][2][2][2], 3)) 0.0 >>> print(round(ibm2.alignment_table[2][2][4][5], 3)) 1.0 >>> test_sentence = bitext[2] >>> test_sentence.words ['das', 'buch', 'ist', 'ja', 'klein'] >>> test_sentence.mots ['the', 'book', 'is', 'small'] >>> test_sentence.alignment Alignment([(0, 0), (1, 1), (2, 2), (3, 2), (4, 3)]) c >[TU]U5 Uc2[USU-5nURUlUR U5 OUSUlUSUl[ SU5HnURU5 M URU5 g)a Train on ``sentence_aligned_corpus`` and create a lexical translation model and an alignment model. Translation direction is from ``AlignedSent.mots`` to ``AlignedSent.words``. :param sentence_aligned_corpus: Sentence-aligned parallel corpus :type sentence_aligned_corpus: list(AlignedSent) :param iterations: Number of iterations to run training algorithm :type iterations: int :param probability_tables: Optional. Use this to pass in custom probability values. If not specified, probabilities will be set to a uniform distribution, or some other sensible value. If specified, all the following entries must be present: ``translation_table``, ``alignment_table``. See ``IBMModel`` for the type and purpose of these tables. :type probability_tables: dict[str]: object Ntranslation_tablealignment_tabler) super__init__rrset_uniform_probabilitiesrrangetrain align_all)selfsentence_aligned_corpus iterationsprobability_tablesibm1n __class__s K/var/www/auris/envauris/lib/python3.13/site-packages/nltk/translate/ibm2.pyrIBMModel2.__init__cs, 01  %4a*nED%)%;%;D "  * *+B C&88K%LD "#56G#HD q*%A JJ. /& ./c[5nUHn[UR5n[UR5nXE4U;dM6UR XE45 SUS-- nU[ R :a%[R"S[U5-S-5 [SUS-5H0n[SUS-5HnX`RUUUU'M M2 M g)NzA source sentence is too long (z& words). Results may be less accurate.r) setlenmotswordsaddrMIN_PROBwarningswarnstrrr) rrl_m_combinationsaligned_sentencelm initial_probijs rr#IBMModel2.set_uniform_probabilitiess5 7 $))*A$**+Av-- $$aV, AE{ ("3"33MM9a&!BC q!a%A"1a!e_;G,,Q/215a8-)!8r c [5nUHnS/UR-nS/UR-n[UR5n[UR5nUR XE5n[ SUS-5H_n XYn [ SUS-5HEn XKn UR XXE5n XU - nURXU 5 URXXU5 MG Ma M URU5 URU5 g)NUNUSEDr"r) Model2Countsr%r&r$prob_all_alignmentsrprob_alignment_pointupdate_lexical_translationupdate_alignment*maximize_lexical_translation_probabilities maximize_alignment_probabilities)rparallel_corpuscountsr- src_sentence trg_sentencer.r/ total_countr2tr1scountnormalized_counts rrIBMModel2.trains /  6$4$9$99L$:(8(>(>>L$))*A$**+A22<NK1a!e_ Oq!a%A$A 55aLWE',1~'=$556F1M++,<qI )%!0* 77? --f5r c~[RnURR5Hup4UR5HvupVUR5H]upxUHRn URUUUU URUUU - n [ X5UR UUUU 'MT M_ Mx M gN)rr( alignmentitemsalignment_for_any_imaxr) rr>r(r1j_sr2src_sentence_lengthsr.trg_sentence_lengthsr/estimates rr<*IBMModel2.maximize_alignment_probabilitiess$$&&,,.FA+.99;'/C/I/I/K+A1",,Q/215a8$88;A>qAB!**1q51!a%8;A>?* *I+,( =  ! !1"; <#:&#,N"; r )rrrH)__name__ __module__ __qualname____firstlineno____doc__rrrr<r7r8r^rra__static_attributes__ __classcell__rs@rr r ;s?%N'0RH(64 S$. O,*&&<&[TU]5 [S5Ul[S5Ulg)Nc[S5$)Nc[S5$)Nc [[5$rHrrSr rKModel2Counts.__init__......3s K...3s 4N(Or rryr rrz'Model2Counts.__init__..3s K OPr c[S5$)Nc [[5$rHrxryr rrzr|6s E(:r rryr rrzr}6s K :;r )rrrrIrK)rrs rrModel2Counts.__init__0s/ $ P $/ ;$  r cfURUU==U- ss'URU==U- ss'grH) t_given_s any_t_given_s)rrDrCrBs rr9'Model2Counts.update_lexical_translation9s1 q!% 1&r c~URUUUU==U- ss'URUUU==U- ss'grHrIrK)rrDr1r2r.r/s rr:Model2Counts.update_alignment=sE q!Q"e+"   #A&q)U2)r r) rkrlrmrnrorr9r:rprqrrs@rr6r6*s  '33r r6) ror) collectionsrnltk.translaterrrrnltk.translate.ibm_modelr r r6ryr rrs7*X#FF+l<l<^363r