fTh8SrSSKrSSKrSSKJrJrJrJr SSKr SSK J r SSK J r JrJrJrJr SSKJrJrJrJrJr SSKJr \"5(aSSKr\R6"\5rS S 0rS rS r \"S S9"SS\55r!S/r"g)z!Tokenization class for Pop2Piano.N)ListOptionalTupleUnion) BatchFeature) AddedToken BatchEncodingPaddingStrategyPreTrainedTokenizerTruncationStrategy) TensorTypeis_pretty_midi_availableloggingrequires_backendsto_numpy)requiresvocabz vocab.jsonc*X - nUb [X!5nU$N)minnumbercutoff_time_idx current_idxs l/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/pop2piano/tokenization_pop2piano.pytoken_time_to_noter(s K"+7 ctX0b.X0nXd:a#UnURXgX/5 US:XaSOUnXU'U$XCU'U$)Nr)append) rcurrent_velocitydefault_velocitynote_onsets_readyrnotes onset_idx offset_idx onsets_readys rtoken_note_to_noter(0s[ ,%-  "$J LL)J K#3q#84kL(4f % L%0&! Lr) pretty_miditorch)backendscV^\rSrSrSrSS/r\rS*U4Sjjr\ S5r Sr S\ S \ 4S jrS+S \ 4S jjrS \R"S \ S\ S\ 4SjrS,S \R"S\R"S \ S\ S\ 4 SjjrS-S \R"S\S\\4SjjrS.S\R"S\R"S\ 4SjjrS-S\S\\S \\4SjjrS/S\\R"\\R<4S\\S\\ S \ 4Sjjr!S/S\\R"\\R<4S\\S\\ S \ 4Sjjr"S0S\\R"\\R<\\\R<4S\\#\\$4S \\#\\4S\\ S!\\ S"\\#S#\\\\%4S$\#S \ 4S%jjr&S1S&\'S'\#4S(jjr(S)r)U=r*$)2Pop2PianoTokenizer?aF Constructs a Pop2Piano tokenizer. This tokenizer does not require training. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab (`str`): Path to the vocab file which contains the vocabulary. default_velocity (`int`, *optional*, defaults to 77): Determines the default velocity to be used while creating midi Notes. num_bars (`int`, *optional*, defaults to 2): Determines cutoff_time_idx in for each token. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `"-1"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to 1): The end of sequence token. pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to 0): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to 2): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. token_idsattention_maskc N>[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUnX lX0l[ US5n [ R"U 5UlSSS5 URR5V V s0sHupX_M sn n Ul [T U]0"SUUUUS.UD6 g!,(df  NX=fs sn n f)NF)lstriprstriprb) unk_token eos_token pad_token bos_token) isinstancestrr r"num_barsopenjsonloadencoderitemsdecodersuper__init__) selfrr"r<r5r6r7r8kwargsfilekv __class__s rrDPop2PianoTokenizer.__init__]sJTT]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir 0 % $99T?DL*.););)=>)=)=>        ?s"D#D! Dc,[UR5$)z-Returns the vocabulary size of the tokenizer.)lenr@rEs r vocab_sizePop2PianoTokenizer.vocab_sizes4<<  rcB[UR40URD6$)z(Returns the vocabulary of the tokenizer.)dictr@added_tokens_encoderrNs r get_vocabPop2PianoTokenizer.get_vocabsDLL>D$=$=>>rtoken_idreturncURRXRS35nURS5nSR USS5[ US5pCX4/$)a Decodes the token ids generated by the transformer into notes. Args: token_id (`int`): This denotes the ids generated by the transformers to be converted to Midi tokens. Returns: `List`: A list consists of token_type (`str`) and value (`int`). _TOKEN_TIME_Nr)rBgetr5splitjoinint)rErVtoken_type_value token_typevalues r_convert_id_to_token'Pop2PianoTokenizer._convert_id_to_tokensc <<++H8H 6TU+11#6HH%5ab%9:C@PQR@S[R"XZ4SS9nMV Uc/$U$)a Converts relative tokens to notes which are then used to generate pretty midi object. Args: tokens (`numpy.ndarray`): Tokens to be converted to notes. beat_offset_idx (`int`): Denotes beat offset index for each note in generated Midi. bars_per_batch (`int`): A parameter to control the Midi output generation. cutoff_time_idx (`int`): Denotes the cutoff time index for each note in generated Midi. N) start_idxrr)axis)rangerMrelative_tokens_ids_to_notesnp concatenate) rErirjrkrr$index_tokens _start_idx_cutoff_time_idx_notess r"relative_batch_tokens_ids_to_notes5Pop2PianoTokenizer.relative_batch_tokens_ids_to_notess*3v;'EmG(>+AA+EEJ.; 66$ 07F 6{aQ?(" =I rbeatstepc\UcSOUnURUUUUS9nURXbX#S9nU$)a Converts tokens to Midi. This method calls `relative_batch_tokens_ids_to_notes` method to convert batch tokens to notes then uses `notes_to_midi` method to convert them to Midi. Args: tokens (`numpy.ndarray`): Denotes tokens which alongside beatstep will be converted to Midi. beatstep (`np.ndarray`): We get beatstep from feature extractor which is also used to get Midi. beat_offset_idx (`int`, *optional*, defaults to 0): Denotes beat offset index for each note in generated Midi. bars_per_batch (`int`, *optional*, defaults to 2): A parameter to control the Midi output generation. cutoff_time_idx (`int`, *optional*, defaults to 12): Denotes the cutoff time index for each note in generated Midi. r)rirjrkr) offset_sec)ry notes_to_midi)rErir{rjrkrr$midis r!relative_batch_tokens_ids_to_midi4Pop2PianoTokenizer.relative_batch_tokens_ids_to_midisP0 /6!O77+)+ 8  !!%h>W!X rrnc @UVs/sHo@RU5PM nnUnSn[[URR 5Vs/sHoR S5PM sn5S-5V s/sHn SPM n n /n UHZupU S:Xa U S:Xa OKMU S:Xa [ XUS9nM'U S:XaU nM1U S :Xa[U UURU UU S 9n MQ[S 5e [U 5HKupUcM UcUS-nO[X?S-5n[UU5nU RUUXR/5 MM [U 5S:Xa/$[R"U 5n U SS2S4S -U SS2S4-nU UR!5n U $s snfs snfs sn f) an Converts relative tokens to notes which will then be used to create Pretty Midi objects. Args: tokens (`numpy.ndarray`): Relative Tokens which will be converted to notes. start_idx (`float`): A parameter which denotes the starting index. cutoff_time_idx (`float`, *optional*): A parameter used while converting tokens to notes. rNOTEr[N TOKEN_SPECIAL TOKEN_TIMErTOKEN_VELOCITY TOKEN_NOTE)rr!r"r#rr$zToken type not understood!)rcrpsumr@keysendswithrr(r" ValueError enumeratemaxr rMrrarrayargsort)rErirnrrfwordsrr!rHir#r$rarpitch note_onsetcutoffr& note_orders rrq/Pop2PianoTokenizer.relative_tokens_ids_to_notess@FFve**51vF +0RVR^R^RcRcRe5fReQjj6HRe5f1gjk1k+lm+laT+lm"' J_,Q;|+0!P[ //#) |+*!%5%)%:%:&7 + !!=>>+#(."++ u:?IHHUOEq!ts*U1a4[8J*,,./EL]G6gmsF F 5 Fr$r}cT[US/5 [R"SSS9n[R"SS9n/nUH8upxp[R"U U X'U- X(U- S9n UR U 5 M: XelURR U5 UR5 U$)as Converts notes to Midi. Args: notes (`numpy.ndarray`): This is used to create Pretty Midi objects. beatstep (`numpy.ndarray`): This is the extrapolated beatstep that we get from feature extractor. offset_sec (`int`, *optional*, defaults to 0.0): This represents the offset seconds which is used while creating each Pretty Midi Note. r)ig^@) resolution initial_tempor)program)velocityrstartend) rr) PrettyMIDI InstrumentNoter r$ instrumentsremove_invalid_notes) rEr$r{r}new_pmnew_inst new_notesr%r&rrnew_notes rr~ Pop2PianoTokenizer.notes_to_midi8s $0''3eL))!4 6; 2I5"''!)J6(:5 H   X &7<#!!(+##% rsave_directoryfilename_prefixc[RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[US5nUR[R"UR55 SSS5 U4$!,(df  U4$=f)aE Saves the tokenizer's vocabulary dictionary to the provided save_directory. Args: save_directory (`str`): A path to the directory where to saved. It will be created if it doesn't exist. filename_prefix (`Optional[str]`, *optional*): A prefix to add to the names of the files saved by the tokenizer. zVocabulary path (z) should be a directoryN-rw) ospathisdirloggererrorr^VOCAB_FILES_NAMESr=writer>dumpsr@)rErrout_vocab_filerGs rsave_vocabulary"Pop2PianoTokenizer.save_vocabularyXsww}}^,, LL,^,<= 7.5` (Volta). return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~file_utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. Returns: `BatchEncoding` containing the token_ids. rr)rrrrrT)r$rr)rrrrrrr9) r:rrndarrayndimlist"_get_padding_truncation_strategiesrrpad)rEr$rrrrrrrrF is_batchedpadding_strategyrr/s r__call__Pop2PianoTokenizer.__call__sd)35"**(E(EUZZ1_:V[\]V^`dKe EIDkDkE !!1 E  E Az ,A,IDOd !..$7% I(($7% IHH $!1"7) rfeature_extractor_output return_midic [[US5=(a [US5=(a [US55nU(d!USRSS:a [S5eU(a[ USSS2S4S:H5USRS:wd&USRSUS RS:waA[S URSS USRSS US RS35eUSRSURS:wa.[S USRSSURS35eO^USRSS:wdUS RSS:wa2[SUSRSSUS RSS35eU(a'[ R "USSS2S4S:H5SnOURS/n/n/nSn[U5GHupXU n U SS2S[ R"[ R "U [UR5:H5S5S-24n USU n US U n U(a|USU nUSU nU S[ R"[ R "US:H5S5S-n U S[ R"[ R "US:H5S5S-n [U 5n [U 5n [U 5n URU U URURS-S-S9nURSRHDnU=R U S- slU=R"U S- slUR%U5 MF UR%U5 XS-- nGM U(a ['XgS.5$['SU05$)a This is the `batch_decode` method for `Pop2PianoTokenizer`. It converts the token_ids generated by the transformer to midi_notes and returns them. Args: token_ids (`Union[np.ndarray, torch.Tensor, tf.Tensor]`): Output token_ids of `Pop2PianoConditionalGeneration` model. feature_extractor_output (`BatchFeature`): Denotes the output of `Pop2PianoFeatureExtractor.__call__`. It must contain `"beatstep"` and `"extrapolated_beatstep"`. Also `"attention_mask_beatsteps"` and `"attention_mask_extrapolated_beatstep"` should be present if they were returned by the feature extractor. return_midi (`bool`, *optional*, defaults to `True`): Whether to return midi object or not. Returns: If `return_midi` is True: - `BatchEncoding` containing both `notes` and `pretty_midi.pretty_midi.PrettyMIDI` objects. If `return_midi` is False: - `BatchEncoding` containing `notes`. r0attention_mask_beatsteps$attention_mask_extrapolated_beatstep beatstepsrr[zattention_mask, attention_mask_beatsteps and attention_mask_extrapolated_beatstep must be present for batched inputs! But one of them were not present.Nextrapolated_beatstepzbLength mistamtch between token_ids, beatsteps and extrapolated_beatstep! Found token_ids length - z, beatsteps shape - z$ and extrapolated_beatsteps shape - z!Found attention_mask of length - z but token_ids of length - zLength mistamtch of beatsteps and extrapolated_beatstep! Since attention_mask is not present the number of examples must be 1, But found beatsteps length - z", extrapolated_beatsteps length - .rm)rir{rkr)r$pretty_midi_objectsr$)boolhasattrshaperrrrwhererrr_r6rrr<rr$rrr r )rEr/rrattention_masks_present batch_idx notes_listpretty_midi_objects_listrnrtend_idxeach_tokens_idsrrrrpretty_midi_objectnotes r batch_decodePop2PianoTokenizer.batch_decodeYs8#' ,.> ? Z02LM Z02XY# '+CK+P+V+VWX+Y\]+]H  #,-=>q!tDIJ+K8>>qAB+K8>>qA+,CDJJ1MN!**3//!*<)==QRjkvRwR}R}~SARBB::RSj:k:q:qrs:t9uw ((89??BiooVWFXX 78PQa8b8h8hij8k7lmHIRIXIXYZI[H\]Y)5;;A>!C+,CDJJ1MQRR 44L[4Y4_4_`a4b3cdFG_`wGxG~G~@GAFBBCD #!9:J!KAqD!QUV!VWXYZI"+,I #%  ' 2NE'':O-a1r266"((?VYZ^ZhZhViCi:jkl:m3nqr3r1r.rsO0=eDI$<=T$UV[$\ !'+CD^+_`e+f(7O:884&&^rxx8PTU8U/VWX/Y(Z]^(^_ (=XbffRXX&Ja&OPQRSTWXX)%'7O +I$,-B$C !!%!G!G&.#}}!%!2a 7 "H" +66q9?? il* IaL(!!$'@ % + +,> ? 1 $IG3J  :!gh hgz233r)rBr"r@r<)Mrz-1102)r)rr r)g)NN)FNNNNNT)T)+__name__ __module__ __qualname____firstlineno____doc__model_input_namesrvocab_files_namesrDpropertyrOrTr_rrcrgrrrryrfloatrrqr~r;rrrrr)rr r rrrr rrrr__static_attributes__ __classcell__)rJs@rr-r-?s$2%&67)   D!!?#S#T#$NcN * ** *  *` !!   **      JX\<jj<-2<EMe_<|2::QT@!c!HSM!]bcf]g!4=A$( E4RZZk&6&6!778E4&&89E4SM E4  E4T=A$( %ERZZk&6&6!778%E&&89%ESM %E  %E\6;;?$(,004;?z JJ !! " k&&' ( * ztS/12z$%778zSMz%SMz (~z!sJ!78zz z@! w4#/w4 w4w4rr-)#rr>rtypingrrrrnumpyrrfeature_extraction_utilsrtokenization_utilsr r r r r utilsrrrrrutils.import_utilsrr) get_loggerrrrrr(r-__all__r9rrrs( //4uu__*   H % \   +,P 4,P 4-P 4f r