fTh|
S r SSKrSSKrSSKrSSKJrJrJrJr SSK J
r
SSK J r J
r
SSKJrJrJrJrJr SSKJr SS KJrJr \\\\\4 \\\\\\4 \\\\\4 \\\\\\4 4 r " S
S \S S
9r " S S\S S
9r " S S\S S
9r " S S\5 r S\\\\\4 S\S\\\4 4S jr S\S\S\4S jr!S r"S r#S r$S S jr%S/r&g) z Processor class for KOSMOS-2. N)ListOptionalTupleUnion ) BatchFeature)
ImageInput
is_batched) ImagesKwargsProcessingKwargsProcessorMixin
TextKwargsUnpack)
AddedToken)
BatchEncoding TextInputc L \ rS rSr% \\\ \S' \\ \S' \\ \S' Sr g)Kosmos2ImagesKwargs% bboxesnum_image_tokensfirst_image_token_id N)
__name__
__module__ __qualname____firstlineno__r r float__annotations__int__static_attributes__r f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/kosmos2/processing_kosmos2.pyr r % s% T%[ !! sm#"3-'r" r F)totalc & \ rS rSr% \\ \S' Srg)Kosmos2TextKwargs+
add_eos_tokenr N)r r r r r boolr r! r r" r# r& r& + s D>!r" r& c
L \ rS rSr% \\S' \\S' SSSSSSSSSS. SS 0S
.rS rg )
Kosmos2ProcessorKwargs/ text_kwargs
images_kwargsTFr ) add_special_tokenspaddingstridereturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mappingreturn_token_type_idsverboser( r @ )r- r. r N) r r r r r& r r _defaultsr! r r" r# r+ r+ / sC ""&& #' ).*/&+%* "
Ir" r+ c ^ \ rS rSrSrSS/rS/rSrSrS U 4S jjr S S \
S
\ \ \
\ 4 S \\ S \4S
jjrS rS r S S\ \ \
\ 4 S \
S\S\\ S \ \\
\ 4 4
S jjrS rS rS S jrS S jr \ S 5 r S
\S\ \
\ \ \
\ \ 4 S \4S jr!S\ \ \\4 \ \ \ \ \ 4 4 S \ \\4 4S jr"S r#U =r$$ )!Kosmos2ProcessorD a
Constructs an KOSMOS-2 processor which wraps a KOSMOS-2 image processor and a KOSMOS-2 tokenizer into a single
processor.
[`Kosmos2Processor`] offers all the functionalities of [`CLIPImageProcessor`] and some functionalities of
[`XLMRobertaTokenizerFast`]. See the docstring of [`~Kosmos2Processor.__call__`] and [`~Kosmos2Processor.decode`]
for more information.
Args:
image_processor (`CLIPImageProcessor`):
An instance of [`CLIPImageProcessor`]. The image processor is a required input.
tokenizer (`XLMRobertaTokenizerFast`):
An instance of ['XLMRobertaTokenizerFast`]. The tokenizer is a required input.
num_patch_index_tokens (`int`, *optional*, defaults to 1024):
The number of tokens that represent patch indices.
image_processor tokenizernum_patch_index_tokens)CLIPImageProcessorCLIPImageProcessorFast
AutoTokenizerc > SUl SU l SU l SU l SU l SU l SU l SU l S U l S
U l S U l
S U l U R U R U R U R U R
U R U R U R U R U R U R / U l X0l
[ U R 5 Vs/ s H! nS
[ U5 R! S5 S3PM# nn/ nU R U- H nUR# [% USSSS95 M UR' U5 [( T U ]U X5 g s snf )NFzzzzzzz z z T)lstriprstrip
normalized)r5 eod_token boi_token eoi_token eoc_token eol_token bop_token eop_token boo_token eoo_token dom_token grd_token
tag_tokensr> rangestrzfillappendr
add_tokenssuper__init__)
selfr< r= r> kwargsxpatch_index_tokens
tokens_to_addtoken __class__s
r# r\ Kosmos2Processor.__init__[ sE */ '!"##"#$#$9&
NN NN NN NN NN NN NN NN NN NN NN
'=#JOPTPkPkJl mJlQ
c!fll1o->a @Jl m
__'99E E$uY^!_ ` : ]+
4 ns 3(E1imagestextr^ returnc
Uc Uc [ S5 eU R " [ 4SU R R 0UD6nUS R SS5 nUS R SS5 nUS R SS5 n US R S
S 5 n
US S n US S
n US R
SS5 n
[ 5 nUb' U R " U40 US D6nUR U5 Ub U R X!XxS9nU ( av U
( do [ U[ 5 ( a U R R U 3nO?[ U[ 5 ( a* U Vs/ s H nU R R U 3PM nnUS S =( a U
US S ' Uc U OS US S
' Uc U
OSUS S' U R " S SU0US D6nUR U5 XS S ' XS S
' XS S' UGbP UGbL U c U R R S- n U n[! U5 S- n[ [# XU- 5 5 nS/S/U- - S/- n/ n/ nUS n[ U[ 5 ( a U/nUS /US' U Hw nUSU U- UUU- S - nUR% U5 [& R& " U5 nU( a S/U- nUS/[) U5 [) U5 -
- -
nUR% U5 My [ U[ 5 ( Ga [+ [- WR. 5 VV s/ s H u nn U[) U 5 4PM sn nS S9n U S u n n U S u nn US S =( a U
US S ' SUS S' U R " S SUU /0US D6n[) UR. S 5 n U U :w GaI U R R0 S:X a U V s/ s H, n U U R R2 /U [) U 5 -
- - PM. nn U V s/ s H n U S/U [) U 5 -
- - PM nn US V s/ s H n U S/U [) U 5 -
- - PM sn US' OU R R0 S:X a U V s/ s H, n U R R2 /U [) U 5 -
- U - PM. nn U V s/ s H n S/U [) U 5 -
- U - PM nn US V s/ s H n S/U [) U 5 -
- U - PM sn US' [ U[ 5 ( a U
c US nUS S US' US nUR [5 UUS US.U
S95 U$ s snf s sn nf s sn f s sn f s sn f s sn f s sn f s sn f ) a
This method uses [`CLIPImageProcessor.__call__`] method to prepare image(s) for the model, and
[`XLMRobertaTokenizerFast.__call__`] to prepare text for the model.
Please refer to the docstring of the above two methods for more information.
The rest of this documentation shows the arguments specific to `Kosmos2Processor`.
Args:
bboxes (`Union[List[Tuple[int]], List[Tuple[float]], List[List[Tuple[int]]], List[List[Tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional* defaults to 64):
The number of (consecutive) places that are used to mark the placeholders to store image information.
This should be the same as `latent_query_num` in the instance of `Kosmos2Config` you are using.
first_image_token_id (`int`, *optional*):
The token id that will be used for the first place of the subsequence that is reserved to store image
information. If unset, will default to `self.tokenizer.unk_token_id + 1`.
add_eos_token (`bool`, defaults to `False`):
Whether or not to include `EOS` token id in the encoding when `add_special_tokens=True`.
Nz*You have to specify either images or text.tokenizer_init_kwargsr. r r r7 r r- r( Fr/ r0 return_tensors)r rf r input_idsattention_maskc U S $ Nr )r_ s r# +Kosmos2Processor.__call__.. s defhdir" )keyrp rightleft)rl rm image_embeds_position_mask)data tensor_typer )
ValueError
_merge_kwargsr+ r= init_kwargspop
setdefaultr r< updatepreprocess_examples
isinstancerW bos_tokenlist unk_token_idr rV rY copylensorted enumeraterl padding_side pad_token_idr )!r] re rf audiovideosr^
output_kwargsr r r r( r/ r0 rj encodingimage_encodings
text_encodingwith_bos start_indeximage_token_ids base_image_embeds_position_maskrl rv
all_input_idstext_idsmaskidxr_
sorted_length_min_len_not_paddedmax_len_paddeds! r# __call__Kosmos2Processor.__call__ sh 8 >dl IJ J ** "
"&.."<"<
/33HdC( 9==>PRTU ,_ = A ABXZ^ _ %m488 %P
*= 9:N O
.y9&}5@@AQSWX > ! 1 1& [M/D d++FJKdt~~ 7 7 8
BH gUZM- ( 3OU~^cgM- ()9 : NN U U
m8T UM OOM *=Om$%9:29m$Y/9Gm$%56 2#+'+~~'B'BQ'F$ *H h-!+K #5)=Vf?f#g hO/0cQC:J4J.JaS.P + I)+ & $[ 1M$$$!.
.67G.H-I)* )#L[1OCh{]mOmOoFpp *yy!@A 3:Ds8}s4y 899*11$7 * $%% &1:=;R;R1ST1SvsAc3q6]1STZi!
)6a(8%%&r*Q!-01EFX= m,-AB BF
m,-=> $ `T#YK `=Q^C_ `
!$]%<% ? ?Ic 3 6 JRRbIc6IcAA~A'> ? ?Ic6!12 44>lu$vlughdnn&A&A%BnWZ[\W]F]%^ab%blu $vIc6IcAQC>CF#: ;a ?Ic 3 6 JRRbIc6IcAQC>CF#: ;a ?Ic6!12
$$$ )? %aL -56F-G-J)*-G-J*
OO %.*23C*D6P
!/
K Lj U %w66 %w66s0 $U U
3U ? U' U+3U$ U U$c Uc g[ U[ 5 ( d [ S5 eU H nUc M [ U[ 5 ( d U/nU Hq n[ U[ 5 ( aP [ U5 S:X a [ S U 5 5 ( a M@ [ U5 S:X a [ S U 5 5 ( a Mh [ S5 e M g)a
Check `bboxes` for a single text example. It could be
- `None`: no bounding box associated to a text.
- A list with each element being the bounding boxes associated to one ` ... ` pair found
in a text. This could be:
- `None`: no bounding box associated to a ` ... ` pair.
- A tuple of 2 integers: A single bounding box specified by patch indices.
- A tuple of 4 float point number: A single bounding box specified by (normalized) coordinates.
- A list containing the above 2 tuple types: Multiple bounding boxes for a
` ... ` pair.
Nz@`bboxes` (for a single text example) should be `None` or a list. c 3 B # U H n[ U[ 5 v M g 7fN)r r .0r_ s r# AKosmos2Processor._check_bboxes_for_single_text..C s .S7az!S/A/A7 rE c 3 B # U H n[ U[ 5 v M g 7fr )r r r s r# r r D s 1XPW1*Q2F2FPWr a' Each element in `bboxes` (for a single text example) should be either `None`, a tuple containing 2 integers or 4 float point numbers, or a list containing such tuples. Also make sure the arguments `texts` and `bboxes` passed to `preprocess_text` are both in batches or both for a single example.)r r ry tupler all)r] r bboxelements r# _check_bboxes_for_single_text.Kosmos2Processor._check_bboxes_for_single_text* s > FD)) _` ` D| d++ v !'511\Q&3.S7.S+S+SG )c1XPW1X.X.X$@ r" c \ UR 5 nUb U SU 3nU R X5 nU$ )N )strip_insert_patch_index_tokens)r] rf imager img_info_tokenss r# _preprocess_single_example+Kosmos2Processor._preprocess_single_exampleM s; zz| %&av.D ..t< r" textsr r c R U R /U- nSR U R /U- U R /- 5 nSn[ U[ 5 ( a SnU/nUc S/[ U5 - nO[
U5 ( d U/n[ U5 [ U5 :w a$ [ S[ U5 S[ U5 S35 eU( d U R U5 U/nOMUb; [ U[ 5 ( d [ S5 eU H nU R U5 M OS/[ U5 - n[ U5 [ U5 :w a$ [ S [ U5 S[ U5 S35 e[ XU5 V V
V s/ s H u pn U R XX5 PM n n
n n U( d U S
n U $ s sn n
n f ) a Add image and bounding box information to `texts` as image and patch index tokens.
Args:
texts (`Union[TextInput, List[TextInput]]`): The texts to be processed.
images (`ImageInput`, *optional*): The images associated to `texts`.
bboxes (`Union[List[Tuple[int]], List[Tuple[float]], List[List[Tuple[int]]], List[List[Tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional*, defaults to 64):
The number of image tokens (used as latent queries). This should corresponds to the `latent_query_num`
attribute in `Kosmos2Config`.
Returns:
`Union[TextInput, List[TextInput]]`: The processed texts with image and patch index tokens.
r TFNzGThe number of examples in `texts` and `images` should be the same. Got v.s. instead.zS`bboxes` should be `None` or a list (as a batch) when `texts` is passed as a batch.zGThe number of examples in `texts` and `bboxes` should be the same. Got r ) rK joinrL r rW r r
ry r r zipr )
r] r re r r
img_tokensr batchedr_ rf r r results
r# r $Kosmos2Processor.preprocess_examplesW s , nn%(88
((DNN#3j#@DNNCS#ST eS ! !GGE >Vc%j(FF## XF u:V $ YZ]^cZdYeeklopvlwkx yB C
. .v 6 XF
fd++ !vww2215 Vc%j(F v;#e* $ YZ]^cZdYeeklopvlwkx yB C
&)%?
%?!T
+ +D O%?
AYF
s 1 F"c : U R R " U0 UD6$ )z
This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please
refer to the docstring of this method for more information.
)r= batch_decoder] argsr^ s r# r Kosmos2Processor.batch_decode s
~~**D;F;;r" c : U R R " U0 UD6$ )z
This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to
the docstring of this method for more information.
)r= decoder s r# r Kosmos2Processor.decode s
~~$$d5f55r" c f UR U R 5 S nU( a [ U5 $ U$ ro )splitrL +clean_text_and_extract_entities_with_bboxes)r] rf cleanup_and_extractcaptions r# post_process_generation(Kosmos2Processor.post_process_generation s, **T^^,R0 >wG Gr" c v U R " U4SU0UD6nU Vs/ s H oPR USS9PM sn$ s snf )ah
Post-process the output of the model to decode the text.
Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.
Returns:
`List[str]`: The decoded text.
skip_special_tokensF)r )r r )r] generated_outputsr r^ generated_textsrf s r# post_process_image_text_to_text0Kosmos2Processor.post_process_image_text_to_text sJ ++,=qSfqjpqZijZiRV,,Tu,MZijjjs 6c U R R nU R R n[ [ R X- 5 5 $ r )r= model_input_namesr< r dictfromkeys)r] tokenizer_input_namesimage_processor_input_namess r# r "Kosmos2Processor.model_input_names s> !% @ @ &*&:&:&L&L#DMM"7"UVWWr" c Ub [ U5 S:X a U$ [ [ R " SUS95 n[ U5 [ U5 :w a$ [ S[ U5 S[ U5 S35 eSn/ n[ X25 H u pgUR
5 u pUR XU 5 U nUc M1 [ U[ 5 ( a U/n/ n
[ S U 5 5 ( d [ S5 eU H, n U R U 5 u pU
R U S U
35 M. [ U
5 S:X a M S
R U
5 nUR S U S 35 M U[ U5 : a UR XS 5 S
R U5 nU$ )Nr z.+?)stringzuThe number of elements in `bboxes` should be the same as the number of ` ... ` pairs in `text`. Got r r c 3 ( # U H oS Lv M
g 7fr r )r boxs r# r >Kosmos2Processor._insert_patch_index_tokens.. s 7$3$$s zTThe multiple bounding boxes for a single phrase should not contain any `None` value.r z z )
r r refinditerry r spanrY r r r #_convert_bbox_to_patch_index_tokensr )r] rf r matched_phrasescurr_posbuffermatchedr r endpatch_index_stringsr
patch_index_1
patch_index_2 position_strs r# r +Kosmos2Processor._insert_patch_index_tokens s >S[A -K r{{+B4 PQ 3v; . H IL M\ I] H^ ^d eh io ep dq qz {
9MG \\^FA MM$, -H|$&& v"$ 7$777 j /3/W/WX[/\,
#**m_Am_+MN &'1,=BBCVWL MMIl^:> ?/ :2 c$i MM$y/ *wwv r" r c [ U5 S:X a Uu p#O6[ [ R " U R 5 5 n[ X5 u p#S[
U5 R S5 S3nS[
U5 R S5 S3nXV4$ )Nr rD rE rF )r r mathsqrtr> coordinate_to_patch_indexrW rX )r] r idx_1idx_2num_patches_per_sidetoken_1token_2s r# r 4Kosmos2Processor._convert_bbox_to_patch_index_tokens s t9> LE5 $'tyy1L1L'M#N 4TPLE!#e*"2"21"5!6a8!#e*"2"21"5!6a8 r" )
rK rQ rO rS rM rJ rL rN rR rP rT r> rU )i )NNNN)NNr7 )T)%r r r r __doc__
attributes valid_kwargsimage_processor_classtokenizer_classr\ r r r r r r+ r r r r BboxInputr r rW r r r r r propertyr r r r r r!
__classcell__)rc s @r# r: r: D s " $[1J,-L L%O+5^ "26 `` ItI./` /0
`
`D!F " *, @YY/0@ @ @
#3- @
sDI~
@F<6 k& X X
+s +E$uSzBRTXY^_dYeTfBf x1 and y2 > y1.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`Tuple[int, int]`: A pair of patch indices representing the upper-left patch and lower-right patch.
zTThe coordinates in `bbox` should be `(x1, y1, x2, y2)` with `x2 > x1` and `y2 > y1`.rk )ry r floorceil) r r x1y1x2y2ul_xul_ylr_xlr_yul_idxlr_idxs r# r r s RR Gopp ::b/ 0D ::b/ 0D 99R.2 3D 99R.2 3D
(4
/F
(4
/F >r" r r c SU- nX- nX- nX- nX- nX:X a XC- nXS- n Xc- U- n
Xs- U- n OIXF:X d XW:X a XC- nXS- n Xc- U- n
Xs- U- n O(XC- US- - nXS- US- - n Xc- US- - n
Xs- US- - n XX4$ )ak
Given a grid of length `num_patches_per_side` and the indices of the upper-left and lower-right corners of a
bounding box, returns the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
Args:
ul_idx (`int`): the index of the grid cell that corresponds to the upper-left corner of the bounding box.
lr_idx (`int`): the index of the grid cell that corresponds to the lower-right corner of the bounding box.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`Tuple[float]`: the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
g ?r r ) r r r cell_sizer
r r r
r r r r s r# patch_index_to_coordinater ' s **I (D )D (D )D
)
)
)
)
A
-
A
-
A
-
A
-
2>r" c
h Sn[ R " X5 n/ nU GH nUR S5 nUR 5 u pgnU( d* SnUR S5 S UR S5 S 4nUR S5 n / n
U H n [ R
" SU 5 n [ R
" SU SS 5 n
U ( d M= U
( d MF U( aE U
R
[ U R S5 5 [ U
R S5 5 45 M U
R
[ U R S5 5 [ U
R S5 5 45 M U( a UR
XuU
45 GMd U
H& nSUS S US S
3nUR
XU/45 M( GM U$ ) a Extract entities contained in `text`. The bounding bboxes is given in the form of patch indices.
This functioin is only intended to be used within `clean_text_and_extract_entities_with_bboxes` where further
processing happens, including converting to normalized coordinates and whitespace character cleaning up.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> entities = extract_entities_with_patch_indices(text)
>>> entities
[(' a snowman', (31, 41), [(44, 863)]), (' a fire', (130, 137), [(5, 911)])]
```z(?:(([^<]+)))?r Nr rC zrk rD z>r N)r r sub)r" rf entity_namestartr adjusted_start adjusted_endadjusted_entitys r# adjust_entity_positionsr, s[ & K %T&5\:;Nrvvgr4:67L"\$BCO r" c U R 5 n[ U 5 [ U R 5 5 -
n/ nU H u nu pgn[ U5 [ UR 5 5 -
n [ U5 [ UR 5 5 -
n
Xc-
U - nXs-
U
-
nUR 5 nUR XVU4U45 M X$4$ )z9Remove the spaces around the text and the entities in it.)r r rG rH rY ) rf entitiesnew_textleading_spaces new_entitiesr' r( r r entity_name_leading_spacesentity_name_trailing_spacess r# _cleanup_spacesr4 s zz|HYT[[]!33NL-5) \e6%(%5K>> text = " An image of a snowman warming himself by a fire."
>>> clean_text, entities = clean_text_and_extract_entities_with_bboxes(text)
>>> clean_text
'An image of a snowman warming himself by a fire.'
>>> entities
[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]
```r% r r r rk )r r& r# r, r rY r4 ) rf r processed_textr r. itemr" r r+ r bboxes_in_coordss r# r r s VVGR.N"Ed"K H+ aDG1&?jpqjpbf 5d1gtAwH\ ]jpq*==> , > 44 rs B) )'r r r r typingr r r r image_processing_utilsr image_utilsr r
processing_utilsr r r
r r tokenization_utilsr tokenization_utils_baser r r r r r r& r+ r: r r r# r, r4 r __all__r r" r# rA s: $ / / 2 1 b b , ?
sCx ueUE) *+eCHo eE5%'( )*,
(,e ( "
% " -U *A ~ A HE%u*D$E ]` ejknpskset >(c (3 (c (Z7't"*5:
r"