fTh$ S r SSKrSSKrSSKJr SSKJrJrJr SSK J
r
SSK J r J
r
Jr SSKJrJr SS KJr " S
S \ S S
9r\R* " \5 r " S S\
5 rS/rg)z
Processor class for Donut.
N)contextmanager)ListOptionalUnion )
ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)loggingc \ rS rSr0 rSrg)DonutProcessorKwargs N)__name__
__module__ __qualname____firstlineno__ _defaults__static_attributes__r b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/donut/processing_donut.pyr r s Ir r F)totalc
^ \ rS rSrSrSS/rSrSrSU 4S jjr SS\ S \
\ \ \
\ \\4 S
\\ 4S jjrS rS
r\S 5 rSS jr\S 5 r\S 5 rSrU =r $ )DonutProcessor% a
Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single
processor.
[`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and
[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and
[`~DonutProcessor.decode`] for more information.
Args:
image_processor ([`DonutImageProcessor`], *optional*):
An instance of [`DonutImageProcessor`]. The image processor is a required input.
tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`], *optional*):
An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input.
image_processor tokenizerAutoImageProcessor
AutoTokenizerc > S nSU; a, [ R " S[ 5 UR S5 nUb UOUnUc [ S5 eUc [ S5 e[
TU ] X5 U R U l SU l g )Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.F)
warningswarn
FutureWarningpop
ValueErrorsuper__init__r current_processor_in_target_context_manager)selfr r kwargsr$ __class__s r r+ DonutProcessor.__init__9 s & ( MM
!'
+> ? -<-H/N_ " HI I AB B
4!%!5!5 */'r imagestextr/ c U R ( a U R " X40 UD6$ Uc Uc [ S5 eU R " [ 4SU R
R 0UD6nUb U R " U40 US D6nUb. Ub US R SS5 U R
" U40 US D6nUc W$ Uc W$ WS WS' US US' U$ ) ax
When used in normal mode, this method forwards all its arguments to AutoImageProcessor's
[`~AutoImageProcessor.__call__`] and returns its output. If used in the context
[`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's
[`~DonutTokenizer.__call__`]. Please refer to the docstring of the above two methods for more information.
zBYou need to specify either an `images` or `text` input to process.tokenizer_init_kwargs
images_kwargs text_kwargsadd_special_tokensF input_idslabels) r- r, r)
_merge_kwargsr r init_kwargsr
setdefault) r. r2 r3 audiovideosr/
output_kwargsinputs encodingss r __call__DonutProcessor.__call__M s * *))&A&A A >dl ab b **
"&.."<"<
))&SM/4RSF ! m,778LeT tL}]/KLI <M
^ ( 5F8 "+K"8F; Mr c : U R R " U0 UD6$ )z
This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer
to the docstring of this method for more information.
)r batch_decoder. argsr/ s r rF DonutProcessor.batch_decodew s
~~**D;F;;r c : U R R " U0 UD6$ )z
This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the
docstring of this method for more information.
)r decoderG s r rK DonutProcessor.decode~ s
~~$$d5f55r c # # [ R " S5 SU l U R U l Sv U R
U l SU l g7f)zq
Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR.
z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r% r& r- r r, r r. s r as_target_processor"DonutProcessor.as_target_processor sG
9
+/'!%
!%!5!5 */'s A Ac Uc U R R 5 n0 nU( Gao [ R " SU[ R 5 nUc GODXR 5 S nSU; a GO*USUR
S5 S- nU[ S5 [ S5 * n[ R " U5 n[ R " SU S3U[ R 5 n U c UR US5 nGOU R 5 n [ R " U5 n
[ R " U 5 n [ R " U
SU 3U[ R [ R - 5 n U b U R S5 R 5 n SU ; a7 SU ; a1 U R U SUS 9n
U
( a [ U
5 S:X a U
S
n
XU' Ou/ XG' U R S 5 HB nUR 5 nX; a US
S :X a US
S S:X a USS
nXG R U5 MD [ XG 5 S:X a XG S
XG' XR! U 5 [ U 5 - S R 5 nUSS S :X a U/U R USS SUS 9- $ U( a GMo [ U5 ( a U( a U/$ U$ U( a / $ SU0$ )zC
Convert a (generated) token sequence into an ordered JSON format.
Nz z
text_sequence)r get_added_vocabresearch
IGNORECASEstartindexlenescapereplacegroupDOTALLstrip
token2jsonsplitappendfind)r. tokensrU rV outputpotential_start start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafs r rg DonutProcessor.token2json s ..88:K ii
FO & !6!6!8!:;K+%%&B (9(9#(>(BCK c%jCH95C ))C.K T+ a"8&"--PI R8 %OO - &(ii &<#$&IIi$8! ))*+51B0CDfbmm^`^g^gNg & %mmA . 4 4 6G (W-? $Ze f "5zQ(-a*/3K&( $+MM)$r sd
% ( ( % H H C +5
H %x$^ x$v
r