o
Zh$ @ s d Z ddlZddlZddlmZ ddlmZmZmZ ddl m
Z
ddl m Z m
Z
mZ ddlmZmZ dd lmZ G d
d d e d d
ZeeZG dd de
ZdgZdS )z
Processor class for Donut.
N)contextmanager)ListOptionalUnion )
ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)loggingc @ s e Zd Zi ZdS )DonutProcessorKwargsN)__name__
__module__ __qualname__ _defaults r r Y/var/www/auris/lib/python3.10/site-packages/transformers/models/donut/processing_donut.pyr s r F)totalc
s e Zd ZdZddgZdZdZd fdd Z d d ed
e e
e e e e
ef d ee fd d
Zdd Zdd Zedd Zd ddZedd Zedd Z ZS ) DonutProcessora
Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single
processor.
[`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and
[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and
[`~DonutProcessor.decode`] for more information.
Args:
image_processor ([`DonutImageProcessor`], *optional*):
An instance of [`DonutImageProcessor`]. The image processor is a required input.
tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`], *optional*):
An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input.
image_processor tokenizerZAutoImageProcessorZ
AutoTokenizerNc sr d }d|v rt dt |d}|d ur|n|}|d u r!td|d u r)tdt || | j| _d| _ d S )Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.F)
warningswarn
FutureWarningpop
ValueErrorsuper__init__r current_processor_in_target_context_manager)selfr r kwargsr __class__r r r 9 s
zDonutProcessor.__init__imagestextr$ c K s | j r
| j||fi |S |du r|du rtd| jtfd| jji|}|dur5| j|fi |d }|durP|durE|d dd | j|fi |d }|du rV|S |du r\|S |d |d < |d |d< |S )
a
When used in normal mode, this method forwards all its arguments to AutoImageProcessor's
[`~AutoImageProcessor.__call__`] and returns its output. If used in the context
[`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's
[`~DonutTokenizer.__call__`]. Please refer to the docstring of the above two methods for more information.
NzBYou need to specify either an `images` or `text` input to process.Ztokenizer_init_kwargsZ
images_kwargsZ text_kwargsZadd_special_tokensFZ input_idslabels) r" r! r Z
_merge_kwargsr r Z init_kwargsr
setdefault) r# r' r( ZaudioZvideosr$ Z
output_kwargsZinputs encodingsr r r __call__M s0 zDonutProcessor.__call__c O | j j|i |S )z
This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer
to the docstring of this method for more information.
)r batch_decoder# argsr$ r r r r. w zDonutProcessor.batch_decodec O r- )z
This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the
docstring of this method for more information.
)r decoder/ r r r r2 ~ r1 zDonutProcessor.decodec c s2 t d d| _| j| _dV | j| _d| _dS )z
Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR.
z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r r r" r r! r r# r r r as_target_processor s
z"DonutProcessor.as_target_processorFc C sD |du r | j }i }|r td|tj}|du rn|| d }d|vr(n|d|dd }|tdtd }t|}td| d|tj} | du rZ| |d}n|
} t|}
t| } t|
d| |tjtj B } | dur|
d } d| v rd| v r| j
| d|d }
|
rt|
dkr|
d
}
|
||< n?g ||< | d D ]%}| }||v r|d
d kr|d
d dkr|dd
}|| | qt|| dkr|| d
||< ||| t| d }|dd d kr
|g| j
|dd d|d S |st|r|r|gS |S |r g S d|iS )zS
Convert a (generated) token sequence into an ordered JSON format.
Nz z Z
text_sequence)r Zget_added_vocabresearch
IGNORECASEstartindexlenescapereplacegroupDOTALLstrip
token2jsonsplitappendfind)r# tokensr8 r9 outputZpotential_startZ start_tokenkeyZ key_escapedZ end_tokenZstart_token_escapedZend_token_escapedcontentvalueleafr r r rH s\
$
,zDonutProcessor.token2jsonc C t dt | jS )Nzg`feature_extractor_class` is deprecated and will be removed in v5. Use `image_processor_class` instead.)r r r image_processor_classr3 r r r feature_extractor_class
z&DonutProcessor.feature_extractor_classc C rR )Nz[`feature_extractor` is deprecated and will be removed in v5. Use `image_processor` instead.)r r r r r3 r r r r rU z DonutProcessor.feature_extractor)NN)NNNN)FN)r r r __doc__
attributesrS Ztokenizer_classr r r r strr r r r
r r, r. r2 r r4 rH propertyrT r
__classcell__r r r% r r % s4
*
:
r )rV r= r
contextlibr typingr r r Z image_utilsr Zprocessing_utilsr r r
Ztokenization_utils_baser r utilsr
r Z
get_loggerr loggerr __all__r r r r s
<