o ZhW @s0ddlmZmZddlmZmZmZddlmZm Z m Z ddl m Z ddl mZz&ddlmZmZmZdd lmZdd lmZdd lmZdd lmZd ZWneefyaeZddZdZYnwe dZ defddZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&dS) )ArgumentParser Namespace)AnyListOptional)Pipelineget_supported_taskspipeline)logging)BaseTransformersCLICommand)BodyFastAPI HTTPException)APIRoute) BaseModel) JSONResponse)runTcOsdS)N)xyrrL/var/www/auris/lib/python3.10/site-packages/transformers/commands/serving.pyr"srFztransformers/servingargscCs:t|j|jr |jnd|j|j|jd}t||j|j|j S)z~ Factory function used to instantiate serving server from provided command line arguments. Returns: ServeCommand N)taskmodelconfig tokenizerdevice) r rrrrr ServeCommandhostportworkers)rZnlprrrserve_command_factory+sr#c@eZdZUdZeed<dS)ServeModelInfoResultz" Expose model information infosN)__name__ __module__ __qualname____doc__dict__annotations__rrrrr%;  r%c@s.eZdZUdZeeed<eeeed<dS)ServeTokenizeResultz Tokenize result model tokens tokens_idsN) r'r(r)r*rstrr,rintrrrrr.Cs  r.c@r$)ServeDeTokenizeResultz! DeTokenize result model textN)r'r(r)r*r1r,rrrrr3Lr-r3c@r$)ServeForwardResultz Forward result model outputN)r'r(r)r*rr,rrrrr5Tr-r5c@seZdZedefddZdedededefdd Z d d Z d d Z e ddde dddfdede fddZe ddde ddde dddfdeede de fddZe dddfddZdS)rparsercCs|jddd}|jdttdd|jdtdd d |jd td d d |jdtddd |jdtdd|jdtdd|jdtdd|jdtddd |jtddS)z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments ZservezFCLI tool to run inference requests through REST and GraphQL endpoints.)helpz--taskzThe task to run the pipeline on)typechoicesr8z--host localhostz$Interface the server will listen on.)r9defaultr8z--porti"z Port the serving will listen to.z --workersr zNumber of http workersz--modelz%Model's name or path to stored model.)r9r8z--configz,Model's config name or path to stored model.z --tokenizerzTokenizer name to use.z--devicezSIndicate the device to run onto, -1 indicates CPU, >= 0 indicates GPU (default: -1))funcN) add_parser add_argumentr1r r2 set_defaultsr#)r7Z serve_parserrrrregister_subcommand]s,z ServeCommand.register_subcommandr r r!r"c Cs||_||_||_||_tstdtd|d|tt d|j t t dgdt d|j tt dgdt d |jtt dgdt d |jtt dgdgd d |_dS) NzUsing serve command requires FastAPI and uvicorn. Please install transformers with [serving]: pip install "transformers[serving]". Or install FastAPI and uvicorn separately.zServing model over :/GET)Zresponse_modelresponse_classmethodsz /tokenizePOSTz /detokenizez/forwardiX)Zroutestimeout) _pipeliner r!r"_serve_dependencies_installed RuntimeErrorloggerinforr model_infor%rtokenizer. detokenizer3forwardr5_app)selfr r r!r"rrr__init__|sR zServeCommand.__init__cCst|j|j|j|jddS)N)r r!r")rrSr r!r"rTrrrrszServeCommand.runcCstt|jjjdS)N)r&)r%varsrJrrrVrrrrOszServeCommand.model_infoNT)ZembedF text_input return_idsc Csjz|jj|}|r|jj|}t||dWSt|dWSty4}z tddt|ddd}~ww)z Tokenize the provided input and eventually returns corresponding tokens id: - **text_input**: String to tokenize - **return_ids**: Boolean flags indicating if the tokens have to be converted to their integer mapping. )r/r0)r/rerror status_codedetailN)rJrrPZconvert_tokens_to_idsr. Exceptionrr1)rTrXrYZ tokens_txtr0errrrPs zServeCommand.tokenizer0skip_special_tokenscleanup_tokenization_spacesc CsPz|jj|||}td|dWSty'}z tddt|ddd}~ww)a: Detokenize the provided tokens ids to readable text: - **tokens_ids**: List of tokens ids - **skip_special_tokens**: Flag indicating to not try to decode special tokens - **cleanup_tokenization_spaces**: Flag indicating to remove all leading/trailing spaces and intermediate ones. r[)rr4rZr\r^N)rJrdecoder3rarr1)rTr0rcrdZ decoded_strrbrrrrQs zServeCommand.detokenizec s\t|dkr tggdSz ||}t|dWSty-}z tddt|id}~ww)zF **inputs**: **attention_mask**: **tokens_type_ids**: r)r6Z attention)r6rZr]N)lenr5rJrarr1)rTZinputsr6rbrrrrRs    zServeCommand.forward)r'r(r) staticmethodrrBrr1r2rUrrOrboolrPrrQrRrrrrr\s$1(    rN)'argparserrtypingrrrZ pipelinesrr r utilsr r[r ZfastapirrrZfastapi.routingrZpydanticrZstarlette.responsesrZuvicornrrK ImportErrorAttributeErrorobjectZ get_loggerrMr#r%r.r3r5rrrrrs0