o Zh(@sddlmZmZmZmZmZddlmZmZm Z m Z m Z ddl m Z mZe r5ddlmZddlmZmZerHddlZdd lmZdd lmZe eZeed d Gd dde ZdS))AnyDictListOptionalUnion)add_end_docstringsis_torch_availableis_vision_availableloggingrequires_backends) ChunkPipelinebuild_pipeline_init_args)Image) load_image valid_imagesN)BaseModelOutput)2MODEL_FOR_ZERO_SHOT_OBJECT_DETECTION_MAPPING_NAMEST)Zhas_image_processorc seZdZdZfddZ ddeedeeee ffde eeeefffdd Z d d Z dd d Z ddZdddZdddeeeffddZZS)ZeroShotObjectDetectionPipelinea Zero shot object detection pipeline using `OwlViTForObjectDetection`. This pipeline predicts bounding boxes of objects when you provide an image and a set of `candidate_labels`. Example: ```python >>> from transformers import pipeline >>> detector = pipeline(model="google/owlvit-base-patch32", task="zero-shot-object-detection") >>> detector( ... "http://images.cocodataset.org/val2017/000000039769.jpg", ... candidate_labels=["cat", "couch"], ... ) [{'score': 0.287, 'label': 'cat', 'box': {'xmin': 324, 'ymin': 20, 'xmax': 640, 'ymax': 373}}, {'score': 0.254, 'label': 'cat', 'box': {'xmin': 1, 'ymin': 55, 'xmax': 315, 'ymax': 472}}, {'score': 0.121, 'label': 'couch', 'box': {'xmin': 4, 'ymin': 0, 'xmax': 642, 'ymax': 476}}] >>> detector( ... "https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png", ... candidate_labels=["head", "bird"], ... ) [{'score': 0.119, 'label': 'bird', 'box': {'xmin': 71, 'ymin': 170, 'xmax': 410, 'ymax': 508}}] ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This object detection pipeline can currently be loaded from [`pipeline`] using the following task identifier: `"zero-shot-object-detection"`. See the list of available models on [huggingface.co/models](https://huggingface.co/models?filter=zero-shot-object-detection). c sFtjdi||jdkrtd|jdt|d|tdS)NtfzThe z is only available in PyTorch.Zvision)super__init__ framework ValueError __class__r Zcheck_model_typer)selfkwargsrr`/var/www/auris/lib/python3.10/site-packages/transformers/pipelines/zero_shot_object_detection.pyr8s   z(ZeroShotObjectDetectionPipeline.__init__Nimagez Image.Imagecandidate_labelsc sd|vr |d}t|ttjfr||d}n"t|ttfr6t|r6ttjddt ||Dfi|S |}tj|fi|}|S)a| Detect objects (bounding boxes & classes) in the image(s) passed as inputs. Args: image (`str`, `PIL.Image` or `List[Dict[str, Any]]`): The pipeline handles three types of images: - A string containing an http url pointing to an image - A string containing a local path to an image - An image loaded in PIL directly You can use this parameter to send directly a list of images, or a dataset or a generator like so: ```python >>> from transformers import pipeline >>> detector = pipeline(model="google/owlvit-base-patch32", task="zero-shot-object-detection") >>> detector( ... [ ... { ... "image": "http://images.cocodataset.org/val2017/000000039769.jpg", ... "candidate_labels": ["cat", "couch"], ... }, ... { ... "image": "http://images.cocodataset.org/val2017/000000039769.jpg", ... "candidate_labels": ["cat", "couch"], ... }, ... ] ... ) [[{'score': 0.287, 'label': 'cat', 'box': {'xmin': 324, 'ymin': 20, 'xmax': 640, 'ymax': 373}}, {'score': 0.25, 'label': 'cat', 'box': {'xmin': 1, 'ymin': 55, 'xmax': 315, 'ymax': 472}}, {'score': 0.121, 'label': 'couch', 'box': {'xmin': 4, 'ymin': 0, 'xmax': 642, 'ymax': 476}}], [{'score': 0.287, 'label': 'cat', 'box': {'xmin': 324, 'ymin': 20, 'xmax': 640, 'ymax': 373}}, {'score': 0.254, 'label': 'cat', 'box': {'xmin': 1, 'ymin': 55, 'xmax': 315, 'ymax': 472}}, {'score': 0.121, 'label': 'couch', 'box': {'xmin': 4, 'ymin': 0, 'xmax': 642, 'ymax': 476}}]] ``` candidate_labels (`str` or `List[str]` or `List[List[str]]`): What the model should recognize in the image. threshold (`float`, *optional*, defaults to 0.1): The probability necessary to make a prediction. top_k (`int`, *optional*, defaults to None): The number of top predictions that will be returned by the pipeline. If the provided number is `None` or higher than the number of predictions available, it will default to the number of predictions. timeout (`float`, *optional*, defaults to None): The maximum time in seconds to wait for fetching images from the web. If None, no timeout is set and the call may block forever. Return: A list of lists containing prediction results, one list per input image. Each list contains dictionaries with the following keys: - **label** (`str`) -- Text query corresponding to the found object. - **score** (`float`) -- Score corresponding to the object (between 0 and 1). - **box** (`Dict[str,int]`) -- Bounding box of the detected object in image's original size. It is a dictionary with `x_min`, `x_max`, `y_min`, `y_max` keys. Z text_queriesr!r"css|] \}}||dVqdS)r#Nr).0Zimglabelsrrr sz;ZeroShotObjectDetectionPipeline.__call__..) pop isinstancestrrlisttuplerr__call__zip)rr!r"rinputsresultsrrr r,As ?  z(ZeroShotObjectDetectionPipeline.__call__cKsNi}d|vr |d|d<i}d|vr|d|d<d|vr"|d|d<|i|fS)Ntimeout thresholdtop_kr)rrZpreprocess_paramsZpostprocess_paramsrrr _sanitize_parameterss    z4ZeroShotObjectDetectionPipeline._sanitize_parametersc cst|d|d}|d}t|tr|d}tj|j|jggtjd}t |D]0\}}|j ||j d}|j ||j d} |j dkrG| |j} |t|dk||d || Vq(dS) Nr!)r0r",)Zdtype)Zreturn_tensorsptr )is_last target_sizecandidate_label)rr(r)splittorchZtensorheightwidthZint32 enumerateZ tokenizerrimage_processortoZ torch_dtypelen) rr.r0r!r"r7ir8Z text_inputsZimage_featuresrrr preprocesss(    z*ZeroShotObjectDetectionPipeline.preprocesscCsB|d}|d}|d}|jdi|}|||d|}|S)Nr7r8r6)r7r8r6r)r'model)rZ model_inputsr7r8r6outputs model_outputsrrr _forwards   z(ZeroShotObjectDetectionPipeline._forward皙?c Csg}|D]>}|d}t|}|jj|||ddd}|dD] }|d|} ||d|d} | || d} || q!qt|dd d d }|rT|d|}|S) Nr8r7)rDr1Z target_sizesrZscoresZboxes)scorelabelboxcSs|dS)NrHr)xrrr sz=ZeroShotObjectDetectionPipeline.postprocess..T)keyreverse)rr>Zpost_process_object_detectionZnonzeroitem_get_bounding_boxappendsorted) rrEr1r2r/Z model_outputrIrDindexrHrJresultrrr postprocesss&    z+ZeroShotObjectDetectionPipeline.postprocessrJz torch.TensorreturncCs8|jdkr td|\}}}}||||d}|S)a% Turns list [xmin, xmax, ymin, ymax] into dict { "xmin": xmin, ... } Args: box (`torch.Tensor`): Tensor containing the coordinates in corners format. Returns: bbox (`Dict[str, int]`): Dict containing the coordinates in corners format. r5zAThe ZeroShotObjectDetectionPipeline is only available in PyTorch.)xminyminxmaxymax)rrinttolist)rrJrWrXrYrZZbboxrrr rPs z1ZeroShotObjectDetectionPipeline._get_bounding_box)N)rGN)__name__ __module__ __qualname____doc__rrr)rrrrr,r3rBrFrUr[rP __classcell__rrrr rs  X  "r)typingrrrrrutilsrr r r r baserrZPILrZ image_utilsrrr:Ztransformers.modeling_outputsrZmodels.auto.modeling_autorZ get_loggerr]loggerrrrrr s