o Zho3@srddlmZmZmZddlZddlmZddlm Z m Z m Z ddl m Z mZddlmZmZmZmZmZddlmZdd lmZdd lmZmZerQdd lmZerder^dd lmZndd l mZerkddl!Z!dd lm"Z"e"#e$Z%dZ&dZ'dZ(dZ)dZ*d#ddZ+de,ddee-fddZ.de-de/e-e-ffddZ0GdddeZ1edd Gd!d"d"eZ2d"gZ3dS)$)ListOptionalUnionN) BatchFeature)IMAGENET_STANDARD_MEANIMAGENET_STANDARD_STDSizeDict)Unpack VideosKwargs) TensorTypeis_torch_availableis_torchvision_availableis_torchvision_v2_availableis_vision_available)requires)BaseVideoProcessor)group_videos_by_shapereorder_videos)PILImageResampling) functional)loggingzYou are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.zgYou are provided the following series of {frame_count} frames from a {video_duration} [H:MM:SS] video. z z Frame from {timestamp}:ic Cs t|dd}|dkrtd|dt|dd}t|dd}|dkr*td|dtt||}t||}|d kr=d }d} |d } |dkr_|d |||kr_t||} t|||} td| } t| |d } | | krvd|d } } tj| | |td } t| } | S) a] Example sampling function which: - Uses `max_frames` (if provided) or calculates it from `fps` and metadata. - Applies a basic center-skip if fewer frames than available, otherwise optionally skips `skip_secs` from both the start and end. - Uniformly samples the desired number of frames between the start and end indices. Args: max_frames (`int`): Maximum number of frames to sample. target_fps (`int`): Target frames to sample per second. metadata (`dict`): Contains video metadata such as "n_frames" and "video_fps". skip_secs (`float`, *optional*, defaults to 1.0): Number of seconds to skip from the start and end if the video is long enough. Returns: numpy.ndarray: An array of unique frame indices to sample. total_num_framesrzInvalid total_num_frames=z in metadata.Zfpsg>@durationzInvalid duration_seconds=Zdtype) getattr ValueErrorintroundminmaxnpZlinspaceunique) metadata max_framesZ target_fpsZ skip_secsrZ native_fpsZduration_secondsZestimated_framesZdesired_framesZ start_idxZend_idxindicesr(c/var/www/auris/lib/python3.10/site-packages/transformers/models/smolvlm/video_processing_smolvlm.pysmolvlm_sample_indices_fnGs.       r*videos torch.TensorreturncCsFtd}}|D]}|dd\}}t||}t||}q||fS)zH Get the maximum height and width across all videos in a batch. z-infN)floatsizer")r+ max_height max_widthvideoheightwidthr(r(r)get_max_height_widths   r6resolution_max_sidecCs|dd\}}tt|}|durt||n|}||}||kr5|}t||}|ddkr4|d7}n||krK|}t||}|ddkrK|d7}t|d}t|d}||fS)a Get the output size of the video after resizing given a dictionary specifying the max and min sizes. Args: video (`np.ndarray`): Video to resize. resolution_max_side (`int`): The longest edge of the video will be resized to this value. The shortest edge will be resized to keep the input aspect ratio. Returns: The output size of the video after resizing. r.Nrrr)r0r!MAX_IMAGE_SIZEr"r)r3r7r4r5Z aspect_ratior(r(r)get_resize_output_image_sizes$       r9c@s eZdZdS)SmolVLMVideoProcessorInitKwargsN)__name__ __module__ __qualname__r(r(r(r)r:s r:)Z torchvision)backendscseZdZejZddiZeZe Z dZ dZ dZ dZdZeZddgZdeeffdd Z d%d d d ed ddedd f ddZ  d&d d deeefdedefddZ d'ded deded ed eddedededed eeeeefd!eeeeefd"eeee ffd#d$Z!Z"S)(SmolVLMVideoProcessor longest_edgeiT pixel_valuespixel_attention_maskkwargsc stjdi|dS)Nr()super__init__)selfrC __class__r(r)rEszSmolVLMVideoProcessor.__init__Nr3r,r0 interpolationzF.InterpolationMode antialiasr-cKs|dur|ntjj}|tjjkrtdtjj}|jr$t||jd}n|j r1|j r1|j |j f}nt d|dtj ||||dS)a9 Resize an video to `(size["height"], size["width"])`. Args: video (`torch.Tensor`): Video to resize. size (`SizeDict`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output video. resample (`InterpolationMode`, *optional*, defaults to `InterpolationMode.BILINEAR`): `InterpolationMode` filter to use when resizing the video e.g. `InterpolationMode.BICUBIC`. Returns: `torch.Tensor`: The resized video. NzYou have used fast image processor with LANCZOS resample which not yet supported for torch.Tensor. BICUBIC resample will be used as an alternative. Please fall back to image processor if you want full consistency with the original model.)r7zHSize must contain 'height' and 'width' keys, or 'longest_edge' key. Got .)rIrJ) FZInterpolationModeZBILINEARLANCZOSloggerZ warning_onceZBICUBICr@r9r4r5rresize)rFr3r0rIrJrCnew_sizer(r(r)rOs  zSmolVLMVideoProcessor.resizer padded_sizefillreturn_pixel_maskc Cs|dd}|d|d}|d|d}|dks |dkr+td|d|d||kr=dd||g}tj|||d}d} |rbtj|d dddddftjd } d| d d|dd|df<|| fS) aPads the sample with empty video to the padded_size Args: video (`torch.Tensor`): Video to pad. padded_size (`Tuple[int, int]`): Height and width to pad. fill (`int`, *optional*): The value to use for the padding. return_pixel_mask (`bool`, *optional*, defaults to `True`): Whether to return a pixel mask. r.NrrzzPadding dimensions are negative. Please make sure that the padded size is larger than the original size. Got padded size: z, original size: rK)rR.r)r0rrLpadtorchZ zeros_likeZint64) rFr3rQrRrSZ original_sizepadding_bottom padding_rightpaddingZ pixel_maskr(r(r)rTs& $zSmolVLMVideoProcessor.padr+do_convert_rgb do_resize do_rescalerescale_factor do_normalizedo_pad image_mean image_stdreturn_tensorsc  KsVt|\}}i}|D]\}}|r||}|r!|j|||d}|||<q t||}t|\}}i}|D]\}}|||||| | }|||<q7t||}| rt|}t|\}}i}i}|D]\}}|j||d\}}|||<|||<qct||}t||}| rtj |ddn|}d|i}| r| r| durtj |ddn||d<t || dS)N)r0rI)rQr)dimrArB)Z tensor_type) ritemsZconvert_to_rgbrOrZrescale_and_normalizer6rTrUstackr)rFr+rYrZr0rIr[r\r]r^r_r`rarCZgrouped_videosZgrouped_videos_indexZresized_videos_groupedshapeZstacked_videosZresized_videosZprocessed_videos_groupedZprocessed_videosZpad_sizeZprocessed_padded_mask_groupedZ padded_masksrBdatar(r(r) _preprocesssL             z!SmolVLMVideoProcessor._preprocess)NT)rT)N)#r;r<r=rrMZresampler0rr_rr`rZr[r]rYr^r:Z valid_kwargsZmodel_input_namesr rEr boolrOtuplerrTrrr/rstrr rg __classcell__r(r(rGr)r?s~ .  3     r?)r)4typingrrrnumpyr#Zimage_processing_utilsrZ image_utilsrrr Zprocessing_utilsr r utilsr r rrrZutils.import_utilsrZvideo_processing_utilsrZ video_utilsrrrZtorchvision.transforms.v2rrLZtorchvision.transformsrUrZ get_loggerr;rNZDEFAULT_SYSTEM_MESSAGEZDEFAULT_VIDEO_INTROZDEFAULT_MEDIA_OUTTROZFRAME_TIMESTAMP_MESSAGEr8r*listrr6rir9r:r?__all__r(r(r(r)sJ        <   (