o ZhrV@sddlZddlmZddlmmZddlmZmZm Z ddl m Z m Z m Z mZmZer4ddlmZer=ddlmZejjdd ZGd d d ejZGd d d ejZ     dddZdS)N)is_scipy_availableis_vision_availablerequires_backends)box_iou dice_lossgeneralized_box_iounested_tensor_from_tensor_listsigmoid_focal_losslinear_sum_assignment)center_to_corners_formatcCsddt||DS)NcSsg|] \}}||dqS))logits pred_boxes).0abrrM/var/www/auris/lib/python3.10/site-packages/transformers/loss/loss_rt_detr.py +sz!_set_aux_loss..)zip) outputs_class outputs_coordrrr _set_aux_loss&srcs0eZdZdZfddZeddZZS)RTDetrHungarianMatcheraThis class computes an assignment between the targets and the predictions of the network For efficiency reasons, the targets don't include the no_object. Because of this, in general, there are more predictions than targets. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). Args: config: RTDetrConfig csztt|dg|j|_|j|_|j|_|j |_ |j |_ |j |_ |j|jkr8|jkr8dkr;tddSdS)NZscipyrz#All costs of the Matcher can't be 0)super__init__rZmatcher_class_cost class_costZmatcher_bbox_cost bbox_costZmatcher_giou_cost giou_costuse_focal_lossZ matcher_alphaalphaZ matcher_gammagamma ValueError)selfconfig __class__rrr9s  " zRTDetrHungarianMatcher.__init__cCs~|djdd\}}|ddd}tdd|D}td d|D}|jrft|ddd}|dd|f}d|j||jd|d  } |jd||j|d  } | | } n|ddd d }|dd|f } tj ||dd } t t |t | } |j| |j| |j| }|||d }d d|D}ddt||d D}dd|DS)aPerforms the matching Params: outputs: This is a dict that contains at least these entries: "logits": Tensor of dim [batch_size, num_queries, num_classes] with the classification logits "pred_boxes": Tensor of dim [batch_size, num_queries, 4] with the predicted box coordinates targets: This is a list of targets (len(targets) = batch_size), where each target is a dict containing: "class_labels": Tensor of dim [num_target_boxes] (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels "boxes": Tensor of dim [num_target_boxes, 4] containing the target box coordinates Returns: A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected targets (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes) rNrrrrcSg|]}|dqS class_labelsrrvrrrrbz2RTDetrHungarianMatcher.forward..cSr)boxesrr,rrrrcr.g:0yE>)pcSg|]}t|dqSr/lenr,rrrrycSsg|] \}}t||qSrr )ricrrrrzscSs0g|]\}}tj|tjdtj|tjdfqS)dtype)torch as_tensorint64)rr7jrrrr|s0)shapeflattenr;catr!Fsigmoidr"r#logZsoftmaxZcdistr rrrr viewcpu enumeratesplit)r%outputstargetsZ batch_sizeZ num_queriesZout_bboxZ target_idsZ target_bboxZout_probZneg_cost_classZpos_cost_classrrr Z cost_matrixsizesindicesrrrforwardHs&&" zRTDetrHungarianMatcher.forward) __name__ __module__ __qualname____doc__rr;no_gradrM __classcell__rrr'rr.s  rcseZdZdZfddZdddZdddZed d Z d d Z d dZ dddZ ddZ ddZdddZddZeddZddZZS) RTDetrLossah This class computes the losses for RTDetr. The process happens in two steps: 1) we compute hungarian assignment between ground truth boxes and the outputs of the model 2) we supervise each pair of matched ground-truth / prediction (supervise class and box). Args: matcher (`DetrHungarianMatcher`): Module able to compute a matching between targets and proposals. weight_dict (`Dict`): Dictionary relating each loss with its weights. These losses are configured in RTDetrConf as `weight_loss_vfl`, `weight_loss_bbox`, `weight_loss_giou` losses (`List[str]`): List of all the losses to be applied. See `get_loss` for a list of all available losses. alpha (`float`): Parameter alpha used to compute the focal loss. gamma (`float`): Parameter gamma used to compute the focal loss. eos_coef (`float`): Relative classification weight applied to the no-object category. num_classes (`int`): Number of object categories, omitting the special no-object category. cs|tt||_|j|_|j|j|jd|_ ddg|_ |j |_ t |jd}|j |d<|d||j|_|j|_dS)N)loss_vfl loss_bbox loss_giouvflr0rr1 empty_weight)rrrmatcherZ num_labels num_classesZweight_loss_vflZweight_loss_bboxZweight_loss_giou weight_dictlossesZeos_coefficientZeos_coefr;ZonesZregister_bufferZfocal_loss_alphar"Zfocal_loss_gammar#)r%r&rYr'rrrs      zRTDetrLoss.__init__TcCstd|vrtdd|vrtd||}|d|}tjddt||Ddd}tt|t|\} } t| } |d} td dt||D} tj | j dd |j tj | j d } | | |<tj| |j d d dddf}tj| | jd}| |j||<|d|}t| }|j||jd ||}tj| ||dd}|d | j d |}d|iS)Nr#No predicted boxes found in outputsrz$No predicted logits found in outputscS g|] \}\}}|d|qSr/rr_target_r7rrrr z.RTDetrLoss.loss_labels_vfl..rdimcSr_r*rr`rrrrrcrr:devicerr[.r1r9none)weight reductionrU)KeyError_get_source_permutation_idxr;rArrrdetachdiagfullr?r[r=rgrBone_hotZ zeros_liker:toZ unsqueezerCr"powr# binary_cross_entropy_with_logitsmeansum)r%rIrJrL num_boxesrDidx src_boxes target_boxesZiousrb src_logitstarget_classes_originaltarget_classestargetZtarget_score_originalZ target_scoreZ pred_scorerjlossrrrloss_labels_vfls0    zRTDetrLoss.loss_labels_vflc Csd|vrtd|d}||}tddt||D}tj|jdd|jtj|j d} || |<t | dd| |j } d | i} | S) zClassification loss (NLL) targets dicts must contain the key "class_labels" containing a tensor of dim [nb_target_boxes] rz#No logits were found in the outputscSr_r*rr`rrrrrcz*RTDetrLoss.loss_labels..Nrrfrloss_ce)rlrmr;rArrpr?r[r=rgrBZ cross_entropy transposeZ class_weight) r%rIrJrLrwrDr{rxr|r}rr]rrr loss_labelss zRTDetrLoss.loss_labelsc Csf|d}|j}tjdd|D|d}|d|jddkd}tj| | } d| i} | S)z Compute the cardinality error, i.e. the absolute error in the number of predicted non-empty boxes. This is not really a loss, it is intended for logging purposes only. It doesn't propagate gradients. rcSr3r*r4r,rrrrr6z/RTDetrLoss.loss_cardinality..)rgr1rZcardinality_error) rgr;r<Zargmaxr?rvnn functionall1_lossfloat) r%rIrJrLrwrrgZtarget_lengthsZ card_predZcard_errr]rrrloss_cardinalityszRTDetrLoss.loss_cardinalityc Csd|vrtd||}|d|}tjddt||Ddd}i}tj||dd} | ||d <d tt t |t |} | ||d <|S) a; Compute the losses related to the bounding boxes, the L1 regression loss and the GIoU loss. Targets dicts must contain the key "boxes" containing a tensor of dim [nb_target_boxes, 4]. The target boxes are expected in format (center_x, center_y, w, h), normalized by the image size. rr^cSr_r/r)rtrbr7rrrrrcz)RTDetrLoss.loss_boxes..rrdrirkrVrrW) rlrmr;rArrBrrvror r) r%rIrJrLrwrxryrzr]rVrWrrr loss_boxess  zRTDetrLoss.loss_boxesc Csd|vrtd||}||}|d}||}dd|D}t|\} } | |} | |} tjj|dddf| j ddddd }|ddd f d }| d } | |j } t || |t || |d } | S) z Compute the losses related to the masks: the focal loss and the dice loss. Targets dicts must contain the key "masks" containing a tensor of dim [nb_target_boxes, h, w]. Z pred_masksz#No predicted masks found in outputscSr))masksrrrrrrrr.z)RTDetrLoss.loss_masks..NZbilinearF)sizemodeZ align_cornersrr)Z loss_maskZ loss_dice)rlrm_get_target_permutation_idxr Z decomposerrrrZ interpolater?r@rEr r) r%rIrJrLrw source_idx target_idxZ source_masksrZ target_masksZvalidr]rrr loss_maskss(       zRTDetrLoss.loss_masksc Cs|d}||}tddt||D}tj|jdd|jtj|jd} || |<t j | |jddddd f} t j || d d d } | d |jd|} d | iS)NrcSr_r*rr`rrrr$rcz.RTDetrLoss.loss_labels_bce..rrfrrh.r1g?rirZloss_bce)rmr;rArrpr?r[r=rgrBrqrtrurv r%rIrJrLrwrDr{rxr|r}r~rrrrloss_labels_bce!s  zRTDetrLoss.loss_labels_bcecC4tddt|D}tdd|D}||fS)NcSs g|] \}\}}t||qSrr;Z full_like)rr7sourcerbrrrr1rcz:RTDetrLoss._get_source_permutation_idx..cSsg|]\}}|qSrr)rrrbrrrr2r.r;rArG)r%rL batch_idxrrrrrm/z&RTDetrLoss._get_source_permutation_idxcCr)NcSs g|] \}\}}t||qSrr)rr7rbr~rrrr7rcz:RTDetrLoss._get_target_permutation_idx..cSsg|]\}}|qSrr)rrbr~rrrr8r.r)r%rLrrrrrr5rz&RTDetrLoss._get_target_permutation_idxc Csd|vrtd|d}||}tddt||D}tj|jdd|jtj|j d} || |<t j | |jddd dd f} t || |j |j} | d|jd|} d | iS) NrzNo logits found in outputscSr_r*rr`rrrrBrcz0RTDetrLoss.loss_labels_focal..rrfrrh.r1Z loss_focal)rlrmr;rArrpr?r[r=rgrBrqr r"r#rurvrrrrloss_labels_focal;s  zRTDetrLoss.loss_labels_focalcCsL|j|j|j|j|j|j|jd}||vrtd|d||||||S)N)labelsZ cardinalityr0rZbceZfocalrXzLoss z not supported)rrrrrrrr$)r%rrIrJrLrwZloss_maprrrget_lossMs zRTDetrLoss.get_lossc Cs|d|d}}dd|D}|ddj}g}t|D]A\}}|dkrItj|tj|d} | |} t||t| ks?J|||| fq|tjdtj|dtjdtj|dfq|S)Ndn_positive_idx dn_num_groupcSr3r*r4rrrrr^r6z6RTDetrLoss.get_cdn_matched_indices..rr+rf) rgrGr;Zaranger=Ztiler5appendZzeros) Zdn_metarJrrZnum_gtsrgZdn_match_indicesr7Znum_gtZgt_idxrrrget_cdn_matched_indices[s  z"RTDetrLoss.get_cdn_matched_indicesc sdd|D}||}tdd|D}tj|gtjtt|j d}tj |dd }i}j D]} |||||fddD|q7d |vrt|d D]8\}||}j D]*}|d krpqi |||||fd dDfd dD|qiq\d |vrd|vrtd|d|}||dd}t|d D]8\}j D]0}|d krqi} j |||||fi| fddDfddD|qq|S)a This performs the loss computation. Args: outputs (`dict`, *optional*): Dictionary of tensors, see the output specification of the model for the format. targets (`List[dict]`, *optional*): List of dicts, such that `len(targets) == batch_size`. The expected keys in each dict depends on the losses applied, see each loss' doc. cSsi|] \}}d|vr||qS)auxiliary_outputsrrkr-rrr }sz&RTDetrLoss.forward..css|] }t|dVqdS)r+Nr4rrrr sz%RTDetrLoss.forward..rfr)minc*i|]}|jvr||j|qSrr\rrl_dictr%rrr*rrcrrrrrrrrrc i|] \}}|d|qS)Z_aux_rrr7rrrrcdn_auxiliary_outputsdenoising_meta_valuesz}The output must have the 'denoising_meta_values` key. Please, ensure that 'outputs' includes a 'denoising_meta_values' entry.rcrrrrrrrrrcr)Z_dn_rrrrrrrc)itemsrZrvr;r<rnextitervaluesrgclampitemr]rupdaterGr$r) r%rIrJZoutputs_without_auxrLrwr]rrkwargsr)r7rr%rrMrsN "         zRTDetrLoss.forward)T)rNrOrPrQrrrr;rRrrrrrmrrr staticmethodrrMrSrrr'rrTs"       rTc Ks t|} | |i} || d<|| d<|jrv| dur2tj|| ddd\} }tj|| ddd\}}t|ddddfdd|ddddfdd}|| d <| d t|g|g| durvt|dd| dd| d <| | d <| | |}t| }|||fS) NrrZ dn_num_splitrrdr1rrrrr) rTrrZauxiliary_lossr;rHrrextendrvr)rrrgrr&rrZenc_topk_logitsZenc_topk_bboxesrr criterionZ outputs_lossZ dn_out_coordZ dn_out_classrZ loss_dictrrrrRTDetrForObjectDetectionLosss( :   r)NNNNN)r;Ztorch.nnrZtorch.nn.functionalrrButilsrrrZloss_for_object_detectionrrr r r Zscipy.optimizer Ztransformers.image_transformsrZjitZunusedrModulerrTrrrrrs*    Q<