JThmSSKrSSKJr SSKJrJrJr SSKrSSKJ r J r SSK J r J r JrJrJrJrJrJrJrJr /SQr\"SS5r"S S \R05rSS \R0S \S \4Sjjr"SS\5r"SS\5rg)N) namedtuple)AnyCallableOptional))sparse_semi_structured_from_dense_cutlass'sparse_semi_structured_to_dense_cutlass) fallback_dispatchersemi_sparse_addmmsemi_sparse_detachsemi_sparse_indicessemi_sparse_linearsemi_sparse_mmsemi_sparse_scaled_mm semi_sparse_tsemi_sparse_valuessemi_sparse_view)SparseSemiStructuredTensor!SparseSemiStructuredTensorCUTLASS$SparseSemiStructuredTensorCUSPARSELTto_sparse_semi_structured_SEMI_STRUCTURED_SPARSE_CONFIGz=sparse_min_rows sparse_min_cols dense_min_rows dense_min_colscF\rSrSr%SrSr\\S'\\ R\ 4\S'Sr \ \S'Sr\ \S'Sr\ \S '\\S '\\\4\S '\\ R&\S '\\ R&\S '\\ R&\S'\\ R&\S'\\ R&\S'\ \S'\\S'/SQr\S*S\ R,S \\ R&S \\ R&S\\ R&S\\ R&S\\ R&S\ S\S\ 4Sjj5rS\4SjrS\\\\\ R,\ \\ 444Sjr\S\\ R,\ \\ 4S\ R&4Sj5r\ R<R>r \S\!4Sj5r"\S+S,Sjj5r#\S\ R&SS4S j5r$\S!\ R&S\ R&4S"j5r%S#r&\S\ R&SS4S$j5r'SS%.S&\ R&S'\\ R&S\ R&4S(jjr(S)r)g)-r&a This class implementes semi-structured sparsity as a Tensor subclass. Semi-structured sparsity describes a sparsity pattern where n in every 2n elements are sparse, depending on the datatype. It is also referred to as 2:4 sparsity or fine-grained structured sparsity. There are two backends available for semi_structred sparsity, either cuSPARSELt or CUTLASS. This class is meant to serve as a base class for both implementations. SparseSemiStructuredCUTLASS and SparseSemiStructuredCUSPARSELT both inherit from this class and define three backend-specific items. Note that as such, this class cannot be insantiated directly. -`_DTYPE_SHAPE_CONSTRAINTS` - A dictionary holding backend specific dense/sparse min shape constraints - `def from_dense()` - backend specific compression routines - `def _mm()` - backend specifc mm op (either torch._cslt_sparse_mm or torch._sparse_semi_structured_(mm|addmm)) r_DEFAULT_ALG_ID_DTYPE_SHAPE_CONSTRAINTSF_FORCE_CUTLASS_FUSE_TRANSPOSE_PROTOTYPE_WARNING_SHOWNBACKENDSPARSE_DISPATCHpackedmetapacked_tmeta_tcompressed_swizzled_bitmaskfuse_transpose_cusparseltalg_id_cusparselt)r!r"r#r$r%shape requires_gradc UR(dQ[R"S[5 SUlUR 5 [ R RU5 UbUn OUbUn O [S5eU RU RU RU S.n [ RR"X40U D6n X,lX[TU5SL$N)getattr)xrFs r<?SparseSemiStructuredTensor.__tensor_flatten__..sWT1-T9r?)listfilter __slots__r(r&r'r))rF inner_tensors tensor_metas` r<__tensor_flatten__-SparseSemiStructuredTensor.__tensor_flatten__sT 94>> J  JJ  * *  " "     ))r?rTc UupVpxU"UURSS5URSS5URSS5URSS5URSS5UUUS9 $)Nr!r"r#r$r% r(r!r"r#r$r%r&r'r))get) r8rSrT outer_size outer_strider(r&r'r)s r<__tensor_unflatten__/SparseSemiStructuredTensor.__tensor_unflatten__sNYJ*; $$Xt4""640"&&z48 $$Xt4(5(9(9-t)'@/'  r?cURUR;a%[URSURS35eURUR"XX45$)NzI only supports a specific set of operations, can't perform requested op (rB)_overloadpacketr NotImplementedErrorrE)r8functypesargsr:s r<__torch_dispatch__-SparseSemiStructuredTensor.__torch_dispatch__sb   s':': :%<<.!//3}}oQ@ ""4#7#78dSSr?Nc[USS5Gc/[RRR[ [RRR [[RRR[[RRR[[RRR[[RRR[[RRR[ [RRR"[$[RRR&[$[RRR([*[RRR,[.[RRR0[[RRR2[40 UlUbUR6R9U5 ggg)zD Loads the op overload sparse dispatch table for the current class. r N)rLr2opsatenvaluesrindicesr is_same_sizer detach_detachr trviewrmmrmatmuladdmmr linearr _to_copy _scaled_mmrr update)r8custom_dispatch_tables r<r1/SparseSemiStructuredTensor._load_dispatch_tablesF 3)4 0 8 %%'9 &&(; ++-@ &&(; %%'9   - ##%5 !!> %%~ $$&7 %%'9 '')< ))+@#C %0##**+@A1! 9r?original_tensorc UR(d[SURS35eUR5S:wa[SUR5S35eUR 5(d [S5eUR UR ;a[SUR SUS 35eURup#UR UR RnUR UR RnX$:dX$-(dX5:d X5-(a[S URS US US 35eg)zO Assert that the given tensor is valid for semi-structured sparse compression. zError original_tensor.device= z= is not supported! Only CUDA tensors are currently supported.zError original_tensor.dim = z; is not supported! Only 2d tensors are currently supported.zXError original_tensor is not contiguous!Only contiguous tensors are currently supported.zError original_tensor.dtype z is not a supported dtype for !zError original_tensor.shape zS is not supported! Both dimensions must be larger or equal than and a multiple of (z, rBN) is_cuda RuntimeErrorr+dim is_contiguousr,rr(sparse_min_rowssparse_min_cols)r8rymnmin_rowsmin_colss r< _validate_device_dim_dtype_shape;SparseSemiStructuredTensor._validate_device_dim_dtype_shapesk &&01G1G0HI==     A %./B/B/D.EF;;  ,,..C   (D(D D./D/D.EEcdgchhij  $$//0E0EFVV//0E0EFVV <1<1<1<./D/D.EFSS[R\\^_g^hhik >> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> A = torch.Tensor([0, 0, 1, 1]).tile((128, 32)).half().cuda() tensor([[0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], ..., [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.]], device='cuda:0', dtype=torch.float16) >>> A_sparse = to_sparse_semi_structured(A) SparseSemiStructuredTensor(shape=torch.Size([128, 128])) >>> A_sparse.values() tensor([[1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], ..., [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.]], device='cuda:0', dtype=torch.float16), >>> A_sparse.indices() tensor([[-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], ..., [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370]], device='cuda:0', dtype=torch.int16)) zSetting transpose from `to_sparse_semi_structured` is deprecated and will be removed in a future release. `SparseSemiStructuredTensor` only support contiguous input tensors.r{) stacklevel) r.r/ FutureWarningrrr2sparserrr)ryrSPARSE_SUBCLASSs r<rr=sab  R    & 4 4  66 \\ > >  % %o 66r?c ^\rSrSrSrSr\R\"SSSS5\R\"SSSS5\R\"SSSS5\R\"SSS S 50r \ S \RS S4S j5rU4S jr\ SS \RS S4Sjj5rSS.S\RS\\RS \R4SjjrSrU=r$)ria This class implements semi-structured sparsity for the CUTLASS backend. In this implementation, the specified elements and metadata are stored seprately, in packed and meta respectively. When _FORCE_CUTLASS is set, or when cuSPARSELt is not available, this subclass calls into _sparse_semi_structured_(mm|addmm) and sparse_semi_structured_from_dense for conversion to the compressed format. cutlass @ryr@c URU5 [U5unnU"URUUSSSURS9$)Nr!r"r#r$r%r))rrr(r))r8rysparse_tensor_cutlassmeta_tensor_cutlasss r<r,SparseSemiStructuredTensorCUTLASS.from_densesU ,,_= 6o F !   ! !($(,)77  r?c>URb URceURRS:Xa [URUR5$[TU]5$)Nr{)r"r!ndimrsuperr)rFrDs r<r*SparseSemiStructuredTensorCUTLASS.to_denses]yy$)@@@ yy~~" 4    !#  r?rc d[R"XSS9unnnnnU"URUUUUUSS9$)a This function takes in a unpruned dense tensor and runs a (branchless) static sort across a 4x4 tile. It greedily picks the largest values in the tile, upholding the 2:4 sparsity constraint across both rows and columns. The algorithm used to prune the matrix is implemented in `_sparse_semi_structured_tile`. Then it creates the packed and meta tensors for the compressed sparse representation of the pruned dense tensor. It also calculates the packed_t and meta_t tensors for the compressed sparse representation of the transposed pruned dense tensor. Since we cannot transpose the compressed representations, we store both for the fw/bw pass respectively. Finally, this function also computes a compressed swizzled bitmask that encodes the sparsity pattern This can be used in the backward pass to mask the gradients. [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to CUTLASS semi-structured -> packed [1 2 6 2] [0 0 6 2] -> metadata -> pack to transposed CUTLASS -> packed_t semi-structured representation -> metadata_t -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same five outputs from the dense tensor can be found below: ``` from torch.sparse import SparseSemiStructuredTensorCUTLASS from torch.sparse._semi_structured_conversions import _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask pruned = _sparse_semi_structured_tile(dense) packed_cutlass, meta_cutlass = sparse_semi_structured_from_dense_cutlass(pruned) packed_t_cutlass, meta_t_cutlass = sparse_semi_structured_from_dense_cutlass(pruned.t().contiguous()) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUTLASS(dense.shape, packed_cutlass, meta_cutlass, packed_t_cutlass, meta_t_cutlass, bitmask) ``` T algorithm use_cutlassFrr2_sparse_semi_structured_tiler(r8ryrr!r"r#r$r%s r<prune_dense_static_sort9SparseSemiStructuredTensorCUTLASS.prune_dense_static_sortsVb  . . d      '   ! !(C  r?Nrrrc [U[5(a [S5eURRnUR S:wdUR S:wa[ SUS35eURb URc[ SUS35eUc-[R"URURU5nO,[R"X RURU5nUSURS$)NZ`SparseSemiStructuredTensor @ SparseSemiStructuredTensor` is not supported by the hardwarer{`)` matmul: Broadcasting is not implemented$` matmul: operation is not supportedr) isinstancerr5rDrErr`r!r"r2_sparse_semi_structured_mm_sparse_semi_structured_addmmr()rFrrr:cls_nameress r<r%SparseSemiStructuredTensorCUTLASS._mms a3 4 4l >>** 99>QVVq[%H:FG  ;; $))"3%H:AB |66t{{DIIqQ99++tyy!A' 'r?r)rErrrrrr2int8rfloat16bfloat16float32rrr6rrrrrr __classcell__)rDs@r<rrs G 22sBC 5b"aC 6r2q!D 5b"aC   #ll ,  $  68< #ll< %< < ~BF(((0(>( ((r?rc \rSrSrSrSr\R\"SSSS5\R\"SSSS5\R\"SSSS5\R\"SSSS50r \ S\RSS4S j5r\ SS\RSS 4S jj5rS S .S\RS\\RS\R4SjjrSrg )ri ab The cuSPARSELt backend expects the specified elements and the metadata to be stored in a single tensor: packed = [ specified elements of original tensor | metadata ] For an original tensor of size (m, k) we expect the first m * k // 2 elements to be the kept elements The rest of the tensor is metadata. Since there is only one tensor, we only use the packed and packed_t attributes respectively. cuSPARSELt also supports transposition fusion, which is necessary for performant 2:4 sparse training, as well as specifying alg_id, a config that affects the performance of the matmul depending on matmul sizes. cusparseltrrrryr@c URU5 U"UR[R"U5SSSS[R [R URS9 $)NrX)rr(r2_cslt_compressrrrr)rs r<r/SparseSemiStructuredTensorCUSPARSELT.from_dense s] ,,_=!''''8(,&@&P&P8HH)77  r?rc d[R"XSS9unnnnnU"URUUUUUSS9$)aZ This function does the same thing as described in SparseSemiStructuredCUTLASS, but uses the cuSPASRELt metadata layout and sparse matmul. The only functional difference is that cuSPARSELt stores `metadata` and `packed` together into a single tensor. [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to cuSPARSELT semi-structured -> packed [1 2 6 2] [0 0 6 2] -> pack to transposed cuSPARSELt -> packed_t semi-structured representation -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same three outputs from the dense tensor can be found below: ``` from torch.sparse import SparseSemiStructuredTensorCUSPARSELT from torch.sparse._semi_structured_conversions import _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask pruned = _sparse_semi_structured_tile(dense) packed_cusparselt = torch._cslt_compress(pruned) packed_t_cusparselt = torch._cslt_compress(pruned.t().contiguous()) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUSPARSELT(dense.shape, packed_cutlass, None, packed_t_cutlass, None, bitmask) ``` Frrrrs r<rQVVq[%DNN++,,UV  77djj %DNN++,,FuTZZGXFYY^_defelel_m^no $ |= BYY    djj 8%DNN++,,FuTZZGXFYY^_defelel_m^no((, |= J\\  ::,, ,%DNN++,,FuTZZGXFYY^_defelel_m^no((, |4``  ;; %DNN++,,PQ '' !%!?!?-- C#<<3557 E# Er?rr)rErrrrrr2rrrrrrrr6rrrrrrr?r<rr s G ;BBK 22r2rB 5b"aC 6r2q!D   #ll /  683 #ll3 %3 3 lBF*F*F(0(>*F *F*Fr?r)F)r. collectionsrtypingrrrr2)torch.sparse._semi_structured_conversionsrr!torch.sparse._semi_structured_opsr r r r r rrrrr__all__rr6rrrrrrr?r<rs"**     ",$C" T"T"rA7\\A7A7 A7HH((BH(VEF+EEFr?