o Zh-@sddlZddlZddlmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZmZe rDdd lmZd d gZGd d d eZGdd d eZGdd d eZdS)N) GeneratorSequence)contextmanager)Path)Optional TYPE_CHECKINGUnion) url_to_fs)StreamTransformExtension)FileSystemBaseFileSystemReaderFileSystemWriter)AbstractFileSystem FsspecWriter FsspecReaderc @s4eZdZdddZedeeejfdede e j ddffddZ deeejfd edeeejffd d Z deeejfdeeejffd d ZdeeejfdeeejfddfddZdeeejfddfddZedeeejfdefddZdeeejfdefddZdeeejfddfddZdS) FileSystemreturnNcCs d|_dSN)fs)selfr^/var/www/auris/lib/python3.10/site-packages/torch/distributed/checkpoint/_fsspec_filesystem.py__init__ s zFileSystem.__init__pathmodec #s|jdusJt|}|j|+}z|VWntfdddDr6z||WYWddS1sBwYdS)Nc3s|]}|vVqdSrr).0chrrr 1sz+FileSystem.create_stream..zw+a)rosfspathopenanyrm_file)rrrstreamrrr create_stream#s    "zFileSystem.create_streamsuffixcCstj||Sr)rrjoin)rrr&rrr concat_path8szFileSystem.concat_pathcKst|fi|\|_}|Sr)r r)rrkwargs_rrr init_path=szFileSystem.init_pathnew_pathcCs|j||dSr)rrename)rrr,rrrr-CszFileSystem.renamecCs|jj|dddS)NT)exist_ok)rmakedirsrrrrrmkdirHszFileSystem.mkdir checkpoint_idcCs2t|trdSzt|WdStyYdSw)NFT) isinstancerr ValueErrorclsr2rrrvalidate_checkpoint_idKs   z!FileSystem.validate_checkpoint_idcCs |j|Sr)rexistsr0rrrr8Ws zFileSystem.existscCs|j|dSr)rZrmr0rrrr#ZszFileSystem.rm_file)rN)__name__ __module__ __qualname__rrrstrrPathLikerioIOBaser%r(r+r-r1 classmethodboolr7r8r#rrrrrsD             rcseZdZdZ      ddeeejfdeded e d e d ed e e e d dffdd Z edeeejfd efddZZS)ra` Basic implementation of StorageWriter using FFspec. This implementation makes the following assumptions and simplifications: * The checkpoint path is an empty or non-existing directory. * File creation is atomic The checkpoint consist of one file per write request plus a `.metadata` file with the serialized metadata. T逖Nrsingle_file_per_rank sync_files thread_countper_thread_copy_ahead overwrite _extensionsrc s<tj|||||||dt|_|jj|fi||_dS)a= Initialize the writer pointing to `path`. Args: path: directory where the checkpoint will be written to. single_file_per_rank: Produce one file per rank instead of one file per tensor/blob. Default to True. sync_files : force files to be synced to permanent storage. Default to True. thread_count: Number of IO threads to use to write. Default to 1. per_thread_copy_ahead: How many bytes to copy from the GPU ahead of saving then. Default 10Mb. overwrite: Whether to allow overwriting existing checkpoints. Defaults to True. _extensions: Extensions to apply to output streams (EXPERIMENTAL) N. B. If sync_files is disabled, there's no guarantee that the checkpoint will be consistent in the case of a failure. )rHrINsuperrrrr+r) rrrDrErFrGrHrIr) __class__rrrms zFsspecWriter.__init__r2cC t|Srrr7r5rrrr7 z#FsspecWriter.validate_checkpoint_id)TTrBrCTN)r9r:r;__doc__rr<rr=rAintrrr rr@r7 __classcell__rrrLrr_s6   %&csPeZdZdeeejfddffdd Zedeeejfde fddZ Z S) rrrNc s.t|t|_|jj|fi||_dSrrJ)rrr)rLrrrs zFsspecReader.__init__r2cCrNrrOr5rrrr7rPz#FsspecReader.validate_checkpoint_id) r9r:r;rr<rr=rr@rAr7rSrrrLrrs &)r>rcollections.abcrr contextlibrpathlibrtypingrrrZ fsspec.corer Z'torch.distributed.checkpoint._extensionr Z'torch.distributed.checkpoint.filesystemr r r Zfsspecr__all__rrrrrrrs"     @8