a h-@sPddlZddlZddlZddlmZddlmZddlmZGdddeZ dS) N)AbstractFileSysteminfer_storage_options) MemoryFilecseZdZdZdZdZdZdZd&fdd Ze d d Z e d'd d Z e ddZ e ddZe ddZd(ddZd)ddZe fddZeddZd*ddZd+d d!Zd,d"d#Zd$d%ZZS)-GithubFileSystemaInterface to files in github An instance of this class provides the files residing within a remote github repository. You may specify a point in the repos history, by SHA, branch or tag (default is current master). For files less than 1 MB in size, file content is returned directly in a MemoryFile. For larger files, or for files tracked by git-lfs, file content is returned as an HTTPFile wrapping the ``download_url`` provided by the GitHub API. When using fsspec.open, allows URIs of the form: - "github://path/file", in which case you must specify org, repo and may specify sha in the extra args - 'github://org:repo@/precip/catalog.yml', where the org and repo are part of the URI - 'github://org:repo@sha/precip/catalog.yml', where the sha is also included ``sha`` can be the full or abbreviated hex of the commit you want to fetch from, or a branch or tag name (so long as it doesn't contain special characters like "/", "?", which would have to be HTTP-encoded). For authorised access, you must provide username and token, which can be made at https://github.com/settings/tokens z9https://api.github.com/repos/{org}/{repo}/git/trees/{sha}zChttps://api.github.com/repos/{org}/{repo}/contents/{path}?ref={sha}Zgithub)<r Nc  stjfi|||_||_|du|duAr6td||_||_|durP||_|durd}tj |j ||dfd|ji|j } | | d}||_|dz ddlm} | fi||_Wntyd|_Yn0dS) Nz%Auth required both username and tokenz)https://api.github.com/repos/{org}/{repo}orgrepotimeoutZdefault_branchr)HTTPFileSystem)super__init__r r ValueErrorusernametokenr requestsgetformatkwraise_for_statusjsonrootlshttprhttp_fs ImportError) selfr r sharrr kwargsurr __class__K/var/www/auris/lib/python3.9/site-packages/fsspec/implementations/github.pyr,s6     zGithubFileSystem.__init__cCs|jrd|j|jfiSiS)Nauth)rrr r'r'r(rJszGithubFileSystem.kwTcCs@tjdddg|d|d|jd}|dd|DS) avList repo names for given org or user This may become the top level of the FS Parameters ---------- org_or_user: str Name of the github org or user to query is_org: bool (default True) Whether the name is an organisation (True) or user (False) Returns ------- List of string zhttps://api.github.com/ZusersZorgs/z/repos)r cSsg|] }|dqSnamer').0r r'r'r( fz*GithubFileSystem.repos..)rrr rr)clsZ org_or_userZis_orgr$r'r'r(reposPs zGithubFileSystem.reposcCsFtjd|jd|jdfd|ji|j}|dd|DS)zNames of tags in the repohttps://api.github.com/repos/r+z/tagsr cSsg|] }|dqSr,r'r.tr'r'r(r/qr0z)GithubFileSystem.tags..rrr r r rrrr r$r'r'r(tagshszGithubFileSystem.tagscCsFtjd|jd|jdfd|ji|j}|dd|DS)zNames of branches in the repor3r+z /branchesr cSsg|] }|dqSr,r'r4r'r'r(r/|r0z-GithubFileSystem.branches..r6r7r'r'r(branchessszGithubFileSystem.branchescCs|j|jdS)z#Named references, tags and branchesr8r9r:r*r'r'r(refs~szGithubFileSystem.refsFc  s|dkr|p|j}|durˆdd}d|p@|j}|D]z}|jd||d}rjd|n|7fdd|D}|st|d}|d d kr|r|gSS|d }qF|jvs||jdfvrftj|j j |j |j |d fd |j i|j} | jdkrt| d ddfdd| dD}||jdfvrp||j<n |j}|rz|Stdd|DSdS)a List files at given path Parameters ---------- path: str Location to list, relative to repo root detail: bool If True, returns list of dicts, one per file; if False, returns list of full filenames only sha: str (optional) List at the given point in the repo history, branch or tag name or commit SHA _sha: str (optional) List this specific tree object (used internally to descend into trees) rNr+T)r!_shacsg|]}|dkr|qSr,r')r.o)so_farr'r(r/r0z'GithubFileSystem.ls..rtypefiler!)r r r!r  directory)Zblobtreec sXg|]P}|dvrr(d|dn|d|d|d|dd|ddqS)r?r+pathmodesizerr!)r-rEr?rFr!)rr.f)rDtypesr'r(r/s   rCcSsg|] }|dqSr,r'rGr'r'r(r/r0)_strip_protocolrrstripsplitrFileNotFoundErrordircacherrurlrr r r r status_coderrsorted) r rDZdetailr!r<r"partspartoutr$r')rDr>rIr(rsP             zGithubFileSystem.lscCs|jdS)N)rNclear)r rDr'r'r(invalidate_cachesz!GithubFileSystem.invalidate_cachecs*t|}d|vrt|S|ddS)NrrDr+)rrrJlstrip)r1rDoptsr%r'r(rJs z GithubFileSystem._strip_protocolcCs>t|}d|vriS|d|dd}|dr:|d|d<|S)Nrpasswordr hostr!r)rDrXrTr'r'r(_get_kwargs_from_urlss z&GithubFileSystem._get_kwargs_from_urlsrbc Ks|dkr t|jj|j|j||p$|jd}tj|fd|ji|j }|j dkrVt || | } | drt| d} | dstdd| S|jdurtd|jj| df|||d |S) Nr\r r rDr!r rAcontents#version https://git-lfs.github.com/zRPlease install fsspec[http] to access github files >1 MB or git-lfs tracked files. download_url)rE block_size cache_options)NotImplementedError content_urlrr r rrrr rrPrMrrbase64 b64decode startswithrrropen) r rDrEr`rar!r"rOr$Z content_jsonr^r'r'r(_opens6     zGithubFileSystem._opencCs0|j|||d}t|D]}|j||dqdS)N) recursivemaxdepth)message)Z expand_pathreversedrm_file)r rDrirjrkpr'r'r(rm s zGithubFileSystem.rmc Ks&|jstd||}||}|s|jj|j|j|d|j d}t j |fd|j i|j }|jdkrrt|||d}|jj|j|j||j d}|j }|pd||d|rd |ini} t j|f| |j d |j }| d d } td | rd} t| |||dS)aW Remove a file from a specified branch using a given commit message. Since Github DELETE operation requires a branch name, and we can't reliably determine whether the provided SHA refers to a branch, tag, or commit, we assume it's a branch. If it's not, the user will encounter an error when attempting to retrieve the file SHA or delete the file. Parameters ---------- path: str The file's location relative to the repository root. message: str, optional The commit message for the deletion. zAuthentication requiredr+r]r rAr!zDelete )rkr!branch)rr rkrzBranch .+ not foundzTRemove only works when the filesystem is initialised from a branch or default (None)N)rrrJ_get_sha_from_cachercrr r rWrrrr rrPrMrrdeleteresearchrV) r rDrkr"r!rOr$Z delete_urlrpdata error_messageerrorr'r'r(rms:     zGithubFileSystem.rm_filecCsL|jD]<}|D]2}|d}|r||krd|vr|dSqq dS)Nr-r!)rNvaluesr)r rDentriesentry entry_pathr'r'r(rqGs  z$GithubFileSystem._get_sha_from_cache)NNNN)T)FNN)N)r\NNN)FNN)N)__name__ __module__ __qualname____doc__rOrcprotocolr rpropertyr classmethodr2r8r9r;rrVrJ staticmethodr[rhrormrq __classcell__r'r'r%r(r s>      B   3  6r) rdrsrspecrutilsrZmemoryrrr'r'r'r(s