7ThASSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r SSK r SSK J r J r SSKJrJr \R""S5r"SS \ 5r"S S \ 5rg) N)suppress)quote)AbstractBufferedFileAbstractFileSystem)infer_storage_optionstokenizewebhdfsc\^\rSrSrSr\"\R"55rSr SU4Sjjr \ S5r Sr S SjrS!Sjr\S 5r\S 5r\S 5rS rS"S jrSrSrSrS#SjrSrSrSrS$SjrSrSr S"Sjr!Sr"S"Sjr#Sr$Sr%Sr&Sr'U=r($)%WebHDFSa Interface to HDFS over HTTP using the WebHDFS API. Supports also HttpFS gateways. Four auth mechanisms are supported: insecure: no auth is done, and the user is assumed to be whoever they say they are (parameter ``user``), or a predefined value such as "dr.who" if not given spnego: when kerberos authentication is enabled, auth is negotiated by requests_kerberos https://github.com/requests/requests-kerberos . This establishes a session based on existing kinit login and/or specified principal/password; parameters are passed with ``kerb_kwargs`` token: uses an existing Hadoop delegation token from another secured service. Indeed, this client can also generate such tokens when not insecure. Note that tokens expire, but can be renewed (by a previously specified user) and may allow for proxying. basic-auth: used when both parameter ``user`` and parameter ``password`` are provided. )r webHDFSc  ">UR(ag[TU]"S0U D6 U (aSOSSUSUS3UlX0lU=(d 0Ul0UlU =(d 0UlUbUcUb [S5eX@R S'XPl X`l UbUc [S 5eOUbXPR S 'UbXpR S 'U(aUb [S 5eXl Xl UR5 S [X53Ulg)ab Parameters ---------- host: str Name-node address port: int Port for webHDFS kerberos: bool Whether to authenticate with kerberos for this connection token: str or None If given, use this token on every call to authenticate. A user and user-proxy may be encoded in the token and should not be also given user: str or None If given, assert the user name to connect with password: str or None If given, assert the password to use for basic auth. If password is provided, user must be provided also proxy_to: str or None If given, the user has the authority to proxy, and this value is the user in who's name actions are taken kerb_kwargs: dict Any extra arguments for HTTPKerberosAuth, see ``_ data_proxy: dict, callable or None If given, map data-node addresses. This can be necessary if the HDFS cluster is behind a proxy, running on Docker or otherwise has a mismatch between the host-names given by the name-node and the address by which to refer to them from the client. If a dict, maps host names ``host->data_proxy[host]``; if a callable, full URLs are passed, and function must conform to ``url->data_proxy(url)``. use_https: bool Whether to connect to the Name-node using HTTPS instead of HTTP session_cert: str or Tuple[str, str] or None Path to a certificate file, or tuple of (cert, key) files to use for the requests.Session session_verify: str, bool or None Path to a certificate file to use for verifying the requests.Session. kwargs Nhttpshttpz://:z /webhdfs/v1z_If passing a delegation token, must not set user or proxy_to, as these are encoded in the token delegationzQIf passing a password, the user must also beset in order to set up the basic-authz user.namedoaszJIf using Kerberos auth, do not specify the user, this is handled by kinit.webhdfs_)_cachedsuper__init__urlkerb kerb_kwargsparsproxy ValueErroruserpassword session_certsession_verify_connectr _fsid)selfhostportkerberostokenr r!proxy_tor data_proxy use_httpsr"r#kwargs __class__s V/var/www/auris/envauris/lib/python3.13/site-packages/fsspec/implementations/webhdfs.pyrWebHDFS.__init__-s,r <<  "6"!*g7s4&${S &," %2  8#7  ',IIl #   | < )- +&   (IIf  (2  ),  456 cUR$N)r%r&s r0fsid WebHDFS.fsids zzr2c[R"5UlUR(aURURlUR URlUR(a'SSKJ n U"S0URD6URl URbAURb3SSKJn U"URUR5URl ggg)Nr)HTTPKerberosAuth) HTTPBasicAuthr)requestsSessionsessionr"certr#verifyrrequests_kerberosr9rauthr r! requests.authr:)r&r9r:s r0r$WebHDFS._connects'')    $ 1 1DLL "11  99 : 0 D43C3C DDLL  99 T]]%> 3 -dii GDLL &? r2c UbURU5OSnURUR[USS9-5nUR 5nUR UR 5 UR5US'[RSXr5 URRUR5UUUUS9n U RS;a_U R5n U SS n U SS n U S ;a [U 5eU S ;a [U 5eU S ;a [!U 5e[#U 5eU R'5 U $![[$4a N$f=f)Nz/=)safeopzsending %s with %s)methodrparamsdataallow_redirects)iiiiiRemoteExceptionmessage exception)IllegalArgumentExceptionUnsupportedOperationException)SecurityExceptionAccessControlException)FileNotFoundException)_strip_protocol _apply_proxyrrcopyupdaterupperloggerdebugr=request status_codejsonrPermissionErrorFileNotFoundError RuntimeErrorKeyErrorraise_for_status) r&rGrHpathrJredirectr.rargsouterrmsgexps r0_call WebHDFS._callsQ-1-=t##D)25D+A AB{{} DIIXXZT  )37ll""<<>$ #  ??7 7 ,hhj+,Y7+,[9WW$S/)KK)#..55+C00&s++  )  s D==EEc `U=(d URn[UUUUURUUUS9$)a Parameters ---------- path: str File location mode: str 'rb', 'wb', etc. block_size: int Client buffer size for read-ahead or write buffer autocommit: bool If False, writes to temporary file that only gets put in final location upon commit replication: int Number of copies of file on the cluster, write mode only permissions: str or int posix permissions, write mode only kwargs Returns ------- WebHDFile instance )mode block_sizetempdir autocommit replication permissions) blocksize WebHDFilero)r&rcrmrnrprqrrr.s r0_open WebHDFS._opens<B 14>>   !LL!##  r2cBUSR5US'USUS'U$)Ntypelengthsize)lower)infos r0 _process_infoWebHDFS._process_infos*F|))+V H~V  r2c[U5S$)Nrc)r)clsrcs r0rTWebHDFS._strip_protocols$T*622r2c[U5nURSS5 URSS5 SU;aURS5US'U$)Nrcprotocolusernamer )rpop)urlpathrfs r0_get_kwargs_from_urlsWebHDFS._get_kwargs_from_urlssG#G,   D!  ''*-CK r2crURSUS9nUR5SnXS'URU5$)N GETFILESTATUSrc FileStatusname)rjr]r})r&rcrfr|s r0r| WebHDFS.info s<jjtj4xxz,'V !!$''r2cURSUS9nUR5SSnUH1nURU5 URS5S-US-US'M3 U(a [ USS 9$[ S U55$) N LISTSTATUSr FileStatusesr/ pathSuffixrc US$)Nrr)is r0WebHDFS.ls..sqyr2)keyc3*# UH oSv M g7f)rNr).0r|s r0 WebHDFS.ls..s954v,5s)rjr]r}rstripsorted)r&rcdetailrfinfosr|s r0ls WebHDFS.lssjjDj1 >*<8D   t $;;s+c1D4FFDL %%89 99599 9r2cHURSUS9nUR5S$)z8Total numbers of files, directories and bytes under pathGETCONTENTSUMMARYrContentSummaryrjr])r&rcrfs r0content_summaryWebHDFS.content_summarys'jj,4j8xxz*++r2cBURSUSS9nSUR;a\URURS5nURR U5nUR 5 UR 5S$UR 5 UR 5S$)z/Checksum info of file, giving method and resultGETFILECHECKSUMF)rcrdLocation FileChecksum)rjheadersrUr=getrbr])r&rcrflocationout2s r0ukey WebHDFS.ukeysjj*jF  $((Z)@AH<<##H-D  ! ! #99;~. .  "88:n- -r2cJURS5nUR5S$)zGet user's home directoryGETHOMEDIRECTORYPathr)r&rfs r0home_directoryWebHDFS.home_directory+s"jj+,xxz&!!r2cU(aURSUS9nOURS5nUR5SnUc [S5eUS$)zRetrieve token which can give the same authority to other uses Parameters ---------- renewer: str or None User who may use this token; if None, will be current user GETDELEGATIONTOKEN)renewerTokenz1No token available for this user/security context urlString)rjr]r)r&rrfts r0get_delegation_tokenWebHDFS.get_delegation_token0sT **17*CC**12C HHJw  9PQ Q~r2cJURSSUS9nUR5S$)z/Make token live longer. Returns new expiry timeRENEWDELEGATIONTOKENputrHr*longr)r&r*rfs r0renew_delegation_tokenWebHDFS.renew_delegation_tokenAs(jj/UjKxxz&!!r2c&URSSUS9 g)z Stop the token from being usefulCANCELDELEGATIONTOKENrrNrj)r&r*s r0cancel_delegation_tokenWebHDFS.cancel_delegation_tokenFs *5 Fr2c&URSSXS9 g)zSet the permission at path Parameters ---------- path: str location to set (file or directory) mod: str or int posix epresentation or permission, give as oct string, e.g, '777' or 0o777 SETPERMISSIONr)rHrc permissionNr)r&rcmods r0chmod WebHDFS.chmodJs ?5t Lr2cN0nUbX$S'UbX4S'UR"SSUS.UD6 g)zChange owning user and/or groupNownergrouprrHrc)SETOWNERr)r&rcrrr.s r0chown WebHDFS.chownWs7  #7O  #7O Ae$A&Ar2c(URSUSUS9 g)z Set file replication factor Parameters ---------- path: str File location (not for directories) replication: int Number of copies of file on the cluster. Should be smaller than number of data nodes; normally 3 on most systems. SETREPLICATIONr)rcrHrqNr)r&rcrqs r0set_replicationWebHDFS.set_replication`s #$u+ Vr2c &URSSUS9 g)NMKDIRSrrrr&rcr.s r0mkdir WebHDFS.mkdirns 8E 5r2crUSLa!URU5(a [U5eURU5 g)NF)existsFileExistsErrorr)r&rcexist_oks r0makedirsWebHDFS.makedirsqs/ u T!2!2!$' ' 4r2c &URSSXS9 g)NRENAMEr)rHrc destinationr)r&path1path2r.s r0mv WebHDFS.mvvs 8E Ir2c :URSSUU(aSOSS9 g)NDELETEdeletetruefalse)rHrc recursiver)r&rcrr.s r0rm WebHDFS.rmys#  )fw  r2c &URU5 gr4)rrs r0rm_fileWebHDFS.rm_files   r2c URU5nSRURU5S[R"S53/5nURUS5n[ R "XF5 SSS5 URXR5 SSS5 g!,(df  N)=f![a< [[5 URU5 SSS5 e!,(df  e=ff=f!,(df  g=f)Nrz.tmp.wb) openjoin_parentsecrets token_hexshutil copyfileobjr BaseExceptionrr_r)r&lpathrpathr.lstream tmp_fnamerstreams r0cp_fileWebHDFS.cp_files YYu $,,u"5w?P?PQS?T>U7V!WXI YYy$/7&&w80 )  0/! /0GGI&110  sM:C3 B*B6B* B' #B**C0C C0 C, (C00C33 DcUR(a-[UR5(aURU5nU$UR(a5URR5Hup#URX#S5nM U$)N)rcallableitemsreplace)r&rkvs r0rUWebHDFS._apply_proxyse ::(4::..zz(+H  ZZ ((*#++A!4+r2) r%rrrr!rr=r"r#rr ) iFNNNNNNFNT)rNNT)rbNTNNFr4)NN))__name__ __module__ __qualname____firstlineno____doc__strtempfile gettempdirrorrpropertyr6r$rjru staticmethodr} classmethodrTrr|rrrrrrrrrrrrrrrr rU__static_attributes__ __classcell__r/s@r0r r s#*(%%'(G#H  c7JH$H+ Z 33( :, ." "" G MB W6 J  r2r cN^\rSrSrSrU4SjrS SjrSrSrSr Sr S r U=r $) rtiz"A file living in HDFS over webHDFSc >[TU]"X40UD6 UR5nURSS5cUR SS5 URSS5cUR SS5 UR SS5UlUR S5nUR SS5SLaSUR Ul[R RU[[R"555Ulgg)NrrrqirorpF) rrrVrrrrrctargetosrruuiduuid4)r&fsrcr.ror/s r0rWebHDFile.__init__s ,V, ::mT * 2 JJ}d + ::mT * 2 JJ}d +!::mS9**Y' ::lE *e 3))DK Wc$**,.?@DI 4r2cURRRURURR 5SS0S9nUR 5 g)zWrite one part of a multi-block file upload Parameters ========== final: bool This is the last block, so should complete file, if self.autocommit is True. content-typeapplication/octet-stream)rJrT)r*r=postrbuffergetvaluerb)r&finalrfs r0 _upload_chunkWebHDFile._upload_chunksRggoo"" MM%%'#%?@#  r2cVURR5nSUR;aSup#O Sup#SUS'URR"X#UR 4SS0UD6nURR URS5nS UR;aURRRUS S 0S 9nUR5 URR"S SUR 4SS0UD6nURR URS5Ul gg)zCreate remote file/uploada)APPENDPOST)CREATEPUTr overwriterdFrwr-r.)rr7r8N) r.rVrmr*rjrcrUrr=rrbr)r&r.rGrHrfrrs r0_initiate_uploadWebHDFile._initiate_uploads!!# $)) )JB(JB"(F; ggmmB LELVL77'' J(?@ $)) 77??&&>3M"N'D  ! ! #77==6499WuWPVWD GG00j1IJDM r2c[US5n[URU5nX:dXR:agURR SUR XU- SS9nUR 5 SUR;aYURSnURRRURRU55nUR$UR$)Nrr2OPENF)rcoffsetryrdr) maxminrzr*rjrcrbrr=rrUcontent)r&startendrfrrs r0 _fetch_rangeWebHDFile._fetch_rangesE1 $))S! <5II-ggmm 5uu    ${{:.H77??&&tww';';H'EFD<< ;; r2cdURRURUR5 gr4)r*rrcr&r5s r0commitWebHDFile.commits  499dkk*r2cNURRUR5 gr4)r*rrcr5s r0discardWebHDFile.discards  499r2)rrcrrr&r) rrrrrrr3r=rGrJrMr!r"r#s@r0rtrts+, A"K( +r2rt)loggingr'rrrr( contextlibr urllib.parserr;specrrutilsrr getLoggerrYr rtrr2r0rUsW   ;3   9 %E EP I$Ir2