o rZhx@sTdZddlZddlZddlmZddlmZmZmZddl m Z GdddeZ dS)z{ A reader for corpora that consist of Tweets. It is assumed that the Tweets have been serialised into line-delimited JSON. N) CorpusReader)StreamBackedCorpusViewZipFilePathPointerconcat)TweetTokenizerc@sNeZdZdZeZ dedfddZdddZddd Z dd d Z d d Z dS)TwitterCorpusReadera7 Reader for corpora that consist of Tweets represented as a list of line-delimited JSON. Individual Tweets can be tokenized using the default tokenizer, or by a custom tokenizer specified as a parameter to the constructor. Construct a new Tweet corpus reader for a set of documents located at the given root directory. If you made your own tweet collection in a directory called `twitter-files`, then you can initialise the reader as:: from nltk.corpus import TwitterCorpusReader reader = TwitterCorpusReader(root='/path/to/twitter-files', '.*\.json') However, the recommended approach is to set the relevant directory as the value of the environmental variable `TWITTER`, and then invoke the reader as follows:: root = os.environ['TWITTER'] reader = TwitterCorpusReader(root, '.*\.json') If you want to work directly with the raw Tweets, the `json` library can be used:: import json for tweet in reader.docs(): print(json.dumps(tweet, indent=1, sort_keys=True)) Nutf8cCsZt||||||jD]}t|trqtj|dkr&t d|dq ||_ dS)a :param root: The root directory for this corpus. :param fileids: A list or regexp specifying the fileids in this corpus. :param word_tokenizer: Tokenizer for breaking the text of Tweets into smaller units, including but not limited to words. rzFile z is emptyN) r__init__abspathsZ_fileids isinstancerospathgetsize ValueError_word_tokenizer)selfrootfileidsZword_tokenizerencodingr rI/var/www/auris/lib/python3.10/site-packages/nltk/corpus/reader/twitter.pyr :s   zTwitterCorpusReader.__init__cs tfdd|ddDS)a( Returns the full Tweet objects, as specified by `Twitter documentation on Tweets `_ :return: the given file(s) as a list of dictionaries deserialised from JSON. :rtype: list(dict) cs$g|]\}}}j|j|dqS))r) CorpusView _read_tweets).0r encZfileidrrr Ysz,TwitterCorpusReader.docs..T)rr )rrrrrdocsNs   zTwitterCorpusReader.docsc CsZ||}g}|D]!}z|d}t|tr||j}||Wq ty*Yq w|S)z Returns only the text content of Tweets in the file(s) :return: the given file(s) as a list of Tweets. :rtype: list(str) text)rr bytesdecoderappendKeyError)rrZ fulltweetstweetsZjsonorrrrstrings_s    zTwitterCorpusReader.stringscs"||}|jfdd|DS)z :return: the given file(s) as a list of the text content of Tweets as as a list of words, screenanames, hashtags, URLs and punctuation symbols. :rtype: list(list(str)) csg|]}|qSr)tokenize)rtZ tokenizerrrr{sz1TwitterCorpusReader.tokenized..)r$r)rrr#rr'r tokenizedrs zTwitterCorpusReader.tokenizedcCs>g}tdD]}|}|s|St|}||q|S)zS Assumes that each line in ``stream`` is a JSON-serialised object. )rangereadlinejsonloadsr!)rstreamr#ilineZtweetrrrr}s   z TwitterCorpusReader._read_tweets)N) __name__ __module__ __qualname____doc__rrrr rr$r(rrrrrrs    r) r4r,r Znltk.corpus.reader.apirZnltk.corpus.reader.utilrrrZ nltk.tokenizerrrrrrs