a dG(b_@s$dZddlZddlZddlZddlZddlmZddlZ ddl m Z ddl mZddlmZeeZddZGdd d ZGd d d eZGd d d eZGdddeZGdddeZGdddeeZGdddeZGdddeZGdddeZGdddejZGdddeZ dS)z~This module contains classes for analyzing the texts of a corpus to accumulate statistical information about word occurrences.N)Counter)utils)Word2VeccCs`|js"t|ddd|jDt}|D].}|j|}t|trP||}q,||q,|S)a[Convert an iterable of ids to their corresponding words using a dictionary. Abstract away the differences between the HashDictionary and the standard one. Parameters ---------- ids: dict Dictionary of ids and their words. dictionary: :class:`~gensim.corpora.dictionary.Dictionary` Input gensim dictionary Returns ------- set Corresponding words. Examples -------- .. sourcecode:: pycon >>> from gensim.corpora.dictionary import Dictionary >>> from gensim.topic_coherence import text_analysis >>> >>> dictionary = Dictionary() >>> ids = {1: 'fake', 4: 'cats'} >>> dictionary.id2token = {1: 'fake', 2: 'tokens', 3: 'rabbids', 4: 'cats'} >>> >>> text_analysis._ids_to_words(ids, dictionary) set(['cats', 'fake']) id2tokencSsi|]\}}||qSr).0kvrrClib/python3.9/site-packages/gensim/topic_coherence/text_analysis.py 9z!_ids_to_words..)rsetattrtoken2iditemsset isinstanceunionadd)ids dictionaryZ top_wordsword_idwordrrr _ids_to_wordss    rc@sdeZdZdZddZeddZejddZddd Zd d Z d d Z ddZ ddZ ddZ dS) BaseAnalyzeraGBase class for corpus and text analyzers. Attributes ---------- relevant_ids : dict Mapping _vocab_size : int Size of vocabulary. id2contiguous : dict Mapping word_id -> number. log_every : int Interval for logging. _num_docs : int Number of documents. cCs8||_t|j|_ddt|jD|_d|_d|_dS)a= Parameters ---------- relevant_ids : dict Mapping Examples -------- .. sourcecode:: pycon >>> from gensim.topic_coherence import text_analysis >>> ids = {1: 'fake', 4: 'cats'} >>> base = text_analysis.BaseAnalyzer(ids) >>> # should return {1: 'fake', 4: 'cats'} 2 {1: 0, 4: 1} 1000 0 >>> print(base.relevant_ids, base._vocab_size, base.id2contiguous, base.log_every, base._num_docs) {1: 'fake', 4: 'cats'} 2 {1: 0, 4: 1} 1000 0 cSsi|]\}}||qSrr)rnrrrr r mr z)BaseAnalyzer.__init__..irN) relevant_idslen _vocab_size enumerate id2contiguous log_every _num_docs)selfrrrr __init__Ws  zBaseAnalyzer.__init__cCs|jSN)r!r"rrr num_docsqszBaseAnalyzer.num_docscCs.||_|j|jdkr*td|jj|jdS)Nrz&%s accumulated stats from %d documents)r!r loggerinfo __class____name__)r"Znumrrr r&us  NcCs tddS)Nz+Base classes should implement analyze_text.NotImplementedError)r"textdoc_numrrr analyze_text}szBaseAnalyzer.analyze_textcCs,t|tst|ds||S|j|SdS)N__iter__)rstrhasattrget_occurrencesget_co_occurrences)r"Z word_or_wordsrrr __getitem__s zBaseAnalyzer.__getitem__cCs||j|SLReturn number of docs the word occurs in, once `accumulate` has been called.)_get_occurrencesrr"rrrr r3szBaseAnalyzer.get_occurrencescCs tddS)Nz)Base classes should implement occurrencesr+r9rrr r8szBaseAnalyzer._get_occurrencescCs||j||j|SOReturn number of docs the words co-occur in, once `accumulate` has been called.)_get_co_occurrencesrr"word_id1word_id2rrr r4szBaseAnalyzer.get_co_occurrencescCs tddS)Nz,Base classes should implement co_occurrencesr+r=rrr r<sz BaseAnalyzer._get_co_occurrences)N)r* __module__ __qualname____doc__r#propertyr&setterr/r5r3r8r4r<rrrr rFs   rcs8eZdZdZfddZddZddZdd ZZS) UsesDictionaryaA BaseAnalyzer that uses a Dictionary, hence can translate tokens to counts. The standard BaseAnalyzer can only deal with token ids since it doesn't have the token2id mapping. Attributes ---------- relevant_words : set Set of words that occurrences should be accumulated for. dictionary : :class:`~gensim.corpora.dictionary.Dictionary` Dictionary based on text token2id : dict Mapping from :class:`~gensim.corpora.dictionary.Dictionary` cs0tt||t|j||_||_|j|_dS)a Parameters ---------- relevant_ids : dict Mapping dictionary : :class:`~gensim.corpora.dictionary.Dictionary` Dictionary based on text Examples -------- .. sourcecode:: pycon >>> from gensim.topic_coherence import text_analysis >>> from gensim.corpora.dictionary import Dictionary >>> >>> ids = {1: 'foo', 2: 'bar'} >>> dictionary = Dictionary([['foo', 'bar', 'baz'], ['foo', 'bar', 'bar', 'baz']]) >>> udict = text_analysis.UsesDictionary(ids, dictionary) >>> >>> print(udict.relevant_words) set([u'foo', u'baz']) N)superrEr#rrrelevant_wordsrrr"rrr)rr r#szUsesDictionary.__init__cCs6z|j|}Wnty$|}Yn0||j|Sr6)rKeyErrorr8rr"rrrrr r3s   zUsesDictionary.get_occurrencescCs0z|j|}Wnty$|}Yn0|j|Sr$)rrJrrKrrr _word2_contiguous_ids   z#UsesDictionary._word2_contiguous_idcCs ||}||}|||Sr:)rLr<)r"word1word2r>r?rrr r4s  z!UsesDictionary.get_co_occurrences) r*r@rArBr#r3rLr4 __classcell__rrrIr rEs  rEcs8eZdZdZfddZddZddZdd ZZS) InvertedIndexBasedz>Analyzer that builds up an inverted index to accumulate stats.cs0tt|j|tddt|jD|_dS)a Parameters ---------- args : dict Look at :class:`~gensim.topic_coherence.text_analysis.BaseAnalyzer` Examples -------- .. sourcecode:: pycon >>> from gensim.topic_coherence import text_analysis >>> >>> ids = {1: 'fake', 4: 'cats'} >>> ininb = text_analysis.InvertedIndexBased(ids) >>> >>> print(ininb._inverted_index) [set([]) set([])] cSsg|] }tqSr)r)r_rrr r z/InvertedIndexBased.__init__..N)rFrPr#npZarrayranger_inverted_indexr"argsrIrr r#szInvertedIndexBased.__init__cCst|j|Sr$)rrUr9rrr r8sz#InvertedIndexBased._get_occurrencescCs"|j|}|j|}t||Sr$)rUr intersection)r"r>r?s1s2rrr r<s  z&InvertedIndexBased._get_co_occurrencescs,dd|jDfddt|jDS)NcSsi|]\}}||qSrr)rrrrrr r r z4InvertedIndexBased.index_to_dict..csi|]\}}||qSrr)rrZ doc_id_setZ contiguous2idrr r r )rrrrUr%rr[r index_to_dictsz InvertedIndexBased.index_to_dict) r*r@rArBr#r8r<r\rOrrrIr rPs  rPc@s"eZdZdZdddZddZdS)CorpusAccumulatorzTGather word occurrence stats from a corpus by iterating over its BoW representation.NcCsDtdd|D}|j|}|D]}|j|j||jq"dS)8Build an inverted index from a sequence of corpus texts.css|]}|dVqdS)rNr)rxrrr r z1CorpusAccumulator.analyze_text..N) frozensetrrXrUrrr!)r"r-r.Z doc_wordsZtop_ids_in_docrrrr r/s zCorpusAccumulator.analyze_textcCs&|D]}|||jd7_q|S)N)r/r&)r"ZcorpusZdocumentrrr accumulate s zCorpusAccumulator.accumulate)N)r*r@rArBr/rcrrrr r]s r]cs8eZdZdZfddZddZddZdd ZZS) WindowedTextsAnalyzerzVGather some stats about relevant terms of a corpus by iterating over windows of texts.cstt||||j|_dS)z Parameters ---------- relevant_ids : set of int Relevant id dictionary : :class:`~gensim.corpora.dictionary.Dictionary` Dictionary instance with mappings for the relevant_ids. N)rFrdr#r _none_tokenrHrIrr r#s zWindowedTextsAnalyzer.__init__cCsH||}tj||ddd}|D]"\}}||||jd7_q |S)NFT)Zignore_below_sizeZinclude_doc_numrb) _iter_textsrZ iter_windowsr/r&)r"texts window_sizeZrelevant_textsZwindowsr.Zvirtual_documentrrr rc"s   z WindowedTextsAnalyzer.accumulatec#s\ttjjjkrtjntj}|D]2}|r$tjfdd|D|t|dVq$dS)Nc3s.|]&}|jvr jj|njVqdSr$)rGrrre)rwr%rr r`0sz4WindowedTextsAnalyzer._iter_texts..)dtypecount) rSZiinfoZuint16maxruint32text_is_relevantZfromiterr)r"rgrjr-rr%r rf,s  z!WindowedTextsAnalyzer._iter_textscCs|D]}||jvrdSqdS)z)Check if the text has any relevant words.TF)rG)r"r-rrrr rn5s z&WindowedTextsAnalyzer.text_is_relevant) r*r@rArBr#rcrfrnrOrrrIr rds    rdc@seZdZdZdddZdS)InvertedIndexAccumulatorr^NcCs*|D] }||jur|j||jqdSr$)rerUrr!)r"windowr.rrrr r/@s z%InvertedIndexAccumulator.analyze_text)N)r*r@rArBr/rrrr ro=srocsneZdZdZfddZddZddZfdd Zdd d Zd dZ ddZ ddZ ddZ ddZ ZS)WordOccurrenceAccumulatorzOAccumulate word occurrences and co-occurrences from a sequence of corpus texts.cs^tt|j|tj|jdd|_tj|j|jfdd|_ tj|jdft d|_ t |_ dS)Nrm)rjrb)rFrqr#rSZzerosr _occurrencesspsZ lil_matrix_co_occurrencesbool _uniq_wordsr_counterrVrIrr r#Is z"WordOccurrenceAccumulator.__init__cCs|jjSr$)r)r*r%rrr __str__Qsz!WordOccurrenceAccumulator.__str__cCs$|j|_|||||Sr$)rtZtolilpartial_accumulate _symmetrize)r"rgrhrrr rcTs  z$WordOccurrenceAccumulator.accumulatecsRd|_d|_|jtt||||jD]\}}|j||7<q2|S)aSMeant to be called several times to accumulate partial results. Notes ----- The final accumulation should be performed with the `accumulate` method as opposed to this one. This method does not ensure the co-occurrence matrix is in lil format and does not symmetrize it after accumulation. N) _current_doc_num_token_at_edgerwclearrFrqrcrrt)r"rgrhZcomborkrIrr ryZs  z,WordOccurrenceAccumulator.partial_accumulateNcCsV||||jdd}|rR|j|d7<|jtt |dddS)Nr{rbr) _slide_windowrvanyrrrwupdate itertools combinationsrSZnonzero)r"rpr.maskrrr r/ns  z&WordOccurrenceAccumulator.analyze_textcCsX||jkr0d|jdd<d|jt|<||_nd|j|j<d|j|d<|d|_dS)NFTr{r)r|rvrSuniquer})r"rpr.rrr rus  z'WordOccurrenceAccumulator._slide_windowcCs6|j}||j||jtj|ddd|_dS)aWord pairs may have been encountered in (i, j) and (j, i) order. Notes ----- Rather than enforcing a particular ordering during the update process, we choose to symmetrize the co-occurrence matrix after accumulation has completed. rrm)ZoffsetsrjN)rtZsetdiagrrTrsZdiagsZdiagonal)r"Zco_occrrr rzs  z%WordOccurrenceAccumulator._symmetrizecCs |j|Sr$)rrr9rrr r8sz*WordOccurrenceAccumulator._get_occurrencescCs|j||fSr$)rtr=rrr r<sz-WordOccurrenceAccumulator._get_co_occurrencescCs4|j|j7_|j|j7_|j|j7_dSr$)rrrtr!)r"otherrrr mergeszWordOccurrenceAccumulator.merge)N)r*r@rArBr#rxrcryr/rrzr8r<rrOrrrIr rqFs    rqc@seZdZdZddZdS) PatchedWordOccurrenceAccumulatorzaMonkey patched for multiprocessing worker usage, to move some of the logic to the master process.cCs|Sr$r)r"rgrrr rfsz,PatchedWordOccurrenceAccumulator._iter_textsN)r*r@rArBrfrrrr rsrcsZeZdZdZfddZddZddZdd Zd d Zd d Z dddZ ddZ Z S)!ParallelWordOccurrenceAccumulatoraAccumulate word occurrences in parallel. Attributes ---------- processes : int Number of processes to use; must be at least two. args : Should include `relevant_ids` and `dictionary` (see :class:`~UsesDictionary.__init__`). kwargs : Can include `batch_size`, which is the number of docs to send to a worker at a time. If not included, it defaults to 64. cs<tt|j||dkr$td|||_|dd|_dS)Nrz9Must have at least 2 processes to run in parallel; got %d batch_size@)rFrr# ValueError processesgetr)r"rrWkwargsrIrr r#sz*ParallelWordOccurrenceAccumulator.__init__cCsd|jj|j|jfS)Nz%s(processes=%s, batch_size=%s))r)r*rrr%rrr rxsz)ParallelWordOccurrenceAccumulator.__str__cCsf||\}}}z||||d}Wn$tyJtd|jd}Yn0|||||}||S)NFz9stats accumulation interrupted; <= %d documents processedT) start_workersqueue_all_textsKeyboardInterruptr'warnr!terminate_workersmerge_accumulators)r"rgrhworkersinput_qoutput_q interrupted accumulatorsrrr rcs  z,ParallelWordOccurrenceAccumulator.accumulatecCsbtj|jd}t}g}t|jD]2}t|j|j}t||||}|| |q$|||fS)aSet up an input and output queue and start processes for each worker. Notes ----- The input queue is used to transmit batches of documents to the workers. The output queue is used by workers to transmit the WordOccurrenceAccumulator instances. Parameters ---------- window_size : int Returns ------- (list of lists) Tuple of (list of workers, input queue, output queue). )maxsize) mpZQueuerrTrrrAccumulatingWorkerstartappend)r"rhrrrrQ accumulatorworkerrrr rs z/ParallelWordOccurrenceAccumulator.start_workersccsDg}||D]&}||t||jkr|Vg}q|r@|VdS)z\Return a generator over the given texts that yields batches of `batch_size` texts at a time.N)rfrrr)r"rgbatchr-rrr yield_batchess z/ParallelWordOccurrenceAccumulator.yield_batchescst||D]r\}}|j|dd|j|j}|jtfdd|D7_||j|jkrtd|d|d|j|jqdS)zSequentially place batches of texts on the given queue until `texts` is consumed. The texts are filtered so that only those with at least one relevant token are queued. Tblockc3s|]}t|dVqdS)rbN)r)rdocrhrr r`r zDParallelWordOccurrenceAccumulator.queue_all_texts..zG%d batches submitted to accumulate stats from %d documents (%d virtual)rbN) rrputr!r sumr'r(r)r"qrgrh batch_numrZbeforerrr rs  z1ParallelWordOccurrenceAccumulator.queue_all_textsFcCs~|s|D]}|jdddqg}t|t|kr@||q tdt||D]}|rT|qT|||S)a_Wait until all workers have transmitted their WordOccurrenceAccumulator instances, then terminate each. Warnings -------- We do not use join here because it has been shown to have some issues in Python 2.7 (and even in later versions). This method also closes both the input and output queue. If `interrupted` is False (normal execution), a None value is placed on the input queue for each worker. The workers are looking for this sentinel value and interpret it as a signal to terminate themselves. If `interrupted` is True, a KeyboardInterrupt occurred. The workers are programmed to recover from this and continue on to transmit their results before terminating. So in this instance, the sentinel values are not queued, but the rest of the execution continues as usual. NTrz+%d accumulators retrieved from output queue) rrrrr'r(Zis_aliveZ terminateclose)r"rrrrrQrrrrr rs z3ParallelWordOccurrenceAccumulator.terminate_workerscCs<t|j|j}|D]}||q|td|j|S)zMerge the list of accumulators into a single `WordOccurrenceAccumulator` with all occurrence and co-occurrence counts, and a `num_docs` that reflects the total observed by all the individual accumulators. z:accumulated word occurrence stats for %d virtual documents)rqrrrrzr'r(r&)r"rrZother_accumulatorrrr rs  z4ParallelWordOccurrenceAccumulator.merge_accumulators)F) r*r@rArBr#rxrcrrrrrrOrrrIr rs    rcs8eZdZdZfddZddZddZdd ZZS) rz.Accumulate stats from texts fed in from queue.cs4tt|||_||_||_tj|j_||_ dSr$) rFrr#rrrsysrr rh)r"rrrrhrIrr r#.s  zAccumulatingWorker.__init__cCslz\z |WnBty6td|jj|jjYntyPt dYn0W| n | 0dS)Nz,%s interrupted after processing %d documentsz'worker encountered unexpected exception) _runrr'r(r)r*rr& ExceptionZ exceptionreply_to_masterr%rrr run6s    zAccumulatingWorker.runcCszd}d}|d7}|jjdd}|dur2tdqd|j||j|t|7}td|||jjqtd||jjdS) Nr{rrbTrz$observed sentinel value; terminatingz7completed batch %d; %d documents processed (%d virtual)z9finished all batches; %d documents processed (%d virtual)) rrr'debugrryrhrr&)r"rZn_docsZdocsrrr rBs"   zAccumulatingWorker._runcCs*td|jj|jddtddS)Nz.serializing accumulator to return to master...Frzaccumulator serialized)r'r(rrrr%rrr rVs z"AccumulatingWorker.reply_to_master) r*r@rArBr#rrrrOrrrIr r+s   rcsReZdZdZdfdd ZddZddZd d Zd d Zd dZ ddZ Z S)WordVectorsAccumulatoraAccumulate context vectors for words using word vector embeddings. Attributes ---------- model: Word2Vec (:class:`~gensim.models.keyedvectors.KeyedVectors`) If None, a new Word2Vec model is trained on the given text corpus. Otherwise, it should be a pre-trained Word2Vec context vectors. model_kwargs: if model is None, these keyword arguments will be passed through to the Word2Vec constructor. Nc s"tt|||||_||_dSr$)rFrr#model model_kwargs)r"rrrrrIrr r#hszWordVectorsAccumulator.__init__cs$tt|}tfdd|DS)Nc3s|]}|jvr|VqdSr$rrrr%rr r`or z6WordVectorsAccumulator.not_in_vocab..)rrZflatten)r"wordsZ uniq_wordsrr%r not_in_vocabmsz#WordVectorsAccumulator.not_in_vocabcCs<z|j|Wnty,|jj|}Yn0|j|dS)r7rk)rrJrrrZ get_vecattr)r"rrrr r3qs  z&WordVectorsAccumulator.get_occurrencescCs tddS)r;z6Word2Vec model does not support co-occurrence countingNr+)r"rMrNrrr r4ysz)WordVectorsAccumulator.get_co_occurrencescCs|jdurtd|S|j}|dur2||d<|dd|d<|dd|d<|dd|d<tfi||_|j||jj||jj |jj d |jj |_|S) Nz3model is already trained; no accumulation necessaryrpZ min_countrbZsgZhwrZhs)Ztotal_examplesepochs) rr'rrcopyrrZ build_vocabZtrainZ corpus_countrZwv)r"rgrhrrrr rc}s     z!WordVectorsAccumulator.accumulatecCs"||}||}|j||Sr$)_words_with_embeddingsrZ n_similarity)r"Zids1Zids2Zwords1Zwords2rrr ids_similaritys  z%WordVectorsAccumulator.ids_similaritycs4t|ds|g}fdd|D}fdd|DS)Nr0csg|]}jj|qSr)rr)rrr%rr rRr zAWordVectorsAccumulator._words_with_embeddings..csg|]}|jvr|qSrrrr%rr rRr )r2)r"rrrr%r rs z-WordVectorsAccumulator._words_with_embeddings)N) r*r@rArBr#rr3r4rcrrrOrrrIr r\s r)!rBrZloggingZmultiprocessingrr collectionsrZnumpyrSZ scipy.sparseZsparsersZgensimrZgensim.models.word2vecrZ getLoggerr*r'rrrErPr]rdrorqrrZProcessrrrrrr s.     -OC(, T 1