a dG(bP@sdZddlZddlZddlZddlZddlZddlZddlm Z m Z m Z e e ZdZz ddlZWneytYn0Gddde jZddZd d ZGd d d e jZGd dde jZGddde jZGddde jZGddde jZdS)a| Compute similarities across a collection of documents in the Vector Space Model. The main class is :class:`~gensim.similarities.docsim.Similarity`, which builds an index for a given set of documents. Once the index is built, you can perform efficient queries like "Tell me how similar is this query document to each document in the index?". The result is a vector of numbers as large as the size of the initial set of documents, that is, one float for each index document. Alternatively, you can also request only the top-N most similar index documents to the query. How It Works ------------ The :class:`~gensim.similarities.docsim.Similarity` class splits the index into several smaller sub-indexes ("shards"), which are disk-based. If your entire index fits in memory (~one million documents per 1GB of RAM), you can also use the :class:`~gensim.similarities.docsim.MatrixSimilarity` or :class:`~gensim.similarities.docsim.SparseMatrixSimilarity` classes directly. These are more simple but do not scale as well: they keep the entire index in RAM, no sharding. They also do not support adding new document to the index dynamically. Once the index has been initialized, you can query for document similarity simply by .. sourcecode:: pycon >>> from gensim.similarities import Similarity >>> from gensim.test.utils import common_corpus, common_dictionary, get_tmpfile >>> >>> index_tmpfile = get_tmpfile("index") >>> query = [(1, 2), (6, 1), (7, 2)] >>> >>> index = Similarity(index_tmpfile, common_corpus, num_features=len(common_dictionary)) # build the index >>> similarities = index[query] # get similarities between the query and all index documents If you have more query documents, you can submit them all at once, in a batch .. sourcecode:: pycon >>> from gensim.test.utils import common_corpus, common_dictionary, get_tmpfile >>> >>> index_tmpfile = get_tmpfile("index") >>> batch_of_documents = common_corpus[:] # only as example >>> index = Similarity(index_tmpfile, common_corpus, num_features=len(common_dictionary)) # build the index >>> >>> # the batch is simply an iterable of documents, aka gensim corpus: >>> for similarities in index[batch_of_documents]: ... pass The benefit of this batch (aka "chunked") querying is a much better performance. To see the speed-up on your machine, run ``python -m gensim.test.simspeed`` (compare to my results `here `_). There is also a special syntax for when you need similarity of documents in the index to the index itself (i.e. queries = the indexed documents themselves). This special syntax uses the faster, batch queries internally and **is ideal for all-vs-all pairwise similarities**: .. sourcecode:: pycon >>> from gensim.test.utils import common_corpus, common_dictionary, get_tmpfile >>> >>> index_tmpfile = get_tmpfile("index") >>> index = Similarity(index_tmpfile, common_corpus, num_features=len(common_dictionary)) # build the index >>> >>> for similarities in index: # yield similarities of the 1st indexed document, then 2nd... ... pass N) interfacesutilsmatutilsFc@sPeZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ dS)Sharda7A proxy that represents a single shard instance within :class:`~gensim.similarity.docsim.Similarity` index. Basically just wraps :class:`~gensim.similarities.docsim.MatrixSimilarity`, :class:`~gensim.similarities.docsim.SparseMatrixSimilarity`, etc, so that it mmaps from disk on request (query). cCsRtj|\|_|_t||_|j|_t d| | | | |_dS)z Parameters ---------- fname : str Path to top-level directory (file) to traverse for corpus documents. index : :class:`~gensim.interfaces.SimilarityABC` Index object. zsaving index shard to %sN)ospathsplitdirnamefnamelenlength __class__clsloggerinfofullnamesave get_indexindex)selfr rr9lib/python3.9/site-packages/gensim/similarities/docsim.py__init__es  zShard.__init__cCstj|j|jS)zuGet full path to shard file. Return ------ str Path to shard instance. )rrjoinr r rrrrrws zShard.fullnamecCs|jS)z Get length.)r rrrr__len__sz Shard.__len__cCs|j}d|vr|d=|S)zSpecial handler for pickle. Returns ------- dict Object that contains state of current instance without `index`. r)__dict__copy)rresultrrr __getstate__s zShard.__getstate__cCsd|jjt||fS)Nz%s Shard(%i documents in %s))r__name__r rrrrr__str__sz Shard.__str__cCs6t|ds0td||jj|dd|_|jS)zLoad & get index. Returns ------- :class:`~gensim.interfaces.SimilarityABC` Index instance. rzmmaping index from %sr)Zmmap)hasattrrdebugrrloadrrrrrrs zShard.get_indexcCs0d|krt|ks"nJd|j|S)a%Get index vector at position `pos`. Parameters ---------- pos : int Vector position. Return ------ {:class:`scipy.sparse.csr_matrix`, :class:`numpy.ndarray`} Index vector. Type depends on underlying index. Notes ----- The vector is of the same type as the underlying index (ie., dense for :class:`~gensim.similarities.docsim.MatrixSimilarity` and scipy.sparse for :class:`~gensim.similarities.docsim.SparseMatrixSimilarity`. rzrequested position out of range)r rr)rposrrrget_document_ids"zShard.get_document_idcCs@|}z|j|_|j|_Wnty6tdYn0||S)a*Get similarities of document (or corpus) `query` to all documents in the corpus. Parameters ---------- query : {iterable of list of (int, number) , list of (int, number))} Document or corpus. Returns ------- :class:`numpy.ndarray` Similarities of document/corpus if index is :class:`~gensim.similarities.docsim.MatrixSimilarity` **or** :class:`scipy.sparse.csr_matrix` for case if index is :class:`~gensim.similarities.docsim.SparseMatrixSimilarity`. zJnum_best and normalize have to be set before querying a proxy Shard object)rnum_best normalize Exception ValueError)rqueryrrrr __getitem__s  zShard.__getitem__N) r __module__ __qualname____doc__rrrrr!rr'r-rrrrr^s rcCs<|\}}td||jt||}td|t|S)aqHelper for request query from shard, same as shard[query]. Parameters --------- args : (list of (int, number), :class:`~gensim.interfaces.SimilarityABC`) Query and Shard instances Returns ------- :class:`numpy.ndarray` or :class:`scipy.sparse.csr_matrix` Similarities of the query against documents indexed in this shard. z+querying shard %s num_best=%s in process %sz(finished querying shard %s in process %s)rr$r(rgetpid)argsr,shardrrrr query_shards r4cCstj|tj|dddS)a(Helper for extracting n documents with maximum similarity. Parameters ---------- n : int Number of elements to be extracted iterable : iterable of list of (int, float) Iterable containing documents with computed similarities Returns ------- :class:`list` List with the n largest elements from the dataset defined by iterable. Notes ----- Elements are compared by the absolute value of similarity, because negative value of similarity does not mean some form of dissimilarity. cSs t|dS)N)abs)itemrrrz_nlargest..)key)heapqnlargest itertoolschain)niterablerrr _nlargestsrAcseZdZdZd&ddZdd Zd d Zd d ZddZddZ ddZ ddZ ddZ ddZ ddZddZd'ddZd d!Zd(fd"d# Zd$d%ZZS)) SimilarityagCompute cosine similarity of a dynamic query against a corpus of documents ('the index'). The index supports adding new documents dynamically. Notes ----- Scalability is achieved by sharding the index into smaller pieces, each of which fits into core memory The shards themselves are simply stored as files to disk and mmap'ed back as needed. Examples -------- .. sourcecode:: pycon >>> from gensim.corpora.textcorpus import TextCorpus >>> from gensim.test.utils import datapath, get_tmpfile >>> from gensim.similarities import Similarity >>> >>> corpus = TextCorpus(datapath('testcorpus.mm')) >>> index_temp = get_tmpfile("index") >>> index = Similarity(index_temp, corpus, num_features=400) # create index >>> >>> query = next(iter(corpus)) >>> result = index[query] # search similar to `query` in index >>> >>> for sims in index[corpus]: # if you have more query documents, you can submit them all at once, in a batch ... pass >>> >>> # There is also a special syntax for when you need similarity of documents in the index >>> # to the index itself (i.e. queries=indexed documents themselves). This special syntax >>> # uses the faster, batch queries internally and **is ideal for all-vs-all pairwise similarities**: >>> for similarities in index: # yield similarities of the 1st indexed document, then 2nd... ... pass See Also -------- :class:`~gensim.similarities.docsim.MatrixSimilarity` Index similarity (dense with cosine distance). :class:`~gensim.similarities.docsim.SparseMatrixSimilarity` Index similarity (sparse with cosine distance). :class:`~gensim.similarities.docsim.WmdSimilarity` Index similarity (with word-mover distance). Nl2cCsx|durtjdd|_n||_td|j||_||_||_t||_ ||_ g|_ gd|_ |_ |durt||dS)a Parameters ---------- output_prefix : str Prefix for shard filename. If None, a random filename in temp will be used. corpus : iterable of list of (int, number) Corpus in streamed Gensim bag-of-words format. num_features : int Size of the dictionary (number of features). num_best : int, optional If set, return only the `num_best` most similar documents, always leaving out documents with similarity = 0. Otherwise, return a full vector with one float for every document in the index. chunksize : int, optional Size of query chunks. Used internally when the query is an entire corpus. shardsize : int, optional Maximum shard size, in documents. Choose a value so that a `shardsize x chunksize` matrix of floats fits comfortably into your RAM. norm : {'l1', 'l2'}, optional Normalization to use. Notes ----- Documents are split (internally, transparently) into shards of `shardsize` documents each, and each shard converted to a matrix, for faster BLAS calls. Each shard is stored to disk under `output_prefix.shard_number`. If you don't specify an output prefix, a random filename in temp will be used. If your entire index fits in memory (~1 million documents per 1GB of RAM), you can also use the :class:`~gensim.similarities.docsim.MatrixSimilarity` or :class:`~gensim.similarities.docsim.SparseMatrixSimilarity` classes directly. These are more simple but do not scale as well (they keep the entire index in RAM, no sharding). They also do not support adding new document dynamically. NZ simserver)prefixz"starting similarity index under %sr)rZ randfname output_prefixrr num_featuresr(normint chunksize shardsizeshards fresh_docs fresh_nnz add_documents)rrGcorpusrHr(rKrLrIrrrr0s$ zSimilarity.__init__cCst|jtdd|jDS)zGet length of index.css|]}t|VqdSNr .0r3rrr gr9z%Similarity.__len__..)r rNsumrMrrrrreszSimilarity.__len__cCsdt|t|j|jfS)NzASimilarity index with %i documents in %i shards (stored under %s))r rMrGrrrrr!iszSimilarity.__str__cCsd}|jr*t|jd||jkr*||D]}t|tjrHt|}n`tj |r\|j }nLt|}|d|j krt t |g|j j|j}nt t ||j |j}|j||j|7_t|j|jkr|t|jddkr.tdt|jq.dS)aExtend the index with new documents. Parameters ---------- corpus : iterable of list of (int, number) Corpus in BoW format. Notes ----- Internally, documents are buffered and then spilled to disk when there's `self.shardsize` of them (or when a query is issued). Examples -------- .. sourcecode:: pycon >>> from gensim.corpora.textcorpus import TextCorpus >>> from gensim.test.utils import datapath, get_tmpfile >>> from gensim.similarities import Similarity >>> >>> corpus = TextCorpus(datapath('testcorpus.mm')) >>> index_temp = get_tmpfile("index") >>> index = Similarity(index_temp, corpus, num_features=400) # create index >>> >>> one_more_corpus = TextCorpus(datapath('testcorpus.txt')) >>> index.add_documents(one_more_corpus) # add more documents in corpus ?333333?'rzPROGRESS: fresh_shard size=%iN)rMr rL reopen_shard isinstancenumpyndarrayscipysparseissparseZnnzrHrunitvec corpus2cscTrI sparse2fullrNappendrO close_shardrr)rrQZ min_ratiodocZdoclenrrrrPns$    zSimilarity.add_documentscCs,|jdrd|j|fSd|j|fSdS)zGet shard file by `shardid`. Parameters ---------- shardid : int Shard index. Return ------ str Path to shard file. .z%s%sz%s.%sN)rGendswith)rshardidrrrshardid2filenames zSimilarity.shardid2filenamecCs|js dSt|j}dd|jt|j|jk}|rTt|j|jt|j|jd}nt|j|jd}td|rrdnd|t | ||}|j |_ |j|_ |j |gd |_|_dS) aForce the latest shard to close (be converted to a matrix and stored to disk). Do nothing if no new documents added since last call. Notes ----- The shard is closed even if it is not full yet (its size is smaller than `self.shardsize`). If documents are added later via :meth:`~gensim.similarities.docsim.MatrixSimilarity.add_documents` this incomplete shard will be loaded again and completed. NrZrX) num_termsnum_docsnum_nnz)rHzcreating %s shard #%sraZdenser)rNr rMrOrHSparseMatrixSimilarityMatrixSimilarityrrrrmr(rprg)rrlrbrr3rrrrhs   zSimilarity.close_shardcCsd|js J|jrtd|jd}|}tdt|t|j|_|j |_ |jd=t ddS)zReopen an incomplete shard.z3cannot reopen a shard with fresh documents in indexrYz-reopening an incomplete shard of %i documentszreopen completeN) rMrNr+rrrr listrrprOr$)rZ last_shardZ last_indexrrrr\s   zSimilarity.reopen_shardcCspt|gt|j|j}trZtdkrZtdttt}|jt |dt|jtd}nd}t t |}||fS)aoApply shard[query] to each shard in `self.shards`. Used internally. Parameters ---------- query : {iterable of list of (int, number) , list of (int, number))} Document in BoW format or corpus of documents. Returns ------- (None, list of individual shard query results) Query results. r5zspawning %i query processes)rKN) zipr rMPARALLEL_SHARDSrr$multiprocessingZPoolZimapr4map)rr,r2poolrrrr query_shardss     zSimilarity.query_shardsc s@||jD]}|j|_|j|_q||\}}|jdurLtt|}nt dgdd|jDfddt |\}}|pt |do|j dko|jddk}|sЇfd d t|D}t|j|}n^g}t|D]&\}fd d|D}||qg}t|D]} t|j| } || q|r<||S) aGet similarities of the document (or corpus) `query` to all documents in the corpus. Parameters ---------- query : {iterable of list of (int, number) , list of (int, number))} A single document in bag-of-words format, or a corpus (iterable) of such documents. Return ------ :class:`numpy.ndarray` or :class:`scipy.sparse.csr_matrix` Similarities of the query against this index. Notes ----- If `query` is a corpus (iterable of documents), return a matrix of similarities of all query documents vs. all corpus document. This batch query is more efficient than computing the similarities one document after another. Examples -------- .. sourcecode:: pycon >>> from gensim.corpora.textcorpus import TextCorpus >>> from gensim.test.utils import datapath >>> from gensim.similarities import Similarity >>> >>> corpus = TextCorpus(datapath('testcorpus.txt')) >>> index = Similarity('temp', corpus, num_features=400) >>> result = index[corpus] # pairwise similarities of each document against each document NrcSsg|] }t|qSrrSrTrrr +r9z*Similarity.__getitem__..csfdd|DS)Ncs g|]\}}||fqSrr)rUZ doc_indexsim)offsetsshard_norrrz.r9z;Similarity.__getitem__..convert..r)r}ri)r|)r}rconvert-sz'Similarity.__getitem__..convertndimr5c3s|]\}}||VqdSrRr)rUr}r)r~rrrV4r9z)Similarity.__getitem__..csg|]}|qSrr)rUri)r~r}rrrz:r9)rhrMr(rIr)ryr^ZhstackrsZcumsumr is_corpusr#rshape enumeraterArgrtZ terminate) rr,r3rxZ shard_resultsrrZresultsZ shard_resultpartsZmergedr)r~r|r}rr-s2     &   zSimilarity.__getitem__cCst|d}|jD]}|t|7}||krq0q|jrF|dksF||krZtd|t|f|||t|}|S)aGet the indexed vector corresponding to the document at position `docpos`. Parameters ---------- docpos : int Document position Return ------ :class:`scipy.sparse.csr_matrix` Indexed vector. Examples -------- .. sourcecode:: pycon >>> from gensim.corpora.textcorpus import TextCorpus >>> from gensim.test.utils import datapath >>> from gensim.similarities import Similarity >>> >>> # Create index: >>> corpus = TextCorpus(datapath('testcorpus.txt')) >>> index = Similarity('temp', corpus, num_features=400) >>> vector = index.vector_by_id(1) rz3invalid document position: %s (must be 0 <= x < %s))rhrMr r+r')rdocposr&r3rrrr vector_by_idGs  zSimilarity.vector_by_idcCs*||}|jd}|_||}||_|S)a Get similarity of a document specified by its index position `docpos`. Parameters ---------- docpos : int Document position in the index. Return ------ :class:`numpy.ndarray` or :class:`scipy.sparse.csr_matrix` Similarities of the given document against this index. Examples -------- .. sourcecode:: pycon >>> from gensim.corpora.textcorpus import TextCorpus >>> from gensim.test.utils import datapath >>> from gensim.similarities import Similarity >>> >>> corpus = TextCorpus(datapath('testcorpus.txt')) >>> index = Similarity('temp', corpus, num_features=400) >>> similarities = index.similarity_by_id(1) F)rrI)rrr,rIrrrrsimilarity_by_idms  zSimilarity.similarity_by_idccsT|jd}|_|D]2}|jddkr>||D] }|Vq0q||Vq||_dS)ajFor each index document in index, compute cosine similarity against all other documents in the index. Uses :meth:`~gensim.similarities.docsim.Similarity.iter_chunks` internally. Yields ------ :class:`numpy.ndarray` or :class:`scipy.sparse.csr_matrix` Similarities of each document in turn against the index. Frr5N)rI iter_chunksr)rrIchunkr{rrr__iter__s     zSimilarity.__iter__ccsn||dur|j}|jD]L}|j}td|jd|D]*}t|jd||}|||}|Vq>> from gensim.corpora.textcorpus import TextCorpus >>> from gensim.test.utils import datapath, get_tmpfile >>> from gensim.similarities import Similarity >>> >>> temp_fname = get_tmpfile("index") >>> output_fname = get_tmpfile("saved_index") >>> >>> corpus = TextCorpus(datapath('testcorpus.txt')) >>> index = Similarity(output_fname, corpus, num_features=400) >>> >>> index.save(output_fname) >>> loaded_index = index.load(output_fname) N)rhrGsuperrBr)rr r2kwargsr rrrs#zSimilarity.savecCs8ddl}||jdD]}td|t|qdS)uaDelete all files under self.output_prefix Index is not usable anymore after calling this method.rN*z deleting %s)globrGrrrremove)rrr rrrdestroys zSimilarity.destroy)NrCrDrE)N)N)r r.r/r0rrr!rPrmrhr\ryr-rrrrrrr __classcell__rrrrrBs", 53O&  (rBc@s>eZdZdZdejdddfddZddZdd Zd d Z dS) rra>Compute cosine similarity against a corpus of documents by storing the index matrix in memory. Unless the entire matrix fits into main memory, use :class:`~gensim.similarities.docsim.Similarity` instead. Examples -------- .. sourcecode:: pycon >>> from gensim.test.utils import common_corpus, common_dictionary >>> from gensim.similarities import MatrixSimilarity >>> >>> query = [(1, 2), (5, 4)] >>> index = MatrixSimilarity(common_corpus, num_features=len(common_dictionary)) >>> sims = index[query] NrCc Cs|dur tddt|}||_||_d|_||_|durHt|}|dur|jdkrbt dt d||t j ||f|d|_ t|D]f\}}|d dkrtd ||t|t jrn,tj|r|}ntt||}||j |<qdS) aR Parameters ---------- corpus : iterable of list of (int, number) Corpus in streamed Gensim bag-of-words format. num_best : int, optional If set, return only the `num_best` most similar documents, always leaving out documents with similarity = 0. Otherwise, return a full vector with one float for every document in the index. num_features : int Size of the dictionary (number of features). corpus_len : int, optional Number of documents in `corpus`. If not specified, will scan the corpus to determine the matrix size. chunksize : int, optional Size of query chunks. Used internally when the query is an entire corpus. dtype : numpy.dtype, optional Datatype to store the internal matrix in. Nz`scanning corpus to determine the number of features (consider setting `num_features` explicitly)r5Trzzcannot index a corpus with zero features (you must specify either `num_features` or a non-empty corpus in the constructor)z1creating matrix with %i documents and %i features)rdtypeizPROGRESS: at document #%i/%i)rZwarningrZ get_max_idrHr(r)rKr r+rr^emptyrrr$r]r_r`rarbtoarrayflattenrrcrf) rrQr(rrHrKZ corpus_lenZdocnoZvectorrrrr s6    zMatrixSimilarity.__init__cCs |jjdS)NrrrrrrrrCszMatrixSimilarity.__len__cst|\}}|r4tjfdd|Djjd}nDtj|rJ| }nt |tj rXnt |j}tj|jjd}tj|jj}|S)aGet similarity between `query` and this index. Warnings -------- Do not use this function directly, use the :class:`~gensim.similarities.docsim.MatrixSimilarity.__getitem__` instead. Parameters ---------- query : {list of (int, number), iterable of list of (int, number), :class:`scipy.sparse.csr_matrix`} Document or collection of documents. Return ------ :class:`numpy.ndarray` Similarity matrix. csg|]}t|jqSr)rrfrH)rUZvecrrrrz\r9z5MatrixSimilarity.get_similarities..r)rrr^asarrayrrr`rarbrr]r_rrfrHdotrerr,rrrrrget_similaritiesFs   z!MatrixSimilarity.get_similaritiescCsd|jjt||jjdfSN%s<%i docs, %i features>r5)r r r rrrrrrr!nszMatrixSimilarity.__str__) r r.r/r0r^float32rrrr!rrrrrrs :(rrc@s2eZdZdZd ddZddZd d Zd d ZdS)SoftCosineSimilaritya&Compute soft cosine similarity against a corpus of documents by storing the index matrix in memory. Examples -------- .. sourcecode:: pycon >>> from gensim.test.utils import common_texts >>> from gensim.corpora import Dictionary >>> from gensim.models import Word2Vec >>> from gensim.similarities import SoftCosineSimilarity, SparseTermSimilarityMatrix >>> from gensim.similarities import WordEmbeddingSimilarityIndex >>> >>> model = Word2Vec(common_texts, vector_size=20, min_count=1) # train word-vectors >>> termsim_index = WordEmbeddingSimilarityIndex(model.wv) >>> dictionary = Dictionary(common_texts) >>> bow_corpus = [dictionary.doc2bow(document) for document in common_texts] >>> similarity_matrix = SparseTermSimilarityMatrix(termsim_index, dictionary) # construct similarity matrix >>> docsim_index = SoftCosineSimilarity(bow_corpus, similarity_matrix, num_best=10) >>> >>> query = 'graph trees computer'.split() # make a query >>> sims = docsim_index[dictionary.doc2bow(query)] # calculate similarity of query to each doc from bow_corpus Check out `the Gallery `__ for more examples. NrCTTcCs<||_t||_||_||_||_d|_tt ||_ dS)a Parameters ---------- corpus: iterable of list of (int, float) A list of documents in the BoW format. similarity_matrix : :class:`gensim.similarities.SparseTermSimilarityMatrix` A term similarity matrix. num_best : int, optional The number of results to retrieve for a query, if None - return similarities with all elements from corpus. chunksize: int, optional Size of one corpus chunk. normalized : tuple of {True, False, 'maintain'}, optional First/second value specifies whether the query/document vectors in the inner product will be L2-normalized (True; corresponds to the soft cosine similarity measure; default), maintain their L2-norm during change of basis ('maintain'; corresponds to query expansion with partial membership), or kept as-is (False; corresponds to query expansion). See Also -------- :class:`~gensim.similarities.termsim.SparseTermSimilarityMatrix` A sparse term similarity matrix built using a term similarity index. :class:`~gensim.similarities.termsim.LevenshteinSimilarityIndex` A term similarity index that computes Levenshtein similarities between terms. :class:`~gensim.similarities.termsim.WordEmbeddingSimilarityIndex` A term similarity index that computes cosine similarities between word embeddings. FN) similarity_matrixrsrQr(rK normalizedr)r^aranger r)rrQrr(rKrrrrrs zSoftCosineSimilarity.__init__cCs t|jSrRr rQrrrrrszSoftCosineSimilarity.__len__csjstSt|\}}|s>t|tjr>fdd|D}jj|jj d}t j |rnt |St|rt|St |dS)aGet similarity between `query` and this index. Warnings -------- Do not use this function directly; use the `self[query]` syntax instead. Parameters ---------- query : {list of (int, number), iterable of list of (int, number)} Document or collection of documents. Return ------ :class:`numpy.ndarray` Similarity matrix. csg|]}j|qSrrQrUirrrrzr9z9SoftCosineSimilarity.get_similarities..)rr)rQr^arrayrrr]r_rZ inner_productrr`rarbrZtodenseZisscalarrrrrrs   z%SoftCosineSimilarity.get_similaritiescCsd|jjt||jjdfS)Nrr)r r r rrrrrrr!szSoftCosineSimilarity.__str__)NrCrr r.r/r0rrrr!rrrrrrs  - rc@s2eZdZdZd ddZddZdd Zd d ZdS) WmdSimilarityaCompute negative WMD similarity against a corpus of documents. Check out `the Gallery `__ for more examples. When using this code, please consider citing the following papers: * `Ofir Pele and Michael Werman, "A linear time histogram metric for improved SIFT matching" `_ * `Ofir Pele and Michael Werman, "Fast and robust earth mover's distances" `_ * `Matt Kusner et al. "From Word Embeddings To Document Distances" `_ Example ------- .. sourcecode:: pycon >>> from gensim.test.utils import common_texts >>> from gensim.models import Word2Vec >>> from gensim.similarities import WmdSimilarity >>> >>> model = Word2Vec(common_texts, vector_size=20, min_count=1) # train word-vectors >>> >>> index = WmdSimilarity(common_texts, model) >>> # Make query. >>> query = ['trees'] >>> sims = index[query] NrCcCs2||_||_||_||_d|_tt||_dS)a Parameters ---------- corpus: iterable of list of str A list of documents, each of which is a list of tokens. kv_model: :class:`~gensim.models.keyedvectors.KeyedVectors` A set of KeyedVectors num_best: int, optional Number of results to retrieve. chunksize : int, optional Size of chunk. FN) rQwvr(rKr)r^rr r)rrQZkv_modelr(rKrrrrs zWmdSimilarity.__init__cCs t|jS)zGet size of corpus.rrrrrrszWmdSimilarity.__len__csttjrfddDr0tdts6gt}g}t|D]<fddjD}t|}dd|}||qJt|dkr|d}n t|}|S)aGet similarity between `query` and this index. Warnings -------- Do not use this function directly; use the `self[query]` syntax instead. Parameters ---------- query : {list of str, iterable of list of str} Document or collection of documents. Return ------ :class:`numpy.ndarray` Similarity matrix. csg|]}j|qSrrrrrrrz2r9z2WmdSimilarity.get_similarities..rcsg|]}j|qSr)rZ wmdistance)rUZdocumentZqidxr,rrrrz;r9rXr5) r]r^r_rsr rrQrrg)rr,Z n_queriesrZqresultrrrrs        zWmdSimilarity.get_similaritiescCs"d|jjt||jjjjdfSr)r r r Z w2v_modelrZsyn0rrrrrr!JszWmdSimilarity.__str__)NrCrrrrrrs  ,rc@s<eZdZdZddddddejdfddZddZd d ZdS) rqaCompute cosine similarity against a corpus of documents by storing the index matrix in memory. Notes ----- Use this if your input corpus contains sparse vectors (such as TF-IDF documents) and fits into RAM. The matrix is internally stored as a :class:`scipy.sparse.csr_matrix` matrix. Unless the entire matrix fits into main memory, use :class:`~gensim.similarities.docsim.Similarity` instead. Takes an optional `maintain_sparsity` argument, setting this to True causes `get_similarities` to return a sparse matrix instead of a dense representation if possible. See also -------- :class:`~gensim.similarities.docsim.Similarity` Index similarity (wrapper for other inheritors of :class:`~gensim.interfaces.SimilarityABC`). :class:`~gensim.similarities.docsim.MatrixSimilarity` Index similarity (dense with cosine distance). NiFc Cs||_d|_||_| |_|durtdz$|j|j|j}}}t dWnt y`Yn0|durn|}|dur~t ddd|D}t j |||||dd j|_|j|_td |jdS) a Parameters ---------- corpus: iterable of list of (int, float) A list of documents in the BoW format. num_features : int, optional Size of the dictionary. Must be either specified, or present in `corpus.num_terms`. num_terms : int, optional Alias for `num_features`, you can use either. num_docs : int, optional Number of documents in `corpus`. Will be calculated if not provided. num_nnz : int, optional Number of non-zero elements in `corpus`. Will be calculated if not provided. num_best : int, optional If set, return only the `num_best` most similar documents, always leaving out documents with similarity = 0. Otherwise, return a full vector with one float for every document in the index. chunksize : int, optional Size of query chunks. Used internally when the query is an entire corpus. dtype : numpy.dtype, optional Data type of the internal matrix. maintain_sparsity : bool, optional Return sparse arrays from :meth:`~gensim.similarities.docsim.SparseMatrixSimilarity.get_similarities`? TNzcreating sparse indexz%using efficient sparse index creationzPrefusing to guess the number of sparse features: specify num_features explicitlycssD|]<}tj|rt|nt|tjr2t|nt |VqdSrR) r`rarbrZ scipy2sparser]r^r_Z full2sparserc)rUvrrrrVsz2SparseMatrixSimilarity.__init__..r[)rnrorprZ printprogressz created %r)r(r)rKmaintain_sparsityrrrnrorpr$AttributeErrorr+rrdrerZtocsr) rrQrHrnrorpr(rKrrrrrrds0    zSparseMatrixSimilarity.__init__cCs |jjdS)zGet size of index.rrrrrrrszSparseMatrixSimilarity.__len__cCst|\}}|r0tj||jjd|jjd}nntj |rD|j }nZt |t j r|jdkrhdt|f|_tjj||jjdj }ntj|g|jjd|jjd}|j|}|jddkr|s|}n|jr|j }n |j }|S)a)Get similarity between `query` and this index. Warnings -------- Do not use this function directly; use the `self[query]` syntax instead. Parameters ---------- query : {list of (int, number), iterable of list of (int, number), :class:`scipy.sparse.csr_matrix`} Document or collection of documents. Return ------ :class:`numpy.ndarray` Similarity matrix (if maintain_sparsity=False) **OR** :class:`scipy.sparse.csc` otherwise r5r)rrrrdrrrr`rarbrer]r^r_rr Z csr_matrixZtocscrrrrrrrrs"    z'SparseMatrixSimilarity.get_similarities) r r.r/r0r^rrrrrrrrrqNs  >rq)r0Zloggingr=rr;r^Z scipy.sparser`ZgensimrrrZ getLoggerr rrurv ImportErrorZSaveLoadrr4rAZ SimilarityABCrBrrrrrqrrrrs0A   xxzom