gnes.indexer.base module

class gnes.indexer.base.BaseChunkIndexer(helper_indexer=None, *args, **kwargs)[source]

Bases: gnes.indexer.base.BaseIndexer

Storing chunks and their vector representations

add(keys, vectors, weights, *args, **kwargs)[source]

adding new chunks and their vector representations

Parameters:
  • keys (List[Tuple[int, int]]) – list of (doc_id, offset) tuple
  • vectors (ndarray) – vector representations
  • weights (List[float]) – weight of the chunks
num_chunks
num_chunks_in_doc(doc_id)[source]
num_docs
query(keys, top_k, *args, **kwargs)[source]
Return type:List[List[Tuple]]
query_and_score(q_chunks, top_k, *args, **kwargs)[source]
Return type:List[ScoredResult]
train(*args, **kwargs)

Train the model, need to be overrided

static update_counter(func)[source]
static update_helper_indexer(func)[source]
class gnes.indexer.base.BaseChunkIndexerHelper(helper_indexer=None, *args, **kwargs)[source]

Bases: gnes.indexer.base.BaseChunkIndexer

A helper class for storing chunk info, doc mapping, weights. This is especially useful when ChunkIndexer can not store these information by itself

add(keys, weights, *args, **kwargs)[source]

adding new chunks and their vector representations

Parameters:
  • keys (List[Tuple[int, int]]) – list of (doc_id, offset) tuple
  • vectors – vector representations
  • weights (List[float]) – weight of the chunks
Return type:

int

query(keys, *args, **kwargs)[source]
Return type:List[Tuple[int, int, float]]
train(*args, **kwargs)

Train the model, need to be overrided

class gnes.indexer.base.BaseDocIndexer(normalize_fn=None, score_fn=None, is_big_score_similar=False, *args, **kwargs)[source]

Bases: gnes.indexer.base.BaseIndexer

Storing documents and contents

Base indexer, a valid indexer must implement add() and query() methods

add(keys, docs, *args, **kwargs)[source]

adding new docs and their protobuf representation

Parameters:
  • keys (List[int]) – list of doc_id
  • docs (List[Document]) – list of protobuf Document objects
query(keys, *args, **kwargs)[source]
Return type:List[Document]
query_and_score(docs, *args, **kwargs)[source]
Return type:List[ScoredResult]
train(*args, **kwargs)

Train the model, need to be overrided

static update_counter(func)[source]
class gnes.indexer.base.BaseIndexer(normalize_fn=None, score_fn=None, is_big_score_similar=False, *args, **kwargs)[source]

Bases: gnes.base.TrainableBase

Base indexer, a valid indexer must implement add() and query() methods

add(keys, docs, weights, *args, **kwargs)[source]
num_chunks
num_docs
query(keys, *args, **kwargs)[source]
Return type:List[Any]
query_and_score(q_chunks, top_k)[source]
Return type:List[ScoredResult]
train(*args, **kwargs)

Train the model, need to be overrided

class gnes.indexer.base.JointIndexer(*args, **kwargs)[source]

Bases: gnes.base.CompositionalTrainableBase

add(keys, docs, *args, **kwargs)[source]
Return type:None
components
query(keys, top_k, *args, **kwargs)[source]
Return type:List[List[Tuple]]
train(*args, **kwargs)

Train the model, need to be overrided