gnes.indexer.base module

class gnes.indexer.base.BaseChunkIndexer(helper_indexer=None, *args, **kwargs)

Bases: gnes.indexer.base.BaseIndexer

Storing chunks and their vector representations

add(keys, vectors, weights, *args, **kwargs)

adding new chunks and their vector representations

Parameters:

• keys (List[Tuple[int, int]]) – list of (doc_id, offset) tuple
• vectors (ndarray) – vector representations
• weights (List[float]) – weight of the chunks

num_chunks

num_chunks_in_doc(doc_id)

num_docs

query(keys, top_k, *args, **kwargs)

Return type:List[List[Tuple]]

query_and_score(q_chunks, top_k, *args, **kwargs)

Return type:List[ScoredResult]

train(*args, **kwargs)

Train the model, need to be overrided

static update_counter(func)

static update_helper_indexer(func)

class gnes.indexer.base.BaseChunkIndexerHelper(helper_indexer=None, *args, **kwargs)

Bases: gnes.indexer.base.BaseChunkIndexer

A helper class for storing chunk info, doc mapping, weights. This is especially useful when ChunkIndexer can not store these information by itself

add(keys, weights, *args, **kwargs)

adding new chunks and their vector representations

Parameters:

• keys (List[Tuple[int, int]]) – list of (doc_id, offset) tuple
• vectors – vector representations
• weights (List[float]) – weight of the chunks

Return type:

int

query(keys, *args, **kwargs)

Return type:List[Tuple[int, int, float]]

train(*args, **kwargs)

Train the model, need to be overrided

class gnes.indexer.base.BaseDocIndexer(normalize_fn=None, score_fn=None, is_big_score_similar=False, *args, **kwargs)

Bases: gnes.indexer.base.BaseIndexer

Storing documents and contents

Base indexer, a valid indexer must implement add() and query() methods

add(keys, docs, *args, **kwargs)

adding new docs and their protobuf representation

Parameters:

• keys (List[int]) – list of doc_id
• docs (List[Document]) – list of protobuf Document objects

query(keys, *args, **kwargs)

Return type:List[Document]

query_and_score(docs, *args, **kwargs)

Return type:List[ScoredResult]

train(*args, **kwargs)

Train the model, need to be overrided

static update_counter(func)

class gnes.indexer.base.BaseIndexer(normalize_fn=None, score_fn=None, is_big_score_similar=False, *args, **kwargs)

Bases: gnes.base.TrainableBase

Base indexer, a valid indexer must implement add() and query() methods

add(keys, docs, weights, *args, **kwargs)

num_chunks

num_docs

query(keys, *args, **kwargs)

Return type:List[Any]

query_and_score(q_chunks, top_k)

Return type:List[ScoredResult]

train(*args, **kwargs)

Train the model, need to be overrided

class gnes.indexer.base.JointIndexer(*args, **kwargs)

Bases: gnes.base.CompositionalTrainableBase

add(keys, docs, *args, **kwargs)

Return type:None

components

query(keys, top_k, *args, **kwargs)

Return type:List[List[Tuple]]

train(*args, **kwargs)

Train the model, need to be overrided