gnes.preprocessor.base module

class gnes.preprocessor.base.BaseAudioPreprocessor(uniform_doc_weight: bool = True, *args, **kwargs)[source]

Bases: gnes.preprocessor.base.BasePreprocessor

doc_type = 4
train(*args, **kwargs)

Train the model, need to be overrided

class gnes.preprocessor.base.BaseImagePreprocessor(uniform_doc_weight: bool = True, *args, **kwargs)[source]

Bases: gnes.preprocessor.base.BasePreprocessor

doc_type = 2
train(*args, **kwargs)

Train the model, need to be overrided

class gnes.preprocessor.base.BasePreprocessor(uniform_doc_weight: bool = True, *args, **kwargs)[source]

Bases: gnes.base.TrainableBase

apply(doc: gnes_pb2.Document) → None[source]
doc_type = 0
train(*args, **kwargs)

Train the model, need to be overrided

class gnes.preprocessor.base.BaseTextPreprocessor(uniform_doc_weight: bool = True, *args, **kwargs)[source]

Bases: gnes.preprocessor.base.BasePreprocessor

doc_type = 1
train(*args, **kwargs)

Train the model, need to be overrided

class gnes.preprocessor.base.BaseVideoPreprocessor(uniform_doc_weight: bool = True, *args, **kwargs)[source]

Bases: gnes.preprocessor.base.BasePreprocessor

doc_type = 3
train(*args, **kwargs)

Train the model, need to be overrided

class gnes.preprocessor.base.PipelinePreprocessor(*args, **kwargs)[source]

Bases: gnes.base.CompositionalTrainableBase

apply(doc: gnes_pb2.Document) → None[source]
train(data, *args, **kwargs)[source]

Train the model, need to be overrided

class gnes.preprocessor.base.RawChunkPreprocessor(uniform_doc_weight: bool = True, *args, **kwargs)[source]

Bases: gnes.preprocessor.base.BasePreprocessor

apply(doc: gnes_pb2.Document) → None[source]
train(*args, **kwargs)

Train the model, need to be overrided

class gnes.preprocessor.base.UnaryPreprocessor(doc_type: int, *args, **kwargs)[source]

Bases: gnes.preprocessor.base.BasePreprocessor

apply(doc: gnes_pb2.Document)[source]
is_trained = True
raw_to_chunk(chunk: gnes_pb2.Chunk, raw_bytes: bytes)[source]
train(*args, **kwargs)

Train the model, need to be overrided