gnes.preprocessor.base module¶

class gnes.preprocessor.base.BaseAudioPreprocessor(uniform_doc_weight=True, *args, **kwargs)[source]¶

Bases: gnes.preprocessor.base.BasePreprocessor

doc_type = 4¶
train(*args, **kwargs)¶

Train the model, need to be overrided

class gnes.preprocessor.base.BaseImagePreprocessor(uniform_doc_weight=True, *args, **kwargs)[source]¶

Bases: gnes.preprocessor.base.BasePreprocessor

doc_type = 2¶
train(*args, **kwargs)¶

Train the model, need to be overrided

class gnes.preprocessor.base.BasePreprocessor(uniform_doc_weight=True, *args, **kwargs)[source]¶

Bases: gnes.base.TrainableBase

apply(doc)[source]¶
Return type:None
doc_type = 0¶
train(*args, **kwargs)¶

Train the model, need to be overrided

class gnes.preprocessor.base.BaseTextPreprocessor(uniform_doc_weight=True, *args, **kwargs)[source]¶

Bases: gnes.preprocessor.base.BasePreprocessor

doc_type = 1¶
train(*args, **kwargs)¶

Train the model, need to be overrided

class gnes.preprocessor.base.BaseVideoPreprocessor(uniform_doc_weight=True, *args, **kwargs)[source]¶

Bases: gnes.preprocessor.base.BasePreprocessor

doc_type = 3¶
train(*args, **kwargs)¶

Train the model, need to be overrided

class gnes.preprocessor.base.PipelinePreprocessor(*args, **kwargs)[source]¶

Bases: gnes.base.CompositionalTrainableBase

apply(doc)[source]¶
Return type:None
train(data, *args, **kwargs)[source]¶

Train the model, need to be overrided

class gnes.preprocessor.base.RawChunkPreprocessor(uniform_doc_weight=True, *args, **kwargs)[source]¶

Bases: gnes.preprocessor.base.BasePreprocessor

apply(doc)[source]¶
Return type:None
train(*args, **kwargs)¶

Train the model, need to be overrided

class gnes.preprocessor.base.UnaryPreprocessor(doc_type, *args, **kwargs)[source]¶

Bases: gnes.preprocessor.base.BasePreprocessor

apply(doc)[source]¶
is_trained = True¶
raw_to_chunk(chunk, raw_bytes)[source]¶
train(*args, **kwargs)¶

Train the model, need to be overrided