BitSwan Features v1906, launched on 1st of July 2019
New features v1906:
KafkaKeyFilter filtering over multiple keys
KafkaKeyFilter also allows filtering of Kafka messages based on multiple keys, which can be specified in the
kafka attribute in the context.
Also, optimizations using
frozenset were added to the KafkaKeyFilter class.
PostgreSQL sink and connection
bspump.postgres was added to BSPump, which allows to connect to PostgreSQL server and output pipeline's events there.
Mongo query in LatchProcessor
The latch processor can be queried using Mongo query to obtain specific data from the internal queue. For more information about Mongo query, see https://pypi.org/project/mongoquery/
Idle time in File sources
An idle time was added to all sources based on FileABCSource, which allows other asynchronous operations to perform their tasks. The number of file lines, after which the source will be in idle state, can be configured together with the length of idle time.
TimeDriftFilter is a processor that allows to drop events, if their timestamp (in seconds) is older than current time minus a specified amount of time, set in the configuration.
TimeZoneNormalizer reads time zone from the inserted local time to normalize the time to UTC automatically.
TimeZoneNormalizer was moved to the
Work on inserting processors to existing pipelines
In pipeline object,
insert_after methods were tested to insert processors to existing pipelines and extended to return True or False based on success.
BSPump installation extension
BSPump can now be installed via setup.py and pip, while installing necessary dependencies.
Default encoding in FileABCSource
The default encoding in FileABCSource and hence in all file-based sources was set to