BitSwan Features v1906

BitSwan Features v1906, launched on 1st of July 2019

New features v1906:

KafkaKeyFilter filtering over multiple keys

KafkaKeyFilter also allows filtering of Kafka messages based on multiple keys, which can be specified in the kafka attribute in the context.

Also, optimizations using frozenset were added to the KafkaKeyFilter class.

PostgreSQL sink and connection

New module bspump.postgres was added to BSPump, which allows to connect to PostgreSQL server and output pipeline's events there.

Mongo query in LatchProcessor

The latch processor can be queried using Mongo query to obtain specific data from the internal queue. For more information about Mongo query, see https://pypi.org/project/mongoquery/

Idle time in File sources

An idle time was added to all sources based on FileABCSource, which allows other asynchronous operations to perform their tasks. The number of file lines, after which the source will be in idle state, can be configured together with the length of idle time.

TimeDriftFilter

TimeDriftFilter is a processor that allows to drop events, if their timestamp (in seconds) is older than current time minus a specified amount of time, set in the configuration.

TimeZoneNormalizer refactorization

TimeZoneNormalizer reads time zone from the inserted local time to normalize the time to UTC automatically.

TimeZoneNormalizer was moved to the bspump.common module.

Work on inserting processors to existing pipelines

In pipeline object, insert_before and insert_after methods were tested to insert processors to existing pipelines and extended to return True or False based on success.

BSPump installation extension

BSPump can now be installed via setup.py and pip, while installing necessary dependencies.

Default encoding in FileABCSource

The default encoding in FileABCSource and hence in all file-based sources was set to utf-8.

About the Author

Premysl Cerny

Premysl is senior developer in LibertyAces

You Might Be Interested in Reading These Articles

BitSwan Features v1904

We've just launched new features: SMTPSink, GeoAnalyzer, Content Filter, MySQLSink bulk operations, Code documentation, BSKibana index patterns name fix. Enjoy and provide us with your feed back.

Continue reading ...

new release features SMTP Geo Analyzer MySQL documentation Kibana

Published on April 30, 2019

BitSwan Features v1905

We've just launched new features: Out-of-band module, File Source Encoding, Kafka key support in KafkaSink, KafkaMultiSink, matrices etc. Enjoy and provide us with your feedback.

Continue reading ...

new release features file encoding oob asynchronous kafka

Published on May 30, 2019

BitSwan Features v1903

We've just launched new features: Jupyter server in BitSwan ecosystem, First implementation of pipeline builder, BlankApp for Telco, Open ID connectivity, MySQL Transaction / Binary log, Session Analyzer to hold session data for a specified time, Throttles are now listed when accessing pipelines via REST call, TimeDriftAnalyzer sends fixed zero metrics when there is no history, Kibana Object Library support in bskibana tool. Enjoy and provide us with your feed back.

Continue reading ...

new release features Jupyter pipeline builder BlankApp Open ID MySQL Session Analyzer TimeDriftAnalyzer Kibana

Published on March 11, 2019