BitSwan Features v1908

BitSwan Features v1908, launched on 15th of September 2019

New features v1908:

PostgreSQLLogicalReplicationSource

Source for PostgreSQL write ahead / logical replication log was added to allow users stream insert, delete, update events from external postgres databases. The idea behind is to change the pattern of periodical SQL SELECT statement for obtaining the data and instead stream database changes right into the pipeline.

ODBCConnection, ODBCSource and ODBCSink

ODBCConnection was added to allow users to asynchronously connect to external databases using the ODBC standard.

ODBCSource is then used to read data objects from the external database, while ODBCSink is used to post individual events there.

Generator refactorization

Generator was refactored to allow to process events asynchronously and inject them to subsequent pipeline depths.

Generator and OOB joined

"Out-of-band" module was replaced by the refactored version of generator to allow process events asynchronously out of the synchronous context of the pipeline. For more information, see OOB examples: https://github.com/LibertyAces/BitSwanPump/tree/master/examples

Add custom ensure_future to pipeline

The pipelines have method ensure_future to run asynchronous coroutines in an queue to limit the number of running coroutines.

Refactoring of analyzers

Analyzers now expect matrix_id before dtype. The default values of dtype were changed for Session and Time Window analyzers.

Added lazy parameters to lookup

All lookups can now be switched to lazy lookups, where user is expected to call the load method on their own or skip it entirely.

KafkaSource handling of disconnections

If a Kafka consumer is disconnected (due to not sending heart beats etc.), the exception is shown to the user and the consumer is recreated automatically.

MySQL timeout fix

The default connect timeout at MySQLConnection was set to 10 seconds.

Files in chunks at GoogleDriveBlockSink

If there are more files to be sent to Google Drive, they are sent in chunks.

Other changes

  • Add tests for matrices
  • Chilldown period in OpportunisticTrigger is now in seconds
  • Add documentation that LRUCacheDict can be used as cache in MySQLLookup
  • Other round of unit tests was added
  • Fixes in dependencies file
  • Fixes in examples and documentation (typos)

About the Author

Premysl Cerny

Premysl is senior developer in LibertyAces

You Might Be Interested in Reading These Articles

BitSwan Features v2001

We've just launched new features: MySQL AsyncLookup, Pipeline link/unlink, DirectSource, etc.

Continue reading ...

new release features mysql async pipeline direct source

Published on February 11, 2020

BitSwan Features v2003

We've just launched new features: MySQL AsyncLookup, Pipeline link/unlink, DirectSource, etc.

Continue reading ...

new release features anomaly async pipeline kafka

Published on March 16, 2020

BitSwan Features v1903

We've just launched new features: Jupyter server in BitSwan ecosystem, First implementation of pipeline builder, BlankApp for Telco, Open ID connectivity, MySQL Transaction / Binary log, Session Analyzer to hold session data for a specified time, Throttles are now listed when accessing pipelines via REST call, TimeDriftAnalyzer sends fixed zero metrics when there is no history, Kibana Object Library support in bskibana tool. Enjoy and provide us with your feed back.

Continue reading ...

new release features Jupyter pipeline builder BlankApp Open ID MySQL Session Analyzer TimeDriftAnalyzer Kibana

Published on March 11, 2019