BitSwan Features v1908

BitSwan Features v1908, launched on 15th of September 2019

New features v1908:

PostgreSQLLogicalReplicationSource

Source for PostgreSQL write ahead / logical replication log was added to allow users stream insert, delete, update events from external postgres databases. The idea behind is to change the pattern of periodical SQL SELECT statement for obtaining the data and instead stream database changes right into the pipeline.

ODBCConnection, ODBCSource and ODBCSink

ODBCConnection was added to allow users to asynchronously connect to external databases using the ODBC standard.

ODBCSource is then used to read data objects from the external database, while ODBCSink is used to post individual events there.

Generator refactorization

Generator was refactored to allow to process events asynchronously and inject them to subsequent pipeline depths.

Generator and OOB joined

"Out-of-band" module was replaced by the refactored version of generator to allow process events asynchronously out of the synchronous context of the pipeline. For more information, see OOB examples: https://github.com/LibertyAces/BitSwanPump/tree/master/examples

Add custom ensure_future to pipeline

The pipelines have method ensure_future to run asynchronous coroutines in an queue to limit the number of running coroutines.

Refactoring of analyzers

Analyzers now expect matrix_id before dtype. The default values of dtype were changed for Session and Time Window analyzers.

Added lazy parameters to lookup

All lookups can now be switched to lazy lookups, where user is expected to call the load method on their own or skip it entirely.

KafkaSource handling of disconnections

If a Kafka consumer is disconnected (due to not sending heart beats etc.), the exception is shown to the user and the consumer is recreated automatically.

MySQL timeout fix

The default connect timeout at MySQLConnection was set to 10 seconds.

Files in chunks at GoogleDriveBlockSink

If there are more files to be sent to Google Drive, they are sent in chunks.

Other changes

  • Add tests for matrices
  • Chilldown period in OpportunisticTrigger is now in seconds
  • Add documentation that LRUCacheDict can be used as cache in MySQLLookup
  • Other round of unit tests was added
  • Fixes in dependencies file
  • Fixes in examples and documentation (typos)

About the Author

Premysl Cerny

Premysl is senior developer in LibertyAces

You Might Be Interested in Reading These Articles

BitSwan Features v1912

We've just launched new features: InterProcess Communication, Kafka Source profiling, flake8, etc.

Continue reading ...

new release features ipc datagram stream kafka profiling flake8

Published on December 19, 2019

BitSwan Analyzers Tutorial

This tutorial is meant to go through all questions the software engineers might encounter during the work with BitSwan Analyzers and Matrices. There is a description of the initialization, parameters to tune, typical usage patterns and recommendations for optimization.

Continue reading ...

analyzer matrix numpy

Published on August 20, 2019

BitSwan Features v2006

We've just launched new features: BSPump Declarative, HyperLogLog, MySQL improvements, WindowsRM, and more!

Continue reading ...

new release bspump-declarative hyperloglog async pipeline kafka windowsrm

Published on June 18, 2020