BitSwan Features v1908

BitSwan Features v1908, launched on 15th of September 2019

New features v1908:


Source for PostgreSQL write ahead / logical replication log was added to allow users stream insert, delete, update events from external postgres databases. The idea behind is to change the pattern of periodical SQL SELECT statement for obtaining the data and instead stream database changes right into the pipeline.

ODBCConnection, ODBCSource and ODBCSink

ODBCConnection was added to allow users to asynchronously connect to external databases using the ODBC standard.

ODBCSource is then used to read data objects from the external database, while ODBCSink is used to post individual events there.

Generator refactorization

Generator was refactored to allow to process events asynchronously and inject them to subsequent pipeline depths.

Generator and OOB joined

"Out-of-band" module was replaced by the refactored version of generator to allow process events asynchronously out of the synchronous context of the pipeline. For more information, see OOB examples:

Add custom ensure_future to pipeline

The pipelines have method ensure_future to run asynchronous coroutines in an queue to limit the number of running coroutines.

Refactoring of analyzers

Analyzers now expect matrix_id before dtype. The default values of dtype were changed for Session and Time Window analyzers.

Added lazy parameters to lookup

All lookups can now be switched to lazy lookups, where user is expected to call the load method on their own or skip it entirely.

KafkaSource handling of disconnections

If a Kafka consumer is disconnected (due to not sending heart beats etc.), the exception is shown to the user and the consumer is recreated automatically.

MySQL timeout fix

The default connect timeout at MySQLConnection was set to 10 seconds.

Files in chunks at GoogleDriveBlockSink

If there are more files to be sent to Google Drive, they are sent in chunks.

Other changes

  • Add tests for matrices
  • Chilldown period in OpportunisticTrigger is now in seconds
  • Add documentation that LRUCacheDict can be used as cache in MySQLLookup
  • Other round of unit tests was added
  • Fixes in dependencies file
  • Fixes in examples and documentation (typos)

About the Author

Premysl Cerny

Premysl is senior developer in LibertyAces

You Might Be Interested in Reading These Articles

BitSwan Features v1907

We've just launched new features: Google Drive Sink, Tableau Sink, KafkaSink, analyzers and lookup refactoring, LatchAnalyzer etc.

Continue reading ...

new release features file google drive matrix kafka latch

Published on July 31, 2019

BitSwan Features v2006

We've just launched new features: BSPump Declarative, HyperLogLog, MySQL improvements, WindowsRM, and more!

Continue reading ...

new release bspump-declarative hyperloglog async pipeline kafka windowsrm

Published on June 18, 2020

BitSwan Features v1911

We've just launched new features: TreeRangeIndex, SliceIndex, bspump.ssh.SFTPSink, bspump.subprocess.SubProcessSource etc.

Continue reading ...

new release features index ssh subprocess

Published on October 31, 2019