BitSwan Features v1908, launched on 15th of September 2019
New features v1908:
Source for PostgreSQL write ahead / logical replication log was added to allow users stream
update events from external postgres databases. The idea behind is to change the pattern of periodical SQL
SELECT statement for obtaining the data and instead stream database changes
right into the pipeline.
ODBCConnection, ODBCSource and ODBCSink
ODBCConnection was added to allow users to asynchronously connect to external databases using the ODBC standard.
ODBCSource is then used to read data objects from the external database, while ODBCSink is used to post individual events there.
Generator was refactored to allow to process events asynchronously and inject them to subsequent pipeline depths.
Generator and OOB joined
"Out-of-band" module was replaced by the refactored version of generator to allow process events asynchronously out of the synchronous context of the pipeline. For more information, see OOB examples: https://github.com/LibertyAces/BitSwanPump/tree/master/examples
ensure_future to pipeline
The pipelines have method
ensure_future to run asynchronous coroutines in an queue to limit the number of running coroutines.
Refactoring of analyzers
Analyzers now expect
dtype. The default values of
dtype were changed for Session and Time Window analyzers.
lazy parameters to lookup
All lookups can now be switched to lazy lookups, where user is expected to call the
load method on their own or skip it entirely.
KafkaSource handling of disconnections
If a Kafka consumer is disconnected (due to not sending heart beats etc.), the exception is shown to the user and the consumer is recreated automatically.
MySQL timeout fix
The default connect timeout at MySQLConnection was set to 10 seconds.
Files in chunks at GoogleDriveBlockSink
If there are more files to be sent to Google Drive, they are sent in chunks.
Add tests for matrices
Chilldown period in OpportunisticTrigger is now in seconds
Add documentation that LRUCacheDict can be used as cache in MySQLLookup
Other round of unit tests was added
Fixes in dependencies file
Fixes in examples and documentation (typos)