BitSwan Features v1908, launched on 15th of September 2019
New features v1908:
PostgreSQLLogicalReplicationSource
Source for PostgreSQL write ahead / logical replication log was added to allow users stream insert, delete, update events from external postgres databases. The idea behind is to change the pattern of periodical SQL SELECT statement for obtaining the data and instead stream database changes
right into the pipeline.
ODBCConnection, ODBCSource and ODBCSink
ODBCConnection was added to allow users to asynchronously connect to external databases using the ODBC standard.
ODBCSource is then used to read data objects from the external database, while ODBCSink is used to post individual events there.
Generator refactorization
Generator was refactored to allow to process events asynchronously and inject them to subsequent pipeline depths.
Generator and OOB joined
"Out-of-band" module was replaced by the refactored version of generator to allow process events asynchronously out of the synchronous context of the pipeline. For more information, see OOB examples: https://github.com/LibertyAces/BitSwanPump/tree/master/examples
Add custom ensure_future to pipeline
The pipelines have method ensure_future to run asynchronous coroutines in an queue to limit the number of running coroutines.
Refactoring of analyzers
Analyzers now expect matrix_id before dtype. The default values of dtype were changed for Session and Time Window analyzers.
Added lazy parameters to lookup
All lookups can now be switched to lazy lookups, where user is expected to call the load method on their own or skip it entirely.
KafkaSource handling of disconnections
If a Kafka consumer is disconnected (due to not sending heart beats etc.), the exception is shown to the user and the consumer is recreated automatically.
MySQL timeout fix
The default connect timeout at MySQLConnection was set to 10 seconds.
Files in chunks at GoogleDriveBlockSink
If there are more files to be sent to Google Drive, they are sent in chunks.
Other changes
Add tests for matricesChilldown period in OpportunisticTrigger is now in secondsAdd documentation that LRUCacheDict can be used as cache in MySQLLookupOther round of unit tests was addedFixes in dependencies fileFixes in examples and documentation (typos)
