BitSwan Features v1907, launched on 1st of August 2019
New features v1907:
Google Drive Sink
Google Drive connection and GoogleDriveBlockSink is used to upload individual events as files to Google Drive.
Tableau Sink
New processor named FileTableauSink was added. FileTableauSink is used to output events to Tableau files using specified data schema.
KafkaSink refactoring
KafkaSink receives the KafkaProducer from the KafkaConnection.
Analyzers refactoring
New parameters were added to the constructor: analyze_on_clock and analyze_period; analyze_on_clock enables analysis by timer, which period can be set by analyze_period or Config["analyze_period"].
The performance of analyzers was improved.
Both predicate and evaluate method contain context attribute.
LatchProcessor fully refactored to LatchAnalyzer
The LatchAnalyzer accumulates events in the Latch variable and provides analyzer interface (analyze, evaluate etc.).
Add new native BSPump endpoint: /lookup/{lookup_id}/meta
The /lookup/{lookup_id}/meta endpoint returns information about the lookup (ID, ETAG, URL of the master).
Matrix refactoring
In the constructor, column_names and column_formats parameters were replaced by dtype. NamedMatrixABC was renamed to NamedMatrix etc.
Introducing bspump.cache
Module bspump.cache was introduced with LRUCacheDict implementing the LRU cache strategy: (https://en.wikipedia.org/wiki/Cache_replacement_policies)[https://en.wikipedia.org/wiki/Cache_replacement_policies]
Introducing bspump.random
New processors named RandomDrop and RandomEnricher were added. RandomDrop simulates random drop of events, while RandomEnricher adds to event only one field with the name from Config['field'] and is mostly meant for testing.
Refactoring positioning of arguments in lookups and connections
Every default lookup and connection's constructors have the following header signature: __init__(self, app, id=None, config=None)
MySQL Lookup improvements
Living cursor was removed from MySQL lookup. The cursor is created when necessary, thus the performance and timeout issues were solved.
The configuration option in MySQLookup labeled "from" can now include a table name or a query string including joins.
