BitSwan Features v1907

BitSwan Features v1907, launched on 1st of August 2019

New features v1907:

Google Drive Sink

Google Drive connection and GoogleDriveBlockSink is used to upload individual events as files to Google Drive.

Tableau Sink

New processor named FileTableauSink was added. FileTableauSink is used to output events to Tableau files using specified data schema.

KafkaSink refactoring

KafkaSink receives the KafkaProducer from the KafkaConnection.

Analyzers refactoring

New parameters were added to the constructor: analyze_on_clock and analyze_period; analyze_on_clock enables analysis by timer, which period can be set by analyze_period or Config["analyze_period"].

The performance of analyzers was improved.

Both predicate and evaluate method contain context attribute.

LatchProcessor fully refactored to LatchAnalyzer

The LatchAnalyzer accumulates events in the Latch variable and provides analyzer interface (analyze, evaluate etc.).

Add new native BSPump endpoint: /lookup/{lookup_id}/meta

The /lookup/{lookup_id}/meta endpoint returns information about the lookup (ID, ETAG, URL of the master).

Matrix refactoring

In the constructor, column_names and column_formats parameters were replaced by dtype. NamedMatrixABC was renamed to NamedMatrix etc.

Introducing bspump.cache

Module bspump.cache was introduced with LRUCacheDict implementing the LRU cache strategy: (https://en.wikipedia.org/wiki/Cache_replacement_policies)[https://en.wikipedia.org/wiki/Cache_replacement_policies]

Introducing bspump.random

New processors named RandomDrop and RandomEnricher were added. RandomDrop simulates random drop of events, while RandomEnricher adds to event only one field with the name from Config['field'] and is mostly meant for testing.

Refactoring positioning of arguments in lookups and connections

Every default lookup and connection's constructors have the following header signature: __init__(self, app, id=None, config=None)

MySQL Lookup improvements

Living cursor was removed from MySQL lookup. The cursor is created when necessary, thus the performance and timeout issues were solved.

The configuration option in MySQLookup labeled "from" can now include a table name or a query string including joins.

Filters refactoring of internal code

More unit tests

OOB example extended to include work with ProactorService, which enables running long running tasks on threads

About the Author

Premysl Cerny

Premysl is senior developer in LibertyAces

You Might Be Interested in Reading These Articles

BitSwan Features v2006

We've just launched new features: BSPump Declarative, HyperLogLog, MySQL improvements, WindowsRM, and more!

Continue reading ...

new release bspump-declarative hyperloglog async pipeline kafka windowsrm

Published on June 18, 2020

BitSwan Features v1910

We've just launched new features: Analyzer profiling, bspump.lookup.MatrixLookup, bspump.lookup.BitMapIndex, MongoDBSink etc.

Continue reading ...

new release features analyzer profiler mongodb

Published on October 31, 2019

BitSwan Features v1911

We've just launched new features: TreeRangeIndex, SliceIndex, bspump.ssh.SFTPSink, bspump.subprocess.SubProcessSource etc.

Continue reading ...

new release features index ssh subprocess

Published on October 31, 2019