BitSwan Features v1907

BitSwan Features v1907, launched on 1st of August 2019

New features v1907:

Google Drive Sink

Google Drive connection and GoogleDriveBlockSink is used to upload individual events as files to Google Drive.

Tableau Sink

New processor named FileTableauSink was added. FileTableauSink is used to output events to Tableau files using specified data schema.

KafkaSink refactoring

KafkaSink receives the KafkaProducer from the KafkaConnection.

Analyzers refactoring

New parameters were added to the constructor: analyze_on_clock and analyze_period; analyze_on_clock enables analysis by timer, which period can be set by analyze_period or Config["analyze_period"].

The performance of analyzers was improved.

Both predicate and evaluate method contain context attribute.

LatchProcessor fully refactored to LatchAnalyzer

The LatchAnalyzer accumulates events in the Latch variable and provides analyzer interface (analyze, evaluate etc.).

Add new native BSPump endpoint: /lookup/{lookup_id}/meta

The /lookup/{lookup_id}/meta endpoint returns information about the lookup (ID, ETAG, URL of the master).

Matrix refactoring

In the constructor, column_names and column_formats parameters were replaced by dtype. NamedMatrixABC was renamed to NamedMatrix etc.

Introducing bspump.cache

Module bspump.cache was introduced with LRUCacheDict implementing the LRU cache strategy: (https://en.wikipedia.org/wiki/Cache_replacement_policies)[https://en.wikipedia.org/wiki/Cache_replacement_policies]

Introducing bspump.random

New processors named RandomDrop and RandomEnricher were added. RandomDrop simulates random drop of events, while RandomEnricher adds to event only one field with the name from Config['field'] and is mostly meant for testing.

Refactoring positioning of arguments in lookups and connections

Every default lookup and connection's constructors have the following header signature: __init__(self, app, id=None, config=None)

MySQL Lookup improvements

Living cursor was removed from MySQL lookup. The cursor is created when necessary, thus the performance and timeout issues were solved.

The configuration option in MySQLookup labeled "from" can now include a table name or a query string including joins.

Filters refactoring of internal code

More unit tests

OOB example extended to include work with ProactorService, which enables running long running tasks on threads

About the Author

Premysl Cerny

Premysl is senior developer in LibertyAces

You Might Be Interested in Reading These Articles

BitSwan Features v1906

We've just launched new features: KafkaKeyFilter filtering over multiple keys, PostgreSQL sink and connection, Mongo query in LatchProcessor, Idle time in File sources, TimeDriftFilter etc.

Continue reading ...

new release features file encoding mongo asynchronous kafka latch time drift

Published on June 30, 2019

BitSwan Features v1904

We've just launched new features: SMTPSink, GeoAnalyzer, Content Filter, MySQLSink bulk operations, Code documentation, BSKibana index patterns name fix. Enjoy and provide us with your feed back.

Continue reading ...

new release features SMTP Geo Analyzer MySQL documentation Kibana

Published on April 30, 2019

BitSwan Analyzers Tutorial

This tutorial is meant to go through all questions the software engineers might encounter during the work with BitSwan Analyzers and Matrices. There is a description of the initialization, parameters to tune, typical usage patterns and recommendations for optimization.

Continue reading ...

analyzer matrix numpy

Published on August 20, 2019