BitSwan Features v20.06, launched on 18th of June 2020

New features v20.06:

Influx connection throttling

• Throttling mechanism was introduced to the InfluxDB Connection, enabling it to become more fluent and reliable.

MySQL features

• It is now possible to use the autocommit feature in MySQL connection's aiomysql.create_pool.
• For more information on this feature, visit the official docs.
• A new MySQL BinaryLogSource was added to extend the general MySQL module usability.

Integrity module

• Integrity module is a new type of Processor, used for enriching JSON data with hashed events.
• The data is encoded by JSON Web Token standards.
• Usage of the new Processor is demonstrated in the example section (bspump-integrity-enricher.py)

Partition specification in the Kafka Source

• It is now possible to specify topic partitions used by the Kafka Source, using configuration.
• Partitions allow Kafka users to parallelize a topic by splitting the data in a particular topic across multiple brokers - each partition can be placed on a separate machine to allow for multiple consumers to read from a topic in parallel.

HyperLogLog

• Implemented a HyperLogLog aggregation/approximation algorithm into the BSpump.
• HyperLogLog is an algorithm for a count-distinct problem.
• It is used to approximate the number of distinct elements in a multiset, which reduces computational load on large datasets.
• An example using this new feature can be found in the examples section (bspump-hyperloglog.py)

BSPump Declarative language

• A huge new feature, that is primarily connected to the BSPump's Logman.io
• For the first time, users are able to build BSPump components (such as Pipelines), using declarative .yml configurations.
• This whole new language is well documented and ready for use, while being extended and improved at the same time.
• Several examples of this exciting new feature can be found in the examples section (bspump-declarative-*.py)

Windows Remote Management

• Addition of a whole new source for Windows Remote Management, a SOAP based protocol enabling interoperation of various vendor's operating systems.
• Usage is demonstrated in the examples (bspump-winrm.py)

Other changes

• Fix in the subprocess source
• Removed Elasticsearch connection retry option
• Introduce changestream mechanic into the mongolookup

• Add a new generator: Aggregator

• Code refactoring