BitSwan Features v2006

BitSwan Features v20.06, launched on 18th of June 2020

New features v20.06:

Influx connection throttling

  • Throttling mechanism was introduced to the InfluxDB Connection, enabling it to become more fluent and reliable.

MySQL features

  • It is now possible to use the autocommit feature in MySQL connection's aiomysql.create_pool.
  • For more information on this feature, visit the official docs.
  • A new MySQL BinaryLogSource was added to extend the general MySQL module usability.

Integrity module

  • Integrity module is a new type of Processor, used for enriching JSON data with hashed events.
  • The data is encoded by JSON Web Token standards.
  • Usage of the new Processor is demonstrated in the example section (bspump-integrity-enricher.py)

Partition specification in the Kafka Source

  • It is now possible to specify topic partitions used by the Kafka Source, using configuration.
  • Partitions allow Kafka users to parallelize a topic by splitting the data in a particular topic across multiple brokers - each partition can be placed on a separate machine to allow for multiple consumers to read from a topic in parallel.

HyperLogLog

  • Implemented a HyperLogLog aggregation/approximation algorithm into the BSpump.
  • HyperLogLog is an algorithm for a count-distinct problem.
  • It is used to approximate the number of distinct elements in a multiset, which reduces computational load on large datasets.
  • An example using this new feature can be found in the examples section (bspump-hyperloglog.py)

BSPump Declarative language

  • A huge new feature, that is primarily connected to the BSPump's Logman.io
  • For the first time, users are able to build BSPump components (such as Pipelines), using declarative .yml configurations.
  • This whole new language is well documented and ready for use, while being extended and improved at the same time.
  • Several examples of this exciting new feature can be found in the examples section (bspump-declarative-*.py)

Windows Remote Management

  • Addition of a whole new source for Windows Remote Management, a SOAP based protocol enabling interoperation of various vendor's operating systems.
  • Usage is demonstrated in the examples (bspump-winrm.py)

Other changes

  • Fix in the subprocess source
  • Removed Elasticsearch connection retry option
  • Introduce changestream mechanic into the mongolookup
  • Add a new generator: Aggregator

  • Code refactoring

You Might Be Interested in Reading These Articles

BitSwan Features v1912

We've just launched new features: InterProcess Communication, Kafka Source profiling, flake8, etc.

Continue reading ...

new release features ipc datagram stream kafka profiling flake8

Published on December 19, 2019

BitSwan Features v1906

We've just launched new features: KafkaKeyFilter filtering over multiple keys, PostgreSQL sink and connection, Mongo query in LatchProcessor, Idle time in File sources, TimeDriftFilter etc.

Continue reading ...

new release features file encoding mongo asynchronous kafka latch time drift

Published on June 30, 2019

BitSwan Features v1911

We've just launched new features: TreeRangeIndex, SliceIndex, bspump.ssh.SFTPSink, bspump.subprocess.SubProcessSource etc.

Continue reading ...

new release features index ssh subprocess

Published on October 31, 2019