Technology

Open-Source Distributed Stream and Batch Processing – hazelcast/hazelcast-jet

Hazelcast Jet is an open-source, in-memory,
distributed data processing engine. You can use it to process both large
volumes of real-time events and batches of potentially huge, static
datasets.
It provides a Java API to build stream and batch processing applications
through the use of a dataflow programming
model. After you deploy your
application to a Jet cluster, Jet will automatically use all the
computational resources on the cluster to run your application.
If you add more nodes to the cluster while your application is running,
Jet automatically scales up your application to run on the new nodes. If
you remove nodes from the cluster, it scales it down seamlessly without
losing the current computational state, providing exactly-once
processing
guarantees.
For example, you can represent the classical word count problem that
reads some local files and outputs the frequency of each word to console
using the following API:
JetInstance jet =Jet.bootstrappedInstance();
Pipeline p =Pipeline.create();
p.readFrom(Sources.files(“/path/to/text-files”))
.flatMap(line -> traverseArray(line.toLowerCase().split(“\\W+”)))
.filter(word ->!word.isEmpty())
.groupingKey(word -> word)
.aggregate(counting())
.writeTo(Sinks.logger());
jet.newJob(p).join();
and then deploy the application to the cluster:
bin/jet submit word-count.jar
Another application, aggregating millions of sensor readings per
second with 10 millisecond resolution from Kafka looks like the
following:
Pipeline p =Pipeline.create();
p.readFrom(KafkaSources.<String, Reading>kafka(kafkaProperties, “sensors”))
.withTimestamps(event -> event.getValue().timestamp(), 10) // use event timestamp, allowed lag in ms
.groupingKey(reading -> reading.sensorId())
.window(sliding(1_000, 10)) // sliding window of 1s by 10ms
.aggregate(averagingDouble(reading -> reading.temperature()))
.writeTo(Sinks.logger());
jet.newJob(p).join();
Jet comes with out-of-the-box support for many kinds of data sources
and sinks, including:

  • Apache Kafka
  • Local Files (Text, Avro, JSON)
  • Apache Hadoop (Azure Data Lake, S3, GCS)
  • Apache Pulsar
  • Debezium
  • Elasticsearch
  • JDBC
  • JMS
  • InfluxDB
  • Hazelcast
  • Redis
  • MongoDB
  • Twitter

When should you use Jet?
Jet is a good fit when you need to process large amounts of data in a
distributed fashion. You can use it to build a variety of
data-processing applications, such as:

  • Low-latency stateful stream processing. For example, detecting trends
    in 100 Hz sensor data from 100,000 devices and sending corrective
    feedback within 10 milliseconds.
  • High-throughput, large-state stream processing. For example,
    tracking GPS locations of millions of users, inferring their velocity
    vectors.
  • Batch processing of big data volumes, for example analyzing a
    day’s worth of stock trading data to update the risk exposure of a
    given portfolio.

Key Features
Predictable Latency Under Load
Jet uses a unique execution model with cooperative
multithreading
and can achieve extremely low
latencies while
processing millions of items per second on just a single node:
The engine is able to run anywhere from tens to thousands of jobs
concurrently on a fixed number of threads.
Fault Tolerance With No Infrastructure
Jet stores computational state in a distributed, replicated in-memory
store and
does not require the presence of a distributed file system nor
infrastructure like Zookeeper to provide high-availability and
fault-tolerance.
Jet implements a version of the
Chandy-Lamport
algorithm to provide exactly-once processing under the face of
failures. When interfacing with external transactional systems like
databases, it can provide end-to-end processing guarantees using
two-phase
commit.
Advanced Event Processing
Event data can often arrive out of
order and Jet has
first-class support for dealing with this disorder. Jet implements a
technique called distributed
watermarks
to treat disordered events as if they were arriving in order.
How Do I Get Started?
Follow the Get Started
guide to start using Jet.
Download
You can download Jet from
https://jet-start.sh.
Alternatively, you can use the latest docker
image:
docker run -p 5701:5701 hazelcast/hazelcast-jet
Use the following Maven coordinates to add Jet to your application:
<groupId>com.hazelcast.jet</groupId>
<artifactId>hazelcast-jet</artifactId>
<version>4.2</version>
Tutorials
See the tutorials for
tutorials on using Jet. Some examples:
Reference
Jet supports a variety of transforms and operators. These include:
Community
Hazelcast Jet team actively answers questions on Stack
Overflow and
Hazelcast Community Slack.
You are also encouraged to join the hazelcast-jet mailing
list if you are
interested in community discussions
How Can I Contribute?
Thanks for your interest in contributing! The easiest way is to just
send a pull request. Have a look at the issues marked as good first
issue
for some guidance.
Building From Source
To build, use:
./mvnw clean package -DskipTests
Use Latest Snapshot Release
You can always use the latest snapshot release if you want to try the
features currently under development.
Maven snippet:
<repositories>
<repository>
<id>snapshot-repository</id>
<name>Maven2 Snapshot Repository</name>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
<snapshots>
<enabled>true</enabled>
<updatePolicy>daily</updatePolicy>
</snapshots>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>com.hazelcast.jet</groupId>
<artifactId>hazelcast-jet</artifactId>
<version>4.3-SNAPSHOT</version>
</dependency>
</dependencies>
License
Source code in this repository is covered by one of two licenses:

  1. Apache License 2.0
  2. Hazelcast Community
    License

The default license throughout the repository is Apache License 2.0
unless the
header specifies another license. Please see the Licensing
section for more information.
Copyright
Copyright (c) 2008-2020, Hazelcast, Inc. All Rights Reserved.
Visit www.hazelcast.com for more info.