Pricewatch: Stream Processing Prices with Reaction
At Reaction we have been continually moving towards a distributed microservice ecosystem. We feel that this offers benefits over the traditional platform that has historically dominated the ecommerce landscape.
In this blog post, we demonstrate using Reaction's powerful data streaming features to re-engage shoppers when product prices change.
Pricewatch is a new project that we've written to monitor changes in prices for products that customers are interested in. When a user indicates the intent to follow a product's price an entry is created in Pricewatch to keep track of this subscription. The actual input trigger is open-ended, but we have it consuming from a "watches" topic that we produce to when someone wants to watch a price. It should be easy to see the many ways in which this could be hooked up to an interface or user-facing panel, but it could also be driven by more arbitrary actions such as an automatic cart inactivity threshold or users staying on a page for a long time. It could even be used internally to monitor changes in competitor pricing for related products!
Pricewatch consumes price changes from Kafka and then detects if those changes are significant enough to notify interested users. We call this a "match". Pricewatch produces match data to an output Kafka topic which could in turn be used to power actions such as emailing or texting users.
We've built Pricewatch as a standalone application that consumes pricing data from Reaction's Pricing Engine, an advanced price manager that we have not yet released. The Pricing Engine publishes all pricing information to Kafka in real-time as prices are set. Pricewatch continually consumes from the
prices-by-id topic that contains an aggregation of all prices for a given product ID.
We realized a few benefits from building this as it's own app, vs. as a Reaction plugin. The application is small, and can be deployed without changing any other Reaction code. This lets us experiment and build quickly. Our scope remained narrow and we didn't have to change Reaction API or Pricing Engine. We've written this app in Clojure to use the event streaming features in Kafka Streams. You could write something similar in any language with Kafka client bindings.
How we store data
We decided to store the state for the Pricewatch app in Kafka topics. The watches are stored in a compacted topic with a composite key of product id and user id. As a result, we don't need an external database to store this state. This reduces the dependencies and complexity of the project. As an added benefit, the watches themselves are streamed and can be consumed by other applications.
We didn't build a management interface to create watches for the demo. We created the records by manually producing to the watches topic using Conduktor. You can also use the Kafka console producer. Here's an example:
docker run --rm -it --network streams.reaction.localhost confluentinc/cp-enterprise-kafka:5.3.1 kafka-console-producer --broker-list=kafka.streams.reaction.localhost:9092 --topic=reaction.pric ewatch.watches.json-gen1 --property "parse.key=true" --property "key.separator=|"
This will open an interactive prompt in which you can send messages. Separate message key and value with a pipe
Kafka Streams provided the tooling to deal with this in our application code. We'll get more into Kafka Streams in a future post.
Ways that the app can be improved from here
This was a fun project that is already somewhat useful in its current state, but there are certainly many more things we could do to improve upon it:
- We could add a user interface to create and manage watches.
- Our Pricewatch rules are hardcoded to a 20% price decrease. We could make this more flexible with dynamic or configurable matching rules.
- Watches are not deleted after a match is found. This made our development easier. In production, we may want to remove a watch after a finding a match.
- The app only works with a single partition because we're using a low-level Kafka Streams processor API to perform the match. Thus the app can scale but can currently only do so vertically, and the entire data set must be on a single application node. We could implement a custom partitioning strategy on the
watchesproducer as a future improvement to make sure
prices-by-idpartitions for a given product are co-located. Then we could horizontally scale Pricewatch nodes to up to the number of
This ended up being a fun, straightforward project. We didn't have to change the Reaction API or any other existing projects or plugins. Since we were not touching other pieces we were able to focus on solutions to the direct problem. This is an indicator to us that we are headed in a good direction when it comes to embracing data streams.
Kafka is a first-class part of the app. We store all data in a compacted Kafka topic instead of using an online database like MongoDB, MySQL, or Redis. With this design, the application's internal state, or "data on the inside" is published as "data on the outside". This could be useful for an instrumentation plane to observe the app. It also opens the application's state to be used by other consumers. The more "open" places you have in your applications, the more you can hook into any time, even if that wasn't intended at the time of writing the application.
We had a great experience composing the existing Reaction Pricing Engine data into a new feature. We've been working on parts of Reaction, like our Pricing Engine, in isolation and for specific use cases. This gave us an opportunity to tap into that data stream to build something that wasn't planned when the Pricing Engine was created. One of our takeaways from doing so is that creating systems to be externally accessible from the beginning is a very helpful pillar, allowing us to build downstream applications like this that use the data in ways the original system never needs to know about. The benefits of accessing a data set over a streaming Kafka topic became tangible, and we are looking forward to doing more of this kind work.
We were able to treat Pricewatch like a greenfield project, which is always fun for developers. We chose the language, databases, dependency selection, etc. We found that this gave us the room to choose tools that we felt were most productive and that gave us a psychological boost.
In terms of Pricewatch specifically, although the app is not quite production ready, we could quite quickly and easily make this a part of Reaction if we wanted to. Although it is already usable in its current state this is a project that could benefit from tighter integration. Perhaps in an admin interface to manage watches, a user settings panel, or a "watch price" button placed directly on product pages.
We hope to get more of the Reaction data flowing over Kafka like in the Pricing Engine so that we have more opportunities to create apps like this. Building services with similar downstream capabilities is very appealing, it opens up opportunities to consume data without modifying producer code (which in this case is our Pricing Engine). We will keep this in mind as we continue to build more systems, and we certainly hope to work on more experiments and create more examples like this.
If you'd like to learn more about how you can use Reaction like this in your org feel free to get in touch with us. If you like to work on software like this on the Reaction team check out our careers page.