Temporal Workflows with ZIO: Introduction

Series: Temporal Workflows with ZIO

Introduction

Do you spend a lot of time making your applications resilient: dealing with distributed state and locks, migrating from sync to async communication, adding retries, tracing?

That’s a usual programming routine that we, as engineers, must care about. On the other hand, it shifts our focus to technical issues instead of the original business problem.

Can we change the status quo? Are there any tools to help us struggle less and to be more productive?

Meet Temporal — a distributed workflow management system for building invincible apps. It handles most technical problems, such as scaling, transactivity, managing state, etc. Temporal lets you focus on business needs and produce value quickly.

Temporal Applications are written using Temporal SDK targetting most mainstream languages, such as Java, Python, Go, JavaScript and so on.

ZIO Temporal is a Scala SDK for Temporal implemented with ZIO . It allows working with Temporal naturally as though it was developed for Scala. It handles most typical programmer errors at compile time. It also brings seamless ZIO interop!

In this series of articles, you will see how to solve kinda real business problems using the concept of Workflows.

About the business problem

We’re going to develop a Content synchronization platform (let’s call it the content sync app).
TL;DR the source code is already available on GitHub

As a regular person, I struggle to check various sources of content (newsletters, videos on YouTube, etc). I wish there were an aggregator fetching content I want and making recommendations for me. For instance, a Telegram Bot

Technical details

While it sounds pretty easy, the content sync app requires a thoughtful design in case reliability and scalability are a requirement.

The components of such a platform can be defined as follows:

  • Data ingestion component that is responsible for fetching the data. For simplicity, let’s call it puller
  • Data processing component that is responsible for getting value out of data and for making recommendations for the end users. Let’s call it processor
  • Frontend API serving the data for users. In our case, it’s gonna be the Telegram Bot interacting with the end user. Let’s call it simply telegram bot.

Those components require:

  • Scheduler to trigger the execution of components
    • Note: that’s the requirement, as many content sources don’t support real-time updates streams
  • Syncrhonization mechamism to run and supervise the content sync process end-to-end
  • Reliability facilities in each component to guarantee proper error handling, retries and isolation (to localize the possible impact of errors, therefore avoiding global outages)
  • Storage layer for raw data and processed data (can be the same or two separate storages)
  • User session store for internal application-level data (for UI-related functionality).

Taking those requirements into account, at the first glance, we might need:

  • Scheduler library like Quartz with a relation database as the state store
  • File system for raw data
  • Relational database (such as Postgres) for processed data
  • Message Queue (like RabbitMQ or Apache Kafka) for intra-component communication
  • Key-value store like Redis for session data

It’s a pretty big list of components to maintain, isn’t it?
No need for this. Temporal platform can reduce this list to just:

  • Temporal cluster
  • File system
  • Relational database

What is Temporal?

Temporal is an open-source workflow management system. Temporal Application implements the business logic in terms of Workflows. Workflows are then executed by the Temporal Server that takes care of retries, persistency and so on.

Workflow vs Activity

Two main building parts of Temporal are Activity and Workflow:

  1. Activity is all the hard work and technical details. Activities perform error-prone operations (such as interactions with external systems and APIs), complex algorithms, etc.
  2. Workflow is the business process definition represented as code. Workflows implement the business logic using Activities. A Workflow can also spawn and supervise Child Workflows.

Workflow definition

The Workflow definition consists of the following particles:

  1. Workflow method that contains the business logic. It is mandatory for the Workflow definition, while other methods are optional.
  2. Signal method that allows to interact with a running workflow from the outside. Workflow definition may have zero or multiple signal methods defined.
  3. Query method that allows inspecting the business process state

The Workflow is defined using the methods above. They are implemented with code, using the Temporal SDK targetting your programming language. You’re free to use any constructs of your favorite programming language as long as it is Deterministic (we will talk about this later).

Workflow execution

Workflows are executed by the Temporal Cluster. The Cluster consists of multiple internal components:

  • External database (like PostgreSQL, MySQL, Cassandra, or SQLite for local development)
  • Internal message queue
  • Components responsible for task management and observability-
  • Server API for Workflow Scheduling
  • Web UI

You can find more information about the Temporal Cluster and how it is deployed in the official Temporal Cluster documentation .

Typically, the Temporal Application consists of the Client-side and the Worker.

Client-side applications schedule Workflow execution using Temporal SDK via the Temporal Server API.

Temporal Server then routes the Workflow to a Task Queue (specific by the client-side), so that a Worker application can pick it up.

Workers execute the Workflow code written with the Temporal SDK. Whenever a Workflow needs to execute an Activity (or Child Workflow), the Temporal Server finds a suitable Worker to execute it (could be the same or another Worker) and stores the result in the Event storage (the database).

Workflow can suspend its execution for some time until a Singal is received or a certain condition is met. The Worker doesn’t waste resources while a Workflow execution is suspended.

This is a very powerful approach, allowing reliable retries of failed Workflows. For instance, in case the Workflow invokes two Activities, if the first Activity invocation succeeds, but the second one fails, only the second is retried. Remember, the result of the first Activity invocation is persisted into an event storage, so there is no need to retry the entire Workflow. The same logic applies to Child Workflows.

Let’s go back to the Determinism restriction for the Workflow logic. In case of failures, the Workflow execution will be retried by the Temporal Server. The steps that succeeded won’t be re-executed (as stated above). Instead, the Workflow code will operate with the results from the Event storage.
Therefore, it is important for the Workflow logic to act deterministically so that the flow goes the same way given historical events.
All non-deterministic code must be encapsulated into Activities.

Content Sync Architecture

Now we can go back to the business task.
Let’s get ahead a little to the Content sync platform architecture using Temporal! Content sync architecture

Important notes here:

  1. The content sources are YouTube API (for videos) and News API (for articles)
  2. The main components of the platform (Puller, Telegram Bot) are implemented using Temporal Workflows and ZIO
  3. The Telegram Bot is used to configure content sources
    • For instance, YouTube integration requires the user to Authenticate with Google OAuth2
    • News API integration requires a list of topics the user is interested in
  4. Content Pullers fetch data from content sources and store it on the file system in Parquet format.
  5. The content is then processed by a Spark Job (Content Processor Job)
    • The Content Processor Job logic is out of the scope of these article series
    • The Job is launched via Temporal Workflow (Content Processor Launcher). It also retries the Job in case of failures
    • The Job performs some magic and stores recommended articles into a Postgres Database
  6. The same Telegram Bot then uses stored recommendations and periodically pushes them into the Telegram chat

What’s next?

In the next article, you will dive deeper into the implementation details of each component.

  1. You will get started with Workflows and Activities using ZIO Temporal
  2. You’ll implement Workflows with Signal methods
  3. You’ll implement more complex business logic using Child Workflows
  4. You’ll dive deeper into Activities by using Activity Heartbeats
  5. …and much more!

Stay tuned for updates ✌️

This is a post in the Temporal Workflows with ZIO series.
Other posts in this series:

comments powered by Disqus