Skip to content

Commit 74065b0

Browse files
andrewwormaldecharrod
andrewwormald
and
echarrod
authored
readme: Add best practices, overview, and run explanation (#65)
* workflow: Add best practices, overview, and run explanation * workflow: Add best practices, overview, and run explanation * workflow: Add best practices, overview, and run explanation * add mentioned in awesome Go * Update README.md Co-authored-by: Ed Harrod <echarrod@users.noreply.github.com> * Update README.md Co-authored-by: Ed Harrod <echarrod@users.noreply.github.com> * Update README.md Co-authored-by: Ed Harrod <echarrod@users.noreply.github.com> * Update README.md Co-authored-by: Ed Harrod <echarrod@users.noreply.github.com> * Update README.md Co-authored-by: Ed Harrod <echarrod@users.noreply.github.com> * remove table of contents * move run states to table --------- Co-authored-by: Ed Harrod <echarrod@users.noreply.github.com>
1 parent 95251c0 commit 74065b0

File tree

1 file changed

+122
-17
lines changed

1 file changed

+122
-17
lines changed

‎README.md

Lines changed: 122 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
<div align="center">
2-
<img src="./logo/logo.png" style="width: 300px; margin: 30px" alt="Workflow Logo">
2+
<img src="./logo/logo.png" style="width: 220px; margin: 30px" alt="Workflow Logo">
33
<div align="center" style="max-width: 750px">
44
<a style="padding: 0 5px" href="https://goreportcard.com/report/github.com/luno/workflow"><img src="https://goreportcard.com/badge/github.com/luno/workflow"/></a>
55
<a style="padding: 0 5px" href="https://sonarcloud.io/summary/new_code?id=luno_workflow"><img src="https://sonarcloud.io/api/project_badges/measure?project=luno_workflow&metric=coverage"/></a>
@@ -10,24 +10,36 @@
1010
<a style="padding: 0 5px" href="https://sonarcloud.io/summary/new_code?id=luno_workflow"><img src="https://sonarcloud.io/api/project_badges/measure?project=luno_workflow&metric=vulnerabilities"/></a>
1111
<a style="padding: 0 5px" href="https://sonarcloud.io/summary/new_code?id=luno_workflow"><img src="https://sonarcloud.io/api/project_badges/measure?project=luno_workflow&metric=duplicated_lines_density"/></a>
1212
<a style="padding: 0 5px" href="https://pkg.go.dev/github.com/luno/workflow"><img src="https://pkg.go.dev/badge/github.com/luno/workflow.svg" alt="Go Reference"></a>
13+
<a style="padding: 0 5px" href="https://github.com/avelino/awesome-go"><img src="https://awesome.re/mentioned-badge-flat.svg" alt="Mentioned in Awesome Go"></a>
1314
</div>
1415
</div>
1516

1617
# Workflow
1718

18-
Workflow is an event driven workflow that allows for robust, durable, and scalable sequential business logic to
19-
be executed in a deterministic manner.
19+
**Workflow** is a distributed event driven workflow framework that runs robust, durable, and
20+
scalable sequential business logic on your services.
21+
22+
**Workflow** uses a [RoleScheduler](https://github.com/luno/workflow/blob/main/rolescheduler.go) to distribute the work
23+
across your instances through a role assignment process (similar to a leadership election process, but with more than
24+
a single role of leader).
25+
26+
**Workflow** expects to be run on multiple instances but can also be run on single
27+
instances. Using the above-mentioned [RoleScheduler](https://github.com/luno/workflow/blob/main/rolescheduler.go),
28+
**Workflow** is able to make sure each process only runs once at any given time
29+
regardless if you are running 40 instances of your service or 1 instance.
2030

2131
---
32+
2233
## Features
2334

2435
- **Tech stack agnostic:** Use Kafka, Cassandra, Redis, MongoDB, Postgresql, MySQL, RabbitM, or Reflex - the choice is yours!
2536
- **Graph based (Directed Acyclic Graph - DAG):** Design the workflow by defining small units of work called "Steps".
26-
- **TDD:** Workflow was built using TDD and remains well-supported through a suit of tools.
37+
- **TDD:** **Workflow** was built using TDD and remains well-supported through a suit of tools.
38+
- **Callbacks:** Allow for manual callbacks from webhooks or manual triggers from consoles to progress the workflow, such as approval buttons or third-party webhooks.
39+
- **Event fusion:** Add event connectors to your workflow to consume external event streams (even if it's from a different event streaming platform).
40+
- **Hooks:** Write hooks that execute on core changes in a workflow Run.
2741
- **Schedule:** Allows standard cron spec to schedule workflows
2842
- **Timeouts:** Set either a dynamic or static time for a workflow to wait for. Once the timeout finishes everything continues as it was.
29-
- **Event fusion:** Add event connectors to your workflow to consume external event streams (even if its from a different event streaming platform).
30-
- **Callbacks:** Allow for manual callbacks from webhooks or manual triggers from consoles to progress the workflow such as approval buttons or third-party webhooks.
3143
- **Parallel consumers:** Specify how many step consumers should run or specify the default for all consumers.
3244
- **Consumer management:** Consumer management and graceful shutdown of all processes making sure there is no goroutine leaks!
3345

@@ -39,9 +51,45 @@ To start using workflow you will need to add the workflow module to your project
3951
go get github.com/luno/workflow
4052
```
4153

42-
### Adapters
43-
Some adapters dont come with the core workflow module such as `kafkastreamer`, `reflexstreamer`, `sqlstore`, and `sqltimeout`. If you
44-
wish to use these you need to add them individually based on your needs or build out your own adapter.
54+
---
55+
56+
## Adapters
57+
Adapters enable **Workflow** to be tech stack agnostic by placing an interface /
58+
protocol between **Workflow** and the tech stack. **Workflow**
59+
uses adapters to understand how to use that specific tech stack.
60+
61+
For example, the Kafka adapter enables workflow
62+
to produce messages to a topic as well as consume them from a topic using a set of predefined methods that wrap the
63+
kafka client. [Reflex](https://github.com/luno/reflex) is an event streaming framework that works very differently
64+
to Kafka and the adapter pattern allows for the differences to be contained and localised in the adapter and not
65+
spill into the main implementation.
66+
67+
### Event Streamer
68+
The [EventStreamer](https://github.com/luno/workflow/blob/main/eventstream.go) adapter interface defines what is needed
69+
to be satisfied in order for an event streaming platform or framework to be used by **Workflow**.
70+
71+
All implementations of the EventStreamer interface should be tested using [adaptertest.TestEventStreamer](https://github.com/luno/workflow/blob/main/adapters/adaptertest/eventstreaming.go)
72+
73+
### Record Store
74+
The [RecordStore](https://github.com/luno/workflow/blob/main/store.go) adapter interface defines what is needed to
75+
satisfied in order for a storage solution to be used by **Workflow**.
76+
77+
All implementations of the RecordStore interface should be tested using [adaptertest.RunRecordStoreTest](https://github.com/luno/workflow/blob/main/adapters/adaptertest/recordstore.go)
78+
79+
### Role Scheduler
80+
The [RoleScheduler](https://github.com/luno/workflow/blob/main/rolescheduler.go) adapter interface defines what is needed to
81+
satisfied in order for a role scheduling solution to be used by **Workflow**.
82+
83+
All implementations of the RoleScheduler interface should be tested using [adaptertest.RunRoleSchedulerTest](https://github.com/luno/workflow/blob/main/adapters/adaptertest/rolescheduler.go)
84+
85+
There are more adapters available but only the above 3 are core requirements to use **Workflow**. To start, use the
86+
in-memory implementations as that is the simplest way to experiment and get used to **Workflow**. For testing other
87+
adapter types be sure to look at [adaptertest](https://github.com/luno/workflow/blob/main/adapters/adaptertest) which
88+
are tests written for adapters to ensure that they meet the specification.
89+
90+
Adapters, except for the in-memory implementations, don't come with the core **Workflow** module such as `kafkastreamer`, `reflexstreamer`, `sqlstore`,
91+
`sqltimeout`, `rinkrolescheduler`, `webui` and many more. If you wish to use these you need to add them individually
92+
based on your needs or build out your own adapter.
4593

4694
#### Kafka
4795
```bash
@@ -62,8 +110,32 @@ go get github.com/luno/workflow/adapters/sqlstore
62110
```bash
63111
go get github.com/luno/workflow/adapters/sqltimeout
64112
```
113+
114+
#### Rink Role Scheduler
115+
```bash
116+
go get github.com/luno/workflow/adapters/rinkrolescheduler
117+
```
118+
119+
#### WebUI
120+
```bash
121+
go get github.com/luno/workflow/adapters/webui
122+
```
123+
65124
---
66-
## Usage
125+
126+
## Connectors
127+
Connectors allow **Workflow** to consume events from an event streaming platform or
128+
framework and either trigger a workflow run or provide a callback to the workflow run. This means that Connectors can act
129+
as a way for **Workflow** to connect with the rest of the system.
130+
131+
Connectors are implemented as adapters as they would share a lot of the same code as implementations of an
132+
EventStreamer and can be seen as a subsection of an adapter.
133+
134+
An example can be found [here](_examples/connector).
135+
136+
---
137+
138+
## Basic Usage
67139

68140
### Step 1: Define the workflow
69141
```go
@@ -171,10 +243,26 @@ Head on over to [./_examples](./_examples) to get familiar with **callbacks**, *
171243

172244
---
173245

174-
## Workflow's RunState
175-
RunState is the state of a Run and can only exist in one state at any given time. RunState is a
176-
finite state machine and allows for control over the Run. A Run is every instance of
177-
a triggered workflow.
246+
## What is a workflow Run
247+
248+
When a **Workflow** is triggered it creates an individual workflow instance called a Run. This is represented as workflow.Run in
249+
**Workflow**. Each run has a lifecycle which is a finite set of states - commonly
250+
referred to as Finite State Machine. Each
251+
workflow Run has the following of states (called RunState in **Workflow**):
252+
253+
| Run State | Value (int) | Description |
254+
|------------------------|-------------|-------------------------------------------------------------------------------------------------------------|
255+
| Unknown | 0 | Has no meaning. Protects against default zero value. |
256+
| Initiated | 1 | State assinged at creation of Run and is yet to be processed. |
257+
| Running | 2 | Has begun to be processed and is currently still being processed by a step in the workflow. |
258+
| Paused | 3 | Temporary stoppage that can be resumed or cancelled. Will prevent any new triggers of the same Foreign ID. |
259+
| Completed | 4 | Finished all the steps configured at time of execution. |
260+
| Cancelled | 5 | Did not complete all the steps and was terminated before completion. |
261+
| Data Deleted | 6 | Run Object has been modified to remove data or has been entirely removed. Likely for PII scrubbing reasons. |
262+
| Requested Data Deleted | 7 | Request state for the workflow to apply the default or custom provided delete operation to the Run Object. |
263+
264+
265+
A Run can only exist in one state at any given time and the RunState allows for control over the Run.
178266
```mermaid
179267
---
180268
title: Diagram the run states of a workflow
@@ -206,8 +294,8 @@ stateDiagram-v2
206294
Hooks allow for you to write some functionality for Runs that enter a specific RunState. For example when
207295
using `PauseAfterErrCount` the usage of the OnPause hook can be used to send a notification to a team to notify
208296
them that a specific Run has errored to the threshold and now has been paused and should be investigated. Another
209-
example is handling a known sentinel error in a Workflow Run and cancelling the Run by calling (where r is *Run)
210-
r.Cancel(ctx) or if a Workflow Run is manually cancelled from a UI then a notifgication can be sent to the team for visibility.
297+
example is handling a known sentinel error in a **Workflow** Run and cancelling the Run by calling (where r is *Run)
298+
r.Cancel(ctx) or if a **Workflow** Run is manually cancelled from a UI then a notifgication can be sent to the team for visibility.
211299

212300
Hooks run in an event consumer. This means that it will retry until a nil error has been returned and is durable
213301
across deploys and interruptions. At-least-once delivery is guaranteed, and it is advised to use the RunID as an
@@ -219,7 +307,6 @@ idempotency key to ensure that the operation is idempotent.
219307
|---------------|---------------------------------|-----------|-------------------------------------------|------------------|
220308
| OnPause | workflow.RunStateChangeHookFunc | error | Fired when a Run enters RunStatePaused | Yes |
221309
| OnCancelled | workflow.RunStateChangeHookFunc | error | Fired when a Run enters RunStateCancelled | Yes |
222-
| OnDataDeleted | workflow.RunStateChangeHookFunc | error | Fired when a Run enters RunStateDeleted | Yes |
223310
| OnCompleted | workflow.RunStateChangeHookFunc | error | Fired when a Run enters RunStateCompleted | Yes |
224311

225312
---
@@ -344,6 +431,7 @@ b.AddStep(
344431
workflow.PauseAfterErrCount(3),
345432
)
346433
```
434+
347435
---
348436

349437
## Glossary
@@ -364,3 +452,20 @@ b.AddStep(
364452
| **RunState** | RunState defines the finite number of states that a Run can be in. This is used to control and monitor the lifecycle of Runs. |
365453
| **Topic** | A method that generates a topic for producing events in the event streamer based on the workflow name and status. |
366454
| **Trigger** | A method in the workflow API that initiates a workflow for a specified foreignID and starting status. It returns a Run ID and allows for additional configuration options. |
455+
456+
---
457+
458+
## Best practices
459+
460+
1. Break up complex business logic into small steps.
461+
2. **Workflow** can be used to produce new meaningful data and not just be used to execute logic. If it is used for this, it's suggested
462+
to implement a CQRS pattern where the workflow acts as the "Command" and the data is persisted into a more queryable manner.
463+
3. Changes to workflows must be backwards compatible. If you need to introduce a non-backwards compatible change
464+
then the non-backwards compatible workflow should be added alongside the existing workflow with
465+
the non-backwards compatible workflow receiving all the incoming triggers. The old workflow should be given time
466+
to finish processing any workflows it started and once it has finished processing all the existing non-finished Runs
467+
then it may be safely removed. Alternatively versioning can be added internally to your Object type that you provide,
468+
but this results in changes to the workflow's Directed Acyclic Graph (map of steps connecting together).
469+
4. **Workflow** is not intended for low-latency. Asynchronous event driven systems are not meant to be low-latency but
470+
prioritise decoupling, durability, distribution of workload, and breakdown of complex logic (to name a few).
471+
5. Ensure that the prometheus metrics that come with **Workflow** are being used for monitoring and alerting.

0 commit comments

Comments
 (0)