Contents
Conductor stores each workflow execution in Postgres and schedules its next step by pushing a message onto a decider queue in Redis. Jepsen’s report on Orkes Conductor, published on 2026-10-20, found that three of its serious safety bugs share one shape: the row and the message are two separate writes, and the process that repairs stuck work only reads one of them. Orkes funded the research, and Jepsen tested versions 5.3.2 through 5.5.0-rc2.
The vendor-fixed items are in 5.5.0 and 5.5.6. The pattern is older than either release, and it is probably present in some system you run.
The sweeper cannot see what it was never told about
Conductor writes a new workflow to Postgres and then appends it to the decider queue. If the process dies, or Redis or Postgres is partitioned, between those two steps, you have a durable workflow with no message. Jepsen’s description of the result: workflows “would remain in the RUNNING state indefinitely, with their tasks never executed,” and restarting nodes did not help.
The reason is the recovery path. Per the report, “the sweeper process which periodically monitors running workflows relies purely on the decider queue, not the database.” A sweeper that iterates a queue can only repair items that made it into the queue. An item that never got enqueued is invisible to it by construction.
Orkes’ fix in 5.5.0 was a background process that re-enqueues abandoned executions. That moves the source of truth for “what should be running” from the queue to the database, which is the only consistent way to close this gap. A reconciler has to enumerate the durable record and compare it against the queue, not the other way round.
An undo that deletes state the client has already seen
The second bug is the same two writes with a different failure handler. If the server committed the workflow to Postgres but could not append it to the decider queue, it tried to be tidy: “the server then decided to undo its write to Postgres by deleting the workflow record.”
The problem is that the first write was already visible. A client calling getWorkflow in that window could see the execution, and then it would be gone for good. Jepsen classes these as lost workflows. In 5.5.0 Conductor stopped deleting executions when a step fails.
A compensating delete is only safe if nobody could have observed the thing you are deleting. Once a record is readable, undoing it is a second externally visible change, and it races with every reader. The safer failure handler for a half-finished two-step write is to leave the row, mark it as needing repair, and let the reconciler from the previous section pick it up.
Retrying a non-idempotent call hides the earlier success
The report’s fourth theme is on the client. During server kills, startWorkflow returned “failed to connect” errors, and many of those workflows had in fact been created. Jepsen counted “1,752 out of 25,920 failed calls to startWorkflow were actually successful” in one run.
The Java client retried the request internally. The first attempt timed out after the server had done the work. The retry then failed to open a connection, and “when that retry failed to open a connection, it returned only that most recent error.” The caller got a definite-sounding “could not connect” for an operation that had already succeeded.
Orkes changed the Java client to stop retrying non-idempotent requests and says it is exploring an isDefinite() method on errors. This item is listed as still under investigation, along with a separate case where startWorkflow returned a null ID after a Redis kill, so do not assume the client is clean after upgrading.
The general rule: an error from a retrying client describes the last attempt, not the operation. If your wrapper collapses a list of attempt errors into the final one, you have turned an indefinite outcome into a definite one, and the caller will act on it, usually by submitting the work again.
Two writes in different layers, one ignored failure
The last serious bug is the same pattern inside Postgres alone. Retrying a timed-out task takes two writes: insert the new task row, then mark the old row superseded. In the report’s words, “Postgres failed between these two writes, but Conductor ignored that failure.” The old task was marked superseded, no retry row existed, a sweeper then saw only terminal tasks, and the workflow was marked COMPLETED without the task ever running.
Jepsen notes that a database transaction would have prevented this, but the writes sat in different code layers and the change would have been large. 5.5.6 stops ignoring a failed write of the retry record. That is a fix for the specific symptom. It does not make the two writes atomic, so a crash between them is still worth testing for.
What I would check in my own system
The recommendations in the report apply beyond Conductor. These are drawn from them, with the checks worded generically since I have not run this against a Conductor deployment.
| Failure shape | Question to ask of your system |
|---|---|
| Row written, message never sent | Does the repair job enumerate the table, or only the queue? |
| Failed second step triggers a delete | Could any reader have seen the row before the delete? |
| Client retries internally | Does a returned error say whether the write may have happened? |
| Status derived from child rows | Can a missing child row look the same as a finished one? |
Jepsen’s own advice for Conductor users is to upgrade to 5.5.6, look for long-idle RUNNING workflows and COMPLETED workflows whose tasks never finished, treat every Conductor error as indefinite, and make tasks idempotent with downstream idempotency keys.
One more finding deserves a mention because it is easy to misread. With default lock lease settings, Jepsen saw concurrent execution of a task about once every six hours in a healthy cluster, and shortening the lease to one second brought that to every few minutes. A check-then-act sweeper made it worse: it could read a task as SCHEDULED after a worker had started it and re-enqueue it. Jepsen’s point is that “mutual exclusion is impossible in asynchronous networks,” so the answer is idempotent tasks, not a longer lease.
Limits of the evidence
Jepsen can show bugs but not their absence. The tests used single-task workflows, one custom worker, Postgres for the database and Redis for the queue. Multi-step workflows, failure workflows, other storage backends, high latency and clock skew were not covered. The fix timeline above is the vendor’s and Jepsen’s account; I did not verify the releases myself.
Sources
- Jepsen: Orkes Conductor 5.3.2, 2026-10-20
- Jepsen findings tracker for Conductor, issues 1 to 9