[Bug] Potential deadlock detected after upgrade to v0.6.0

[laurynas]

What are you really trying to do?

I have upgraded sdk-ruby from v0.5.0 to v0.6.0 and ran benchmark workflow locally in dev env.

Describe the bug

Benchmark workflow consistently works fine with v0.5.0, but with v0.6.0 it consistently gets stuck with error [TMPRL1101] Potential deadlock detected: workflow didn't yield within 2.0 second(s)..

Minimal Reproduction

• Start worker
• Start workflow
• Wait
• Get error

Environment/Versions

• OS and processor: M3 MacBook
• Temporal Version: 1.18
• Temporal server running on Docker

Additional context

Workflow:

class Benchmarks::AsyncActivityTemporalioWorkflow < TemporalioWorkflow
  def execute(iterations = 1000)
    futures = iterations.times.map do |i|
      Temporalio::Workflow::Future.new do
        TemporalioExt::Activities.execute(HelloWorldActivityV2, "Test #{i}")
      end
    end

    Temporalio::Workflow::Future.all_of(*futures).wait

    true
  end
end

Activity:

class HelloWorldActivityV2 < TemporalioActivity
  def execute(name)
    "Hello #{name}"
  end
end

[cretz]

Thanks for the replication! We will try to replicate and investigate.

[cretz]

I am unable to replicate this on any version including 0.6.0. Here is a full standalone script:

# frozen_string_literal: true

require 'temporalio/activity'
require 'temporalio/client'
require 'temporalio/testing'
require 'temporalio/worker'
require 'temporalio/workflow'

class ManyActivitiesActivity < Temporalio::Activity::Definition
  def execute(name)
    "Hello #{name}"
  end
end

class ManyActivitiesWorkflow < Temporalio::Workflow::Definition
  def execute(iterations = 1000)
    futures = iterations.times.map do |i|
      Temporalio::Workflow::Future.new do
        Temporalio::Workflow.execute_activity(ManyActivitiesActivity, "Test #{i}", start_to_close_timeout: 30)
      end
    end

    Temporalio::Workflow::Future.all_of(*futures).wait

    'done'
  end
end

# Run local, full Temporal server (same as `temporal server start-dev` from CLI)
Temporalio::Testing::WorkflowEnvironment.start_local do |env|
  # Run worker
  Temporalio::Worker.new(
    client: env.client,
    task_queue: 'my-task-queue',
    activities: [ManyActivitiesActivity],
    workflows: [ManyActivitiesWorkflow]
  ).run do
    # Run workflow
    result = env.client.execute_workflow(
      ManyActivitiesWorkflow,
      id: 'my-workflow',
      task_queue: 'my-task-queue'
    )
    puts "Workflow result: #{result}"
  end
end

I wonder if your worker is otherwise overloaded or there is something else happening inside of TemporalExt not shown here. Is it possible to provide a fully-runnable standalone replication? Can alter the script I provided to demonstrate replication if needed.

Also, as a general note, you might consider batching activity work where you can. Calling 20 activities that each do 50 things is often more reasonable than 1000 activities doing 1 thing, though the latter may make sense if needing independent retryability (but note there are maximum activity limits per workflow).

[laurynas]

Hey @cretz, thanks for the tips. It was just my experiment checking activity performance. It is strange that there is a difference between v0.5.0 and v0.6.0, maybe related with this change (I didn't check):
#309

I'll try to experiment more, maybe with plain ruby, without any TemporalioExt helpers for clarity (though they just set some defaults to reduce boilerplate).

I'll post the results once I have something.

[laurynas]

Hi again! I have created plain ruby example to run on docker:
https://github.com/laurynas/temporalio-ruby-sdk-experiments

Wrote short instructions how to run it in the README file.

With this example I can reproduce the issue quite consistently on my MacBook M3 Pro.

v0.5.0 works fine with the same workflow.

[cretz]

Thanks! I think I was able to replicate just by changing my script to use child workflows as you did in your repo there. We are investigating...

[laurynas]

I ran my docker experiment on much slower Dell Latitude 7290 with Ubuntu 24.04 and I couldn't reproduce the issue. The workflow runs slowly, but no deadlocks.

[cretz]

After investigation, this is a case where, with the illegal call tracer on, it actually does take over 2s to process things. There is also a bug preventing the disabling of the call tracer we will fix. The only current workaround for this is to set debug_mode: true when creating the worker that disables deadlock detector (and therefore allows CPU-bound work to take longer than 2s).

We are investigating more permanent fixes here including reducing the scope of the illegal call tracer. I will update when we have more info.

[cretz]

Added a PR that will hopefully reduce the call tracing scope down enough to not cause issues here: #332. If you'd like to test it, can rebuild the SDK (see bottom of README) on that branch. EDIT: Though make sure it is in "ready for review". We may have to turn it to a draft while we debug some garbage collector issues.

[cretz]

This should be now solved with #332. Basically, we just run less validation things so as not to slow down enough to hit this deadlock detection.