graphql-ruby: Circular dependency issue with AsyncDataloader
Describe the bug My existing schema, which works well with the classic Dataloader, fails for certain requests with AsyncDataloader
Versions
graphql version: master commit becd60008606
rails (or other framework): 7.1.2
other applicable versions (graphql-batch, etc)
- graphql-c_parser (1.0.7)
- graphql-sources (1.2.0)
GraphQL schema
Simplified schema to repro:
module Types
class CategoryType < BaseObject
field :id, ID, null: false
field :offerings, [Types::OfferingType], null: false
def offerings(selected: nil, hidden: nil)
dataloader.with(Loaders::Category::OfferingsLoader, selected, hidden).load(object.id)
end
end
end
module Loaders::Offering
class CategoryLoader < GraphQL::Dataloader::Source
def fetch(record_ids)
@record_ids = record_ids
record_ids.map do |id|
records.find do |record|
record.offerings.any? { |offering| offering.id == id }
end
end
end
private
def records
@records ||= Category
.joins(:offerings)
.includes(:offerings)
.where(offerings: {id: @record_ids})
.distinct
end
end
end
module Types
class OfferingType < BaseObject
field :id, ID, null: false
field :category, Types::CategoryType, null: true
private
def category
dataloader.with(Loaders::Offering::CategoryLoader).load(object.id)
end
end
end
class Loaders::Category::OfferingsLoader < GraphQL::Dataloader::Source
attr_reader :selected, :hidden
def initialize(selected, hidden)
@selected = selected
@hidden = hidden
end
def fetch(record_ids)
@record_ids = record_ids
record_ids.map do |id|
records.select { |record| record.category_id == id }
end
end
private
def records
@records ||=
Offering.where(category_id: @record_ids)
.tap { |query| query.where!(selected:) unless selected.nil? }
.tap { |query| query.where!(hidden:) unless hidden.nil? }
.order_by_position
end
end
GraphQL query
Simplified schema to repro:
mutation InitializeUI($input: CreateSessionInput!) {
createSession(input: $input) {
user {
categories {
id
offerings {
id
}
}
offerings {
id
}
}
}
}
{
"data": {
"createSession": {
"user": {
"categories": [
{
"id": "5fe31061-9db0-4c5e-8602-4540f2ab385e",
"offerings": [
{
"id": "d99c5112-854e-43a0-8e75-3c4e936eff72"
},
{
"id": "2df4632a-fdb2-4775-a1bb-4d258223c091"
},
{
"id": "9fac4393-f2a9-4883-91aa-ab0392e6a1da"
}
]
},
{
"id": "e91b2429-f4aa-44ff-9909-1f57f48afac0",
"offerings": [
{
"id": "44885e32-4056-479f-b053-759c68ebb4c4"
},
{
"id": "a9a9116d-d346-46bd-9d6e-96ac28c0804f"
},
{
"id": "03c91409-d4b3-452b-820e-6aa69c742bd5"
}
]
}
],
"offerings": [
{
"id": "9271ab6c-089c-4094-bc56-454d6ada98e7"
},
{
"id": "409bf2cd-3bf9-44f6-9387-61456713920e"
},
{
"id": "d86b33f0-5545-4b38-a072-1ba1ac0b44dd"
},
{
"id": "2df4632a-fdb2-4775-a1bb-4d258223c091"
},
{
"id": "6b02a1b1-0111-4a93-9d55-ccb83ee964db"
},
{
"id": "8acd96b7-17d1-4576-bf53-fda7b1faf2db"
},
{
"id": "0610c3bc-9725-4761-a0f2-be35835b7a8d"
},
{
"id": "d99c5112-854e-43a0-8e75-3c4e936eff72"
},
{
"id": "44885e32-4056-479f-b053-759c68ebb4c4"
},
{
"id": "a374b451-23ea-42a3-b7d3-65e18aae7c29"
},
{
"id": "a9a9116d-d346-46bd-9d6e-96ac28c0804f"
},
{
"id": "a1aaba60-0ef3-4d63-a3a9-d1e926800c4d"
},
{
"id": "03c91409-d4b3-452b-820e-6aa69c742bd5"
},
{
"id": "eff97220-2ce1-4803-a32a-25e18e0e4bb1"
},
{
"id": "df337b0b-c000-4672-a943-5c3de506cca2"
},
{
"id": "9fac4393-f2a9-4883-91aa-ab0392e6a1da"
},
{
"id": "991ad8a0-6b5d-4bae-bac4-4403b478b19d"
},
{
"id": "59b7bf3d-0780-42c7-ad0a-7d27504e8b26"
},
{
"id": "a86a4b6f-538f-4915-8df4-6fa568f7ade1"
},
{
"id": "a8924cad-9981-43a5-8188-03f0dc0427e3"
},
{
"id": "054a2473-5631-457d-ad6e-55185481ca9b"
},
{
"id": "32b2a064-8f44-441c-a04c-8660519d40ef"
},
{
"id": "4c38fcef-1a99-477b-b5b1-42ddb1465c6b"
},
{
"id": "03437e1d-63f2-4af7-9d2b-b519817444bf"
},
{
"id": "d44655a5-08ad-4f0f-b855-7026f2b383dd"
},
{
"id": "dfdb1300-849c-443a-b8f7-d9b2581f00db"
},
{
"id": "612783f9-1eff-43a3-9917-8b6de28c9ffa"
},
{
"id": "012a04eb-db2f-4236-9c0e-f53baaeb55cf"
},
{
"id": "efb5aa0c-5a06-433b-900d-9a01849c505d"
},
{
"id": "0333dd63-a93f-442d-ba1d-196150cf1b49"
},
{
"id": "c9079b40-2f1f-4a09-8b5d-bcdea11efcbd"
},
{
"id": "988f9639-b6f0-456c-9621-4431a8ce070d"
},
{
"id": "260e110e-34a4-4350-8c9b-d50a67be57d3"
},
{
"id": "f49e8771-4f40-42d5-b792-60b0df40c606"
},
{
"id": "a619ff9b-0f58-4b61-a6ac-7d1fed9cfbab"
},
{
"id": "be387d8d-1044-45ff-91b4-ae646cf5d72f"
},
{
"id": "0aecc5ed-576b-49fe-96bd-e5a4f96aec75"
},
{
"id": "6e1144ed-24d0-4cb9-8791-9ccd6e454d54"
},
{
"id": "e882da2a-edc9-4dd7-af22-9776e6b7713d"
},
{
"id": "eb9f3e23-3aab-4f64-ab3e-d5b9b9a4dd72"
}
]
}
}
}
}
Steps to reproduce
Change from stock Dataloader to AsyncDataloader
Expected behavior
Queries complete the same as they did before
Actual behavior
RuntimeError: Loaders::Category::OfferingsLoader#sync tried 1000 times to load pending keys (["5fe31061-9db0-4c5e-8602-4540f2ab385e"]), but they still weren't loaded. There is likely a circular dependency.
Place full backtrace here (if a Ruby exception is involved):
Click to view exception backtrace
warn: Async::Task [oid=0xc328] [ec=0xc33c] [pid=24436] [2023-12-21 17:41:48 -0500]
| Task may have ended with unhandled exception.
| RuntimeError: Loaders::Category::OfferingsLoader#sync tried 1000 times to load pending keys (["5fe31061-9db0-4c5e-8602-4540f2ab385e"]), but they still weren't loaded. There is likely a circular dependency.
| → bundler/gems/graphql-ruby-becd60008606/lib/graphql/dataloader/source.rb:101 in `sync'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/dataloader/source.rb:57 in `load'
| app/graphql/types/category_type.rb:8 in `offerings'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field.rb:700 in `public_send'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field.rb:700 in `block (2 levels) in resolve'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field.rb:843 in `block in with_extensions'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field.rb:879 in `block (2 levels) in run_extensions_before_resolve'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field.rb:882 in `run_extensions_before_resolve'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field.rb:879 in `block in run_extensions_before_resolve'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field_extension.rb:134 in `resolve'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field.rb:866 in `run_extensions_before_resolve'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field.rb:838 in `with_extensions'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field.rb:669 in `block in resolve'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/execution/interpreter/runtime.rb:770 in `minimal_after_lazy'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/query.rb:372 in `after_lazy'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/schema/field.rb:667 in `resolve'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/execution/interpreter/runtime.rb:358 in `block (2 levels) in evaluate_selection_with_resolved_keyword_args'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/tracing/trace.rb:52 in `execute_field'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/execution/interpreter/runtime.rb:357 in `block in evaluate_selection_with_resolved_keyword_args'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/execution/interpreter/runtime.rb:709 in `call_method_on_directives'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/execution/interpreter/runtime.rb:354 in `evaluate_selection_with_resolved_keyword_args'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/execution/interpreter/runtime.rb:264 in `evaluate_selection'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/execution/interpreter/runtime.rb:214 in `block (2 levels) in evaluate_selections'
| bundler/gems/graphql-ruby-becd60008606/lib/graphql/dataloader/async_dataloader.rb:65 in `block in spawn_job_task'
| gems/async-2.6.5/lib/async/task.rb:160 in `block in run'
| gems/async-2.6.5/lib/async/task.rb:330 in `block in schedule'
Loaders::Category::OfferingsLoader#sync tried 1000 times to load pending keys (["5fe31061-9db0-4c5e-8602-4540f2ab385e"]), but they still weren't loaded. There is likely a circular dependency.
app/graphql/types/category_type.rb:8:in `offerings'
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 29 (25 by maintainers)
Thanks for giving it a try – glad to hear it. I really think the conditions for this error are gone. It wasn’t actually a circular dependency, but rather, a Source was trying 1000 times to be resolved but it still wasn’t resolved. This was due to the unpredictable scheduling of Fibers, where yielding might go right back to the source itself without trying other fibers. (And as previously described, I couldn’t find a way of controlling that flow better :S). So, I’ve reduced the scope where the fiber scheduler is applied, so that there won’t be as many open, unrelated fibers to choose between. I’m going to call this closed 😅
I think these may both be due to the switch to
.transfer-based Fiber handling. It has an unexpected (and maybe buggy?) result, which I wrote up here: https://github.com/rmosolgo/graphql-ruby/pull/4748#issuecomment-1868340432 and reported/asked about here: https://bugs.ruby-lang.org/issues/20081