async-graphql: Possible deadlock with dataloader
Hey guys, my dataloader seems to be locking up and not accepting any new request after using the request several times.
This seems to be the only dataloader with this issue, but it is also the only dataloader that i use reqwest rather than my tiberius sql db.
It happens after receiving several interupted request from the client, but never happens if i send a single request using the playground.
That dataloader will not process any further request, and the rest of the query can succeed if i take that part out of the query
The expected behavior is that it does not hang ever. Please let me know if you require the implementation of any of the functions shown.
#[derive(
Default,
Clone,
PartialEq,
serde_derive::Serialize,
serde_derive::Deserialize,
SimpleObject,
Debug,
)]
#[graphql(complex)]
#[serde(rename_all = "camelCase")]
pub struct ServiceNowRelCi {
#[serde(rename = "connection_strength")]
pub connection_strength: ConnectionStrength,
pub parent: Parent,
#[serde(rename = "sys_mod_count")]
pub sys_mod_count: SysModCount,
#[serde(rename = "sys_updated_on")]
pub sys_updated_on: SysUpdatedOn,
#[serde(rename = "sys_tags")]
pub sys_tags: SysTags,
#[serde(rename = "type")]
pub type_field: Type,
#[serde(rename = "sys_id")]
pub sys_id: SysId,
#[serde(rename = "sys_updated_by")]
pub sys_updated_by: SysUpdatedBy,
pub port: Port,
#[serde(rename = "sys_created_on")]
pub sys_created_on: SysCreatedOn,
#[serde(rename = "percent_outage")]
pub percent_outage: PercentOutage,
#[serde(rename = "sys_created_by")]
pub sys_created_by: SysCreatedBy,
pub child: Child,
}
#[ComplexObject]
impl ServiceNowRelCi {
pub async fn children(&self, ctx: &Context<'_>) -> FieldResult<Option<Vec<ServiceNowRelCi>>> {
let data: Option<Vec<ServiceNowRelCi>> = ctx
.data_unchecked::<DataLoader<ServiceNowRelCiChildrenLoader>>()
.load_one(self.child.value.clone())
.await?;
Ok(data)
}
//rest
}
pub struct ServiceNowRelCiChildrenLoader(Client);
impl ServiceNowRelCiChildrenLoader {
pub fn new(pool: Client) -> Self {
Self(pool)
}
}
fn query_rel_ci_from_sn_url(keys: &[String]) -> String {
format!("https://nb.service-now.com/api/now/table/cmdb_rel_ci?sysparm_query=parentIN{}&sysparm_display_value=all",keys.join(","))
}
///DataLoader with an array as child.
///DataLoader using a rest endpoint
#[async_trait]
impl Loader<String> for ServiceNowRelCiChildrenLoader {
type Value = Vec<ServiceNowRelCi>;
type Error = FieldError;
async fn load(&self, keys: &[String]) -> Result<HashMap<String, Self::Value>, Self::Error> {
let query = query_rel_ci_from_sn_url(keys);
let stream: Vec<ServiceNowRelCi> = CmdbCiService::query_service_now_get_builder::<RootServiceNowRelCi>(
self.0.clone(),
query.as_str(),
)
.await?
.result;
let stream = stream
.clone()
.into_iter()
.map(|data: ServiceNowRelCi| {
(
data.parent.value.clone(),
stream
.clone()
.into_iter()
.filter(|filtered| &data.parent.value == &filtered.parent.value)
.collect(),
)
})
.collect::<HashMap<String, Vec<ServiceNowRelCi>>>();
Ok(stream)
}
}
///Get request from service now, just requires url
async fn query_service_now_get_builder<T>(
client: reqwest::Client,
url: &str,
) -> anyhow::Result<T>
where
T: DeserializeOwned,
{
dotenv().ok();
let username = env::var("SN_API_KEY_USER").expect("SN_API_KEY_USER must be set");
let password = env::var("SN_API_KEY_PASS").expect("SN_API_KEY_PASS must be set");
client
.get(url)
.basic_auth(username, Some(password))
.send()
.await?
.json()
.await
.map_err(Into::into)
}
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 28 (1 by maintainers)
Commits related to this issue
- Fix deadlock in dataloader. #555 — committed to async-graphql/async-graphql by sunli829 3 years ago
It works!!! Thank you!
Release in
v3.0.12
πI really appreciate your help with this. The library is too awesome man.
@sunli829 thank you - I just pulled down the latest commit in Cargo and can confirm Iβm not seeing the locks anymore! This is with tokio
I fixed this in the
master
branch.Now the
dataLoader::new
function adds a parameter to specify the function that spawning task, because I donβt wantasync-graphql
to depend on a specific runtime.Hey there,
I appreciate you guys taking this up, as I am starting to see the error again.
As the individual said above, it is only occurring after canceling a request. I am also using Apollo client, so I wonder if there is a proper cancel mechanism on that library.
Has there been any update to this issue?
Thank you, these can definitely help me. @tombowditch
I think I have narrowed it down:
action
inside the dataloader isDelay
, whereas other requests it isStartFetch
cancelled screenshot: all localhost requests are POSTs to the graphql endpoint, you can see the first is cancelled which makes the third pending forever
In this code here in the dataloader: https://github.com/async-graphql/async-graphql/blob/master/src/dataloader/mod.rs#L359-L366
When a request is successful, the
action
isStartFetch
when I get a deadlock, theaction
isDelay
I added a println here https://github.com/tombowditch/async-graphql/blob/dataloader-race/src/dataloader/mod.rs#L379
Working request:
Deadlocked request:
Which leads me to believe the await here https://github.com/async-graphql/async-graphql/blob/master/src/dataloader/mod.rs#L368 is awaiting forever?
@sunli829 your input would be appreciated, this is where my lack of knowledge of the codebase fails me π I hope that is helpful at least π