async-graphql: Possible deadlock with dataloader

Hey guys, my dataloader seems to be locking up and not accepting any new request after using the request several times.

This seems to be the only dataloader with this issue, but it is also the only dataloader that i use reqwest rather than my tiberius sql db.

It happens after receiving several interupted request from the client, but never happens if i send a single request using the playground.

That dataloader will not process any further request, and the rest of the query can succeed if i take that part out of the query

The expected behavior is that it does not hang ever. Please let me know if you require the implementation of any of the functions shown.

#[derive(
    Default,
    Clone,
    PartialEq,
    serde_derive::Serialize,
    serde_derive::Deserialize,
    SimpleObject,
    Debug,
)]
#[graphql(complex)]
#[serde(rename_all = "camelCase")]
pub struct ServiceNowRelCi {
    #[serde(rename = "connection_strength")]
    pub connection_strength: ConnectionStrength,
    pub parent: Parent,
    #[serde(rename = "sys_mod_count")]
    pub sys_mod_count: SysModCount,
    #[serde(rename = "sys_updated_on")]
    pub sys_updated_on: SysUpdatedOn,
    #[serde(rename = "sys_tags")]
    pub sys_tags: SysTags,
    #[serde(rename = "type")]
    pub type_field: Type,
    #[serde(rename = "sys_id")]
    pub sys_id: SysId,
    #[serde(rename = "sys_updated_by")]
    pub sys_updated_by: SysUpdatedBy,
    pub port: Port,
    #[serde(rename = "sys_created_on")]
    pub sys_created_on: SysCreatedOn,
    #[serde(rename = "percent_outage")]
    pub percent_outage: PercentOutage,
    #[serde(rename = "sys_created_by")]
    pub sys_created_by: SysCreatedBy,
    pub child: Child,
}

#[ComplexObject]
impl ServiceNowRelCi {
    pub async fn children(&self, ctx: &Context<'_>) -> FieldResult<Option<Vec<ServiceNowRelCi>>> {
        let data: Option<Vec<ServiceNowRelCi>> = ctx
            .data_unchecked::<DataLoader<ServiceNowRelCiChildrenLoader>>()
            .load_one(self.child.value.clone())
            .await?;

        Ok(data)
    }
//rest
}

pub struct ServiceNowRelCiChildrenLoader(Client);

impl ServiceNowRelCiChildrenLoader {
    pub fn new(pool: Client) -> Self {
        Self(pool)
    }
}

fn query_rel_ci_from_sn_url(keys: &[String]) -> String {
    format!("https://nb.service-now.com/api/now/table/cmdb_rel_ci?sysparm_query=parentIN{}&sysparm_display_value=all",keys.join(","))
}

///DataLoader with an array as child.  
///DataLoader using a rest endpoint
#[async_trait]
impl Loader<String> for ServiceNowRelCiChildrenLoader {
    type Value = Vec<ServiceNowRelCi>;
    type Error = FieldError;

    async fn load(&self, keys: &[String]) -> Result<HashMap<String, Self::Value>, Self::Error> {
        let query = query_rel_ci_from_sn_url(keys);

        let stream: Vec<ServiceNowRelCi> = CmdbCiService::query_service_now_get_builder::<RootServiceNowRelCi>(
            self.0.clone(),
            query.as_str(),
        )
        .await?
        .result;

        let stream = stream
            .clone()
            .into_iter()
            .map(|data: ServiceNowRelCi| {
                (
                    data.parent.value.clone(),
                    stream
                        .clone()
                        .into_iter()
                        .filter(|filtered| &data.parent.value == &filtered.parent.value)
                        .collect(),
                )
            })
            .collect::<HashMap<String, Vec<ServiceNowRelCi>>>();

        Ok(stream)
    }
}

    ///Get request from service now, just requires url
    async fn query_service_now_get_builder<T>(
        client: reqwest::Client,
        url: &str,
    ) -> anyhow::Result<T>
    where
        T: DeserializeOwned,
    {
        dotenv().ok();
        let username = env::var("SN_API_KEY_USER").expect("SN_API_KEY_USER must be set");
        let password = env::var("SN_API_KEY_PASS").expect("SN_API_KEY_PASS must be set");
        client
            .get(url)
            .basic_auth(username, Some(password))
            .send()
            .await?
            .json()
            .await
            .map_err(Into::into)
    }


About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 28 (1 by maintainers)

Commits related to this issue

Most upvoted comments

It works!!! Thank you!

Release in v3.0.12 πŸ™‚

Please upgrade to 3.x grin

It looks like the book is still on 2.x, do you have any migration docs?

I forgot to update the version in the book, it has been changed now. πŸ™‚

I really appreciate your help with this. The library is too awesome man.

@sunli829 thank you - I just pulled down the latest commit in Cargo and can confirm I’m not seeing the locks anymore! This is with tokio

I fixed this in the master branch.

Now the dataLoader::new function adds a parameter to specify the function that spawning task, because I don’t want async-graphql to depend on a specific runtime.

DataLoader::new(MyLoader, tokio::spawn)
DataLoader::new(MyLoader, async_std::task::spawn)

Thank you, these can definitely help me. @tombowditch

Hey there,

I appreciate you guys taking this up, as I am starting to see the error again.

As the individual said above, it is only occurring after canceling a request. I am also using Apollo client, so I wonder if there is a proper cancel mechanism on that library.

Has there been any update to this issue?

Thank you, these can definitely help me. @tombowditch

I think I have narrowed it down:

  • the deadlock seems to happen when a pending request is cancelled (see screenshot below) (I am using apollo client in react for example)
  • I can call the GQL service with 10+ requests/sec using curl with no deadlocks
  • the action inside the dataloader is Delay, whereas other requests it is StartFetch

cancelled screenshot: all localhost requests are POSTs to the graphql endpoint, you can see the first is cancelled which makes the third pending forever Screenshot 2021-11-15 at 19 59 59

In this code here in the dataloader: https://github.com/async-graphql/async-graphql/blob/master/src/dataloader/mod.rs#L359-L366

When a request is successful, the action is StartFetch when I get a deadlock, the action is Delay

I added a println here https://github.com/tombowditch/async-graphql/blob/dataloader-race/src/dataloader/mod.rs#L379

Working request: Screenshot 2021-11-15 at 20 03 15

Deadlocked request: Screenshot 2021-11-15 at 20 03 45

Which leads me to believe the await here https://github.com/async-graphql/async-graphql/blob/master/src/dataloader/mod.rs#L368 is awaiting forever?

@sunli829 your input would be appreciated, this is where my lack of knowledge of the codebase fails me πŸ˜ƒ I hope that is helpful at least πŸ˜ƒ