milvus: [Bug]: [nightly] query returns not empty after delete operation

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 
- Deployment mode(standalone or cluster): standalone
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

query returns not empty after delete operation

[2022-05-12T20:50:02.188Z] [2022-05-12 19:02:15 - DEBUG - ci_test]: (api_request)  : [Collection.delete] args: ['int64 in [1]', None, 20], kwargs: {} (api_request.py:55)

[2022-05-12T20:50:02.188Z] [2022-05-12 19:02:16 - DEBUG - ci_test]: (api_response) : (insert count: 0, delete count: 1, upsert count: 0, timestamp: 433162062718828546)  (api_request.py:27)

[2022-05-12T20:50:02.188Z] [2022-05-12 19:02:16 - DEBUG - ci_test]: (api_request)  : [Collection.query] args: ['int64 in [0, 1]', None, None, 20], kwargs: {} (api_request.py:55)

[2022-05-12T20:50:02.188Z] [2022-05-12 19:02:16 - DEBUG - ci_test]: (api_response) : [{'int64': 1}]  (api_request.py:27)

Expected Behavior

query returns empty after delete operation

Steps To Reproduce

Running nightly: https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/milvus-nightly-ci/detail/master/526/pipeline/124

cases:

1. testcases/test_delete.py::TestDeleteOperation::test_delete_entity_loop

    def test_delete_entity_loop(self):
        """
        target: test delete all entities one by one in a loop
        method: delete data one by one for a loop
        expected: No exception
        """
        # init an auto_id collection and insert tmp_nb data, flush and load
        collection_w, _, _, ids = self.init_collection_general(prefix, nb=tmp_nb, insert_data=True, auto_id=True)[0:4]

        for del_id in ids:
            expr = f'{ct.default_int64_field_name} in {[del_id]}'
            res = collection_w.delete(expr)[0]
            assert res.delete_count == 1

        # query with all ids
        expr = f'{ct.default_int64_field_name} in {ids}'
        collection_w.query(expr, check_task=CheckTasks.check_query_empty)

1. testcases/test_delete.py::TestDeleteOperation::test_delete_merge_ids_channel_and_sealed

    def test_delete_merge_ids_channel_and_sealed(self):
        """
        target: test merge deleted ids come from both channel and sealed
        method: 1.create, insert ids [0, tmp_nb) with shard_num=1
                2.delete id 0 and flush
                3.load and query with id 0
                4.delete id 1 (merge same segment deleted ids 0 and 1)
                5.query with id 0 and 1
        expected: Empty query result
        """
        # init collection and insert data without flush
        collection_w = self.init_collection_wrap(name=cf.gen_unique_str(prefix), shards_num=1)
        df = cf.gen_default_dataframe_data(tmp_nb)
        collection_w.insert(df)

        # delete id 0 and flush
        del_res, _ = collection_w.delete(tmp_expr)
        assert del_res.delete_count == 1
        assert collection_w.num_entities == tmp_nb

        # load and query id 0
        collection_w.load()
        collection_w.query(tmp_expr, check_task=CheckTasks.check_query_empty)

        # delete id 1 and query id 0 and 1
        collection_w.delete(expr=f'{ct.default_int64_field_name} in {[1]}')
        collection_w.query(expr=f'{ct.default_int64_field_name} in {[0, 1]}',
                           check_task=CheckTasks.check_query_empty)

Milvus Log

https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/milvus-nightly-ci/detail/master/526/artifacts/

name: artifacts-milvus-standalone-master-526-pymilvus-e2e-logs.tar.gz

Anything else?

1. for failed test case: testcases/test_delete.py::TestDeleteOperation::test_delete_entity_loop 1.1 collection name: delete_xvx6Sd46 1.2 FAILED timeline: 2022-05-12T19:02:09.856Z] [gw3] [ 12%] FAILED testcases/test_delete.py::TestDeleteOperation::test_delete_entity_loop

2. for failed test case: testcases/test_delete.py::TestDeleteOperation::test_delete_merge_ids_channel_and_sealed 2.1 collection name: delete_qQwOG19e 2.2 FAILED timeline 2022-05-12T19:02:18.008Z] [gw3] [ 12%] FAILED testcases/test_delete.py::TestDeleteOperation::test_delete_merge_ids_channel_and_sealed