katib: Go Test failed sometimes

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

One of github action for Go Test failed sometimes, but it succeed when retest.

I’ve check for some cases:

• https://github.com/kubeflow/katib/runs/3444147420?check_suite_focus=true
• https://github.com/kubeflow/katib/runs/3451003535?check_suite_focus=true
• https://github.com/kubeflow/katib/runs/3451166076?check_suite_focus=true
• https://github.com/kubeflow/katib/runs/3435251159?check_suite_focus=true
• https://github.com/kubeflow/katib/runs/3413375074?check_suite_focus=true

All case failed for either one of these:

    trial_controller_test.go:261: 
        Timed out after 40.000s.
        Expected
            <bool>: false
        to be true
FAIL

    experiment_controller_test.go:350: 
        Timed out after 40.000s.
        Expected
            <bool>: false
        to be true

What did you expect to happen:

If these issue occured just from the network issue, what about increasing the timeout ? Does it make sense ? WYDT ?

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Kubeflow version (kfctl version):
• Minikube version (minikube version):
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):