pkg: Controller Pods are too slow to become the new leader after the previous leader releases the ownership

Expected Behavior

We run leader election mode for Serving Autoscaler, Controller, Webhook. If there are multiple pods, one pod is expected to be take over the ownership of a bucket immediately after the previous leader pod releases the ownership.

Actual Behavior

It could take more than 20 seconds for a bucket to be occupied by a new owner.

Steps to Reproduce the Problem

1. Run watch kubectl -n knative-serving get leases.
2. Choose a lease and find its holder, get the pod name as the prefix and delete the pod.
3. Watch the outputs from step 1. It usually takes 15 seconds for the change with a new leader. 15 seconds is the default value of leaseDuration.

Additional Info

I think we need to tweak https://github.com/knative/pkg/blob/master/leaderelection/config.go#L102