diffusers: Loading cached pipeline takes too long

Describe the bug

When loading a pipeline that is cached lot of time is wasted by calling the Hub unnecessarily. We should try to avoid this by trying to call the Hub only once if the model is cached.

See internal slack thread for more context.

Reproduction

Run the following script:

#!/usr/bin/env python3
from diffusers import StableDiffusionPipeline
import time
import sys

local_file_only = bool(int(sys.argv[1]))

start_time = time.time()
pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", local_files_only=local_file_only)
print(f"Local files only {str(local_file_only)}: {time.time() - start_time}")

with the args 0 and 1, one gets:

Local files only False: 2.1148853302001953
Local files only True: 0.31396007537841797

=> We do too many calls to the Hub when everything is already cached. We should only do one call to the Hub to check that everything is cached ideally.

Logs

No response

System Info

Main