diffusers: The load_lora_weights is not working offline

Describe the bug

The load_lora_weights is not working offline.

Reproduction

Sample code to reproduce this issue. (Turn off the internet)

from diffusers import DiffusionPipeline, LCMScheduler

pipeline = DiffusionPipeline.from_pretrained(
    "Lykon/dreamshaper-8",
    local_files_only=True,
)
pipeline.load_lora_weights("latent-consistency/lcm-lora-sdv1-5")

Logs

Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:00<00:00, 13.51it/s]
Traceback (most recent call last):
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\urllib3\connection.py", line 203, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\urllib3\util\connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Rupesh\AppData\Local\Programs\Python\Python311\Lib\socket.py", line 961, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno 11001] getaddrinfo failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\urllib3\connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\urllib3\connectionpool.py", line 491, in _make_request
    raise new_e
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\urllib3\connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\urllib3\connectionpool.py", line 1096, in _validate_conn
    conn.connect()
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\urllib3\connection.py", line 611, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\urllib3\connection.py", line 210, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x00000198E0086790>: Failed to resolve 'huggingface.co' ([Errno 11001] getaddrinfo failed)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\requests\adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\urllib3\connectionpool.py", line 844, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\urllib3\util\retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/latent-consistency/lcm-lora-sdv1-5 (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x00000198E0086790>: Failed to resolve 'huggingface.co' ([Errno 11001] getaddrinfo failed)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\dev\push\faster\unify\fastsdcpu\lora_test.py", line 7, in <module>
    pipeline.load_lora_weights("latent-consistency/lcm-lora-sdv1-5")
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\diffusers\loaders.py", line 1200, in load_lora_weights
    state_dict, network_alphas = self.lora_state_dict(pretrained_model_name_or_path_or_dict, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\diffusers\loaders.py", line 1351, in lora_state_dict
    weight_name = cls._best_guess_weight_name(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\diffusers\loaders.py", line 1401, in _best_guess_weight_name
    files_in_repo = model_info(pretrained_model_name_or_path_or_dict).siblings
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\huggingface_hub\utils\_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\huggingface_hub\hf_api.py", line 1697, in model_info
    r = get_session().get(path, headers=headers, timeout=timeout, params=params)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\requests\sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\huggingface_hub\utils\_http.py", line 63, in send
    return super().send(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\dev\push\faster\unify\fastsdcpu\env\Lib\site-packages\requests\adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: (MaxRetryError('HTTPSConnectionPool(host=\'huggingface.co\', port=443): Max retries exceeded with url: /api/models/latent-consistency/lcm-lora-sdv1-5 (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x00000198E0086790>: Failed to resolve \'huggingface.co\' ([Errno 11001] getaddrinfo failed)"))'), '(Request ID: 50f553fe-20fc-4790-a449-39f185542a17)')

System Info

• diffusers version: 0.23.0
• Platform: Windows-10-10.0.22631-SP0
• Python version: 3.11.0
• PyTorch version (GPU?): 2.0.1+cpu (False)
• Huggingface_hub version: 0.17.3
• Transformers version: 4.35.0
• Accelerate version: 0.23.0
• xFormers version: not installed
• Using GPU in script?: <fill in>
• Using distributed or parallel set-up in script?: <fill in>

Who can help?

@sayakpaul @patrickvonplaten