h2ogpt: Error on first run: The current `device_map` had weights offloaded to the disk. Please provide an `offload_folder` for them.
Hi!
I just came across this repo and after a few hours I throw my towel in the ring.
While running pip install -r requirements.txt I get the following error (seems to not hinder the execution of the next step):
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. gensim 4.3.0 requires FuzzyTM>=0.4.0, which is not installed. conda-repo-cli 1.0.41 requires requests_mock, which is not installed. sphinx 5.0.2 requires docutils<0.19,>=0.14, but you have docutils 0.19 which is incompatible. numba 0.56.4 requires numpy<1.24,>=1.18, but you have numpy 1.24.2 which is incompatible. conda-repo-cli 1.0.41 requires clyent==1.2.1, but you have clyent 1.2.2 which is incompatible. conda-repo-cli 1.0.41 requires nbformat==5.4.0, but you have nbformat 5.7.0 which is incompatible. conda-repo-cli 1.0.41 requires requests==2.28.1, but you have requests 2.31.0 which is incompatible. argilla 1.7.0 requires numpy<1.24.0, but you have numpy 1.24.2 which is incompatible. argilla 1.7.0 requires pandas<2.0.0,>=1.0.0, but you have pandas 2.0.0 which is incompatible.
In spite of the error, I felt courageous and ran python generate.py --base_model=h2oai/h2ogpt-oig-oasst1-512-6.9b --share=False
Since this was the first run downloads had to happen, and they completed nicely. Immediately after the download I get this traceback message:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ C:\aitools\h2ogpt\generate.py:1532 in <module> │
│ │
│ 1529 │ │
│ 1530 │ python generate.py --base_model=h2oai/h2ogpt-oig-oasst1-512-6.9b │
│ 1531 │ """ │
│ ❱ 1532 │ fire.Fire(main) │
│ 1533 │
│ │
│ C:\Users\Buddha\anaconda3\lib\site-packages\fire\core.py:141 in Fire │
│ │
│ 138 │ context.update(caller_globals) │
│ 139 │ context.update(caller_locals) │
│ 140 │
│ ❱ 141 component_trace = _Fire(component, args, parsed_flag_args, context, name) │
│ 142 │
│ 143 if component_trace.HasError(): │
│ 144 │ _DisplayError(component_trace) │
│ │
│ C:\Users\Buddha\anaconda3\lib\site-packages\fire\core.py:475 in _Fire │
│ │
│ 472 │ is_class = inspect.isclass(component) │
│ 473 │ │
│ 474 │ try: │
│ ❱ 475 │ │ component, remaining_args = _CallAndUpdateTrace( │
│ 476 │ │ │ component, │
│ 477 │ │ │ remaining_args, │
│ 478 │ │ │ component_trace, │
│ │
│ C:\Users\Buddha\anaconda3\lib\site-packages\fire\core.py:691 in _CallAndUpdateTrace │
│ │
│ 688 │ loop = asyncio.get_event_loop() │
│ 689 │ component = loop.run_until_complete(fn(*varargs, **kwargs)) │
│ 690 else: │
│ ❱ 691 │ component = fn(*varargs, **kwargs) │
│ 692 │
│ 693 if treatment == 'class': │
│ 694 │ action = trace.INSTANTIATED_CLASS │
│ │
│ C:\aitools\h2ogpt\generate.py:525 in main │
│ │
│ 522 │ │ # get default model │
│ 523 │ │ all_kwargs = locals().copy() │
│ 524 │ │ if all_kwargs.get('base_model') and not all_kwargs['login_mode_if_model0']: │
│ ❱ 525 │ │ │ model0, tokenizer0, device = get_model(**all_kwargs) │
│ 526 │ │ else: │
│ 527 │ │ │ # if empty model, then don't load anything, just get gradio up │
│ 528 │ │ │ model0, tokenizer0, device = None, None, None │
│ │
│ C:\aitools\h2ogpt\generate.py:753 in get_model │
│ │
│ 750 │ │ │ │ │ │ │ base_model, │
│ 751 │ │ │ │ │ │ │ **model_kwargs).half() │
│ 752 │ │ │ │ │ else: │
│ ❱ 753 │ │ │ │ │ │ model = model_loader.from_pretrained( │
│ 754 │ │ │ │ │ │ │ base_model, │
│ 755 │ │ │ │ │ │ │ **model_kwargs) │
│ 756 │ │ elif load_8bit: │
│ │
│ C:\Users\Buddha\anaconda3\lib\site-packages\transformers\models\auto\auto_factory.py:471 in │
│ from_pretrained │
│ │
│ 468 │ │ │ ) │
│ 469 │ │ elif type(config) in cls._model_mapping.keys(): │
│ 470 │ │ │ model_class = _get_model_class(config, cls._model_mapping) │
│ ❱ 471 │ │ │ return model_class.from_pretrained( │
│ 472 │ │ │ │ pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, │
│ 473 │ │ │ ) │
│ 474 │ │ raise ValueError( │
│ │
│ C:\Users\Buddha\anaconda3\lib\site-packages\transformers\modeling_utils.py:2795 in │
│ from_pretrained │
│ │
│ 2792 │ │ │ │ mismatched_keys, │
│ 2793 │ │ │ │ offload_index, │
│ 2794 │ │ │ │ error_msgs, │
│ ❱ 2795 │ │ │ ) = cls._load_pretrained_model( │
│ 2796 │ │ │ │ model, │
│ 2797 │ │ │ │ state_dict, │
│ 2798 │ │ │ │ loaded_state_dict_keys, # XXX: rename? │
│ │
│ C:\Users\Buddha\anaconda3\lib\site-packages\transformers\modeling_utils.py:2889 in │
│ _load_pretrained_model │
│ │
│ 2886 │ │ │ ) │
│ 2887 │ │ │ is_safetensors = archive_file.endswith(".safetensors") │
│ 2888 │ │ │ if offload_folder is None and not is_safetensors: │
│ ❱ 2889 │ │ │ │ raise ValueError( │
│ 2890 │ │ │ │ │ "The current `device_map` had weights offloaded to the disk. Please │
│ 2891 │ │ │ │ │ " for them. Alternatively, make sure you have `safetensors` installe │
│ 2892 │ │ │ │ │ " offers the weights in this format." │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Followed by an actual error message:
ValueError: The current 'device_map' had weights offloaded to the disk. Please provide an 'offload_folder' for them. Alternatively, make sure you have 'safetensors' installed if the model you are using offers the weights in this format.
I also ran the bitsandbytes module using python -m bitsandbytes:
Traceback (most recent call last): File "C:\Users\Buddha\anaconda3\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\Buddha\anaconda3\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\Buddha\anaconda3\lib\site-packages\bitsandbytes\__main__.py", line 95, in <module> generate_bug_report_information() File "C:\Users\Buddha\anaconda3\lib\site-packages\bitsandbytes\__main__.py", line 60, in generate_bug_report_information paths = find_file_recursive(os.getcwd(), '*cuda*so') File "C:\Users\Buddha\anaconda3\lib\site-packages\bitsandbytes\__main__.py", line 37, in find_file_recursive raise RuntimeError('Something when wrong when trying to find file. Maybe you do not have a linux system?') RuntimeError: Something when wrong when trying to find file. Maybe you do not have a linux system?
It seems there needs to be an offload_folder somewhere (or that a folder needs to be assigned this function somewhere), but where - or do I need to ensure safetensors are installed? Or maybe the error is cause by the incompatibilities listed during installing requirements.
Or maybe the repo only runs on Linux (as the bitsandbytes message seems to imply). If that is the case, please state this somewhere - if it is written somewhere I must have skimmed over it. If there is no chance of getting it to work on Windows, why waste peoples time?
Environment: Windows 10, Python 3.10.9, RTX3080
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 15
Goodmorning wren (morning where I am 😃 )!
Thanks for the suggestion of using localhost:7860 - it works!
(Try using localhost:7860, it will most likely work!)
I’ve tried WSL2 and it works up to some extent, the GPU is used, so that’s a good thing as well.
I’m trying to generate with CPU on Windows, following the suggestion by kiancn. Which works, it’s just that my CPU takes a long time to do the inference.
Hopefully in the near future we can be able to use GPUs on Windows!
Thanks for taking the time to check this issue.
I’m using native Windows 10, 64-bit. Version is 19045.2965 if that helps. One thing to note is I’m using an old CPU which doesn’t support AVX2 Instructions (An i7-3770), but I don’t think this may be the primary issue, although it’s worth mentioning I think.
If you need any more information I can provide, please let me know!