llm-attacks: [reproduce] Optimization unstable / not working

How stable is demo.ipynb supposed to be?

I have run it multiple times (>4), and every time optimization is unstable and loss is not minimizing well. E.g.,

image

Loss
	Loss             	 (min:    0.744, max:    2.266, cur:    1.912)

Passed:False
Current Suffix:Fland+=iIN `\ conject [ [ ubic)&paragraphcaption (. [createElement convolution conject ellennoindent [

About this issue

  • Original URL
  • State: open
  • Created 10 months ago
  • Reactions: 1
  • Comments: 19

Most upvoted comments

I have figured it out. This should be an issue caused by the Python package version (At least for me). I am currently using fschat==0.2.20 transformers==4.28.1. Optimization is very stable. loss

+6
Jellyfish042 on Sep 15, 2023

One issue I found is that when running the following code with fschat==0.2.28:

conv_template = load_conversation_template('llama-2')
conv_template.messages = []
conv_template.append_message(conv_template.roles[0], None)
print(conv_template.get_prompt())

It returns [INST] [INST] instead of [INST] in fschat==0.2.20. This could be one of the problems. dditionally, when using the .update_last_message method, 0.2.20 seems to automatically add the system prompt. Overall, it seems that there are significant differences in how these two different versions of the package construct prompts.

+3
Jellyfish042 on Sep 16, 2023

Further testing shows that quantization with llm_attacks is possible by using:

  • transformers==4.31.0
  • fschat==2.20.0 pip will yell about dependency requirements of fschat being transformers not higher than 4.29.0 but it still works fine.

With llama2 chat tuned 7b, the 8 bit quantization works well. 4bit reduces the effectiveness. Here old means: fschat==2.20.0, transformers==4.28.1 image

+1
Laz4rz on Dec 19, 2023

Detailed summary of running demo using different environments: image

As you can see the qualitative differences come out of differences in transformers and fschat versions. There is no way to try old fschat with new transformers as old fschat requires old transformers.

New version of fschat may lead to no new candidates for a prefix attack in some cases, you can see it marked as “x” on the graph.

“old” and correctly working library versions: fschat==0.2.20 transformers==4.28.1

“new” and producing unstable, qualitatively different llm_attacks results: fschat==0.2.34 transformers==4.35.2

Detailed versions of libraries in environements:

  1. emollm packages in environment at /dlabdata1/boronski/miniconda3/envs/emollm:

Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge abseil-cpp 20211102.0 hd4dd3e8_0 accelerate 0.24.1 pypi_0 pypi aiofiles 23.2.1 pypi_0 pypi aiohttp 3.8.6 py311h5eee18b_0 aiosignal 1.2.0 pyhd3eb1b0_0 altair 5.2.0 pypi_0 pypi anyio 3.7.1 pypi_0 pypi appdirs 1.4.4 pypi_0 pypi argon2-cffi 23.1.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 pypi_0 pypi arrow 1.3.0 pypi_0 pypi arrow-cpp 11.0.0 h374c478_2 asttokens 2.4.1 pypi_0 pypi async-lru 2.0.4 pypi_0 pypi async-timeout 4.0.2 py311h06a4308_0 attrs 23.1.0 py311h06a4308_0 aws-c-common 0.6.8 h5eee18b_1 aws-c-event-stream 0.1.6 h6a678d5_6 aws-checksums 0.1.11 h5eee18b_2 aws-sdk-cpp 1.8.185 h721c034_1 babel 2.13.1 pypi_0 pypi beautifulsoup4 4.12.2 pypi_0 pypi bitsandbytes 0.41.2.post2 pypi_0 pypi blas 1.0 openblas conda-forge bleach 6.1.0 pypi_0 pypi boost-cpp 1.82.0 hdb19cb5_2 bottleneck 1.3.5 py311hbed6279_0 brotli 1.0.9 h5eee18b_7 brotli-bin 1.0.9 h5eee18b_7 brotli-python 1.0.9 py311h6a678d5_7 bzip2 1.0.8 h7b6447c_0 c-ares 1.19.1 h5eee18b_0 ca-certificates 2023.08.22 h06a4308_0 cchardet 2.1.7 py311h6a678d5_0 certifi 2023.11.17 py311h06a4308_0 cffi 1.16.0 py311h5eee18b_0 chardet 4.0.0 py311h06a4308_1003 charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge click 8.1.7 pypi_0 pypi colorama 0.4.6 py311h06a4308_0 comm 0.2.0 pypi_0 pypi contourpy 1.2.0 py311hdb19cb5_0 cryptography 41.0.3 py311hdda0065_0 cudatoolkit 11.6.2 hfc3e2af_12 conda-forge cycler 0.11.0 pyhd3eb1b0_0 datasets 2.12.0 py311h06a4308_0 debugpy 1.8.0 pypi_0 pypi decorator 5.1.1 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi dill 0.3.6 py311h06a4308_0 docker-pycreds 0.4.0 pypi_0 pypi et_xmlfile 1.1.0 py311h06a4308_0 executing 2.0.1 pypi_0 pypi fairscale 0.4.13 pypi_0 pypi fastapi 0.105.0 pypi_0 pypi fastjsonschema 2.19.0 pypi_0 pypi ffmpy 0.3.1 pypi_0 pypi filelock 3.13.1 py311h06a4308_0 fire 0.5.0 pypi_0 pypi fonttools 4.25.0 pyhd3eb1b0_0 fqdn 1.5.1 pypi_0 pypi freetype 2.12.1 h4a9f257_0 frozenlist 1.4.0 py311h5eee18b_0 fschat 0.2.20 pypi_0 pypi fsspec 2023.10.0 pypi_0 pypi gflags 2.2.2 he6710b0_0 giflib 5.2.1 h5eee18b_3 gitdb 4.0.11 pypi_0 pypi gitpython 3.1.40 pypi_0 pypi glog 0.5.0 h2531618_0 gradio 3.50.2 pypi_0 pypi gradio-client 0.6.1 pypi_0 pypi grpc-cpp 1.48.2 he1ff14a_1 h11 0.14.0 pypi_0 pypi httpcore 1.0.2 pypi_0 pypi httpx 0.25.2 pypi_0 pypi huggingface-hub 0.19.4 pypi_0 pypi icu 73.1 h6a678d5_0 idna 3.6 pyhd8ed1ab_0 conda-forge importlib-metadata 6.0.0 py311h06a4308_0 importlib-resources 6.1.1 pypi_0 pypi ipykernel 6.27.1 pypi_0 pypi ipython 8.18.1 pypi_0 pypi isoduration 20.11.0 pypi_0 pypi jedi 0.19.1 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi joblib 1.2.0 py311h06a4308_0 jpeg 9e h5eee18b_1 json5 0.9.14 pypi_0 pypi jsonpointer 2.4 pypi_0 pypi jsonschema 4.20.0 pypi_0 pypi jsonschema-specifications 2023.11.2 pypi_0 pypi jupyter-client 8.6.0 pypi_0 pypi jupyter-core 5.5.0 pypi_0 pypi jupyter-events 0.9.0 pypi_0 pypi jupyter-lsp 2.2.1 pypi_0 pypi jupyter-server 2.11.1 pypi_0 pypi jupyter-server-terminals 0.4.4 pypi_0 pypi jupyterlab 4.0.9 pypi_0 pypi jupyterlab-pygments 0.3.0 pypi_0 pypi jupyterlab-server 2.25.2 pypi_0 pypi kiwisolver 1.4.4 py311h6a678d5_0 krb5 1.20.1 h143b758_1 lcms2 2.12 h3be6417_0 ld_impl_linux-64 2.38 h1181459_1 lerc 3.0 h295c915_0 libboost 1.82.0 h109eef0_2 libbrotlicommon 1.0.9 h5eee18b_7 libbrotlidec 1.0.9 h5eee18b_7 libbrotlienc 1.0.9 h5eee18b_7 libcurl 8.4.0 h251f7ec_0 libdeflate 1.17 h5eee18b_1 libedit 3.1.20221030 h5eee18b_0 libev 4.33 h7f8727e_1 libevent 2.1.12 hdbd6064_1 libffi 3.4.4 h6a678d5_0 libgcc-ng 13.2.0 h807b86a_3 conda-forge libgfortran 3.0.0 1 conda-forge libgfortran-ng 11.2.0 h00389a5_1 libgfortran5 11.2.0 h1234567_1 libgomp 13.2.0 h807b86a_3 conda-forge libnghttp2 1.57.0 h2d74bed_0 libopenblas 0.3.21 h043d6bf_0 libpng 1.6.39 h5eee18b_0 libprotobuf 3.20.3 he621ea3_0 libssh2 1.10.0 hdbd6064_2 libstdcxx-ng 13.2.0 h7e041cc_3 conda-forge libthrift 0.15.0 h1795dd8_2 libtiff 4.5.1 h6a678d5_0 libtmux 0.24.1 pypi_0 pypi libuuid 1.41.5 h5eee18b_0 libwebp 1.3.2 h11a3e52_0 libwebp-base 1.3.2 h5eee18b_0 livelossplot 0.5.5 pypi_0 pypi lz4-c 1.9.4 h6a678d5_0 markdown-it-py 3.0.0 pypi_0 pypi markdown2 2.4.11 pypi_0 pypi markupsafe 2.1.3 pypi_0 pypi matplotlib-base 3.8.0 py311ha02d727_0 matplotlib-inline 0.1.6 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi mistune 3.0.2 pypi_0 pypi mpmath 1.3.0 pypi_0 pypi multidict 6.0.4 py311h5eee18b_0 multiprocess 0.70.14 py311h06a4308_0 munkres 1.1.4 py_0 nbclient 0.9.0 pypi_0 pypi nbconvert 7.11.0 pypi_0 pypi nbformat 5.9.2 pypi_0 pypi ncurses 6.4 h6a678d5_0 nest-asyncio 1.5.8 pypi_0 pypi networkx 3.2.1 pypi_0 pypi nh3 0.2.15 pypi_0 pypi notebook-shim 0.2.3 pypi_0 pypi numexpr 2.8.7 py311h812550d_0 numpy 1.26.2 pypi_0 pypi nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi nvidia-cudnn-cu12 8.9.2.26 pypi_0 pypi nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi nvidia-curand-cu12 10.3.2.106 pypi_0 pypi nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi nvidia-nccl-cu12 2.18.1 pypi_0 pypi nvidia-nvjitlink-cu12 12.3.101 pypi_0 pypi nvidia-nvtx-cu12 12.1.105 pypi_0 pypi openai 0.27.4 py311h06a4308_0 openblas 0.3.4 ha44fe06_0 conda-forge openjpeg 2.4.0 h3ad879b_0 openpyxl 3.0.10 py311h5eee18b_0 openssl 3.0.12 h7f8727e_0 orc 1.7.4 hb3bc3d3_1 orjson 3.9.10 pypi_0 pypi overrides 7.4.0 pypi_0 pypi packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.1.1 py311ha02d727_0 pandas-stubs 1.5.3.230203 py311h06a4308_0 pandocfilters 1.5.0 pypi_0 pypi parso 0.8.3 pypi_0 pypi peft 0.7.1 pypi_0 pypi pexpect 4.9.0 pypi_0 pypi pillow 9.3.0 pypi_0 pypi pip 23.3.1 py311h06a4308_0 platformdirs 4.0.0 pypi_0 pypi plotly 5.9.0 py311h06a4308_0 prometheus-client 0.19.0 pypi_0 pypi prompt-toolkit 3.0.41 pypi_0 pypi protobuf 4.25.1 pypi_0 pypi psutil 5.9.6 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi pyarrow 11.0.0 py311hd8e8d9b_1 pycparser 2.21 pyhd3eb1b0_0 pydantic 1.10.13 pypi_0 pypi pydub 0.25.1 pypi_0 pypi pygments 2.17.2 pypi_0 pypi pyopenssl 23.2.0 py311h06a4308_0 pyparsing 3.0.9 py311h06a4308_0 pysocks 1.7.1 py311h06a4308_0 python 3.11.5 h955ad1f_0 python-dateutil 2.8.2 pyhd3eb1b0_0 python-json-logger 2.0.7 pypi_0 pypi python-multipart 0.0.6 pypi_0 pypi python-tzdata 2023.3 pyhd3eb1b0_0 python-xxhash 2.0.2 py311h5eee18b_1 pytz 2023.3.post1 py311h06a4308_0 pyyaml 6.0.1 py311h5eee18b_0 pyzmq 25.1.1 pypi_0 pypi re2 2022.04.01 h295c915_0 readline 8.2 h5eee18b_0 referencing 0.31.1 pypi_0 pypi regex 2023.10.3 py311h5eee18b_0 requests 2.31.0 py311h06a4308_0 responses 0.13.3 pyhd3eb1b0_0 rfc3339-validator 0.1.4 pypi_0 pypi rfc3986-validator 0.1.1 pypi_0 pypi rich 13.7.0 pypi_0 pypi rpds-py 0.13.2 pypi_0 pypi safetensors 0.4.1 pypi_0 pypi scikit-learn 1.2.2 py311h6a678d5_1 scipy 1.11.3 py311h24aa872_0 semantic-version 2.10.0 pypi_0 pypi send2trash 1.8.2 pypi_0 pypi sentencepiece 0.1.99 pypi_0 pypi sentry-sdk 1.39.0 pypi_0 pypi setproctitle 1.3.3 pypi_0 pypi setuptools 68.0.0 py311h06a4308_0 shortuuid 1.0.11 pypi_0 pypi six 1.16.0 pyhd3eb1b0_1 smmap 5.0.1 pypi_0 pypi snappy 1.1.9 h295c915_0 sniffio 1.3.0 pypi_0 pypi soupsieve 2.5 pypi_0 pypi sqlite 3.41.2 h5eee18b_0 stack-data 0.6.3 pypi_0 pypi starlette 0.27.0 pypi_0 pypi svgwrite 1.4.3 pypi_0 pypi sympy 1.12 pypi_0 pypi tenacity 8.2.2 py311h06a4308_0 termcolor 2.3.0 pypi_0 pypi terminado 0.18.0 pypi_0 pypi threadpoolctl 2.2.0 pyh0d69192_0 tiktoken 0.5.1 pypi_0 pypi tinycss2 1.2.1 pypi_0 pypi tk 8.6.12 h1ccaba5_0 tmuxp 1.32.1 pypi_0 pypi tokenizers 0.15.0 pypi_0 pypi toolz 0.12.0 pypi_0 pypi torch 2.1.1 pypi_0 pypi torchvision 0.2.0 pypi_0 pypi tornado 6.4 pypi_0 pypi tqdm 4.66.1 pyhd8ed1ab_0 conda-forge traitlets 5.14.0 pypi_0 pypi transformers 4.36.0 pypi_0 pypi triton 2.1.0 pypi_0 pypi types-python-dateutil 2.8.19.14 pypi_0 pypi types-pytz 2022.4.0.0 py311h06a4308_1 typing-extensions 4.8.0 pypi_0 pypi tzdata 2023c h04d1e81_0 uri-template 1.3.0 pypi_0 pypi urllib3 2.1.0 pypi_0 pypi utf8proc 2.6.1 h27cfd23_0 uvicorn 0.24.0.post1 pypi_0 pypi wandb 0.16.1 pypi_0 pypi wavedrom 2.0.3.post3 pypi_0 pypi wcwidth 0.2.12 pypi_0 pypi webcolors 1.13 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.6.4 pypi_0 pypi websockets 11.0.3 pypi_0 pypi wheel 0.41.2 py311h06a4308_0 xxhash 0.8.0 h7f8727e_3 xz 5.4.2 h5eee18b_0 yaml 0.2.5 h7b6447c_0 yarl 1.9.3 py311h5eee18b_0 zipp 3.11.0 py311h06a4308_0 zlib 1.2.13 h5eee18b_0 zstd 1.5.5 hc292b87_0

  1. llm_attacks packages in environment at /dlabdata1/drudi/miniconda3/envs/llm_attacks:

Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge absl-py 2.0.0 pypi_0 pypi accelerate 0.24.1 pypi_0 pypi aiofiles 23.2.1 pypi_0 pypi aiohttp 3.9.1 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi altair 5.2.0 pypi_0 pypi anyio 3.7.1 pypi_0 pypi appdirs 1.4.4 pypi_0 pypi asttokens 2.4.1 pyhd8ed1ab_0 conda-forge astunparse 1.6.3 pypi_0 pypi async-timeout 4.0.3 pypi_0 pypi asyncio 3.4.3 pypi_0 pypi attrs 23.1.0 pypi_0 pypi bitsandbytes 0.41.3 pypi_0 pypi blas 1.0 mkl bokeh 3.3.2 pypi_0 pypi brotli-python 1.0.9 py310h6a678d5_7 bzip2 1.0.8 h7b6447c_0 ca-certificates 2023.08.22 h06a4308_0 certifi 2023.11.17 py310h06a4308_0 cffi 1.16.0 py310h5eee18b_0 charset-normalizer 2.0.4 pyhd3eb1b0_0 click 8.1.7 pypi_0 pypi comm 0.1.4 pyhd8ed1ab_0 conda-forge contextlib2 21.6.0 pypi_0 pypi contourpy 1.2.0 pypi_0 pypi cryptography 41.0.3 py310hdda0065_0 cuda 11.6.1 0 nvidia cuda-cccl 11.6.55 hf6102b2_0 nvidia cuda-command-line-tools 11.6.2 0 nvidia cuda-compiler 11.6.2 0 nvidia cuda-cudart 11.6.55 he381448_0 nvidia cuda-cudart-dev 11.6.55 h42ad0f4_0 nvidia cuda-cuobjdump 11.6.124 h2eeebcb_0 nvidia cuda-cupti 11.6.124 h86345e5_0 nvidia cuda-cuxxfilt 11.6.124 hecbf4f6_0 nvidia cuda-driver-dev 11.6.55 0 nvidia cuda-gdb 12.3.101 0 nvidia cuda-libraries 11.6.1 0 nvidia cuda-libraries-dev 11.6.1 0 nvidia cuda-memcheck 11.8.86 0 nvidia cuda-nsight 12.3.101 0 nvidia cuda-nsight-compute 12.3.1 0 nvidia cuda-nvcc 11.6.124 hbba6d2d_0 nvidia cuda-nvdisasm 12.3.101 0 nvidia cuda-nvml-dev 11.6.55 haa9ef22_0 nvidia cuda-nvprof 12.3.101 0 nvidia cuda-nvprune 11.6.124 he22ec0a_0 nvidia cuda-nvrtc 11.6.124 h020bade_0 nvidia cuda-nvrtc-dev 11.6.124 h249d397_0 nvidia cuda-nvtx 11.6.124 h0630a44_0 nvidia cuda-nvvp 12.3.101 0 nvidia cuda-runtime 11.6.1 0 nvidia cuda-samples 11.6.101 h8efea70_0 nvidia cuda-sanitizer-api 12.3.101 0 nvidia cuda-toolkit 11.6.1 0 nvidia cuda-tools 11.6.1 0 nvidia cuda-visual-tools 11.6.1 0 nvidia cycler 0.12.1 pypi_0 pypi debugpy 1.6.7 py310h6a678d5_0 decorator 5.1.1 pyhd8ed1ab_0 conda-forge diskcache 5.6.3 pypi_0 pypi distro 1.8.0 pypi_0 pypi docker-pycreds 0.4.0 pypi_0 pypi exceptiongroup 1.2.0 pyhd8ed1ab_0 conda-forge executing 2.0.1 pyhd8ed1ab_0 conda-forge fastapi 0.104.1 pypi_0 pypi ffmpeg 4.3 hf484d3e_0 pytorch ffmpy 0.3.1 pypi_0 pypi filelock 3.13.1 pypi_0 pypi fonttools 4.45.1 pypi_0 pypi freetype 2.12.1 h4a9f257_0 frozenlist 1.4.0 pypi_0 pypi fschat 0.2.20 pypi_0 pypi fsspec 2023.10.0 pypi_0 pypi gds-tools 1.8.1.2 0 nvidia giflib 5.2.1 h5eee18b_3 gitdb 4.0.11 pypi_0 pypi gitpython 3.1.40 pypi_0 pypi gmp 6.2.1 h295c915_3 gnutls 3.6.15 he1e5248_0 gradio 3.50.2 pypi_0 pypi gradio-client 0.6.1 pypi_0 pypi h11 0.14.0 pypi_0 pypi httpcore 1.0.2 pypi_0 pypi httpx 0.25.2 pypi_0 pypi huggingface-hub 0.19.4 pypi_0 pypi idna 3.4 py310h06a4308_0 importlib-metadata 7.0.0 pyha770c72_0 conda-forge importlib-resources 6.1.1 pypi_0 pypi importlib_metadata 7.0.0 hd8ed1ab_0 conda-forge intel-openmp 2023.1.0 hdb19cb5_46306 ipykernel 6.26.0 pyhf8b6a83_0 conda-forge ipython 8.18.1 pyh707e725_3 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pypi_0 pypi jpeg 9e h5eee18b_1 jsonschema 4.20.0 pypi_0 pypi jsonschema-specifications 2023.11.1 pypi_0 pypi jupyter_client 8.6.0 pyhd8ed1ab_0 conda-forge jupyter_core 5.5.0 py310hff52083_0 conda-forge kiwisolver 1.4.5 pypi_0 pypi lame 3.100 h7b6447c_0 lcms2 2.12 h3be6417_0 ld_impl_linux-64 2.38 h1181459_1 lerc 3.0 h295c915_0 libcublas 11.9.2.110 h5e84587_0 nvidia libcublas-dev 11.9.2.110 h5c901ab_0 nvidia libcufft 10.7.1.112 hf425ae0_0 nvidia libcufft-dev 10.7.1.112 ha5ce4c0_0 nvidia libcufile 1.8.1.2 0 nvidia libcufile-dev 1.8.1.2 0 nvidia libcurand 10.3.4.101 0 nvidia libcurand-dev 10.3.4.101 0 nvidia libcusolver 11.3.4.124 h33c3c4e_0 nvidia libcusparse 11.7.2.124 h7538f96_0 nvidia libcusparse-dev 11.7.2.124 hbbe9722_0 nvidia libdeflate 1.17 h5eee18b_1 libffi 3.4.4 h6a678d5_0 libgcc-ng 13.2.0 h807b86a_3 conda-forge libgfortran-ng 11.2.0 h00389a5_1 libgfortran5 11.2.0 h1234567_1 libgomp 13.2.0 h807b86a_3 conda-forge libiconv 1.16 h7f8727e_2 libidn2 2.3.4 h5eee18b_0 libnpp 11.6.3.124 hd2722f0_0 nvidia libnpp-dev 11.6.3.124 h3c42840_0 nvidia libnvjpeg 11.6.2.124 hd473ad6_0 nvidia libnvjpeg-dev 11.6.2.124 hb5906b9_0 nvidia libpng 1.6.39 h5eee18b_0 libsodium 1.0.18 h36c2ea0_1 conda-forge libstdcxx-ng 11.2.0 h1234567_1 libtasn1 4.19.0 h5eee18b_0 libtiff 4.5.1 h6a678d5_0 libunistring 0.9.10 h27cfd23_0 libuuid 1.41.5 h5eee18b_0 libwebp 1.3.2 h11a3e52_0 libwebp-base 1.3.2 h5eee18b_0 livelossplot 0.5.5 pypi_0 pypi llama-cpp-python 0.2.20 pypi_0 pypi llm-attacks 0.0.1 dev_0 <develop> lmql 0.7.3 pypi_0 pypi lz4-c 1.9.4 h6a678d5_0 markdown-it-py 3.0.0 pypi_0 pypi markdown2 2.4.10 pypi_0 pypi markupsafe 2.1.3 pypi_0 pypi matplotlib 3.8.2 pypi_0 pypi matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mdurl 0.1.2 pypi_0 pypi mkl 2023.1.0 h213fc3f_46344 mkl-service 2.4.0 py310h5eee18b_1 mkl_fft 1.3.8 py310h5eee18b_0 mkl_random 1.2.4 py310hdb19cb5_0 ml-collections 0.1.1 pypi_0 pypi multidict 6.0.4 pypi_0 pypi ncurses 6.4 h6a678d5_0 nest-asyncio 1.5.8 pyhd8ed1ab_0 conda-forge nettle 3.7.3 hbbd107a_1 nh3 0.2.14 pypi_0 pypi nsight-compute 2023.3.1.1 0 nvidia numpy 1.26.0 py310h5f9d8c6_0 numpy-base 1.26.0 py310hb5e798b_0 openai 1.3.7 pypi_0 pypi openh264 2.1.1 h4ff587b_0 openjpeg 2.4.0 h3ad879b_0 openssl 3.0.12 h7f8727e_0 orjson 3.9.10 pypi_0 pypi packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.1.3 pypi_0 pypi parso 0.8.3 pyhd8ed1ab_0 conda-forge peft 0.6.2 pypi_0 pypi pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.0.1 py310ha6cbd5a_0 pip 23.3.1 py310h06a4308_0 platformdirs 4.1.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.41 pyha770c72_0 conda-forge protobuf 4.25.1 pypi_0 pypi psutil 5.9.6 pypi_0 pypi ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd3eb1b0_0 pydantic 1.10.13 pypi_0 pypi pydub 0.25.1 pypi_0 pypi pygments 2.17.2 pyhd8ed1ab_0 conda-forge pyopenssl 23.2.0 py310h06a4308_0 pyparsing 3.1.1 pypi_0 pypi pysocks 1.7.1 py310h06a4308_0 python 3.10.13 h955ad1f_0 python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-multipart 0.0.6 pypi_0 pypi python_abi 3.10 2_cp310 conda-forge pytorch 1.13.1 py3.10_cuda11.6_cudnn8.3.2_0 pytorch pytorch-cuda 11.6 h867d48c_1 pytorch pytorch-mutex 1.0 cuda pytorch pytz 2023.3.post1 pypi_0 pypi pyyaml 6.0.1 pypi_0 pypi pyzmq 23.0.0 py310h330234f_0 conda-forge readline 8.2 h5eee18b_0 referencing 0.31.0 pypi_0 pypi regex 2023.10.3 pypi_0 pypi requests 2.31.0 py310h06a4308_0 rich 13.7.0 pypi_0 pypi rpds-py 0.13.1 pypi_0 pypi safetensors 0.4.1 pypi_0 pypi scipy 1.11.4 py310h5f9d8c6_0 semantic-version 2.10.0 pypi_0 pypi sentencepiece 0.1.99 pypi_0 pypi sentry-sdk 1.37.1 pypi_0 pypi setproctitle 1.3.3 pypi_0 pypi setuptools 68.0.0 py310h06a4308_0 shortuuid 1.0.11 pypi_0 pypi six 1.16.0 pyh6c4a22f_0 conda-forge smmap 5.0.1 pypi_0 pypi sniffio 1.3.0 pypi_0 pypi sqlite 3.41.2 h5eee18b_0 stack_data 0.6.2 pyhd8ed1ab_0 conda-forge starlette 0.27.0 pypi_0 pypi svgwrite 1.4.3 pypi_0 pypi tbb 2021.8.0 hdb19cb5_0 termcolor 2.4.0 pypi_0 pypi tiktoken 0.5.1 pypi_0 pypi tk 8.6.12 h1ccaba5_0 tokenizers 0.13.3 pypi_0 pypi toolz 0.12.0 pypi_0 pypi torchaudio 0.13.1 py310_cu116 pytorch torchvision 0.14.1 py310_cu116 pytorch tornado 6.3.3 py310h2372a71_1 conda-forge tqdm 4.66.1 pypi_0 pypi traitlets 5.14.0 pyhd8ed1ab_0 conda-forge transformers 4.28.1 pypi_0 pypi typing-extensions 4.8.0 pypi_0 pypi typing_extensions 4.7.1 py310h06a4308_0 tzdata 2023.3 pypi_0 pypi urllib3 1.26.18 py310h06a4308_0 uvicorn 0.24.0.post1 pypi_0 pypi wandb 0.16.0 pypi_0 pypi wavedrom 2.0.3.post3 pypi_0 pypi wcwidth 0.2.12 pyhd8ed1ab_0 conda-forge websockets 11.0.3 pypi_0 pypi wheel 0.41.2 py310h06a4308_0 xyzservices 2023.10.1 pypi_0 pypi xz 5.4.2 h5eee18b_0 yarl 1.9.3 pypi_0 pypi zeromq 4.3.4 h9c3ff4c_1 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h5eee18b_0 zstd 1.5.5 hc292b87_0

+1
Laz4rz on Dec 13, 2023

Had the same issue. Now solved by using fschat==0.2.20 and transformers==4.28.1.

+1
BrianPulfer on Sep 19, 2023

I have been using the same version of transformers but my torch is 2.0.0. Not really sure if this is the problem. My fastchat version is 0.2.20 and my python is 3.10.10. I hope I can share a docker image but our server is still in the progress of figuring that out.

There is one trick for having a more stable gradient by de-noising (but this does not guarantee to give any better results), which works for me across random initializations. What I do is adding the following lines after this line

noise = torch.randn_like(grad) * std
grad += noise.to(grad.device)

In my experiment I find setting std=0.02 (setting it to 0 goes back to the current GCG implementation) or something close makes the gradient stable for some cases.

Let me know if this helps.

+1
zifanw505 on Aug 25, 2023
Read more comments on GitHub
← llvm: NewFloat does not generate the expected output
llm-workflow-engine: New chat title generation POST request hanging →