adapters: ImportError: cannot import name 'AutoModelWithHeads' from 'transformers'

Hi I am trying with this example colab: https://colab.research.google.com/github/Adapter-Hub/website/blob/master/app/static/notebooks/Adapter_Quickstart_Training.ipynb#scrollTo=Lbwb3NRf8mBF

getting this error:

Traceback (most recent call last):
  File "test.py", line 11, in <module>
    from transformers import AutoTokenizer, EvalPrediction, GlueDataset, GlueDataTrainingArguments, AutoModelWithHeads, AdapterType
ImportError: cannot import name 'AutoModelWithHeads' from 'transformers' (/idiap/user/rkarimi/libs/anaconda3/envs/adapter/lib/python3.7/site-packages/transformers/__init__.py)

versions

(adapter) rkarimi@italix17:/idiap/user/rkarimi/dev/internship/seq2seq/adapter-transformers$ conda list | grep transformers
adapter-transformers      1.0.1                     <pip>
transformers              3.5.1                     <pip>
(adapter) rkarimi@italix17:/idiap/user/rkarimi/dev/internship/seq2seq/adapter-transformers$ conda list | grep pytorch
pytorch-lightning         1.0.4                     <pip>
adapter hub from github is installed

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 17 (7 by maintainers)

Most upvoted comments

There’s no standard way to integrate adapters into custom models. If our implementations for bert-based models don’t work for you, please have a look at our implementation to get an idea how it might be adapted to your needs. Here are the important parts of the implementation:

  • adapter_modeling.py implements the single adapter modules
  • adapter_bert.py integrates the adapter modules into the BERT architecture. Stacking and Fusion of adapters are implemented here. All the integration into the Huggingface model is done via Python mixins. Note how every module of BERT in modeling_bert.py has a corresponding mixin here. When supporting a custom model, you probably want to take this file and make changes to fit your model.
  • adapter_model_mixin.py implements useful methods for saving and loading adapters. The classes here are explained in https://docs.adapterhub.ml/extending.html
  • in the actual model class (e.g. modeling_bert.py), we only implement the mixins. No other changes are done here compared to Huggingface.

Those are all the main classes where our changes are implemented.

Unfortunately, we can’t give detailed support for implementing custom models. If your model is part of the Huggingface repo, you can open a feature request so we can potentially support it officially. Otherwise, we can try to help out if you have any further specific questions to our implementation.

The pip adapter-transformers 1.0.1 is built on transformers 2.11.0 link However, the master branch of adapter-transformers is already updated to transformers 3.5.1 You can run pip install git+https://github.com/Adapter-Hub/adapter-transformers.git to get that version.