RoBERTa instead of BERT: what changes in SPECTER scripts?

[piegu]

Hi @armancohan.

The SPECTER scripts use BERT.

Could you give the changes to do in order to use a model RoBERTa (for example: there is no file vocab.txt for a model RoBERTa, the tokenizer is not WordPiece, etc.)? Thank you.

List of scripts:

• create_training_files.py
• pytorch_lightning_training_script/train.py
• other scripts to change?

About the script create_training_files.py, here the lines where "something" needs to be changed, no?

• 17
  from allennlp.data.token_indexers import SingleIdTokenIndexer, PretrainedBertIndexer
• 19
  from allennlp.data.tokenizers.word_splitter import WordSplitter, SimpleWordSplitter, BertBasicWordSplitter
• 47
  "do_lowercase": "true",
• 48
  "pretrained_model": "data/scivocab_scivocab_uncased/vocab.txt",
• 121
  _tokenizer = WordTokenizer(word_splitter=BertBasicWordSplitter(do_lower_case=bert_params["do_lowercase"]))
• 122
  _token_indexers = {"bert": PretrainedBertIndexer.from_params(Params(bert_params))}
• 417

def main(data_files, train_ids, val_ids, test_ids, metadata_file, outdir, n_jobs=1, njobs_raw=1,
         margin_fraction=0.5, ratio_hard_negatives=0.3, samples_per_query=5, comment='', bert_vocab='',
         concat_title_abstract=False, included_text_fields='title abstract'):

• 502
  ap.add_argument('--bert_vocab', help='path to bert vocab', default='data/scibert_scivocab_uncased/vocab.txt')
• 519

main([data_file], [train_ids], [val_ids], [test_ids], metadata_file, args.outdir, args.njobs, args.njobs_raw,
         margin_fraction=args.margin_fraction, ratio_hard_negatives=args.ratio_hard_negatives,
         samples_per_query=args.samples_per_query, comment=args.comment, bert_vocab=args.bert_vocab,
         concat_title_abstract=args.concat_title_abstract, included_text_fields=args.included_text_fields
         )

About the script pytorch_lightning_training_script/train.py, as the classes AutoTokenizer and AutoModel are used, there is nothing to change?