Summary: google-research/bert: TensorFlow code and pre-trained models for BERT

Pre-trained models

We are releasing the BERT-Base and BERT-Large models from the paper.

Using BERT has two stages: Pre-training and fine-tuning.

Pre-training with BERT

We are releasing code to do “masked LM” and “next sentence prediction” on an arbitrary text corpus.

python run_pretraining.py \ –input_file=/tmp/tf_examples.tfrecord \ –output_dir=/tmp/pretraining_output \ –do_train=True \ –do_eval=True \ –bert_config_file= $BERT_BASE_DIR /bert_config.json \ –init_checkpoint= $BERT_BASE_DIR /bert_model.ckpt \ –train_batch_size=32 \ –max_seq_length=128 \ –max_predictions_per_seq=20 \ –num_train_steps=20 \ –num_warmup_steps=10 \ –learning_rate=2e-5

This will produce an output like this:

***** Eval results ***** global_step = 20 loss = 0.0979674 masked_lm_accuracy = 0.985479 masked_lm_loss = 0.0979328 next_sentence_accuracy = 1.0 next_sentence_loss = 3.45724e-05

Note that since our sample_text.txt file is very small, this example training will overfit that data in only a few steps and produce unrealistically high accuracy numbers.

Model SQUAD 1.1 F1/EM Multi NLI Accuracy BERT-Large, Uncased (Original) 91.0/84.3 86.05 BERT-Large, Uncased (Whole Word Masking) 92.8/86.7 87.07 BERT-Large, Cased (Original) 91.5/84.8 86.09 BERT-Large, Cased (Whole Word Masking) 92.9/86.7 86.46

***** New February 7th, 2019: TfHub Module *****

BERT has been uploaded to TensorFlow Hub.

Similar Articles

google-research/bert: TensorFlow code and pre-trained models for BERT

TensorFlow code and pre-trained models for BERT. Contribute to google-research/bert development by creating an account on GitHub.

Read the complete article at: github.com