electra-italian-xxl-cased-squad-it

Electra model for (Extractive) Question Answering on Italian texts

Model description

This model has been fine-tuned on squad_it dataset, starting from the pre-trained model dbmdz/electra-base-italian-xxl-cased-discriminator.

It can be used for Extractive Q&A on Italian texts.

Evaluation

Metric	Value
EM	0.660
F1	0.775

Evaluation notebook

Usage in Transformers 🤗

Model checkpoints are available for usage in PyTorch. They can be used directly with pipelines as:

from transformers import pipelines

qa = pipeline('question-answering', model='anakin87/electra-italian-xxl-cased-squad-it')
qa(question="Qual è il soprannome di Vasco Rossi?", context="Vasco Rossi, noto anche semplicemente come Vasco e in passato con l'appellativo Blasco (Zocca, 7 febbraio 1952), è un cantautore italiano")
>>> {'score': 0.93, 'start': 80, 'end': 86, 'answer': 'Blasco'}

Usage in Haystack 🚀🚀🚀

With the Haystack NLP framework, you can use this model and create a scalable Question Answering system that works across millions of documents.

For a complete walkthrough, see this notebook.

...
print_answers(prediction, details="medium")

>>> Query: Con chi ha parlato di vaccini il premier Mario Draghi?
Answers:
[   {   'answer': 'Von der Leyen',
        'context': " vaccino dell'azienda britannica. Durante la telefonata "
                   'tra Draghi e Von der Leyen, la presidente della '
                   'Commissione Ue ha annunciato al presidente del',
        'score': 0.9663902521133423},
    {   'answer': 'Ursula Von der Leyen',
        'context': 'colloquio telefonico con la presidente della Commissione '
                   'europea Ursula Von der Leyen. Secondo fonti di Palazzo '
                   'Chigi, dalla conversazione è emerso ch',
        'score': 0.9063920974731445},
    {   'answer': 'Mario Draghi, ha tenuto un lungo discorso alla 76esima '
                  'Assemblea Generale delle Nazioni Unite',
        'context': 'Il presidente del Consiglio, Mario Draghi, ha tenuto un '
                   'lungo discorso alla 76esima Assemblea Generale delle '
                   'Nazioni Unite, nella notte italiana. Tant',
        'score': 0.5243796706199646}]

Comparison ⚖️

Model	EM	F1	Model size (PyTorch)	Architecture
it5/it5-large-question-answering	69.10	78.00	3.13 GB	encoder-decoder
anakin87/electra-italian-xxl-cased-squad-it (this one)	66.03	77.47	437 MB	encoder
it5/it5-base-question-answering	66.30	76.10	990 MB	encoder-decoder
it5/mt5-base-question-answering	66.30	75.70	2.33 GB	encoder-decoder
antoniocappiello/bert-base-italian-uncased-squad-it	63.80	75.30	440 MB	encoder
luigisaetta/squad_it_xxl_cased_hub1	63.95	75.27	440 MB	encoder
it5/it5-efficient-small-el32-question-answering	64.50	74.70	569 MB	encoder-decoder
mrm8488/bert-italian-finedtuned-squadv1-it-alfa	62.51	74.16	440 MB	encoder
mrm8488/umberto-wikipedia-uncased-v1-finetuned-squadv1-it	60.50	72.41	443 MB	encoder
it5/it5-small-question-answering	61.90	71.60	308 MB	encoder-decoder
it5/mt5-small-question-answering	56.00	66.00	1.2 GB	encoder-decoder
DrQA-it trained on SQuAD-it	56.10	65.90	?	?

Training details 🏋️‍

Training notebook

Hyperparameters

learning_rate: 2e-05
batch_size: 8
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 2
mixed_precision_training: Native AMP

Created by Stefano Fiorucci/anakin87

Made with ♥ in Italy