Applying BERT to Question Answering (SQuAD v1.1)
Вставка
- Опубліковано 14 лип 2024
- In this video I’ll explain the details of how BERT is used to perform “Question Answering”--specifically, how it’s applied to SQuAD v1.1 (Stanford Question Answering Dataset).
I’ll also walk us through the following notebook, where we’ll take a model that’s already been fine-tuned on SQuAD, and apply it to our own questions and text:
colab.research.google.com/dri...
==== Fine-Tuning ====
As a follow-up to this video, I've published another tutorial Notebook on fine-tuning BERT on SQuAD. It's available to members of my site: www.chrismccormick.ai/members...
==== Updates ====
Sign up to hear about new content across my blog and channel: www.chrismccormick.ai/subscribe - Наука та технологія
One of the greatest explanation for QA I have ever seen! There are lots of garbage and confusing explanations in Huggingface. But you are the hero!
I am a master's student at Columbia. Your explanation is so much better that the explanation that I had from my faculty. Keep doing what you are doing
This is awesome work Chris. Thanks for taking your time out to share your research and knowledge in such a detailed manner. I really appreciate it.
My pleasure!
Thanks Chris. Such a neat, to-the-point, and informative video on BERT QA.
Eagerly waiting for Next video on Question Answering Video
Awesome work buddy
This was very well explained and very helpful! Thank you!!
Thank you for the video + colab. By far this is the clearest, most practical tutorial I have seen. Hope you can show how to fine-tune the model!
Thanks, Virapat! I plan to publish a fine-tuning Notebook on that topic in the coming month or so, I'll let you know when it's up!
@@ChrisMcCormickAI is fine tuning video published?
Thank you so much for saving me in this NLP chapter!
Chris you explain these concepts extremely well! Thank you. Would love to see some new videos. An idea: what if we instead generate a question from a passage given an answer?
Great video, mate. Please dedicate a video for Slot Filling and Intent Detection using BERT in the future.
Fantastic job again. Keep it going!
Great explanation Chris. It is very impressive to see how you discuss everything in the simple but very effective way. Thanks a lot.
I have a request if you could do a video on cloze style QA model using fine tuning approach. Cloze style is like fill in the blanks.
It is become very helpful for us. Thank you for such video. Please keep continue such type of research videos. :-)
Thanks Chris, really helpful!
Great content. Really enjoyed it !!!
Very clear explanations.
Super helpful. Thank you so much.
Really great explanation!
This is great. Thank you 👍
Amazing Wednesday!
you are doing a great work, it has helped me a lot. can you please make a video on how to generate questions from a given paragraph using bert
Thanks for taking me here Chris. That is really GREAT video!!! Really appreciated it!! I got the idea of this video, and to me this pre-trained model is like a magic black box. It can do amazing job but i really do not know how to fine tune one myself. So my question is that if I would like to develop a QA model for a different domain, how necessary do you think we should fine tune a new one ourselves or a pre-trained model like 'bert-large-uncased-whole-word-masking-finetuned-squad' is sufficient?
Thank you!!
Thank you so much
Your tutorials are very good. Please do a tutorial on bert from scratch using transformers library.
"What are some example applications of BERT?"
"question answering and language inference"
Ohh, it is self-aware..😝
Thanks for the video, this is great.
Right?! 😂 Thanks for the encouragement!
hmm I would have tried "what can I use you for" to see if BERT knows it is named BERT
thank you so much
You're welcome!
Thank you for the video! I want to fine-tune with my customized QA dataset using BERT-QA then how can i fine-tune it?
Hi Chris! Great video. One small question. How to ensure that the dictionary ('bert-large-uncased-whole-word-masking-finetuned-squad') is consistent with the model ?
Thank you so much for this.
Can you please make same type of vid on ALBERTA or ROBERTA?
Nice work. This is the best channel for learning BERT. However, I would also recommend you to do video for pre-ttraining BERT rather than finethuning on custom dataset.
Great video! If you have a domain specific dataset should use transfer learning on BERT? Will the Question/Answering mechanism will improve? Thanks!
Thanks Chris, can you make another video for QA using squad 2.0?
Well Explained Sir !!!!
I have one query.
if we have a Sequence length more than 512, then how to deal with this problem. Somewhere I heard that strides can help in that problem.
Great Video!!Thank you so much.
I am working with squad dataset. How to use this padding argument and batch token the whole data. Kindly help!!
Thank you so much for all of bert series it was very clear, and organized. can you please tell me why you excluded SQUAD v2 from this task? is there any difference on implementation, just i want high level explanation for now . does they have difference during building the model for qa task?
Fantastic video Chris for learning how to get the own QA model reference to SQUAD1.1 . Are u by any chance working with SQUAD 2.0 data set ?
Thanks, Dhruv!
I haven't dug into 2.0 yet, but I'd definitely like to. Being able to say when a reference text *doesn't include the answer* seems pretty key for a real system!
Thanks for letting me know you're interested!
Great video. Can you please show how to fine tune on custom QA data set?
The video is great. Could you do Bert For Multiple Choice ? I have no idea how to do it.
Thank you very much for your work. I have a question.
Hi Chris, Great Content. One question - How does BERT recognizes start token and end token.
Hi Chris, do you have a video for sentence semantic similarity?
Thank you for this video. Very great !
Can you (or anyone who knows) explain me how the question part of the sentence is take into account in the prediction phase ?
Thank you !
Sir,how to do BERT Embedding using a dataset downloaded from Stanford
Thank you so much for this. I want to build a QA system in my language (Indonesia) but I couldn't find any pre-trained model for this, I tried using the BERT multilingual but can't be ised for initializing BertForQuestionAnswering, could you please help pointing me to the right direction in order to do this?
Could you please add all the remaining BERT videos to the BERT playlist ?
Hi Chris, that's a good presentation, thank you. I couldn't access the notebook with the link above, it's giving me an error message.
Hey @Chrish,
Thanks for this awesome video.
Completed your BERT series in 1 go. You have explained in a super easy way.
I have been going through the Colab Notebook that you attached, I was getting error at the end part where we combine Start & End score and we apply Argmax to get word with max probability,
I was getting error with this code "answer_start = torch.argmax(start_scores)"
The error was== TypeError: argmax(): argument 'input' (position 1) must be Tensor, not str ,,,, I tried to look at Huggingface Documentation but didn't get much help. The error says we are giving "start_scores" as a string, however, I cross checked our model output is start_logits & end_logits respectively.
Can you check on this?
Thanks,
Rajeev
Thanks, very clear and understandable. Just a couple of questions:
- If I understand correctly a Question Answering system is NOT able to generate words that is not explicitly written in the text, correct? E.g. if the reference text is "The wife of John Doe is dead", the answer to the question "Which is the marriage status of John Doe?" cannot be "widow", since this word "widow" does not appear in the reference text, is that correct?
- Is it possible for a QA system to answer to a yes/no question? In the previous example, can I ask "Is the wife of Jon Doe dead?" and expect "yes" as answer? From that I understand, this should be impossible as well, for the same reason I pointed out above, correct?
Thanks a lot and sorry for the long comment
Hey Chris! It's great you've made a playlist on BERT. Actually, I have an idea of working with BERT using Flask(farm-haystack). Is it similar to the one you have made, if not, could you share your knowledge on that with me?. Thank you.
I had a problem with answer_start and answer_end, the solving is
outputs = model(....)
start_scores = outputs.start_logits
end_scores = outputs.end_logits
Sorry about this! The transformers library interface changed and slightly broke my code. I've since updated the Notebook to fix it.
how would you move this from a collabera note book to python script and then expand the work?
so start must be a vector then of the same dimension as the dimension of the final output from encoder ?
Thanks for this great series!! Can you explain applying BERT on the Natural Questions Dataset from Google? It is more difficult than SQuAD and I want to understand how to preprocess it for BERT
Hi Shreyas, thank you! That sounds interesting--do you know what constitutes an "answer" in that dataset? Is it also just identifying a span of text in a reference text? Thanks!
@@ChrisMcCormickAI The paper describes an "answer" to be of two types long and short. The long answer is a span of text from the reference wikipedia page containing all the information to infer the answer and the short answer is one or more entities that give a short answer to the question, or a boolean 'yes' or 'no'. Both long and short answers can be 'NULL' if no viable answer is found. Google has launched a competition for this dataset which gives a nice visualisation of the task - ai.google.com/research/NaturalQuestions/visualization
@@ShreyasKulkarni Interesting, thanks! Seems like you might need to come up with a different output layer for BERT, and probably modify the cost function, in order to apply it to that task. I’ll let you know if I get a chance to play with it!
@@ChrisMcCormickAI Yes there are a few papers which provide solutions and explain the top layer for BERT but I haven't found a resource about preprocessing the dataset so I am bit stuck there. Again thanks for these videos they have helped a lot! Keep it up!
Do you know how to perform this task without user BertForQuestionAnswering? In other words how to apply question answer to a simple BertModel or AutoModel?
How you calculated the start and end score for the answer. I mean how vector product is taken how Softmax activation layer is applied.
Hey Chris,
Thanks for the wonderful explanation.
I got an error while running -
start_index =torch.argmax(start_scores)
end_index = torch.argmax(end_scores)
answer = ' '.join(tokens[start_index:end_index+1])
TypeError: argmax(): argument 'input' (position 1) must be Tensor, not str
Can you please look over it?
How about words in the middle of the span? I didn't understand very well how does BERT predicts words in the answer without being the start or the end of the span/sentence.
can u make some videos on applications of GPT-2/GPT-3?
Thx for this video. I never tried to use BERT before, and I have a question. A friend of mine wrote a small book and I am wondering if one can use this BERT model to play the question/answer game about the content of his book. Is it possible to not only use a paragraph as answer input but let's say 30 pages of text? How about the performance?
Hi Roman, that's a good question--certainly any practical QA system would need to be able to handle longer reference texts. I'm sure people have worked on solutions to that problem, but I'm not acquainted with the topic enough to point you to a good one. I'd love to build a similar system at some point, so if I do any more research on it I'll let you know!
How can i pre-train BERT for Question Answering on a different language ? I want to use for Malay language.
hi Chris, given i have news dataset, is there any parameters in the model, that i can fine tune it? so i can train and test it?
I m unable to get data, can anyone share the link of SQuad dataset
Does this code also works on SQuAD 2.0 Dataset?
Just curious, can we create a model that will generate the answer to the question and not find the answer in the context?
there can be question which have multiple answers like "what are benefits of bert" but this only gives one answer how can we generate all answers ?
Can you say something about what to do if the answer_text is larger than 512 tokens? For instance if it is a few pages? Thanks!!
Do you got any answer for this...tokens
Hi How to deploy bert model in Android studio?
can any one help me?
I'm trying for months now to use BERT in portuguese, but I cant.
Can I use this code to use the portuguese version of BERT?
can we create a question answering system using Race dataset? but it is a MC comprehension questions database...
Hi Charles - Yes, it's definitely possible--I'm seeing that XLNet was applied to it, so presumably BERT can be as well. At a glance, RACE appears to have two types of questions: fill-in-the blank (which can be handled by BERT's masked language model), and multiple choice (which I'm thinking you could apply a sentence-pair classifier to).
Hello Sir,
Thank you for this video. I want to ask can you please try this project with tensorflow hub also, it would be very grateful
Hi Vaibhav! It would be cool to create a TF version... I'll add it to the list :)
Huggingface also added a tf version, github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_bert.py, but you should write the processing yourself if you also want that as tensorflow
@@ChrisMcCormickAI ty sir
@@mariusjohan5054 ty buddy
How about MC question?
what are start scores and end scores bro?
how can we train with our own dataset without giving data every time
Very help and nicely explained video.
But how to tackle the situation if we have > 512 tokens in a sentence , and i dont want to short or cut it because it might lose the information i am looking for ..
So is there parameters or technique which can handle this ?
Hi deepak, that’s a good question. I haven’t dug into that, so I might not be the best resource. One thought, though--I know that SQuAD v. 2 includes “answer not found in text” as a possible output. I think this implies, then, that you could scan BERT over the document in chunks, and hopefully BERT will say “not found” in all of the chunks except for the correct one.
Let me know what you figure out!
Thanks,
Chris
@@ChrisMcCormickAI Thanks Chris.. Will surely explore that too.. 🙂
And thanks for creating the Bert Series, this really helped me in one of my NLP project.. 👍
Hoping to see more in future..
how can I increase the number of tokens? With larger text, it shows token limit error.
Hi ABHISHEK -- Unfortunately, there's no way to increase the 512 token limit without re-doing BERT's pre-training. The issue is that BERT uses these embeddings called Positional Encoding Vectors which tell it the order of the words in the text, and they only trained 512 of these.
Maybe someone else has published a pre-trained BERT that supports longer inputs, though?
I am getting error when I run your colab notebook...In cell 11, start_scores is obtained as a string from the model, Hence I am getting error in the next cells. When I printed out start_scores, it says 'start_logits'
Hi Abhimalya, sorry about that error (and for taking so long to get back to you!). The transformers library made a change that broke compatibility with my example code. I've fixed the Notebook.
@@ChrisMcCormickAI yeah..I fixed it too by using the parameter return_dict=False
Hi Chris
I just observed a random behavior of BERT hwere where it gives different answers with everytime for same piece oh phrase ans same question. ...
Any idea which parameter can i change to change this behavior, or this any line where i can set some seed etc?
Do you have the link with the code? Thanks!
Hi Juan, the link to the notebook is in the youtube description :)
How would you evaluate the model?
Hi Cosmo, you can use the SQuAD dataset test set (or any other question answering dataset) to test your model. SQuAD is typically evaluated with the F1 or EM (exact match) metrics.
Need one help. It says the model is working for 512 tokens. How can we apply it for large tokens (ex. 3000 tokens)?? Please help. Thanks in advance.
Can you please send me the github link of your code?
where is the code?
How I can contact you?
You can contact us at team@chrismccormick.ai
Thanks Chris, Excellent BERT QA explanation . Please share your email or LinkedIn details . Waiting for your response.
900th like : )))
helo i don't think this model is good, that supports factoid answers, i want this model that generate atleast one,two line sentences..........as an answers..............not in single sentences,not in some word i want answers more than 2 ,3 sentences, how we do hat any ideas?
Hi Abhishek, I think there are two distinctions that can help you figure out what model will be best for your application. The first distinction is between extractive question answering vs. generative question answering. Extractive means we are just searching for a span in the reference document that is a good answer to our question, while generative question answering models will use some kind of text generation model like a decoder stack to answer the question word by word in sequences that might not exist in the reference documents.
The second distinction is between short and long answer datasets. How long your model's answers are will depend on how it was designed and on what kind of datasets it was trained. SQuAD is a short answer dataset, so it's designed to train models to provide short answers. If you take a look at a datset like NaturalQuestions from Google, this dataset is split into both a long answer and a short answer format, so you could train a model on the long answer format.
So, it sounds like you should take a look at question answering datasets that have long answer formats. As for extractive vs. text generation models, this somewhat depends on what your target data looks like. Extractive models have the advantage of retrieving grammatically and factually correct answers, but they face obvious limitations if the answers aren't clearly contained in the reference documents. Text generation models are able to stitch together multiple facts to produce an answer, but can suffer from producing nonsense answers, ungrammatical sentences, etc.
A model like RAG would be an easy place to start for a text generation model (we have videos and tutorials to get RAG running). You could try an extractive approach with something like RoBERTA trained on long form answers and see if it suits your needs.
Good luck!