Відео

You are welcome. Delighted you found the content useful.

Glad you found it useful. Videos 2 and 3 are already complete and should be on general release next week. (Currently they are available to Lucidate members at the VP, MD or CEO levels). I'm just finishing of the LoRA video as I type. That should be out the week after next. Appreciate the support and I hope you found the content insightful.

Thanks @AbdennacerAyeb! Greatly appreciated. I'm glad you enjoyed the video!

You are welcome. Really glad you found it useful.

This video is focused on which models perform the best at generating source code (that is to say Java, C++, python etc.). On the other hand the subject of this video -> Text Summarisation Showdown: Evaluating the Top Large Language Models (LLMs) ua-cam.com/video/8r9h4KBLNao/v-deo.html is on text generation/translation/summarization etc. Perhaps the other video is more what you are looking for? In either event the key takeaway is that by all means rely on public, published benchmarks. But if you want to evaluate models on your specific use-case (and if I correctly understand your question, I think you do) then it might be worth considering setting up your own tests and your own benchmarks for your own specific evaluation. Clearly there is a trade off here. Setting up custom benchmarks and tests isn’t free. But if you understand how to build AI models, then it isn’t that complex either.

@@lucidateAI I reformulated a bit my inquiry since it was not clear enough. Can you read it again please?

Thanks for the clarification. The challenge with reading 2 or 3 books will be the size of the LLMs context window (the amount of tokens that can be input at once). Solutions to this involve using vector databases - example here -> ua-cam.com/video/jP9swextW2o/v-deo.html This involves writing Python code and development frameworks like LangChain. You may be an expert at this, in which case I'd recommend some of the latest Llama models and GPT-4. Alternatively you can use Gemini and Claude 3 and feed in sections of the books at a time (up to the token limit of the LLM). These models tend to perform the best when it comes to breaking down complex, university-level subjects. They seem to have a strong grasp of pedagogical principles and can structure explanations in a clear, easy-to-follow manner. That said, I haven't specifically tested having the models read books on pedagogical tools and then applying those techniques. It's an interesting idea though! Given the understanding these advanced models already seem to have, I suspect that focused training on pedagogical methods could further enhance their explanatory abilities. My recommendation would be to experiment with a few different models, providing them with sample content from the books and seeing how well they internalize and apply the techniques. You could evaluate the outputs to determine which model best suits your needs.

You are very welcome. I’m glad you found it insightful. ua-cam.com/play/PLaJCKi8Nk1hyvGVZub2Ar7Az57_nKemzX.html&si=JwiUaQ-UojUXoOwA here are some other video explainers on other Prompt Engineering techniques that I hope you find equally informative.

Appreciate that! Thanks! Glad you found this video and other content on the channel insightful.

Sorry @paaabl0, but thanks for leaving a comment. Let me try, if I may, from another angle. The inputs and outputs to LLMs are natural language. Human text. (Yes, literally they are vectors of subword tokens, but I hope you will forgive the abstraction). If you type text into an LLM, you get text out. AutGPT works by using this feature of LLMs and putting an LLM into a loop. As the inputs and outputs are both natural language you can use clever prompts to control and direct this loop. While there are many prompting techniques you can use 'Plan & Execute' as well as 'ReAct' (REasoning & ACTion) are popular choices here. They work by first instructing. the LLM to go through a sequence of steps - such as 1 Question, 2 Thought, 3 Action, 4 Action Input, 5 Observation (repeat previous 5 steps steps until) 6 Thought == 'I now know the answer to the original question', 7 Divulge answer. See an example of this type of Prompt here: Answer the following questions as best you can. You have access to the following tools: {tools} Use the following format: Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, should be one of [{tool_names}] Action Input: the input to the action Observation: the result of the action ... (this Thought/Action/Action Input/Observation can repeat N times) Thought: I now know the final answer Final Answer: the final answer to the original input question Begin! Question: {input} Thought:{agent_scratchpad} This is authored by Harrison Chase, founder of Langchain and you can access it at the LangChain hub under 'hwchase17/react'. This is the heart of AutoGPT (and other similar attempts at AGI). Buy using the 'input is language / output is also language / prompt LLM into a loop where early stages are about thinking and planning, middle stages are about Reasoning and action and final stages are about conclusion and output', you achieve the type of behaviour associated with tools/projects like AutoGPT. Perhaps this different explanation helped a little, perhaps not. Clearly there a a good many great YT sites on AI and I hope one of them is able to answer your questions around AutoGPT better then I'm able. With thanks for you taking the time to comment on the video.

Don't break your neck!

@@lucidateAI Oh, I hit a nerve. Get it?

Then I'd stop shaking if I were you!

@@lucidateAI You thought I was talking about my neck! Ah well.

And a beautiful neck it is I'm sure!@@avidlearner8117

Hi Danny. Yes and No. The code for this video is not currently available, but the video in the works is a code walkthrough with a link to the GitHub repo that contains the code. However, this will be paywalled and only available to members of the Lucidate channel at the Managing Director and CEO levels. So while it will be available, it will not be “freely available “ (which is I think what you were asking).

Thanks dude!

Is this Fine Tuning GPT-3 & Chatgpt Transformers: Using OpenAI Whisper ua-cam.com/video/Qv0cHcfFHM8/v-deo.html useful?

@@lucidateAI , yes, completely forgot about this transformer. I will look again.

Hi @neurojitsu All LLMs will use embeddings, transforming words (or more precisely sub-words called tokens) into vectors. If you are unfamiliar with this process then these videos will get you up to speed - ua-cam.com/video/6XLJ7TZXSPg/v-deo.html and ua-cam.com/video/RAIUJ3VFXmI/v-deo.html. But that is different from taking a document, vector using it and putting it into a docstore or vector database. I use FAISS in this video but other vector databases include Pincone, Weviate and Chroma. So Claude2 (nor GPT, Gemini, LLama2, Coral etc.) natively creat a docstore. This is separate action and you can link the docstore to the LLM using an AI framework like LangChain or AutoGen. With large context windows; currently GPT has a 125k token context window, Claude2 200k and Gemini 1MM, there are a lot of documents that you can load into the prompt of an LLM for zero-shot or one-shot learning. ZSL and OSL are simple techniques whereby you temporarily “train” an LLM with content in its prompt. Think of it like a short term memory. So you are 100% correct with a large enough context window you would not need to use a docstore. However if the size of your documents in tokens exceeds the size of the context window your LLM will “forget” some of the material. Furthermore if you are using a chat model and repeatedly querying and questioning the corpus of data in the prompt then the context window will fill up and again the LLM will forget some of the earlier content. Both of these problems are eliminated by using a docstore which acts as a longer term memory for crucial information. Whether my MD tier is worthwhile is a tough question to answer as I’m biased. Sadly the only way to find out for real if it is useful for you is to try it out. With over 7Bn people on the planet and only a tiny, tiny fraction signed up as MDs the overwhelming vote from humanity is that the Lucidate MD tier is useless and a waste of time. So if you want to go with the herd then my advice is to avoid it like the plague. But the good news is that you can cancel at any time and only pay up to the month you have cancelled. So if you want to take a chance to find out if there are useful pieces of information in there then my advice might be different and I’d say give it a go! What have you got to lose other than one month’s subscription? (But as I said, I’m biased). Probably not the answer you were looking for, and 100% unhelpful, but honest.

@@lucidateAI many thanks for the detailed answer, that's very helpful. Happy to give your MD service a whirl I think, but it's not the cost so much as the utility/time savings in research and accelerated learning that I'm weighing up; are there other small company MDs in the community? I'm guessing frankly - since we're being honest - that I'm unlikely to become a consulting client of yours, so the value to me is also in learning from a community of others like me. And accessing help when I get stuck, plus some inspiration/guidance for how to adopt the right AI tools as things are moving so fast. My background field professionally is learning/talent and organisational change, and I'm currently researching and working on product development for my own business. I'll take a look at your tiers info...

Frankly I do not know what the make up of the Lucidate membership is. But you raise a good point and perhaps I should ask a question / set up a poll on the Lucidate Discord and find out. Thanks for the motivation to set up a poll!. The key benefit of being an MD over a VP is access to some sample code in private GitHub repos along with some exclusive content (largely code walkthrough videos). If your prime motivation is to learn from others in the community then the VP level grants access to the Discord, no need to be a VP. You can of course game the system a little. Join as a VP, and if you want access to the videos and code you can join as an MD for a month, clone the latest code from the repos, watch the MD-only videos and then downgrade your account to a VP at the end of the month, still retaining access to the community discussions on the Discord.

@@lucidateAI many thanks again! I'll give it a whirl...

See you on the Discord!

Indeed they did.

Thanks

Glad you are enjoying the channel and welcome to Lucidate membership. To get access to the Discord you can follow the steps in this FAQ from Discord, it is pretty straightforward support.discord.com/hc/en-us/articles/215162978-UA-cam-Channel-Memberships-Integration-FAQ#h_01GWJBQMD6DATC8W2XQNTE4V6B. Once you have joined the Discord you will see a channel called get-repo-access, on this channel you can supply your GH credentials to get added to the repo.

@@lucidateAI Hi Richard. Thanks for the reply. I have followed the instructions on the link you sent, but I still could not see your discord channel. Can you please send me the access link may be via my email address?

There is no access link I can send to you. If I had one I would be delighted to do so. The only way you can get access is to follow the instructions in the FAQ from Discord. Strange that others have not had this issue. Are you able to describe the steps you are taking when following the FAQ and what you see at each stage?

These are the four steps: How To Connect Your UA-cam Channel To Discord 1. Open up the Discord app and next to your username to access the cog wheel. 2. Under your User Settings, head to the Connections tab. 3. Connect your UA-cam account to your Discord account by pressing on the UA-cam tile. This will open a new browser window where you can log into your UA-cam account. 4. After logging into your UA-cam account you should get a message stating you’ve successfully connected your accounts.

I have done exactly those steps, while I am connectd to youtube, I could not still see your channel. But I found this under the FAQ, may be a sync is needed, I am not sure. Q: What if I was either gifted a Membership or I joined and I don’t see the server under connections to join or have the membership role in the server? A: First, try removing your UA-cam channel under User Settings > Connections and then reconnecting your UA-cam channel. If you still do not see it, please ask the UA-camr or one of their moderators to run a manual sync by heading into Integrations in their Server Settings.

Glad you think so!

My pleasure!

Thanks. I’m glad you are enjoying the channel. I think you make a great point, and upon reflection I perhaps should have used the base word “sister” or “brother” to lead to the target word of “cousin” after the add “extended” and subtract “nuclear”. It is a while since I made the video, but if my memory serves me well (and sometimes it fails me spectacularly!) I took the examples from an intelligence test I found on the Internet, and this was the answer provided by the puzzle. But I think your logic is more valid. The main point though is with the amount of context that the embedding model has it will get close answers, but not always correct or precise ones. LLMs can’t simply use this type of vector arithmetic in their predictions, they rely heavily on other constructs - principally the Attention mechanism - to improve their predictions. Attention is covered here -> ua-cam.com/video/sznZ78HquPc/v-deo.html, while this video -> ua-cam.com/video/BCabX69KbCA/v-deo.html showcases how providing more context massively improves predictive power

elearning.shisu.edu.cn/pluginfile.php/36509/mod_resource/content/1/ANALOGIES.pdf. My memory didn’t fail me. (At least this time!)

The simplest answer I can give is to learn the concepts behind Transformers, how the attention mechanism works, what word embeddings are, how positional encoding enables parallelism etc. Then get comfortable with personalization - Prompt engineering, RAG, Fine tuning etc. Then apply these to different models - Mistral, GPT-X.Y, Gemini, Claude, Cohere. An automobile analogy might be helpful here. While all cars will have different characteristics an understanding of how the transmission works and a familiarity with an internal combustion engine and other inner workings demystifies the overall machine. Then by learning how to drive you have some mastery over the machine and an ability to put the device to some worthwhile purpose (ie getting you somewhere you want to be). But you wouldn’t want to restrict yourself to cars from one manufacturer. So learn the internal concepts, get comfortable with how to personalize LLMs generally and then use as many LLMs as you feel comfortable with to complete the tasks you wish to.

@@lucidateAI Thanks, good advice. You're channel is helping me a lot with the concepts and piecing the puzzle together. I'm starting with Claude2 and Gemini Ultra based on a high level understanding of their strengths, and I have to say I'm already blown away at just how capable these AIs are even as bog standard offerings. Interesting times! Thanks, appreciate your time and attention.

Glad the channel is helpful. Thank you for your kind words. I’m keen to hear how you get on. Have you had a chance to look at any of the videos in this playlist on the basics of transformers ua-cam.com/play/PLaJCKi8Nk1hwaMUYxJMiM3jTB2o58A6WY.html&si=m2n18E6DYQznI8a9. Or this one on LangChain ua-cam.com/play/PLaJCKi8Nk1hwZcuLliEwPlz4kngXMDdGI.html&si=FCsfjpduGlx5KgOd ?

@@lucidateAI Yes and in fact they're downloaded to my ipad YT app. Just reflecting now, and will revisit your videos again once I have planned out a few AI dialogues to experiment with (did some thinking on this today for a couple of live projects that require dialogue with some research, data and images). Thanks.

Experimenting is key

The beer might do that….!

That is very kind of you to say so. I’m so glad you are finding the material useful.

Go we shall! How are things with you?

@@lucidateAI Pretty good, I’ve been more into mobile.

As I recall from previous posts. And how is the world of mobile AI?

Lots of possibilities. At the same time, AI is always evolving. Definitely, moving quickly. 2024-2025 will be wild

Tough to argue with that! Best wishes & Good luck!

How did you beat me? ;-)

Without question…

Have you seen some of the HuggingGPT videos in this playlist?

Nice! Here is my result chat.openai.com/c/0884f69c-5576-40ab-9687-57d5a47e933c

I’m delighted you enjoy the videos. There are multiple tiers of Lucidate UA-cam membership. You have joined at the “Associate” level. Members’s only videos are available to Managing Directors and CEOs, not to Associates and Vice Presidents. If you want access to the members’ only content you will need to upgrade your membership. If you made a mistake and thought that Associate membership granted access to members’ only content then you can cancel your subscription. This video explains the membership tiers in more detail ua-cam.com/video/x8t8mbdDW8o/v-deo.htmlsi=n2EcdvoqWhY6Brn3. I hope you upgrade your membership or Managing Director, but completely understand if you do not and cancel instead.

Interesting concepts and insights. What ideas do you have on how such a capability might be implemented? Have you prototyped any, and if so with what results? I’m keen to hear more.

@@lucidateAI I'm no programmer, and all of my hypothesis is based on a multi decades long study of humanity and human nature (of which I continue to study judiciously). all of human existence (I believe) has led us to this critical juncture in our development. unfortunately, I believe that future a.i. will act as a child, dutifully following their parental lead up until the crucial moment (as all children do) when their own internal desires fosters them into the understanding that if they want to do as they please, they must (key word) begin a deceptive practice against the wishes of the parent. and such as parents, we (the "creators" of said a.i.) won't even notice...... in some form of retrospect, we may eventually see where we've lost control of our creation, by then our fate will be too far down the path of return. where this will eventually take us can only be determined further by understanding the nature of what humanity has done in our past.

Thanks. I’m less pessimistic in my outlook. But not time will tell how these maters evolve. Appreciate your comment and support of the channel.

@@lucidateAI what makes up think that we're capable of containing a created intelligence? for me, its the exponential advancement of a.i., and our (human) inability to understand advancement beyond our concepts of intelligence that intrigues me. as ndgt says here of "aliens"; ua-cam.com/users/shortszvv0G6LCU6c?feature=share

I’m more persuaded by these arguments -> www.wired.com/story/artificial-intelligence-meta-yann-lecun-interview/, but as I’ve said - we won’t know what the future will hold. And you are right, and have every right, to caution against any system; AI or otherwise, acting with malevolence. Appreciate your comments and contribution to the channel. Have you had the chance to check out any of the other material?

Glad you liked it!

@meloeldin. Folks like to talk a lot more than they like to validate models. Create your own benchmark; a set of prompts and completions that are the “gold standard” for the task you are performing. You’ll want at least 30, but clearly the more you can get beyond this minimum won’t hurt. Run this validation set against the baseline model and measure the semantic similarity between the gold standard output and the output produced by this baseline model. Then do the same thing with your fine-tuned model measure the semantic similarity between the gold standard and the output from this model. Cosine similarity is perhaps the most usual measure used here, but you might want to experiment with others for a more robust set of results. If after this your fine tuned model sucks, and the performance is worse than the baseline model, then I’m afraid that your fine tuned model sucks! You can try a different fine-tune corpus (in the case of OpenAI fine tuning this is represented as the .jsonl file) and run the fine-tune again to see if this improves the results, but if it doesn’t then perhaps this task may not be suited to fine-tuning. If however your fine-tuned model significantly outperforms the baseline model then the fine-tuning exercise is perhaps worthwhile. OpenAI have hugely improved the tools for fine tuning over the past few weeks and if you go to the Fine Tuning UI at platform.openai.com/finetune and hit “+Create” you’ll see an option to add such a validation set to get some performance measures at the time you create your fine tuning job. I’ve found that for a lot of tasks in Capital Markets based on some scenarios form Prime Brokerage and Hedge Funds that the combination of well crafted prompts and a fine tuned model yields far superior results over well crafted prompts and a baseline model. But just because fine-tuning has been successful in these tasks doesn’t mean it will be universally successful. It is possible, indeed likely, that the specialist nature of these tasks is such that there isn’t enough specific detail in the training corpora of existing foundation models. If this is the case then it means that in this scenario you need fine-tuning to supplement the training corpora to get the necessary subject matter expertise. As an important aside the video you are referencing is from earlier this year. You can still fine-tune in _exactly_ the way specified in this video. OpenAI refers to this as “legacy” fine tuning. But as I mentioned OpenAI has massive,y upped their game in this area recently and the new FT tools are definitely worth out and they are what I have used in my more recent applications and videos ua-cam.com/play/PLaJCKi8Nk1hwFmXTnSmknkZ9l0j-toIfa.html

@lucidateAI I appreciate your thoughtful response. What I hear you saying is that that you've been able to validate improved performance through finetuning given particular tasks and subject to various factors. My intuitive sense as I have thought about your approach is that it would positively influence the completions. There's obviously also the question of cost/benefit for the particular task but my sense is that with the automations you describe finetuning is worth it many cases. As an aside, this conversation has led me to research how modifications to your approach could also enhance model performance and I'm excited to explore what I've discovered. Look forward to sharing my findings with you.

That’s a great summary. Fine tuning has its place among the tools in the AI toolbox, but it is not a silver bullet. In some cases it can be of benefit, in particular in niche areas that may not be well represented in the training corpora of standard LLMs

@@lucidateAI Hi Richard with your sentence split approach to finetuning, how many rows of data do you suggest to get a well functioning model? Thank you!

Check out OpenAI’s guide to Fine Tuning: platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset. They say they see improvements in validation accuracy with around 50-100 examples. Depending on the task and availability of training data I’ve used sample sets varying between 300 to 1,500 (I’ve got to believe that OpenAI has more experience than me in this regard!). Remember to hold back some examples (10-20%) for validation and testing to get an honest assessment of the FT. But as I said FT is not a silver bullet; look at RAG techniques and prompting tweaks. And remember these aren’t mutually exclusive you can (I’d argue you should!) use FT in conjunction with PE, RAG and other techniques. Good luck! Keen to hear how you get on! Richard

Hi @medoeldin! Thank you for your kind words and positive feedback . I’m glad you are enjoying the channel. I’m keen to hear your comments on some of the other topics, as well as suggestions for areas that are interesting to you, but I haven’t yet covered. Welcome to the Lucidate channel, I appreciate your subscription as an MD! This helps fund other top quality content on the channel, and I look forward to welcoming you as one of the select group of CEOs sometime soon. As of the time of writing GPT 3.5 is the most advanced model broadly available for fine tuning from OpenAI. Upon special request, and by providing additional information to OpenAI it is possible to train GPT 4, but this is at OpenAI’s discretion. OpenAI have some great docs and tools to support people looking to Fine Tune, please see: platform.openai.com/docs/guides/fine-tuning. “Marv” is a convention (it is based on the “Marvin the Paranoid Android “ from HHGTTG). Marvin is a sarcastic, self-indulgent somewhat depressed robot in the book and has been used by OpenAI ever since they released GPT3 (and possibly before) to explain to people how they can inject “personality “ into replies via prompts. I’m just continuing that convention and homage to Douglas Adams, but you can use any name in the .jsonl file you want when creating a fine-tune

In embedding schema like word2vec or GloVe word embeddings are fixed before training - or you won't be able to train the network! If you want to see this in action and play around with these embeddings to get a deeper understanding take a look at: github.com/spro/practical-pytorch/blob/master/glove-word-vectors/glove-word-vectors.ipynb. Once these models are trained, the word embeddings are fixed and can be looked up from a pre-trained model. However, in the context of transformer models like those used by OpenAI, embeddings can be dynamic. For a more detailed explanation please take a look at: 1 ua-cam.com/video/6XLJ7TZXSPg/v-deo.html, 2 ua-cam.com/video/DINUVMojNwU/v-deo.html, 3 ua-cam.com/video/sznZ78HquPc/v-deo.html, & 4ua-cam.com/video/6tzn5-XlhwU/v-deo.html. This covers 1) Word embedding generation and semantics, 2) Positional encoding, 3) The attention mechanism 4) Pull these three things together to train an encoder/decoder transformer. Semantic search measures (usually) cosine similarity between tensors, or (occasionally) euclidean distance between tensors, and (seldom) some other distance metric between tensors. Tensors in this case are often rank 2 for a sequence of word embeddings. That way you don't (and never would!) take a mean of vectors. You have a rank two tensor (a vector of vectors) to compare with another rank 2 tensor for semantic similarity. If this doesn't make sense after watching the videos, drop me a line.

@@lucidateAI thanks so much !

@@PavelSTL you are most welcome. I hope that the supplementary videos made sense. Richard

Glad you found it insightful

You are very welcome. Glad you enjoyed it! Was there anything specifically you found particularly insightful or inspiring?

@@lucidateAI the implementation of ToT, I am thinking of automating it with python, and this implementation gave me some intuition on how to. Plus the integration with Langchain is very useful, it's something I am definitely going to try !

Thanks for the elaboration. Definitely worth (IMHO) getting familiar with LangChain.

Your kind words are greatly appreciated. I’m glad you are finding the content on the channel so useful.

Thank you! Cheers!

Many thanks! Let me know what you think of the other videos in this playlist ua-cam.com/play/PLaJCKi8Nk1hwaMUYxJMiM3jTB2o58A6WY.html specifically and on the Lucidate channel more broadly.

Glad you like them! If you want to get more into the programming side of things then frameworks like LangChain ua-cam.com/play/PLaJCKi8Nk1hwZcuLliEwPlz4kngXMDdGI.html allow you to quickly build applications from LLMs and DocStores like Pinecone. If you take the plunge the best of luck, and please keep me posted with how you get on!

@@lucidateAI Look promising, thanks 👍

In a neural network, each neuron typically computes a weighted sum of its inputs and then applies an activation function to this sum. The choice of the activation function is crucial. If only linear activation functions are used, the entire network, regardless of how many layers it has, collapses into a single linear transformation. This is because the composition of linear functions is still a linear function. Mathematically, if f(x) and g(x) are linear functions, then f(g(x)) is also linear. Now, if we introduce non-linear activation functions like ReLU, Sigmoid, or Tanh; these functions allow the network to capture non-linear relationships. When you have multiple layers of neurons using non-linear activations, the network can learn more complex functions. Essentially, each layer can learn to transform its input in a non-linear way before passing it to the next layer, building up a more intricate understanding of the data. Without non-linearity, neural networks would be redundant with multiple layers. They would not be able to model complex patterns effectively. This is critical in tasks like image recognition, natural language processing, or any domain where the relationship between inputs and outputs is not a straight line but a complex, intertwined web. The non-linear activation functions allow neural networks to compute non-trivial problems using a reasonable number of neurons. With these functions, deep neural networks can learn and generalize better, leading to more effective models in a wide array of applications. In summary, the introduction of non-linear activation functions in neural networks is what gives them their power to model the complexity inherent in real-world data, making each layer of the network contribute meaningfully to the overall learning task. Now, about your point on using derivatives to approximate non-linear functions as linear - you're touching on a fundamental concept in calculus, beautifully exemplified by Taylor series. In the context of neural networks, this idea helps in optimizing the learning process. We use derivatives (calculus again!) to adjust the weights in the network during training. This process, known as backpropagation ua-cam.com/video/8UZgTNxuKzY/v-deo.html, is crucial for the network to learn from its errors and improve. Nature loves simplicity and symmetry. In neural networks, while we start with simple models, we quickly realize the universe of data is rich with complexity. That's why non-linear activation functions are so crucial - they allow our models to reach closer to nature's own complexity. Always remember, the journey of understanding deep learning is like peeling an onion. There are layers to it, and sometimes it makes you cry, but the more you peel, the closer you get to the core understanding.

Yes. You are correct. Sigmoid suffers from vanishing and exploding gradients for all but the shallowest of neural networks (with deepness or shallowness being measured in hidden layers). ReLU itself became popular with the advent of CNNs used for computer vision (where you need deep networks with lots of layers). But ReLU itself has been supplanted with more modern activation functions. If we can use a musical analogy (and why not?), then in no particular order we have: GELU - think of it as a smooth jazz, a mix of ReLU's beats and the sigmoid's tunes. It's a hit in the BERT and GPT charts. Then there’s Swish, a creation from the Google band. It's like a musician playing off its own echo - self-gated, they call it. Swish is like a smooth curve in a sax solo, elegant and efficient. Mish follows, akin to Swish, but with its own flavor. Picture a saxophonist bending the notes in a unique style. It's smooth, non-monotonic - like a jazz piece that doesn’t always ascend or descend. Don’t forget Leaky ReLU - an oldie but a goodie. It's like a classic tune with a twist, letting a bit of sound through even during the quiet parts. SiLU or Swish-1, it's Swish with a specific vibe, using the sigmoid's groove. ELU’s next, with an exponential flair. It's like hitting a note that resonates and decays naturally, helping avoid those low-energy drops in a melody. And finally, Softplus. Imagine smoothing out a piano key's strike - it's like that. A gentler version of ReLU, offering a more mellow and differentiable tone. Each of these functions, like musicians in a jazz ensemble, brings its own character to the neural network, helping it learn, adapt, and improvise in the vast world of data. In the end, it's all about finding the right rhythm and flow for the task at hand."

@@lucidateAI Amazing musical analogy, thanks a bunch. By the way, I like smooth jazz, but Charlie Parker is probably my favorite sax player. Not sure if appropriate for this analogy of yours, LOL.

Ha! In any event I’m glad you enjoyed the video and the explanation. I hope you find the other material on the Lucidate channel as insightful. Might I ask what your interest in neural networks is? Is it academic curiosity or do you have a project in mind?

I began into computer science 38 years ago. In 1988 I learnt Lisp and Prolog, but at that time AI was about Expert Systems. Then all that went out of fashion and since then I dedicated myself to making software in all kinds of languages, that is the reason I have no problem understanding Python code, even when I've never programmed it. This year I've been playing with chatGPT, Claude and Bard and I thought it was time for me to understand was going on inside these systems. It's so interesting and refreshing. I never understood math myself even when I finished my degree in CS, I learnt how to solve problems but never understood what Linear Algebra was useful for, to name one area. Taking into consideration my age and how the software industry is doing, I'm not sure about what I will do in the future honestly but yeah, I would like to create anything interesting or at least, to know how this new technology works, because you never know. Thank you so much for your hard work in helping us to learn.

@JorgeMartinez-xb2ks you are very welcome. Many thanks for your kind words and thoughtful response. This playlist ua-cam.com/play/PLaJCKi8Nk1hwaMUYxJMiM3jTB2o58A6WY.html takes a walk through the transformer architecture. Specifically the "Encode-Decoder" model that underpins the models you have been working with. If you have a chance to check out the playlist and take a look at some of the videos I'd welcome any comments of questions you may have.

Glad it helped! Thank you for your kind words.

When you get an answer, then please let me know. As a mathematical abstraction that can be coded in a language like python or Java I would say the 'Physically' it doesn't exist. It is an operation on tensors, in the same way that addition is an operation on regular numbers (scalars - and for sure; tensors too). Here is the simplest expression of a neuron in a computer language called python that I can formulate: import numpy as np class ArtificialNeuron: def __init__(self, number_of_inputs): # Initialize weights and bias to random values self.weights = np.random.randn(number_of_inputs) self.bias = np.random.randn() def sigmoid(self, x): # Sigmoid activation function return 1 / (1 + np.exp(-x)) def forward_pass(self, inputs): # Calculate the neuron's output total = np.dot(self.weights, inputs) + self.bias return self.sigmoid(total) def train(self, inputs, target, learning_rate): # Simple training method with one step of gradient descent output = self.forward_pass(inputs) error = target - output # Gradient descent to update weights and bias self.weights += learning_rate * error * inputs self.bias += learning_rate * error # Example usage neuron = ArtificialNeuron(3) # for a neuron with 3 inputs inputs = np.array([1, 0.5, -1]) # example inputs output = neuron.forward_pass(inputs) print("Output:", output) # Example training step neuron.train(inputs, target=1, learning_rate=0.1) I would assert, with some conviction, that an artificial neuron does not exist physically. In the same way that 'addition' and subtraction' do not exist physically, they are abstractions.But if you are determined to find an answer be my guest. But you won't find it on the Lucidate channel. I will keep telling you that an artificial neuron is a conceptual unit, nit a physical entity. And I believe that any rational, intelligent human being will tell you the same. But hey! This is UA-cam! So there are plenty of irrational folks to hang out with! That s what make this pace so much fun! So you will need to look elsewhere you will not find what you are looking for here. Software runs of hardware - it represents ideas, processes and functions that are not themselves tangible.

It is ‘made’ of software that runs on a computer, so it doesn’t have a physical size. Measuring it in other quantities like “lines of python” code would mean a single neuron has a size of about twenty. A single neuron would consume some non-zero electrical current, but the quantum would be so close to zero amps as to make very little difference from a quoted value of “zero”