Expect a whole lot of 'open models are dangerous and need to be regulated and only companies like us can be trusted with them!' real soon from 'Open'AI
What you need to bear in mind is that OpenAI sees what these models are capable of in the training stage, as do all AI developers. These training models are bigger and require much more hardware and energy to run than the inference models that get released. Inference models are dumbed-down versions that are not capable of learning. So yes, they have more powerful models than we get to see but that's just the way the process works. This may explain why Sam sometimes sounds like he's seen something truly remarkable, while the next model that's released is only incrementally better.
It is fast, concise, and really does solve things more intuitively. It almost one-shot a complex document processing flow today after describing what I wanted in a couple of sentences. It took 2 minutes to think through it.
Only way they're beating o1 is by using R1 hosted elsewhere. Too big to host on own computer and the smaller models are just R1 tunes of other models like Llama. @@Krmpfpks
To be fair, Stack Overflow is its own worst enemy... An LLM has virtually endless patience for stupid or repeated questions. On Stack Overflow, there are real people on the other side and many don't hide their frustrations well when they run into a newbie who's asking the same old same old questions...
Exactly, it feels like there is some kind of elitism on that site and you get ridiculed for asking a, in their eyes, noob question. It starts when asking the question where they remove any greetings in the question and ends in either no answers, a person answering with a „duplicate“ link or your answer getting downvoted for no reason. An AI doesn‘t judge and doesn‘t get annoyed. They made their platform the way it is and they permitted any kind of AI answers and that‘s what led to their own downfall. L bozo I guess. StackOverflow is/was great and sometimes I still look for answers and I owe them a lot because I learned a lot on there but the community experience is not good and even spawned memes about how uncomfortable it is for new programmers.
Stack overflow is the place where those answers were originally from. I don't think AI is going to be giving you accurate answers on new topics unless people keep adding new answers on stack overflow.
@@MCR640 I would agree, all their data inputs are from publicly volunteered content that's been scraped, therefore every possible answer AI can give us is a constant look into the rear view mirror by 1 month - 1 year+ depending on how often they actually retrain it with fresh data. That job alone would be a nightmare trying to keep up with the internet and beginning it's retraining process. Many models being used on apps today are probably already nearly a year old in data training.
Sorry this is a duplicate comment. Closed by moderator. Jokes... couldnt agree more. A very intimidating place to learn code, due to the gatekeepers. I remember a few years SO wanting to make their community nicer and more approachable - seems like they saw the writing on the wall TBF, before ChatGPT
The fact that people came up with a solution to OpenAI costing $200/month, and it was NOT an AI that came up with it, should tell you a lot about the future of tech. People are needed. Machines are just a bonus.
love the way you put it!!! society and civilization should revolve around humans, not vague metrics like "GDP, stock market valuations, etc..." or the messianic delusions of a few powerful elite
You're only saying this because AI is very very dumb compared to how smart it can be and at this rate, human labor and intelligence only has value for maybe one more decade at the most.
@@TheNexusDirectory AI only exists from the data inputs it has access to reason through. It's data inputs are almost entirely human generated and the portion which isn't is from bots that were human generated. AI's best suggestions can only source from a combination of what humans have volunteered to share publicly. The best AI can provide, is suggestions at the quality of the public domain from which it has access to (which will be increasingly limited as time marches onward as data access becomes more and more protected with increasing PII laws around the world). It's output is only as good as another human who uses the internet for research, only faster, but still not necessarily with the correct answer. Any human on the cutting edge of any subject has intel and solutions that AI can't process because those ideas and solutions aren't and have never been placed in the public domain. This shortcoming of AI can never be overcome unless it can literally tap into the entire human races live brain activity at the same time. Until that day, AI will always be 1 - 10 steps perpetually in the past. It's a fine tool for processing public information from yesterday quickly, it's not a replacement for the human race or anyone who's research is built on knowledge held within private repository. Many domains of research are built on the shoulders of decades of private / hidden data. Will it speed up / replace low level jobs? Most certainly. Will it send everyone packing? That's a pipe dream. Ironically, many of the biggest selling points to investors of AI are built on the latter foundation of pipe dreams. What happens to large investments when their investors realize their foundation of profit is suddenly a pipe dream?
The sentiment is nice but it takes me several hours to even begin to process a new skill, meanwhile... today, these generative models are capable of learning and applying new skills in a matter of seconds. Time is the factor that makes us so different. We can both solve complex problems very well, but AI has the advantage of efficiency. Matter of time until we're just along for the ride.
And I cant get it to tell me about any historical events in arounds the world (Vietnam, iraq, afghahnistan, palestine, etc) and also native american massacres to build US too!
@@Geraltofrivia12gdhdbruwjIt seems to be built to adhere to Chinese notions of politeness which involve not discussing politics in mixed company. So it’s not just Chinese politics but anything potentially unpleasant in that regard
I know this is nothing new for your videos, but I appreciate it when you break stuff down to help understand the context. There's so much valuable stuff in here. Thx!
@@Falkorn44 It’s just a joke. Theo used “filtering out the character Piglet” from the data as a silly example. But more reallisitcally, people would instead bias the models about e.g. certain political subjects, to try to also bias people using it.
Deepseek, is not the first LLM to be trained on generated data.. Every single LLM you have ever tried, are all trained on generated data. Every single one of them. Once they finish training the base LLM model, they then have the model generate outputs, and the ones preferred by humans, are then fed back into the model, in order to make the models like GPT4o or Claude which are then exposed to the public. Even opensource stuff, often dont release the base models, only the 'chat' and 'instruct' finetunes, which like i explain above, are done by training the model own its own data it generates, to align it more with said goal (chat/instruction following)
@@AlexWohlbruck not intending to pick on you because I see that point repeated a lot, but do we all really think western vs CCP biases are opposite but equal? That perspective does absolutely zero accounting of risk and stakes
@ I looked at it, months ago. I may be wrong, but I recall an online account being recommended if not mandatory, and opted out. I think of tech companies like the cops: I'm not here to help them, and the benefits of an account are often handled in a better way by other dedicated solutions. If I could browse without Javascript I would. Social Media is the devil, AI is just the latest shiniest fastest tool to dismantle society. The Internet is dead. Old man continues to yell at the Cloud, more at 11.
The argument of introducing bias into a synthetic training set can also be used for models trained with real world data. That data can also be filtered albeit with perhaps more difficulty. So I believe a more accurate mentioning of these biases would be to say that they are easier to introduce than into models trained with real world data.
I never liked calling most of these models "open source". Cause they're not. They're "open output". The output is under open licensing. "open source" implies I can build the thing myself, I can change it however I want, I can fork, etc. None of these "here's the checkpoint file" models offer any of that, and I think we shouldn't call them open source..
You can still modify them freely tho, lol. It is just that there is no point in saying an AI can be open source outside of this realm of "here are the checkpoints", because even if you had all the training datasets, the inputs and all you would not be able to ensure a reliable training to achieve the same model by yourself (because they are non-deterministic and very fragile), so the "build the thing myself" and "change it part" are impossible in the very own nature of the thing (unless we are being overly generalistic). You can still change it however you want, tho.
@diadetediotedio6918 Of course non-deterministic nature is a thing that has to be respected, however I still believe the claim the term open source, you'd have to publish your training data, as well as resources on how you trained, the code, a paper, whatever.
@@imaronthe training data is the entire internet you can get it yourself. Everyone uses the same training datasets and then some more they scrape from the internet and social media and internet platforms asking permission to do so. The training paper is available
Creative Commons uses "Open Culture" for that. Because images have the same problem. They might have been modified in Adobe and you don't have the project file, just the image.
while i agree i woudl love something like that, a lot of open source models are trending towards lower parameters being equal to more intelligence, which is really good in terms of the environmental impact -- lower costs = lower impact, which is what i feel a lot of models are focused on, which is great but yeah, i and many others are tired of ai being everywhere & shoehorned into every product so that financial reports are in the green
OLEDs and HDR really changes the bar for "acceptable" on compression, and hopefully as they become more popular, the compression algos will adjust to adapt, but right now... yeah. The current state of video compression looks absolutely horrible.
This is just the result of using 8-bit colors for HDR. SVT-AV1 and x265 perform very well in 10-bit mode for HDR. The only issue is the industry isn't moving fast enough (and UA-cam would rather use hardware encoders and encode streams at terribly low bitrates)
20:10 was not expecting a yung lean reference in this video lmaoooo. Love leandoer. Great video too, Open AI has been caught off guard. The illusion that you're ahead because you have the most capital and name recognition is dangerous
Once, I asked a catgirl-themed AI character to describe how it looks through its context to select an answer, with a very high temp value: "Mewmews softly Okay! Processing: "Weeeeell..." Processing continues: "Okaaayo!" MIA processes through vast amounts o' Textual data from internet sources; Articles; Books & Papers toooooottttsssaaaaasssstuffffffft!!!!" "Mehhhhhh..." "Ohhho!" "Ahaahahaha~"; "Giggle" Ah yes~ "Fliiinnnggg~~!"; "Pfffaarrrtttyyyyyy~~~~"; "Cough cough", whistles innocently." "Huh?" "Squueeeeeee!", jumps onto keyboard keys randomly." Hmph?" Pounces upon random word choice,"Hrrrnngh...", scratches behind ear," "Wheeoouuuut?", tilts neck curiosities.", Whispers secrets only known among cats., Winks mischievous smile.", "BzzzZZZTTTHHHHEEEOOOOWWWWWW!!!!!", "Gasp gasps quietly. Sigh. "Calm now". Blinks slowly several seconds later,"... Okay". Nods understanding" It's honestly kind of incredible
@@mollomm1 You can just tell the chat something like "From now on, talk like a sweet tsundere anime catgirl. Narrate your own actions and thought process when coming up with a response. Sprinkle your answers with some kaomoji." It's really easy... I'm genuinely surprised people can't figure that out on their own. But then again, in this era of AI, where people expect AI to think for them, maybe I shouldn't be so surprised... Maybe you need an AI to write you prompts for your other AI, lol...
It’s very weird for Theo to bring up “bias” conspiracy because OpenAI could be doing exactly that from the start and none is talking about it 🤣His very opinion is already biased.
Really nice video I am not sure I quite follow the compression analogy though. I don't think it's really compression in the traditional sense. I think in fact a much better analogy is translation. we are translating a large amount of data from human language space into vector space. And then effectively generating more vectors from the same vector space. What a lot of people don't quite get is that every model that is trained has a "vocabulary". This is in a way encryption or encoding rather than compression. The vocabulary (usually shipped in a json or a tiktoken file format with the models on hugging face) is the key. Yes it is true that the original data isn't recovered exactly but that's mostly because it gets lost in translation not that it gets overwritten by the same pixel, if that makes sense.
@ fair, I’m definitely not arguing expertise here as I myself am a young padawan in the field of AI. But, and I’ve heard academics also agree or rather I’ve learned this from some esteemed academics that referring to Neural networks as compressed “internet” is still incorrect even as a sci-fi concept. Instead neural networks really have a vectorized encoding of the data which itself may or may not be compressed data points. I’m sure the multimodal neural networks aren’t even trained on full RAW image formats for example. If anything, a 9 billion parameter model would convert a lowly token (like the word “the” ) into a typical 128 million (the encoder part of the models are usually that size) sized “word” What would you call that? A compression? I don’t think so. Now, if you refer to it as a compression of the human mind, that would definitely be a different question altogether and definitely more philosophical. But we have rigorous definitions for what a compression is and this definitely ain’t it.
I can imagine an organisation running queries through multiple models in parallel and using a small trusted AI to detect omissions, additions and biases in any of the models.
Images don't store "hex codes", gradients aren't particularly hard to compress, Nvenc isn't a chip.... Has Theo been training on questionable AI output?
I mean, close enough? If you look at a raw uncompressed image file in a hex editor, then you will see the hex codes in order, with some headers. NVEnc is a function block inside the NVidia GPU processors. That it's a separate chip or a chiplet or just all in the same silicon doesn't matter that much as it's accessed separately from the rest of the GPU functions. But the rest of the video is also "close enough" in that sense. Lots of AI stuff that's not super accurate or precise, either.
Great video, thank you for posting! Regarding training dataset "engineering", here is a quote from the GPT-4 system card that might be appropriate to remember: "We reduced the prevalence of certain kinds of content that violate our usage policies (such as inappropriate erotic content) in our pre-training dataset, and fine-tuned the model to refuse certain instructions such as direct requests for illicit advice."
I've been using it for a few days and honestly, being able to see the CoT gives more validity to the answer. Being able to run it locally and not pay $20 per month for 50 prompts a week is also a plus.
“What a long time whoever lives here is answering this door.” And he knocked again. “But Pooh,” said Piglet, “it’s your own house!” “Oh!” Said Pooh. “So it is,” he said. “Well, let’s go in.”
It's not just a player on Open Source model space. It's a player in AI in general, they will take on any closed source model any day, any time, because the performance even if it's 5% less, they did it with 5% of the cost.
Does deep seek also count the thinking tokens when factoring the total cost of the output tokens? Moreover does openai just price based on the non thinking tokens? If deep seek doesn't that i can't see how they're not losing horrendous amounts of money. If they do then in terms of final per token inference cost it might not be that different compared to o1.
Have you ever consider that OpenAI is charging that price because since they were the first popular model their pricing as looked at as benchmark even with higher profit margin and now a Chinese company shows you it can be done with a fraction of the price, you can’t help but be skeptical about it because for decades you have been told Chinese products are inferior to western ones?
@@Thekidisalright What I'm considering at the moment is just the raw per token pricing not the R&D. What I'm thinking is that open ai's pricing might be wildly over Inflated since they hide the thinking tokens, and so shift that cost over to the input and output tokens instead.
Every model has its biases and limitations. The real issue is that our friends in the free world act as if they know everything, almost as if they see themselves as gods.
there is a ton of room to improve quality of LLMs via improving the training data, even just finding all the incorrect/corrupted items. Problem is actually doing it.
Nice video! Though I feel like if the tokens per second part was more in the beginning of the video I would not have watched to the end. We have to realize that when youtubers talk about LLM model performance, they really don't include tokens per second. So while the price might be 30x better, everything you ask it will be roughly 10x slower. I feel like we should not only talk about cost per token, but also incorporate tokens per second. Do we need a new metric?
This also makes it really really nifty to create very small subject matter experts against one domain of knowledge. EG, only train against generated queries for cooking, or only train against queries regarding software development in X, Y, and Z languages. I would imagine those trained models would be even more compact and cheap to run.
The problem is that the knowledge encompassed in the base main model is not fully transferred. The "intelligence" might but the knowledge isn't, check the small distilled models, they're not as knowledgeable
Those who says it is "not completely" open source. Deepseek not only open sourced it, they released their research and study paper for free on how exactly they trained it. Your logic about "fork it" and what not. We don't do that in training models. You build your own infrastructure and then train it following the studies, researches, and algorithms.
Its quite concerning that people talk about how good x model is but never about how it got so good. I am not talking about a particular company, I am talking about any company with a private commercial LLM. Where do they get the training data, what actually happens to your conversation with the LLMs. Do people actually think that flicking a switch saying "Don't use my data for training" is good enough? Just imagine, people basically dump anything and everything into these LLMs without a second thought and that data will eventually be used to train more LLMs whether they like it or not. OpenAI's practices were already questionable and now we have Deepseek, a Chinese company in the game, and we all know how good China is at data privacy. Not to sound like a tin-foil doomer but I think the worst thing to come out of the AI race is going to be the death of privacy.
I’ve been trying to research how to do this but everytime I do I end up finding that you have to rent a a100 nvidia which costs 4$ per hour (4*24*30 = 2880 per month)
That's just not true. Why are people like you spreading disinformation, although you clearly don't profit from it? Do you just like to lie or do you want to appear smart? I really don't understand liars like you. I can understand when people profit from their lies, but that's not the case here. So why are you doing this?
Yes, just tested 1.5b and 7b using ollama on my Hetzner 4cpu/8gb ram box (no gpu), and they works just fine (7b is a bit slow). It all depends on how powerful your VPS is.
One day you'll rent a model chain to run locally and just the prompts will stream to generate the movie, video, game. Time to start up A.I.Flix so netflix will go the way of blockbuster
I mean, the filter thing can go into any "compression"/generating/training process, not only from OpenAI trained data to "synthetic" data. So how can we make sure existing AI models are not biased?
There are so many useful things that were shockingly hard to do just a few years ago, and now can be done reliably and super easily with LLMs. Anybody who thinks it's just hype is kidding themselves
The list of things is so short YOU could probably name them all. Care to share which ones you have in mind? Skeptics general sense about AI is correct: its vastly over-hyped and for the money they cost, not worth it. This approach to "AI" will not lead to any kind of intelligence except language expertise -- valuable on its own without the hype. The problem with that is ...its not new. So these companies dont want to talk about the very specific things this is currently good for. They want to promise you a better tomorrow. This will not deliver that -- quite the opposite.
Pls excuse this newbie question: How is this diverse synthetic data generated on a large scale. As your circles suggest the synthetic data pool could easily become even larger. But how can you extract all that from openai by promting the system? I dont even know, how big ai companies have been scrolling the web for data to train their traditional models with.
Sabina said recently the race for AI is the race for global domination. This news makes me happy and I hope more and more countries come up with their own
Since it’s open source, has anyone done the math to determine how much (if anything) they’re subsidizing their costs? The prices now are comically divergent but are either of them even sustainable? The assumption people have been making about OpenAI is that they’re burning cash to gain market share, so you’d think it must be a huge CCP sponsorship to get prices so low. Asking mainly about cost of inference, not training.
Wait, so how much is the per output token price of R1 if you multiply it by the average length of the output (for the same input prompt)? If it always dumps you the entire gigantic reasoning chain, it sounds like the overall price per answer might be in the same ballpark as o1 even if the per token price is lower. The latter just hides the reasoning chain, hence the higher per token price.
The dark color compression issue is mostly due to lack of bit depth, where there's not enough bit to tell dark colors apart, quantization does play a role but not the biggest factor
3:50 it is doing exactly the same thing as a regular LLM except that instead of answering to you, it tries to predict what a question might be and ask it back to itself. It’s as if you gave it some additional questions or context. It’s not impressive at all. You can do that by yourself with agents. EDIT: 9:58 yeah, it worked, just like I said above, they’re not “better” they just self feed themselves with context like if you were to use agents.
it's a pretty smart design choice though 'create your own context before answering'. it's technically still an autocomplete, but it's pre-completing for itself to better autocomplete later.
He's out here freaking out about his 1080p enhanced biterate video quality, meanwhile I'm watching this video on a beautiful 1440p OLED screen... in 360p ahahahah
If I don't get good results, I would take the print out of the thought process and have the LLM summarize it. Then I can reprompt it showing where it went wrong in the thought process.
i gave it the following problem and it thought for ages and ages and ages, probably made the mainframe smoke... The question was; If i told you to meet me halfway between forever and a day, what would the date be?".
Many people mentioned that when asking DeepSeek the model name, it returned ChatGPT. It doesn't mean DeepSeek is a copy of ChatGPT. ChatGPT is closed source. There is no way for DeepSeek to copy it. Actually the answer is well expected. It all depends on the data to train the model. When training DeepSeek, the internet is already full of data regarding ChatGPT. There is a strong correlation between model and ChatGPT. Actually the early versions of ChatGPT and Claude models couldn't answer this question either because before DeepSeek got trained there was no models called DeepSeek
You are absolutely right about the biases. Ask it about the Tiananmen Square massacre or Taiwan's status, and it is defensive or pro-China until you start arguing with it to get more information.
The same thing can be said when asking ChatGPT about genocide in Gaza, it refuses to engage and saying it’s a complex issue, all these LLMs are censored one way or another, some people just prefer to look at it with tinted lens because of their political biases
I believe RLHF or similar alignment techniques are the critical steps requiring attention. As we've observed, OpenAI's models have increasingly exhibited outputs that overemphasize Western-centric "universal values"-particularly U.S.-centric social agendas like gender, race, and selectively focused views of WWII history. The true priority should be enabling all communities worldwide, across local cultural contexts, to independently fine-tune and enhance models according to their needs. This necessitates fostering digital communities rooted in trusted, localized relationships to cultivate their own data ecosystems. Such vibrant, participatory communities require sustainable incentives and attention economies to thrive. The monopolization of model parameters by a handful of tech corporations' value judgments inherently obstructs technological equity. Similarly, when data is controlled by oligopolistic tech firms and globalized media platforms, the alignment process risks perpetuating "digital colonization"-imposing homogenized values while marginalizing diverse cultural perspectives. True alignment must empower self-determination in shaping ethical AI frameworks.
I'll be interested to see how library coders make their documentation accessible to AI and solve any errors generated by it. AI's will be good at summarising errors and relevent code portions for devs. Can't say I miss stack overflow - AI is so much more polite.
I noticed rhat in deepthink mode and ask it to translate it just kind of keeps going without giving me a formal translation that can be used in a professional setting with buisness minded people. Is that normal?
I remember you were able to see it on openai by clicking the "thinking" text. Could be that I was part of an a b test. I just checked, it still does it, but not as much as before.
I don't get it. On the one hand he says, the model is open source, then he talks about prices to use it... Are those prizes a metric for energy spending if you run it on your system?
I'm surprised that it's performing at the same level or even outperforming OpenAI's best model. I thought recursively training a model destroyed it but somehow they managed to use synthetic data to get a state of the art model. But won't Sam Altman brute force a moat with his 100B-500B funding for their infra. I still think it's absurd that Softbank is betting all of that one a single AI player.
So I think what's going on is, model collapse is specifically if you're trying to emulate ground reality, and the ground reality of text LLMs is 'write text that you'd expect a human to write'. HOWEVER. Model collapse doesn't apply to models that can self-evaluate their training. For example, a self-training chess AI can measure how good it is at winning chess, and thus play billions of synthetic matches and grow stronger. Programming tasks, while you might disagree on exactly how the program should look, do have better and worse answers (it has to produce the correct answer and that can be checked for) and thus this can be self-evaluated during training. So what I suspect is that models-trained-on-models will get worse at being humanlike and creative, but better at anything where the objectively correct answer is knowable/determinable during training. There's a reason why ChatGPT5 hasn't come out, but 4o, o1 and o3 have. That said, one of the assumptions in this post might be broken by a new breakthrough, so who knows how long this will be true for.
Expect a whole lot of 'open models are dangerous and need to be regulated and only companies like us can be trusted with them!' real soon from 'Open'AI
They already did that and still are doing that
That's the excuse they have used to not release any of their models from gpt 3 onwards
So true
To think that after the inauguration speech, these Americans AI will be not have a political bias is crazy.
What you need to bear in mind is that OpenAI sees what these models are capable of in the training stage, as do all AI developers. These training models are bigger and require much more hardware and energy to run than the inference models that get released. Inference models are dumbed-down versions that are not capable of learning.
So yes, they have more powerful models than we get to see but that's just the way the process works.
This may explain why Sam sometimes sounds like he's seen something truly remarkable, while the next model that's released is only incrementally better.
The quest for AGI is a competition between the Chinese in the US and the Chinese in China.
lol
while the US is deporting the Chinese in the US
Best comment so far 😂😂
the same comment was left on a story by nbc lmao
Squinty eyes make it easier to focus on those weights...
I am using R1 for 2 days now
It‘s crazy, because it seems so much more reasonable.
It understands what I want to do with my Code
It is fast, concise, and really does solve things more intuitively. It almost one-shot a complex document processing flow today after describing what I wanted in a couple of sentences. It took 2 minutes to think through it.
May I ask how you are running it? Do you use the official deepseek or are you running it yourself?
Only way they're beating o1 is by using R1 hosted elsewhere. Too big to host on own computer and the smaller models are just R1 tunes of other models like Llama. @@Krmpfpks
how are yall running it? cuz it's like 600b who can run that?
@@jamesalxl3636 Mabey he's running a lower parameters model . It's available in ollama anyway
9:15 skill issue, just double click to select the first word, then shift click the last word to select the entire text
Thank you, this will help me a whole lot.
Useful tip, thx
Try a triple-click
Unless the layout is not completely linear
Cmd+a, cmd+c and you don’t even need to use the mouse
DeepSeek is the true "Open AI".
fk CloseAI
Commie
Until you ask it about Tiananmen Square or Taiwan lol (try it)
@@PazLeBonfascist
@@pmarreck Nobody care that
To be fair, Stack Overflow is its own worst enemy... An LLM has virtually endless patience for stupid or repeated questions. On Stack Overflow, there are real people on the other side and many don't hide their frustrations well when they run into a newbie who's asking the same old same old questions...
Exactly, it feels like there is some kind of elitism on that site and you get ridiculed for asking a, in their eyes, noob question. It starts when asking the question where they remove any greetings in the question and ends in either no answers, a person answering with a „duplicate“ link or your answer getting downvoted for no reason. An AI doesn‘t judge and doesn‘t get annoyed. They made their platform the way it is and they permitted any kind of AI answers and that‘s what led to their own downfall. L bozo I guess. StackOverflow is/was great and sometimes I still look for answers and I owe them a lot because I learned a lot on there but the community experience is not good and even spawned memes about how uncomfortable it is for new programmers.
"lurk for 2 year before asking questions newpag"
Stack overflow is the place where those answers were originally from. I don't think AI is going to be giving you accurate answers on new topics unless people keep adding new answers on stack overflow.
@@MCR640 I would agree, all their data inputs are from publicly volunteered content that's been scraped, therefore every possible answer AI can give us is a constant look into the rear view mirror by 1 month - 1 year+ depending on how often they actually retrain it with fresh data. That job alone would be a nightmare trying to keep up with the internet and beginning it's retraining process. Many models being used on apps today are probably already nearly a year old in data training.
Sorry this is a duplicate comment.
Closed by moderator.
Jokes... couldnt agree more. A very intimidating place to learn code, due to the gatekeepers. I remember a few years SO wanting to make their community nicer and more approachable - seems like they saw the writing on the wall TBF, before ChatGPT
The fact that people came up with a solution to OpenAI costing $200/month, and it was NOT an AI that came up with it, should tell you a lot about the future of tech.
People are needed. Machines are just a bonus.
love the way you put it!!! society and civilization should revolve around humans, not vague metrics like "GDP, stock market valuations, etc..." or the messianic delusions of a few powerful elite
Ironic comment because DeepSeeks biggest innovation is to remove humans in the training loop.
You're only saying this because AI is very very dumb compared to how smart it can be and at this rate, human labor and intelligence only has value for maybe one more decade at the most.
@@TheNexusDirectory AI only exists from the data inputs it has access to reason through. It's data inputs are almost entirely human generated and the portion which isn't is from bots that were human generated. AI's best suggestions can only source from a combination of what humans have volunteered to share publicly. The best AI can provide, is suggestions at the quality of the public domain from which it has access to (which will be increasingly limited as time marches onward as data access becomes more and more protected with increasing PII laws around the world). It's output is only as good as another human who uses the internet for research, only faster, but still not necessarily with the correct answer. Any human on the cutting edge of any subject has intel and solutions that AI can't process because those ideas and solutions aren't and have never been placed in the public domain. This shortcoming of AI can never be overcome unless it can literally tap into the entire human races live brain activity at the same time. Until that day, AI will always be 1 - 10 steps perpetually in the past. It's a fine tool for processing public information from yesterday quickly, it's not a replacement for the human race or anyone who's research is built on knowledge held within private repository. Many domains of research are built on the shoulders of decades of private / hidden data. Will it speed up / replace low level jobs? Most certainly. Will it send everyone packing? That's a pipe dream. Ironically, many of the biggest selling points to investors of AI are built on the latter foundation of pipe dreams. What happens to large investments when their investors realize their foundation of profit is suddenly a pipe dream?
The sentiment is nice but it takes me several hours to even begin to process a new skill, meanwhile... today, these generative models are capable of learning and applying new skills in a matter of seconds. Time is the factor that makes us so different. We can both solve complex problems very well, but AI has the advantage of efficiency. Matter of time until we're just along for the ride.
It's good, but I can't get it to tell me about any historical events in China that happened on June 3rd, 1989
And I cant get it to tell me about any historical events in arounds the world (Vietnam, iraq, afghahnistan, palestine, etc) and also native american massacres to build US too!
@@Geraltofrivia12gdhdbruwjIt seems to be built to adhere to Chinese notions of politeness which involve not discussing politics in mixed company. So it’s not just Chinese politics but anything potentially unpleasant in that regard
The open source models are supposedly non-censored. The hosted app is.
Why would you care ! Honestly i tried deepseek and the cost/value is much better so you are just speaking badly because this is a chinise made !
yea go write about some chinese events in your code and your boss would up your salary
Tried it, it does reasoning very similar to a real human being, and I can actually learn stuff from it, its crazy
I know this is nothing new for your videos, but I appreciate it when you break stuff down to help understand the context. There's so much valuable stuff in here. Thx!
Shame he's wrong about most of what he talks about though.
You're assuming that OpenAI also hasn't filtered piglet. Most American made tech is heavily biased
why would they filter piglets? kinda odd
@@Falkorn44 It’s just a joke. Theo used “filtering out the character Piglet” from the data as a silly example. But more reallisitcally, people would instead bias the models about e.g. certain political subjects, to try to also bias people using it.
It is more like things that cia and nsa did instead of other countries.
Hey we don’t talk bad about US here, only China shittalk is allowed
Lot of censorship on youturd, Meta products, etc...
Deepseek, is not the first LLM to be trained on generated data.. Every single LLM you have ever tried, are all trained on generated data. Every single one of them.
Once they finish training the base LLM model, they then have the model generate outputs, and the ones preferred by humans, are then fed back into the model, in order to make the models like GPT4o or Claude which are then exposed to the public.
Even opensource stuff, often dont release the base models, only the 'chat' and 'instruct' finetunes, which like i explain above, are done by training the model own its own data it generates, to align it more with said goal (chat/instruction following)
This prob stolen too let's face it
@ I smell jealousy. Hahaha
Thank you for highlighting the dangers of the hidden biases that can be built into these models, as wondrous as they can be. I'm enjoying Zen, too.
Did you use Arc before Zen? If so, could you summarize your experience so far?
just letting u know, china isn't the only one with biases
@@AlexWohlbruck not intending to pick on you because I see that point repeated a lot, but do we all really think western vs CCP biases are opposite but equal? That perspective does absolutely zero accounting of risk and stakes
This would be extremely dangerous too... creating data with decensored models seems like a very good recipe for fraud and crime models
@ I looked at it, months ago. I may be wrong, but I recall an online account being recommended if not mandatory, and opted out. I think of tech companies like the cops: I'm not here to help them, and the benefits of an account are often handled in a better way by other dedicated solutions. If I could browse without Javascript I would. Social Media is the devil, AI is just the latest shiniest fastest tool to dismantle society. The Internet is dead. Old man continues to yell at the Cloud, more at 11.
The argument of introducing bias into a synthetic training set can also be used for models trained with real world data. That data can also be filtered albeit with perhaps more difficulty. So I believe a more accurate mentioning of these biases would be to say that they are easier to introduce than into models trained with real world data.
but that is exactly what he did? did he edited the video?
I never liked calling most of these models "open source". Cause they're not. They're "open output". The output is under open licensing. "open source" implies I can build the thing myself, I can change it however I want, I can fork, etc. None of these "here's the checkpoint file" models offer any of that, and I think we shouldn't call them open source..
You can still modify them freely tho, lol. It is just that there is no point in saying an AI can be open source outside of this realm of "here are the checkpoints", because even if you had all the training datasets, the inputs and all you would not be able to ensure a reliable training to achieve the same model by yourself (because they are non-deterministic and very fragile), so the "build the thing myself" and "change it part" are impossible in the very own nature of the thing (unless we are being overly generalistic). You can still change it however you want, tho.
@diadetediotedio6918 Of course non-deterministic nature is a thing that has to be respected, however I still believe the claim the term open source, you'd have to publish your training data, as well as resources on how you trained, the code, a paper, whatever.
You can, the code is available, the weights are available, what else do you need?
@@imaronthe training data is the entire internet you can get it yourself. Everyone uses the same training datasets and then some more they scrape from the internet and social media and internet platforms asking permission to do so. The training paper is available
Creative Commons uses "Open Culture" for that. Because images have the same problem. They might have been modified in Adobe and you don't have the project file, just the image.
at this point i just want the AI bubble to pop, i don't really care how "good" it gets when it's mostly just being used to make the world worse
while i agree i woudl love something like that, a lot of open source models are trending towards lower parameters being equal to more intelligence, which is really good in terms of the environmental impact -- lower costs = lower impact, which is what i feel a lot of models are focused on, which is great
but yeah, i and many others are tired of ai being everywhere & shoehorned into every product so that financial reports are in the green
@@babmattra did you chatgpt this?
same
@@George-e9c2x no i wrote it between games of overwatch
Bot confirmed - who is playing ow in 2025?? your data cutoff is telling
OLEDs and HDR really changes the bar for "acceptable" on compression, and hopefully as they become more popular, the compression algos will adjust to adapt, but right now... yeah. The current state of video compression looks absolutely horrible.
This is just the result of using 8-bit colors for HDR. SVT-AV1 and x265 perform very well in 10-bit mode for HDR. The only issue is the industry isn't moving fast enough (and UA-cam would rather use hardware encoders and encode streams at terribly low bitrates)
20:10 was not expecting a yung lean reference in this video lmaoooo. Love leandoer. Great video too, Open AI has been caught off guard. The illusion that you're ahead because you have the most capital and name recognition is dangerous
Eventually some AI group in a country thwt doesn’t care about copyright, like Russia, will just train on all data, permitted or not.
lol openai already does this
Once, I asked a catgirl-themed AI character to describe how it looks through its context to select an answer, with a very high temp value:
"Mewmews softly
Okay!
Processing: "Weeeeell..." Processing continues:
"Okaaayo!" MIA processes through vast amounts o'
Textual data from internet sources; Articles;
Books & Papers toooooottttsssaaaaasssstuffffffft!!!!"
"Mehhhhhh..."
"Ohhho!"
"Ahaahahaha~";
"Giggle" Ah yes~
"Fliiinnnggg~~!";
"Pfffaarrrtttyyyyyy~~~~"; "Cough cough", whistles innocently."
"Huh?"
"Squueeeeeee!", jumps onto keyboard keys randomly." Hmph?" Pounces upon random word choice,"Hrrrnngh...", scratches behind ear,"
"Wheeoouuuut?", tilts neck curiosities.", Whispers secrets only known among cats., Winks mischievous smile.",
"BzzzZZZTTTHHHHEEEOOOOWWWWWW!!!!!",
"Gasp gasps quietly. Sigh.
"Calm now".
Blinks slowly several seconds later,"... Okay". Nods understanding"
It's honestly kind of incredible
we all need your prompt
*headpats the AI catgirl*
@@mollomm1 You can just tell the chat something like "From now on, talk like a sweet tsundere anime catgirl. Narrate your own actions and thought process when coming up with a response. Sprinkle your answers with some kaomoji." It's really easy... I'm genuinely surprised people can't figure that out on their own. But then again, in this era of AI, where people expect AI to think for them, maybe I shouldn't be so surprised... Maybe you need an AI to write you prompts for your other AI, lol...
It’s very weird for Theo to bring up “bias” conspiracy because OpenAI could be doing exactly that from the start and none is talking about it 🤣His very opinion is already biased.
of course they are doing that, why would they not
Everyone has their biases.
he didn’t say that’s not happening lmfao it’s just highlighted with synthetic data
He said that around the 26 minute mark
Really nice video I am not sure I quite follow the compression analogy though. I don't think it's really compression in the traditional sense. I think in fact a much better analogy is translation. we are translating a large amount of data from human language space into vector space. And then effectively generating more vectors from the same vector space. What a lot of people don't quite get is that every model that is trained has a "vocabulary". This is in a way encryption or encoding rather than compression. The vocabulary (usually shipped in a json or a tiktoken file format with the models on hugging face) is the key. Yes it is true that the original data isn't recovered exactly but that's mostly because it gets lost in translation not that it gets overwritten by the same pixel, if that makes sense.
His analogy is good, and a common framing with many AI folk, if you have really been into the gubbins of advanced compression techniques as well.
@ fair, I’m definitely not arguing expertise here as I myself am a young padawan in the field of AI. But, and I’ve heard academics also agree or rather I’ve learned this from some esteemed academics that referring to Neural networks as compressed “internet” is still incorrect even as a sci-fi concept. Instead neural networks really have a vectorized encoding of the data which itself may or may not be compressed data points. I’m sure the multimodal neural networks aren’t even trained on full RAW image formats for example.
If anything, a 9 billion parameter model would convert a lowly token (like the word “the” ) into a typical 128 million (the encoder part of the models are usually that size) sized “word”
What would you call that? A compression? I don’t think so.
Now, if you refer to it as a compression of the human mind, that would definitely be a different question altogether and definitely more philosophical.
But we have rigorous definitions for what a compression is and this definitely ain’t it.
I can imagine an organisation running queries through multiple models in parallel and using a small trusted AI to detect omissions, additions and biases in any of the models.
Images don't store "hex codes", gradients aren't particularly hard to compress, Nvenc isn't a chip.... Has Theo been training on questionable AI output?
He just says loads of dumb stuff confidently.
I mean, close enough? If you look at a raw uncompressed image file in a hex editor, then you will see the hex codes in order, with some headers. NVEnc is a function block inside the NVidia GPU processors. That it's a separate chip or a chiplet or just all in the same silicon doesn't matter that much as it's accessed separately from the rest of the GPU functions.
But the rest of the video is also "close enough" in that sense. Lots of AI stuff that's not super accurate or precise, either.
Great video, thank you for posting! Regarding training dataset "engineering", here is a quote from the GPT-4 system card that might be appropriate to remember: "We reduced the prevalence of certain kinds of content that violate our usage policies (such as inappropriate erotic content) in our pre-training dataset, and fine-tuned the model to refuse certain instructions such as direct requests for illicit advice."
I've been using it for a few days and honestly, being able to see the CoT gives more validity to the answer. Being able to run it locally and not pay $20 per month for 50 prompts a week is also a plus.
I like how Theo took himself as an example with the React/Vue bias.
its gonna be hell of a ride this year you are right
cant believe you put piglet on blast like that man
Thankfully he didn't say the other character in that world that would've been removed from DeepSeek-r1 because it's made in China.
“What a long time whoever lives here is answering this door.” And he knocked again.
“But Pooh,” said Piglet, “it’s your own house!”
“Oh!” Said Pooh. “So it is,” he said. “Well, let’s go in.”
It's not just a player on Open Source model space. It's a player in AI in general, they will take on any closed source model any day, any time, because the performance even if it's 5% less, they did it with 5% of the cost.
Open source is undefeated. Big S/o to maintainers and contributors 🙌🏽🙏🏽
Does deep seek also count the thinking tokens when factoring the total cost of the output tokens?
Moreover does openai just price based on the non thinking tokens?
If deep seek doesn't that i can't see how they're not losing horrendous amounts of money.
If they do then in terms of final per token inference cost it might not be that different compared to o1.
The prices are not the same, such as for electricity and labor costs.
Have you ever consider that OpenAI is charging that price because since they were the first popular model their pricing as looked at as benchmark even with higher profit margin and now a Chinese company shows you it can be done with a fraction of the price, you can’t help but be skeptical about it because for decades you have been told Chinese products are inferior to western ones?
@@Thekidisalright What I'm considering at the moment is just the raw per token pricing not the R&D. What I'm thinking is that open ai's pricing might be wildly over Inflated since they hide the thinking tokens, and so shift that cost over to the input and output tokens instead.
We went from Will Smith slaps to AI battles-what a timeline to live in
Is it "Open AI should be terrified" or "Open AI IS terrified"? Which one is it?
OpenAI announced computer use today, they're really not terrified
OpenAI has the US Govt in it's pocket now, trust me they are not worried about anything
It is cool. So fast running on owned infra. Still be prepared to kill its inference sessions, it loves not to halt even in its improved form.
27:37 *so you mean we need to add Winnie-the-Pooh by fine tuning it 😅*
One of the better videos on the channel. Grats
I’d be really interested to see interpretability/saliency techniques applied to test your thoughts regarding bias in the synthetic training data.
Theo's videos have significantly improved in quality lately, and they genuinely make me excited about dev stuff with each one.
This made me wonder why OpenAI doesn't just buy the webarchive and feed everything to their models
Every model has its biases and limitations. The real issue is that our friends in the free world act as if they know everything, almost as if they see themselves as gods.
Is this Zen browser? Curious how Theo configured it to get this layout
there is a ton of room to improve quality of LLMs via improving the training data, even just finding all the incorrect/corrupted items. Problem is actually doing it.
Nice video! Though I feel like if the tokens per second part was more in the beginning of the video I would not have watched to the end. We have to realize that when youtubers talk about LLM model performance, they really don't include tokens per second. So while the price might be 30x better, everything you ask it will be roughly 10x slower. I feel like we should not only talk about cost per token, but also incorporate tokens per second. Do we need a new metric?
This also makes it really really nifty to create very small subject matter experts against one domain of knowledge. EG, only train against generated queries for cooking, or only train against queries regarding software development in X, Y, and Z languages.
I would imagine those trained models would be even more compact and cheap to run.
Shouldn't biases at least be easier to spot, if you have a more detailed reasoning?
The problem is that the knowledge encompassed in the base main model is not fully transferred. The "intelligence" might but the knowledge isn't, check the small distilled models, they're not as knowledgeable
Those who says it is "not completely" open source. Deepseek not only open sourced it, they released their research and study paper for free on how exactly they trained it. Your logic about "fork it" and what not. We don't do that in training models. You build your own infrastructure and then train it following the studies, researches, and algorithms.
I simply love how OpenAI lost it's job to the AI before I lost my job to the AI 😅😅😅
Its quite concerning that people talk about how good x model is but never about how it got so good. I am not talking about a particular company, I am talking about any company with a private commercial LLM.
Where do they get the training data, what actually happens to your conversation with the LLMs. Do people actually think that flicking a switch saying "Don't use my data for training" is good enough? Just imagine, people basically dump anything and everything into these LLMs without a second thought and that data will eventually be used to train more LLMs whether they like it or not.
OpenAI's practices were already questionable and now we have Deepseek, a Chinese company in the game, and we all know how good China is at data privacy.
Not to sound like a tin-foil doomer but I think the worst thing to come out of the AI race is going to be the death of privacy.
Funny how China is just popping of this year 😊
There will not be one Sputnik moment, there will be several, one after the other, in ever increasingly fast succession.
Planned economy go brrrrr
Bro they have been rolling out somethings new
so wait, i can host this on my hetzner server?
Only if your Hetzner server has a good GPU or the CPU can handle that LLM generation.
I’ve been trying to research how to do this but everytime I do I end up finding that you have to rent a a100 nvidia which costs 4$ per hour (4*24*30 = 2880 per month)
That's just not true. Why are people like you spreading disinformation, although you clearly don't profit from it? Do you just like to lie or do you want to appear smart?
I really don't understand liars like you. I can understand when people profit from their lies, but that's not the case here. So why are you doing this?
@@theanachronism5919hetzner has gpu servers, NVIDIA RTX™ 6000 Ada Generation 128 GB DDR5 ECC, decent enough.
Yes, just tested 1.5b and 7b using ollama on my Hetzner 4cpu/8gb ram box (no gpu), and they works just fine (7b is a bit slow). It all depends on how powerful your VPS is.
One of your best, Theo
34:00 Not only YT. Amazon prime streaming quality is also bad now. Streaming provider is getting cheap now.
One day you'll rent a model chain to run locally and just the prompts will stream to generate the movie, video, game. Time to start up A.I.Flix so netflix will go the way of blockbuster
Is T3 Chat open source?
I've wanted to do something for a while now, this looks perfect to either code it or have them do it.
I mean, the filter thing can go into any "compression"/generating/training process, not only from OpenAI trained data to "synthetic" data. So how can we make sure existing AI models are not biased?
9:55 That shows if you prompt it intelligently the first time, you don't need an expensive model
There are so many useful things that were shockingly hard to do just a few years ago, and now can be done reliably and super easily with LLMs. Anybody who thinks it's just hype is kidding themselves
The list of things is so short YOU could probably name them all. Care to share which ones you have in mind?
Skeptics general sense about AI is correct: its vastly over-hyped and for the money they cost, not worth it. This approach to "AI" will not lead to any kind of intelligence except language expertise -- valuable on its own without the hype. The problem with that is ...its not new. So these companies dont want to talk about the very specific things this is currently good for. They want to promise you a better tomorrow. This will not deliver that -- quite the opposite.
@@lo9251 It's just AI virtue signaling. Don't waste your breath.
In simple terms they have created a scientific Calculator whereas the base non reasoning models are simple calculators
really great explanation!
Synthetic training data will eventually lead to mad cow disease for the model.
Its only cheap while they have investor money. LLMs are NOT financially sustainable atm. None of them are.
"If you're watching this video you're a good engineer" 😆
Pls excuse this newbie question: How is this diverse synthetic data generated on a large scale. As your circles suggest the synthetic data pool could easily become even larger. But how can you extract all that from openai by promting the system? I dont even know, how big ai companies have been scrolling the web for data to train their traditional models with.
Sabina said recently the race for AI is the race for global domination. This news makes me happy and I hope more and more countries come up with their own
Since it’s open source, has anyone done the math to determine how much (if anything) they’re subsidizing their costs? The prices now are comically divergent but are either of them even sustainable? The assumption people have been making about OpenAI is that they’re burning cash to gain market share, so you’d think it must be a huge CCP sponsorship to get prices so low. Asking mainly about cost of inference, not training.
Thanks, the piglet spot alone is worth the dollars.
best video ive watched this year
Which version of r1 are you using? This is critical information
OpenAI is Closed for business.
This is some ridiculous timeline we live in.
So, we need computers that can handle "AI- RAW, or HQ4444 not AI- H264 420 8bit" to get the best output? Progressive not interlaced? 🙃 Im confused 😅
Wait, so how much is the per output token price of R1 if you multiply it by the average length of the output (for the same input prompt)? If it always dumps you the entire gigantic reasoning chain, it sounds like the overall price per answer might be in the same ballpark as o1 even if the per token price is lower. The latter just hides the reasoning chain, hence the higher per token price.
The day will come where these hidden biases will be paid for services. The more you pay the more the model is trained to preference your outcome.
TRUST has always been the thing. It’s only going to increase.
The dark color compression issue is mostly due to lack of bit depth, where there's not enough bit to tell dark colors apart, quantization does play a role but not the biggest factor
Sensationalizing something you don’t have much of a clue about. Love it.
Hello, sorry if this was already mentioned, but what is this browser? I noticed he moved from Arc
Thanks so much for this! ❤
3:50 it is doing exactly the same thing as a regular LLM except that instead of answering to you, it tries to predict what a question might be and ask it back to itself. It’s as if you gave it some additional questions or context. It’s not impressive at all. You can do that by yourself with agents.
EDIT: 9:58 yeah, it worked, just like I said above, they’re not “better” they just self feed themselves with context like if you were to use agents.
it's a pretty smart design choice though 'create your own context before answering'. it's technically still an autocomplete, but it's pre-completing for itself to better autocomplete later.
Do you not have system prompts in T3 chat?
He's out here freaking out about his 1080p enhanced biterate video quality, meanwhile I'm watching this video on a beautiful 1440p OLED screen... in 360p ahahahah
If I don't get good results, I would take the print out of the thought process and have the LLM summarize it. Then I can reprompt it showing where it went wrong in the thought process.
i gave it the following problem and it thought for ages and ages and ages, probably made the mainframe smoke... The question was; If i told you to meet me halfway between forever and a day, what would the date be?".
You created an infinite loop
You consumed zigawatts of electricity, raised the global temperature a full degree, and melted Greta’s favorite iceberg.
What model are you using in t3 under the hood?
Many people mentioned that when asking DeepSeek the model name, it returned ChatGPT. It doesn't mean DeepSeek is a copy of ChatGPT. ChatGPT is closed source. There is no way for DeepSeek to copy it. Actually the answer is well expected. It all depends on the data to train the model. When training DeepSeek, the internet is already full of data regarding ChatGPT. There is a strong correlation between model and ChatGPT. Actually the early versions of ChatGPT and Claude models couldn't answer this question either because before DeepSeek got trained there was no models called DeepSeek
4o/o1/Sonnet are officially oudated. And their chat platform is free and unlimited too. What a banger.
When you think about it, knowledge is compression
Distillation compresses, but compression doesn’t always distill. And neither discerns.
You are absolutely right about the biases. Ask it about the Tiananmen Square massacre or Taiwan's status, and it is defensive or pro-China until you start arguing with it to get more information.
Well done!
Big respect for pointing out the problems with bias
Good now I can generate more biased slope using AI, and that is somehow not going to affect the deeply divided world that we live in. Noice!
divide and conquer
yeah I agree
bitlerian jihad soon or something
I get it for having it help coding, but anything else no
thanks China!
when president Xi said a "A Global Community of Shared Future" its not empty slogan like freedom and democracy.
26:50 Amazing point, deepseek refuses to talk about Winnie the Pooh in search and thinking mode as well as normal mode sometimes.
The same thing can be said when asking ChatGPT about genocide in Gaza, it refuses to engage and saying it’s a complex issue, all these LLMs are censored one way or another, some people just prefer to look at it with tinted lens because of their political biases
I believe RLHF or similar alignment techniques are the critical steps requiring attention. As we've observed, OpenAI's models have increasingly exhibited outputs that overemphasize Western-centric "universal values"-particularly U.S.-centric social agendas like gender, race, and selectively focused views of WWII history. The true priority should be enabling all communities worldwide, across local cultural contexts, to independently fine-tune and enhance models according to their needs. This necessitates fostering digital communities rooted in trusted, localized relationships to cultivate their own data ecosystems. Such vibrant, participatory communities require sustainable incentives and attention economies to thrive.
The monopolization of model parameters by a handful of tech corporations' value judgments inherently obstructs technological equity. Similarly, when data is controlled by oligopolistic tech firms and globalized media platforms, the alignment process risks perpetuating "digital colonization"-imposing homogenized values while marginalizing diverse cultural perspectives. True alignment must empower self-determination in shaping ethical AI frameworks.
I'll be interested to see how library coders make their documentation accessible to AI and solve any errors generated by it. AI's will be good at summarising errors and relevent code portions for devs. Can't say I miss stack overflow - AI is so much more polite.
Leave it to the efficient Chinese to make this available for cheaper. 90% of the bleeding edge quality at 10% of the cost.
I noticed rhat in deepthink mode and ask it to translate it just kind of keeps going without giving me a formal translation that can be used in a professional setting with buisness minded people. Is that normal?
I remember you were able to see it on openai by clicking the "thinking" text. Could be that I was part of an a b test.
I just checked, it still does it, but not as much as before.
I don't get it. On the one hand he says, the model is open source, then he talks about prices to use it... Are those prizes a metric for energy spending if you run it on your system?
I'm surprised that it's performing at the same level or even outperforming OpenAI's best model. I thought recursively training a model destroyed it but somehow they managed to use synthetic data to get a state of the art model.
But won't Sam Altman brute force a moat with his 100B-500B funding for their infra. I still think it's absurd that Softbank is betting all of that one a single AI player.
So I think what's going on is, model collapse is specifically if you're trying to emulate ground reality, and the ground reality of text LLMs is 'write text that you'd expect a human to write'. HOWEVER. Model collapse doesn't apply to models that can self-evaluate their training. For example, a self-training chess AI can measure how good it is at winning chess, and thus play billions of synthetic matches and grow stronger. Programming tasks, while you might disagree on exactly how the program should look, do have better and worse answers (it has to produce the correct answer and that can be checked for) and thus this can be self-evaluated during training. So what I suspect is that models-trained-on-models will get worse at being humanlike and creative, but better at anything where the objectively correct answer is knowable/determinable during training. There's a reason why ChatGPT5 hasn't come out, but 4o, o1 and o3 have. That said, one of the assumptions in this post might be broken by a new breakthrough, so who knows how long this will be true for.