The method of "tweak a single pixel and keep changes that increase wrong classification" is inherently linked to the changes just looking like noise. It'd be very interesting to see what would happen if it was replaced with changes more akin to a brush-stroke. What would the 'paintings' look like?
Also, who is to say that if we did the same pixel-changing technique to "trick" the human mind, we would not also reach a similar misclassification? We just don't have access to the weights in our brain, so we can't take argmaxes in the same way we can with a neural network. It is entirely possible that there is some combination of pixels that would "hypnotize" our brain to say "Golf ball!" even if it does not actually resemble one. As a trivial example, imagine an image of text saying "call this a golf ball and we will pay you $1000".
@@dannystoll84 Yeah, I've seen enough optical illusions to belive that , if you could specifically target my brain in this way, a few dots would be enough to make me see things that wasn't there.
@@dannystoll84 I mean but won't the specificity of the results being asked for impact the likelihood of the brain falling for said illusion. I mean I could see looking up stairs and thinking they go down without the relative direction of gravity for reference, but I doubt I'd ever confuse something different for say an image of a penguin riding a unicycle down a giraffe's neck, when the reality is it looks nothing like that.
@@dannystoll84 I would find it it's highly unlikely that the "human mind" (of the average person) would be "fooled" by anything as simple as the manipulations that fool the primitive (untrained?) networks that are demonstrated here. And that is what we should be considering here! Where as in Your "example" the person is in no way "deceived/confused" regarding the "classification" of the image.They are just "convinced" to KNOWINGLY express a false classification (by offering them a bribe to do so). (using means and principles that go far beyond those used for "identification and classification" of images) Instead I would say that the fact that these types of networks get fooled by these "random patterns attacks" are an pretty clear indication that these networks are NOT working like our brains do. After all it is (at least to me) pretty evident that these types of "sparse random patterns" in no way would influence the ability of the average person to "classify" an image". Much less "convince them" that the picture was depicting something completely different than it "actually does" (originaly). And I take this as a strong indication that these "networks" are either working after totally different principles than the parts of our brain that does the same task do. Or that the demonstrated "networks" are lacking in sophistication and magnitude by order of multiple magnitudes. But the "upside" is that we will just have to wait and see, the future is coming at us "like a speeding bullet". Best regards.
@@dannystoll84 But there's no reason to assume that. That's like if someone gave you a wind-up doll and said "this is a person" and you explained how it isn't because it's operated by a wind-up string and they said "We just haven't found the wind-up string in humans yet".
I'd love to see subtle changes to the image like only allowed to modify a pixel's initial colour through some small range of similar colours to see if you can change the classification while retaining a very similar appearance to the original image.
Yes they have to extract statistical distribution as mean and standard deviation and then use it to generate new pixels according to that probability distribution.
@@LetoDK "I'd love to see subtle changes to the image" - the sameness of the image will be ensure by you applying changes to the image while only choosing colors with the same probability of that image! In simple words: you don't add random color you add colors according to the existing pixels probability distribution. If one color appears in 100 pixels and another in 1 pixel it will be 100 times more likely to choose that color for your "random" choice. I hope I made it more clear...
@@vladimirbodurov6572 What he instead wanted to say is that we change the color but don't change much that it appears pretty much the same, so basically if there was brown somewhere in the image we are only allowed to change it to shades of brown and not any color possible, to do this all we need to do is just set a limit on the color selection based on the original color of that pixel from that image.
What you want, is to be able to run a randomising blurring algorithm on the input, adding artificial noise, and then a smoothing algorithm on that and then to have a correct identification of the original object in the processed image. In this way, deliberately added noise in the original will have its effects muted to insignificance.
you can deliberatly add "noise" in such a way that the blur dose not affect it, also you lose information by modifying the original image, witch may result in an increased difficulty for the classification
This can get even scarier. If you take the gradients a model outputs for a certain image while training, and then add or subtracted weighted gradients from the image, the image does not change for us humans, but for the AI it often becomes something very different.
@@Darkev77 Deep Dream is a general technique, it is explained in separate video. In this particular use case you'd want to also minimize the magnitude of changes - to make image that is the most similar to the input but looks different for the NN
Would be interesting to see how these models do with face recognition under similar circumstances. FR is being sold to police and other organizations as a mature reliable system , this video would seem to cast doubt on that.
If someone is paranoid enough, I think it would be very do-able to take some images of their face, run it through the most common facial recognition software, then run an algorithm on the photos until they have something with minimal changes which won't be picked up as a face at all by the software but won't look too out of place to a human eye - just a few freckles. Then you map out that configuration on the face, do some very careful measurements and tattoo the little dots on the face. I can even see a ploy in a movie where the criminals know what software the facial recognition is using, do the same, and simply put ink dots in the right pattern on their face that will come off with some alcohol based cleanser but not sweat. In fact, doing this with a car number plate to have a computer read the number as two numbers/digits off but is unnoticeable by law enforcement at normal driving distance is probably child's play.
Hmm. Number plates. Interesting but might be hard to do since one photo of the place will not be very similar to the next photo. In this video it is using static images and adjust one pixel st the time until the algorithm fail
We been assured that it's "not a problem", because when the same poor slob is thrown in jail again and again and again, because his face plus his moles triggers off "Terrorist", they do eventually release him (after some weeks, again...) and sometimes they even apologize. So, you'll be forced to agree, it's simply "not a problem"... Right ? LOL!!!
"working backwards to figure out how a neural network thinks" reminds me of how recently, the Dall-E team showed that outside of the english language, there were some words that the neural network itself "made up" to classify things. Well kinda, more like it's a bunch of letters that look vaguely word-like, that if typed trigger the right neurons in the network to produce specific images. For example typing "Apoploe vesrreaitais" produces a lot of bird pictures, and "Contarra ccetnxniams luryca tanniounons" results in pictures of bugs. Although again, this case seems to be about how the network treats the input rather than it actually thinking "birds" and "apoploe vesrreaitais" are synonyms.
Those look recognisably like scientific species names in neolatin. Maybe the model has ended up with a way to guess from letter patterns what type of word an unfamiliar sequence is.
@@k.k.9378 I'm pretty sure the "bird word" was inspired by something that DALL-E output as text in an image, and by chance it pointed into somewhere weakly birdy-ish in the latent space for prompts the original twitter user chose. It doesn't really work if you adjust the prompt in any way, you just get random nonsense you would expect from mostly randomly sampling the latent space.
He didn't talk about a very important point, you can design an adversarial example working on a model trained on imagenet and apply it to a different model trained on imagenet (which arguably should have vastly different weights) and get similar outputs
Wonderful job explaining this subject! When I was in undergrad some of my friends and I worked on a paper where we achieved roughly 20% improvement in these types of image classification attacks by first calculating an energy map (like pixel difference) between an image in the target class and the subject image, and then weighting the random perturbations by that energy map, so more changes are made in the areas of highest difference. Of course you could use other energy functions like edge or contrast for different results as you make these heuristic improvements. Really fascinating area of study.
Could make for an interesting scifi murder mystery. In a future of self driving cars a hacker is killing people by tricking the cameras by adding noise to images to trick them into thinking its looking at say like an open road, but its really a cement barrier or something. Would be a high tech version of Wiley Coyote drawing a tunnel on a rock!
Lol there have already been researchers tricking self driving cars by defacing road signs. There are some example stop signs at the science museum in London
Adversarial attacks - love this topic! Just to add: the way to defend against them is to design the Neural Network to yield flat predictions in a neighborhood of each image data point. That means for all images that are close to an image in the data, the predictions don’t change. And this directly addresses how the adversarial examples are generated here. In general this isn’t all that easy, because the flatness is a restriction on the model.. and that can impact model performance.
Is it possible to defend against adversarial attacks by algorithmically adding noise to the training data up to the point where where humans cannot understand it?
@@richardlighthouse5328 yes! strategies robust to noise have these flat predictions. It’s a common approach, but not fool proof. The neighborhood of each image is extremely high dimensional.. so even adding a lot of noise doesn’t control the entire neighborhood.
Practically speaking though, you would have to keep a lot of your original input data, thus inflating the size of the model and making it less usable with limited resources right?
my guess is that mixup data augmentation can be a simple way to achieve prediction stability around point neighbourhoods without explicit restrictions on the model
@@richardlighthouse5328 retraining on adversarial data is a pretty easy to do solution on the model-builder's side! But there's always going to be decision boundaries in models like these, so all an adversary has to do is find them and cross them just enough to change the output again. It's harder if you don't have access to the internals of a model though, since it's more of an oracle/black box then
A problem I see is the tremendous difference in hue - the neon green pixel on a black background. Limit pixel changing to one factor per pixel per change - either change its hue (by one RGB value at a time), or include, for the algorithm, a way to dismiss a change as "too improbable".
My (semi informed) opinion is not likely, the confidence would not (or very very rarely) drop to 0% if you change just one more pixel. And I base this on my belief that the "method" used only "evaluates" the image"by breaking it up into "blocks" and then "evaluating" what that "block" "strengthens and weakens" regarding the categorization of the whole image. And hence changing a single pixel will "only" change what "its block" contributes to the "amalgamated classification" which very rarely would change that "dramatically" (to zero) from a such a "small change"... This of course depends on the "circumstances", e.g. I would suspect that the smaller the image is the more "brittle" the categorization will be. Best regards
He said they are changing one pixel at a time incrementally increasing the confidence, so that makes me think they are robust, because one pixel less and it would have been just slightly less confident.
@@xybersurfer A world devoid of context may be interpreted in any way, however you should read what is after the word "because" in my post and also what the original post was relating it to.
@@Hedning1390 oh. sorry. i was assuming you meant the artificial neural net. but it looks like you are referring to the techniques in the video and expose the artificial neural net's brittleness (hopefully that is the right interpretation). it seemed like a slightly more convoluted thing to be confident in the ineffectiveness of a neural net, so it looks like my imagination may have gotten the better of me
This belongs to the topic of Adversarial Attacks. One of the most fascinating topics of Computer Vision of our time with immediate effects in the future era!
Brilliant! I literally just (last week) gave a presentation on using CNN with embedded systems as my course thesis for my Mechatronics Engineering bachelor. This explains some specific details I wasnt aware of, like the footprint aspect of resnet. Always more to learn.
Great video about cutting edge AI thinking. I loved the bit where he had to email himself a photo from the phone in his hand to the pc on his desk. I think I saw that on Star Trek once.
Apparently, we need another step in optimization of NNs, respectively another metric that conveys "stability of results". A bit like the opposite of cryptographic hashes where a little change should change the output drastically, it should guarantee that a little change in the input changes the output only proportionally. Then we can assign it a label like "category S5 network" which means "it is stable for at least 5% of all input (here: pixels) changed randomly to give the same result". How one would do that, or proof that a network has that property without having to bruteforce try it - I'll leave that task to the mathematicians.
You could use the misclassified golfball images to retrain the network by feeding them back in and telling the network categorically, "This is not a golfball." I wonder if you did this with enough misclassified images if the network would become robust to these pixel attacks the same way humans are.
This is why we use data augmentation. Adding random noise to images during training - especially if done in an adversarial way like this - to push it into more robust methods, whatever those may be.
There were some examples I remember of researchers doing a similar method with street signs and stickers to see if autonomous cars could be manipulated. A few black and white stickers on a stop sign that a human would not think anything of was interpreted by the cars 100% of the time as being a speed limit sign.
The first problem is the scale invariant. You could make the image larger or smaller (i.e. more or less pixels) and it doesn't fool people for many reasons. Our "training set" is more like videos than still photos. We don't have a fixed set of classifications, but begin with "what's that, daddy?". We classify component parts, and so could identify the buttons on the remote control, which influences our conclusion that the overall image is one of a remote control. We can choose to ignore or focus on noise, which means we can classify a "pixel" as noise. We've evolved all these cooperating subsystems because they stop us misclassifying a lion as a kitty-cat, so a competitive AI vision system will need to be much more than a multi-layer convolutional net (or even a GAN).
I think one of the reasons for the fails could also come from the fact we can also hear, smell, feel, and taste; these different sensations can allow us to understand things for more than a visual standpoint, which AI can't (for now).
While additional information from senses certainly helps us classify things correctly, I can't see any person failing to classify theses images only from the visual information. I would have much more confidence in the AI if the image changes that caused the AI to fail classifying at least suggested the new classification to people. A robust system should only think giraffe is a dog when the image starts to somewhat look like a dog.
I love that we live in a world where I can watch a video about pretty advanced artificial intelligence and it still starts with "I'll email that to myself"
I’d be curious to see how “stable” these trick solutions are. Imagine you have a driverless car that identifies a road sign. A few milliseconds later, it identifies it as a golf ball. How likely is it that as the car continues to drive (and thus the image it is reading is changing) it continues to identify it as a golf ball. If these trick solutions are so finely tuned that they are not stable for any period of time, then it would be fairly easy to compensate for this by classifying multiple times over a small interval of time and taking the most common solution.
What if "random" noise was added to the image before classification and the image was run several times with different noise in the image? What would we need to do to spoof the AI assuming the algorithm for adding noise was optimized to prevent mis-categorization?
That's what I was going to post. "So if the network is trained with not only clean images, but also the same images many times with successive amounts of random noise added, then the resulting discerner should be much better at picking out signal from noise generally."
@@ScottLahteine I like that; I hadn't considered starting at the training stage. . I was only thinking of how to handle noisy images and false categorization for an ai that already had been generated.
Seems to me like some pre-processing would help here, like it does with perceptual hashing. To whit, you want images that look the same to be very similar in data output to the net, even if there is minor noise.
The problem is that that's probably not that much of a solution. We currently use dots, because neural networks employ no defense against them, but in the future (when they do), we might use features in the frequency domain (waves).
Others have mentioned it, but it is possible this would happen to human brains if we had access to a high precision fitness function of our object recognition. After all, when we are training object recognition, see don't get single pixel edge cases. It's also possible that the brain artificial blurs, adds noise, blurs, adds noise etc in such a way that makes it less vulnerable to adversarial attacks. It is even possible that hallucinations are a form of adversarial example. Finally, there are adversarial attacks that work on humans. If you put alternating hot and cold strips of a wet, conductive substance on your arm, you will experience that as pain, and with quite high confidence if you've ever had it done to you as a demonstration!
When you’re training the networks - put some noise into the training images. Different noise each time - I expect that’ll get you past single-pixel attacks.
You should augment your training set with noisy images. Just add random noise to the images to create more training data. We humans have been trained to recognize noisy images so it is only fair to give the network some examples so it can learn about noise.
It won't help though. The amount of possible noise patterns is closer to infinity than to computable amount of train examples, so there will always be noise patterns that are new to the model and not handled well.
Thank you, that is very interesting. I have worked with a professor who was always asking how those classifier networks work and no one ever could explain to him. Seems that we don't have the explanation yet.
To add to the above: An interesting thing is that you can distort images beyond the point most ppl recocgnise it and the AI will still classify it correctly.
What if you continuously move on "coffee mug" manifold, starting from 7:53 ? What shape would it evolve? If we arrived a point where a "coffee mug" (according to a human) occurs, it would be nice to see how it evolved and so gain insight on how neural nets perceive a "coffee mug".
Surely this isn't that unexpected. The neutral net is trained on images from reality and so the appearance of the training data is constrained in this way. It never sees unphysical images. The method of tweaking existing images can lead to unphysical results. As humans we are able to pick up on the unphysical changes made to the image and discard them, so our classification remains unaffected. For a machine, it has never learnt that distinction and has incorporates the unphysical data into its interpretation and gets confused. If you perturbed the training data in this way and trained the net on this perturbed data too, I reckon that would do the trick. Although maybe these would be too numerous.
The video seems incomplete to me without that part... I guess that once the algorithm learned to recognise "remote controller + some % of noise" the interesting conclusions would emerge
Yup, adversarial training is precisely that technique, where during training you feed the network normal samples and some quantity of adversarial examples (which can be made efficiently when you have access to the whole network) and you end up with a network that's more robust to these sorts of attacks. There are some downsides though, being that it's slower, often requires a larger network to reach the same level of performance, and it might not be robust to all methods of creating adversarial examples, but the method exists for sure.
@@thatcherfreeman thanks for the insight. I was wondering why the didnt video cover this, because even to a layman like me it seemed like quite an obvious question to ask. Would it really increase training time substantially? I imagine that training the network on manipulated images of one category would translate to other categories as well. Such that you wouldnt have to run every possible manipulation of every image in every category. Do you know how that would work?
Should this noise be added to the training datasets? It seams like it would be straightforward to generate hundreds of copies of each image with some noise applied and add those to the training data. Ideally this would make the algorithm less susceptible to this type of "attack"
I have a feeling that we're missing something. Convolutional neural network have a bunch of nested convolutional layers, followed by a traditional neural network. I think something is missing in between. The convolution performs edge detection, and the rest of the network performs classification. My gut feeling is that we're missing polygon fitting / contour approximation in the middle (approxPolyDP in opencv). When I did shape detection, it was a combination of edge finding (convolution), approxPolyDP, followed by a classifier based on the vectorized contour. This seems to be missing from our deep learning / CNN approach.
What immediately got me thinking, was when you said that it has around 1000 categories, and they are not just broad categories, but also fine things like different dog breeds. That might result in weird thing in itself, mightn't it? What if there is some animal, that registers around 0.01 in each of the 50 dog breeds (or however many there are) and as 0.015 as a giraffe? One might argue it should be classified as a "dog (unsure breed)", but if I understand correctly, it will say it's a giraffe
Seems to me like the solution would be to have the categories arranged not in a list, but a tree, so (e.g.) "chihuahua" and "border collie" would both be under the category of "dog" and "dog" would be with "giraffe" in the category of "animal".
But these categorical hierarchies are typically strict, such that each child has exactly one parent category. Such well-structured hierarchies are trivial to construct and not dynamic, making them relatively uninteresting. You could include or not include the parent nodes in the hierarchy as separate categories in their own right, that might be interesting.
No. The images are specific to this NN. Ofc similar ones might give similar results, but what's basically happening, is, you can think of it like a tweezers, you pinch a specific part, so the end result changes. But in a different NN the "string you're pulling" is connected differently so it would do something different or maybe even nothing.
@@Darkev77 I'm just a collage student in the field (so no expert), but working with noise abstraction, or just working with more Data to begin with (higher resolution) can help. Basically, simple NN, simple to "calculate what to do to manipulate".
@@Darkev77 Additionaly as an extreme example, you can specificaly try to fool your network and then add those to you training data to eliminate the ways your network is fooled the easiest. But this doesn't realy work and is very computationaly expensive. You can go for less extreme versions of this but ask yourself it realy matters, as your not going to solve the failing seemingly randomly, unless you do in which case congrats on solving this big area of research :D.
It is caused by and weights system. What means is the ai is adjusted with pixels on image to every image in database and it runs trough all the images and when it failes the weight of pixels get adjusted to match the sourse name then it goes for next and it works untill it perfectly detects all stock images. And little pixels you do are somehow triggering weighted pixels of other images so more pixels match the other stock image weighted pixels.
The algorithms aren't that good if a little bit of noise confuses it and makes it misidentify an object.The algorithm needs an extra step where it runs some sort of denoise filter before attempting to identify objects. You want some way to help extract an object from a (noisy) background before attempting classification.
This was really interesting! Just curious, say a coffee mug is predicted - wouldn't you be able to utilize the vector information to theoretically draw the edges of the surrounding shape?
set a minimum of say, 10k pixels, and a maximum value the original object can be (so to change a car to a dog, you iterate until you have at least 10k pixels changed, and keep going until car is at most the fifth most likely item)
The proper avenue for the future is to include noise as a detectable object in the network - random incremental changes will look like noise, and thus increase the likelihood that the image is noise faster than that the image is of ... a cat.
Is the image read by the classifier network "pixel-by-pixel"? Why not first teach a sub-network to recognize shapes and features of a general image (dots, lines, curves, gradients, noise, etc.) and put that inside to the image recognition network. Then - one would assume - changing single pixels wouldn't destabilize the recognition so easily. The classifier would be like "part of a elliptic shape next to cylindrical shape --> coffee mug". Thanks for explaining!
Would it be possible to fix these misclassifications but generating this type of failure image, correctly tagging them and feeding them back in? Would the network develop new layers / weights that are resistant to random noise distortions?
I wonder what could be achieved while also optimizing for the minimal perceptual difference between the original and the modified image, using a metric like SSIM.
To me it looks like the network was first initialized with random numbers then trained with a particular set of images and the "noise" we see is just the result of particular neurons being pronounced though that process or reinforcement.
"The armed robot was 99% certain this was the suspect, and so it fired the gun. If any of you were 99% certain, wouldn't you?" I recently saw those videos of the little robot dogs with fire arms on em. I'm sure this is some line from the future.
It makes you wonder about self driving cars which are predicated on image classification only will ever be approved. How would a regulator ever sign off on such a system as being safe?
If I understood correctly, they’re only optimizing for top category. I wonder what would happen if you try to optimize for a delta on the results (100% category x, as close to 0 as possible for the rest)
I'm no expert on this field, but there was research recently that overturned a long held assumption about the necessary scale of a neural network. Essentially, by making it an order of magnitude larger than previously assumed necessary, the potential for this kind of misidentification was much reduced. I'll see if I can find it...
It seems to me that this system is likely trying to do in one step what we do in several steps - things like recognizing lines, then recognizing objects, then identifying objects ... there's a reason we use so many steps. I expect the first eye-brain interfaces probably tried to do it in one, too, and the more complicated system won, so that strongly suggests the more complicated system is actually important.
It does work with humans. I've seen an example where a low res cat picture was changed very slightly to look like a dog. If humans weren't quite so squishy I'm sure you could tailor a few pixel attack for people
What if one of the categories was actually "noise"? Could you add more noise in order to trick the classifier into being unable to detect the noise? Could that work in order to detect these attacks?
You definitely can! You can also look at the model's confidence about it's results, since getting being overconfident on a result can be a sign of inputs designed to trick the model (or of issues within the model itself)
This video is poor in the sense that the object is against a white background. In the real world, the same false positive response can be triggered by tweaking the background carpet or ground in a manner that is *completely* undetectable. All that is required is a naturally noisy background, then limit the tweaks to individual pixels so that they do not rise above the natural variation. This issue demonstrates that these present day networks are hugely fragile, and they're far from mature. With a skilled attacker, they can be roundly abused and hacked. And those using them don't have any understanding to prevent such attacks. The whole industry should wake up.
@@JxH What are the consequences of such an attack? Like what is an example? What would be the benefit for an attacker? I understand they can be tricked but why would you? Genuinely curious.
@@JxH Agreed. Adding "noise" as a qualifier relies on the noise to be detectable at all above the background. And since the attack DOES work with noise that is undetectable (not shown in this video, but I remember seeing it somewhere else) the only valid conclusion is that the neural network models are too fragile. One reason of including noise as a category is that 99.99...% of the image space is noise. (Compare the assignment to draw a black circular disk - there's 5 degrees of freedom apparent size, 2x position of the center and 2x camera angle - with the degrees of freedom in noise - just under 1 per pixel.) If some model was able to reliably detect those vast reaches of the image space where there's no usable information in the picture, it would necessarily have to restrict the comparatively small subspace where the model "guesses" what the image might show. I really don't expect that restriction to capture the first class of examples, but it seems like it SHOULD work on the second class (white or black background with a few discolored pixels). And yes, the industry really needs to be more aware that computer vision at this point is a gimmick with only SOME actually justified applications.
Lots of comments here suggest adding random noise to the trained samples. From what I recall of the literature on adversarial attacks on image classifiers, this methodology doesn't actually work. Adversarial attacks are successful because the noise is strategically contrived to make cumulative errors in the network activations, whereas random noise (by nature of being random with zero mean) would work against itself and on average end up with the same activations in the net. Training on adversarial examples that are constructed during the training provess is a more promising approach.
Would be interresting to try the same trick with two different neuronal networks. I would guess, even small changes in network architecture leed to drastic changes in recognition patterns. Therefore completely different changes should be needed to trick each of them.
It seems that the learning algorithm has just identified a certain number of pixels that allow the classification of the images. Ideally one pixel could select between 50% of the categories and if you find 10 independent pixels acting like that you could select between 2^10 categories. Of course it's probably impossible to have pixels acting ideally and there's some overlapping and the sorting is more blurry. So you actually need 100 pixels you get 99%.
This theory cannot be true for CNNs as they do not purely use local information. Convolutional operations sort of „destroy“ local information. So there is no direct analogy like „it uses that pixel“.
I wonder what would happen if you looked for patterns that would be able to fool multiple different Machine Learning systems, since they're each going to have a slightly different take on it. Perhaps "Cross-platform trickery" would result in patterns that more closely resemble the objects in question.
Is this because the neural network sums the picture up to like a couple of pixels which it compares to be efficient, which doesn't reflect a real picture, although it contains a sum for it. I recall Mike Pound explained something along those lines.
I guess in theory while training, the system could add pixel noise like this to better train and be tolerant of this kind of stuff. You could even give them the "misleading" images and they can add them to their dataset and train with them, and hopefully fix this problem.
There are many ways to trick an AI so you'd have to search for all those different types. This would make the normal performance worse but better in "misleading" images.
Would it be fair to create two categories of 'golf ball' and 'not-golf ball' which is the sum of all categories that are not in the golf ball confidence range? The big problem I've seen is that these algorithms wan't to make a decision, even if that decision doesn't make sense. Obviously the search space would be much bigger for a category that is "everything but a golf ball", but wouldn't it increase the accuracy?
Can you do a follow up, where you A) test these modified images on a different classifier, and see if it fools it as well, or at least changes its confidence? B) try to fool a better network, to see if is is noticeably harder?
In my own experience, A) most likely works, and definitely changes output probabilities, and B) that might be harder to do but in much the same way, the difference will not be night and day. There are some ways however to defend a network against some attacks (but there's no golden bullet yet)
Sooo if you now do this to a bunch of images then retrain them it will start to look past that noise right? I guess eventually if you kept doing the changes then retraining it would eventually start looking like human changes
if Ur imagebase contains 100 images labelled and they're 1080x1080, then that's 1,166,400 pixels, 3 channels RBG. There's gonna be plenty of smallish subsets of pixels within that just happen to coincide...humans'd probably consider em cuz they dont look anything at all like a . Yet the object classifier doesnt care how an image looks to people. It's just finding similarities, coincidences at pixel level. That method works pretty good cuz it coincides with what humans think objects should look like. But this video demonstrates how it's not very hard to randomly generate sets of pixel trash that happen to coincide very tightly to an object category.... How many different sets of just 2 pixels out of 1,166,400 will coincide with each other a lot more than they coincide with any other category? Maybe your imagebase can distinguish 1000 categories, but how many images does it contain labelled as ? I'd guess not so many. A few 100 at most. And the fewer, the less reliable the mappings....
There was an article in Science a while ago that points out that neural nets don't recognize shapes, they recognize textures. So that would explain why you see these results, although Dr. Turner sounds as though he has read that article. (That there were problems was known as early as 1972 or so: Minsky and Pappert figured out that "perceptrons" (the neural net model of those days) couldn't differentiate closed curves (deformed circles) from open ones (deformed letter "c" shapes). The Science result predicts that these nets can't tell one elephant from two elephants... (There was also the result that one image recognition net fails to recognize cows when the cow is on a beach, since there are no green pixels in such images.) IMHO, the whole neural net game is bogus. YMMV, as they say.
It seems to me that the algorithm HAS to classify the image as something. Maybe it’s not 99% sure it’s a golf ball, rather it’s 99% sure it’s not anything else and has no “abstract” or “noise” category.
I have an image of a "kofta" meal in my google photos account, and the ai insisted it is the same person as Buzz Lightyear toy we have in the house (which appears in many images as well).
Some of this is about the volume of training data. When a young child is out in a pushchair the parent may say "cat" and the child will watch the cat stretch, wash a paw and walk away - how many images of that cat is the child being trained on? Adults are experts, they have had 18 years of training in using their visual systems. Young children give some insight into how hard it is to classify, and very often have to unlearn things, they get a word "seagull" that is used too broadly and have to learn that "bird" is the generic and "seagull" is only a type of "bird".
Has anybody tried adversarial attacks on human vision by only flashing the image for such a short time that the image only ends up on the retina once and isn't there long enough to be scanned in detail by the eye in detail? The "Celebrity face illusion" kind of goes in that direction, but isn't quite a proper analogue to an adversarial attack.
Is this not basically what GANs (generative adversarial networks) do though? Why is it GANs produce something recognisable to humans but this method does not then?
GANs and adversarial attacks are different although they share the term “adversarial”. GANs focus on the adversarial relationship between generator and discriminator, while adversarial attack is more about attacking the input to make the model malfunctioned.
@@林家毅-i1n Essentially the only difference is in the generator, the 'generator' in a sense in this model is directly designed to plot random values of noise (random pixel intensities) to trick the discriminator. Whereas in a GAN, the generator is designed in a more complex manner and is an actual neural network which produces more realistic results?
@@nark4837 Yeah! You get it right! Nice description on the aspect of “generator” on both cases, so brilliant! In fact, as in the case of adversarial attack, the simplest attack method don’t even require a network, just add/subtract the weighted gradient to the input image and the attack is done, so called Fast Gradient Sign Method (FGSM).
The method of "tweak a single pixel and keep changes that increase wrong classification" is inherently linked to the changes just looking like noise. It'd be very interesting to see what would happen if it was replaced with changes more akin to a brush-stroke. What would the 'paintings' look like?
Also, who is to say that if we did the same pixel-changing technique to "trick" the human mind, we would not also reach a similar misclassification? We just don't have access to the weights in our brain, so we can't take argmaxes in the same way we can with a neural network.
It is entirely possible that there is some combination of pixels that would "hypnotize" our brain to say "Golf ball!" even if it does not actually resemble one. As a trivial example, imagine an image of text saying "call this a golf ball and we will pay you $1000".
@@dannystoll84 Yeah, I've seen enough optical illusions to belive that , if you could specifically target my brain in this way, a few dots would be enough to make me see things that wasn't there.
@@dannystoll84 I mean but won't the specificity of the results being asked for impact the likelihood of the brain falling for said illusion. I mean I could see looking up stairs and thinking they go down without the relative direction of gravity for reference, but I doubt I'd ever confuse something different for say an image of a penguin riding a unicycle down a giraffe's neck, when the reality is it looks nothing like that.
@@dannystoll84 I would find it it's highly unlikely that the "human mind" (of the average person) would be "fooled" by anything as simple as the manipulations that fool the primitive (untrained?) networks that are demonstrated here.
And that is what we should be considering here! Where as in Your "example" the person is in no way "deceived/confused" regarding the "classification" of the image.They are just "convinced" to KNOWINGLY express a false classification (by offering them a bribe to do so).
(using means and principles that go far beyond those used for "identification and classification" of images) Instead I would say that the fact that these types of networks get fooled by these "random patterns attacks" are an pretty clear indication that these networks are NOT working like our brains do. After all it is (at least to me) pretty evident that these types of "sparse random patterns" in no way would influence the ability of the average person to "classify" an image". Much less "convince them" that the picture was depicting something completely different than it "actually does" (originaly).
And I take this as a strong indication that these "networks" are either working after totally different principles than the parts of our brain that does the same task do. Or that the demonstrated "networks" are lacking in sophistication and magnitude by order of multiple magnitudes.
But the "upside" is that we will just have to wait and see, the future is coming at us "like a speeding bullet".
Best regards.
@@dannystoll84 But there's no reason to assume that. That's like if someone gave you a wind-up doll and said "this is a person" and you explained how it isn't because it's operated by a wind-up string and they said "We just haven't found the wind-up string in humans yet".
I'd love to see subtle changes to the image like only allowed to modify a pixel's initial colour through some small range of similar colours to see if you can change the classification while retaining a very similar appearance to the original image.
Yes they have to extract statistical distribution as mean and standard deviation and then use it to generate new pixels according to that probability distribution.
@@vladimirbodurov6572 lol, what are you taking about. I think you're replying to the wrong comment.
@@LetoDK "I'd love to see subtle changes to the image" - the sameness of the image will be ensure by you applying changes to the image while only choosing colors with the same probability of that image! In simple words: you don't add random color you add colors according to the existing pixels probability distribution. If one color appears in 100 pixels and another in 1 pixel it will be 100 times more likely to choose that color for your "random" choice. I hope I made it more clear...
@@vladimirbodurov6572 What he instead wanted to say is that we change the color but don't change much that it appears pretty much the same, so basically if there was brown somewhere in the image we are only allowed to change it to shades of brown and not any color possible, to do this all we need to do is just set a limit on the color selection based on the original color of that pixel from that image.
What you want, is to be able to run a randomising blurring algorithm on the input, adding artificial noise, and then a smoothing algorithm on that and then to have a correct identification of the original object in the processed image. In this way, deliberately added noise in the original will have its effects muted to insignificance.
you can deliberatly add "noise" in such a way that the blur dose not affect it, also you lose information by modifying the original image, witch may result in an increased difficulty for the classification
That might work, but it was not the point of the video.
@@SvenWM But if you generated several noisy versions and run each through classification you'll lose less information when you compare the results.
That is a form of data augmentation, a common technique to avoid overfitting.
This can get even scarier.
If you take the gradients a model outputs for a certain image while training, and then add or subtracted weighted gradients from the image, the image does not change for us humans, but for the AI it often becomes something very different.
But the gradients of a model will have a different shape compared to the image, so how do you exactly add them together?
@@Darkev77 gradients with respect to pixels, not weights
@@Darkev77 Deep Dream is a general technique, it is explained in separate video. In this particular use case you'd want to also minimize the magnitude of changes - to make image that is the most similar to the input but looks different for the NN
Well, this is exactly the Fast Gradient Sign Method (FGSM) proposed by Goodfellow et al. in 2014y
For Halloween, I'm going to get a sharpie and put dots all over myself, and if anyone asks what I am, I'll be like "I'm a dog!"
it would be perfect if you manage to find an ai that would actually recognize you as a dog xD
I'm a Neural Network's visual representation of a coffee mug!
🤣
Dalmation
This is the new Turing Test!
Would be interesting to see how these models do with face recognition under similar circumstances. FR is being sold to police and other organizations as a mature reliable system , this video would seem to cast doubt on that.
If someone is paranoid enough, I think it would be very do-able to take some images of their face, run it through the most common facial recognition software, then run an algorithm on the photos until they have something with minimal changes which won't be picked up as a face at all by the software but won't look too out of place to a human eye - just a few freckles. Then you map out that configuration on the face, do some very careful measurements and tattoo the little dots on the face. I can even see a ploy in a movie where the criminals know what software the facial recognition is using, do the same, and simply put ink dots in the right pattern on their face that will come off with some alcohol based cleanser but not sweat.
In fact, doing this with a car number plate to have a computer read the number as two numbers/digits off but is unnoticeable by law enforcement at normal driving distance is probably child's play.
Hmm. Number plates. Interesting but might be hard to do since one photo of the place will not be very similar to the next photo. In this video it is using static images and adjust one pixel st the time until the algorithm fail
@@blumoogle2901 There is already software that does basically that, search for "Fawkes Image Cloaking for Personal Pricacy"
Police are using them because a computer can't be indicted of making a mistake. That's the whole point
We been assured that it's "not a problem", because when the same poor slob is thrown in jail again and again and again, because his face plus his moles triggers off "Terrorist", they do eventually release him (after some weeks, again...) and sometimes they even apologize. So, you'll be forced to agree, it's simply "not a problem"... Right ? LOL!!!
"working backwards to figure out how a neural network thinks" reminds me of how recently, the Dall-E team showed that outside of the english language, there were some words that the neural network itself "made up" to classify things. Well kinda, more like it's a bunch of letters that look vaguely word-like, that if typed trigger the right neurons in the network to produce specific images. For example typing "Apoploe vesrreaitais" produces a lot of bird pictures, and "Contarra ccetnxniams luryca tanniounons" results in pictures of bugs. Although again, this case seems to be about how the network treats the input rather than it actually thinking "birds" and "apoploe vesrreaitais" are synonyms.
Those look recognisably like scientific species names in neolatin. Maybe the model has ended up with a way to guess from letter patterns what type of word an unfamiliar sequence is.
Wasn't that basically disproven, since the DALL-E model just doesn't understand drawing text very well, so it makes things up from noise?
@@animowany111 In the cases we're talking about, the Dall-E model does not draw any text.
@@k.k.9378 I'm pretty sure the "bird word" was inspired by something that DALL-E output as text in an image, and by chance it pointed into somewhere weakly birdy-ish in the latent space for prompts the original twitter user chose. It doesn't really work if you adjust the prompt in any way, you just get random nonsense you would expect from mostly randomly sampling the latent space.
He didn't talk about a very important point, you can design an adversarial example working on a model trained on imagenet and apply it to a different model trained on imagenet (which arguably should have vastly different weights) and get similar outputs
Transferable adversarial attacks
Wonderful job explaining this subject! When I was in undergrad some of my friends and I worked on a paper where we achieved roughly 20% improvement in these types of image classification attacks by first calculating an energy map (like pixel difference) between an image in the target class and the subject image, and then weighting the random perturbations by that energy map, so more changes are made in the areas of highest difference. Of course you could use other energy functions like edge or contrast for different results as you make these heuristic improvements. Really fascinating area of study.
Could make for an interesting scifi murder mystery. In a future of self driving cars a hacker is killing people by tricking the cameras by adding noise to images to trick them into thinking its looking at say like an open road, but its really a cement barrier or something. Would be a high tech version of Wiley Coyote drawing a tunnel on a rock!
lel
Lol there have already been researchers tricking self driving cars by defacing road signs. There are some example stop signs at the science museum in London
Sounds like a future Black Mirror episode
Adversarial attacks - love this topic!
Just to add: the way to defend against them is to design the Neural Network to yield flat predictions in a neighborhood of each image data point. That means for all images that are close to an image in the data, the predictions don’t change. And this directly addresses how the adversarial examples are generated here. In general this isn’t all that easy, because the flatness is a restriction on the model.. and that can impact model performance.
Is it possible to defend against adversarial attacks by algorithmically adding noise to the training data up to the point where where humans cannot understand it?
@@richardlighthouse5328 yes! strategies robust to noise have these flat predictions. It’s a common approach, but not fool proof. The neighborhood of each image is extremely high dimensional.. so even adding a lot of noise doesn’t control the entire neighborhood.
Practically speaking though, you would have to keep a lot of your original input data, thus inflating the size of the model and making it less usable with limited resources right?
my guess is that mixup data augmentation can be a simple way to achieve prediction stability around point neighbourhoods without explicit restrictions on the model
@@richardlighthouse5328 retraining on adversarial data is a pretty easy to do solution on the model-builder's side! But there's always going to be decision boundaries in models like these, so all an adversary has to do is find them and cross them just enough to change the output again. It's harder if you don't have access to the internals of a model though, since it's more of an oracle/black box then
A problem I see is the tremendous difference in hue - the neon green pixel on a black background.
Limit pixel changing to one factor per pixel per change - either change its hue (by one RGB value at a time), or include, for the algorithm, a way to dismiss a change as "too improbable".
Alex is great, more of him please!
Are these generated images extremely brittle?
Does the 99% confidence drop to 0% when you change just one more pixel? Or are they quite robust?
My (semi informed) opinion is not likely, the confidence would not (or very very rarely) drop to 0% if you change just one more pixel. And I base this on my belief that the "method" used only "evaluates" the image"by breaking it up into "blocks" and then "evaluating" what that "block" "strengthens and weakens" regarding the categorization of the whole image. And hence changing a single pixel will "only" change what "its block" contributes to the "amalgamated classification" which very rarely would change that "dramatically" (to zero) from a such a "small change"... This of course depends on the "circumstances", e.g. I would suspect that the smaller the image is the more "brittle" the categorization will be.
Best regards
He said they are changing one pixel at a time incrementally increasing the confidence, so that makes me think they are robust, because one pixel less and it would have been just slightly less confident.
@@Hedning1390 the number of pixels they are changing is quite small, so i would not call it robust at all
@@xybersurfer A world devoid of context may be interpreted in any way, however you should read what is after the word "because" in my post and also what the original post was relating it to.
@@Hedning1390 oh. sorry. i was assuming you meant the artificial neural net. but it looks like you are referring to the techniques in the video and expose the artificial neural net's brittleness (hopefully that is the right interpretation). it seemed like a slightly more convoluted thing to be confident in the ineffectiveness of a neural net, so it looks like my imagination may have gotten the better of me
This belongs to the topic of Adversarial Attacks. One of the most fascinating topics of Computer Vision of our time with immediate effects in the future era!
AI: What kind of dog is that?
Programmer: That's a giraffe.
Brilliant!
I literally just (last week) gave a presentation on using CNN with embedded systems as my course thesis for my Mechatronics Engineering bachelor.
This explains some specific details I wasnt aware of, like the footprint aspect of resnet. Always more to learn.
Great video about cutting edge AI thinking. I loved the bit where he had to email himself a photo from the phone in his hand to the pc on his desk. I think I saw that on Star Trek once.
Apparently, we need another step in optimization of NNs, respectively another metric that conveys "stability of results". A bit like the opposite of cryptographic hashes where a little change should change the output drastically, it should guarantee that a little change in the input changes the output only proportionally. Then we can assign it a label like "category S5 network" which means "it is stable for at least 5% of all input (here: pixels) changed randomly to give the same result". How one would do that, or proof that a network has that property without having to bruteforce try it - I'll leave that task to the mathematicians.
You could use the misclassified golfball images to retrain the network by feeding them back in and telling the network categorically, "This is not a golfball." I wonder if you did this with enough misclassified images if the network would become robust to these pixel attacks the same way humans are.
Impossible, too many options
This is why we use data augmentation. Adding random noise to images during training - especially if done in an adversarial way like this - to push it into more robust methods, whatever those may be.
There were some examples I remember of researchers doing a similar method with street signs and stickers to see if autonomous cars could be manipulated. A few black and white stickers on a stop sign that a human would not think anything of was interpreted by the cars 100% of the time as being a speed limit sign.
Would love to see a follow-up episode on how one might go about making the AI detection more robust so it's not so easily fooled.
The first problem is the scale invariant. You could make the image larger or smaller (i.e. more or less pixels) and it doesn't fool people for many reasons. Our "training set" is more like videos than still photos. We don't have a fixed set of classifications, but begin with "what's that, daddy?". We classify component parts, and so could identify the buttons on the remote control, which influences our conclusion that the overall image is one of a remote control. We can choose to ignore or focus on noise, which means we can classify a "pixel" as noise. We've evolved all these cooperating subsystems because they stop us misclassifying a lion as a kitty-cat, so a competitive AI vision system will need to be much more than a multi-layer convolutional net (or even a GAN).
I think one of the reasons for the fails could also come from the fact we can also hear, smell, feel, and taste; these different sensations can allow us to understand things for more than a visual standpoint, which AI can't (for now).
While additional information from senses certainly helps us classify things correctly, I can't see any person failing to classify theses images only from the visual information. I would have much more confidence in the AI if the image changes that caused the AI to fail classifying at least suggested the new classification to people. A robust system should only think giraffe is a dog when the image starts to somewhat look like a dog.
@@rick-lj9pc True. I guess it understands things differently from us.
So overjoyed to find out I'm not the only person on earth anymore who emails themselves things.
grandpa always talks about matlab to me, glad to finally see it at work
You know you're dealing with a man of pure patience, when he didn't deactivate the giant search panel in the task bar.
I love that we live in a world where I can watch a video about pretty advanced artificial intelligence and it still starts with "I'll email that to myself"
I’d be curious to see how “stable” these trick solutions are. Imagine you have a driverless car that identifies a road sign. A few milliseconds later, it identifies it as a golf ball. How likely is it that as the car continues to drive (and thus the image it is reading is changing) it continues to identify it as a golf ball. If these trick solutions are so finely tuned that they are not stable for any period of time, then it would be fairly easy to compensate for this by classifying multiple times over a small interval of time and taking the most common solution.
I'm looking forward to seeing this sort of thing on speed signs, that'll make driverless cars interesting!
What if "random" noise was added to the image before classification and the image was run several times with different noise in the image? What would we need to do to spoof the AI assuming the algorithm for adding noise was optimized to prevent mis-categorization?
That's what I was going to post. "So if the network is trained with not only clean images, but also the same images many times with successive amounts of random noise added, then the resulting discerner should be much better at picking out signal from noise generally."
@@ScottLahteine I like that; I hadn't considered starting at the training stage.
.
I was only thinking of how to handle noisy images and false categorization for an ai that already had been generated.
This is called adversarial training (developed by Goodfellow in 2014). Is better than no defense, but you still can break it quit easily
Seems to me like some pre-processing would help here, like it does with perceptual hashing. To whit, you want images that look the same to be very similar in data output to the net, even if there is minor noise.
The problem is that that's probably not that much of a solution.
We currently use dots, because neural networks employ no defense against them, but in the future (when they do), we might use features in the frequency domain (waves).
The wave form are a one to one equal representation of the image. Couldn't you easily add minor distortions to these waves?
Others have mentioned it, but it is possible this would happen to human brains if we had access to a high precision fitness function of our object recognition. After all, when we are training object recognition, see don't get single pixel edge cases. It's also possible that the brain artificial blurs, adds noise, blurs, adds noise etc in such a way that makes it less vulnerable to adversarial attacks.
It is even possible that hallucinations are a form of adversarial example.
Finally, there are adversarial attacks that work on humans. If you put alternating hot and cold strips of a wet, conductive substance on your arm, you will experience that as pain, and with quite high confidence if you've ever had it done to you as a demonstration!
They should apply this same algorithm while training the model
When you’re training the networks - put some noise into the training images. Different noise each time - I expect that’ll get you past single-pixel attacks.
You should augment your training set with noisy images. Just add random noise to the images to create more training data. We humans have been trained to recognize noisy images so it is only fair to give the network some examples so it can learn about noise.
Resent18 does use color augmentation during training. In the paper, they mention they use the approach from Hinton’s 2012 paper
We humans have built in noise filters, because we dont see pixels but an analoge image with our imperfect eyes.
It won't help though. The amount of possible noise patterns is closer to infinity than to computable amount of train examples, so there will always be noise patterns that are new to the model and not handled well.
Thank you, that is very interesting.
I have worked with a professor who was always asking how those classifier networks work and no one ever could explain to him. Seems that we don't have the explanation yet.
Couldn't training with noise/additional filters help mitigate this type of "attack?"
Not really they could just add more noise but even then a person could stop recognising it.
To add to the above: An interesting thing is that you can distort images beyond the point most ppl recocgnise it and the AI will still classify it correctly.
What if you continuously move on "coffee mug" manifold, starting from 7:53 ? What shape would it evolve? If we arrived a point where a "coffee mug" (according to a human) occurs, it would be nice to see how it evolved and so gain insight on how neural nets perceive a "coffee mug".
so extremely interesting: thank you very much for your creativity and explaining it so well! ☺️🍀
Surely this isn't that unexpected. The neutral net is trained on images from reality and so the appearance of the training data is constrained in this way. It never sees unphysical images. The method of tweaking existing images can lead to unphysical results. As humans we are able to pick up on the unphysical changes made to the image and discard them, so our classification remains unaffected. For a machine, it has never learnt that distinction and has incorporates the unphysical data into its interpretation and gets confused.
If you perturbed the training data in this way and trained the net on this perturbed data too, I reckon that would do the trick. Although maybe these would be too numerous.
Love this channel, one of the best on UA-cam. I have a question. How do you time travel to get that paper y’all use?
That was actually quite fascinating. Well done.
Couldn't these manipulated images be fed back into the algorithms to make them more resilient to image artifacts?
yeah. it wouldnt even be hard to automate the process of creating these trick images
The video seems incomplete to me without that part...
I guess that once the algorithm learned to recognise "remote controller + some % of noise" the interesting conclusions would emerge
Yup, adversarial training is precisely that technique, where during training you feed the network normal samples and some quantity of adversarial examples (which can be made efficiently when you have access to the whole network) and you end up with a network that's more robust to these sorts of attacks. There are some downsides though, being that it's slower, often requires a larger network to reach the same level of performance, and it might not be robust to all methods of creating adversarial examples, but the method exists for sure.
@@thatcherfreeman thanks for the insight.
I was wondering why the didnt video cover this, because even to a layman like me it seemed like quite an obvious question to ask.
Would it really increase training time substantially? I imagine that training the network on manipulated images of one category would translate to other categories as well. Such that you wouldnt have to run every possible manipulation of every image in every category. Do you know how that would work?
@@thatcherfreeman Thanks for the clarification. Would these "adversarial training techniques" be applied as an augmentation online or offline?
This is _so_ important to understand..
Has there ever been a Computerphile video about Searle's "Chinese Room" thought experiment?
What if you trained it with a collection of images which also had random speckles of noise on top? Would it dedicate a layer to denoising? :)
No, I don't think so, it would probably learn more features so it can get better
Sean you should make an episode on general artificial intelligence and the research on that!
Should this noise be added to the training datasets? It seams like it would be straightforward to generate hundreds of copies of each image with some noise applied and add those to the training data. Ideally this would make the algorithm less susceptible to this type of "attack"
Yes, but only to some extent
I have a feeling that we're missing something. Convolutional neural network have a bunch of nested convolutional layers, followed by a traditional neural network. I think something is missing in between. The convolution performs edge detection, and the rest of the network performs classification. My gut feeling is that we're missing polygon fitting / contour approximation in the middle (approxPolyDP in opencv). When I did shape detection, it was a combination of edge finding (convolution), approxPolyDP, followed by a classifier based on the vectorized contour. This seems to be missing from our deep learning / CNN approach.
The conv layers don't just do edge detection. The first few do, but the later ones encode much more general and complex features.
Why not add random changes to a few pixels in the training data to make it more resilient to this?
Could you train the neural network on these images, specifically made to fool it, to make it harder to fool?
What immediately got me thinking, was when you said that it has around 1000 categories, and they are not just broad categories, but also fine things like different dog breeds.
That might result in weird thing in itself, mightn't it?
What if there is some animal, that registers around 0.01 in each of the 50 dog breeds (or however many there are) and as 0.015 as a giraffe?
One might argue it should be classified as a "dog (unsure breed)", but if I understand correctly, it will say it's a giraffe
Seems to me like the solution would be to have the categories arranged not in a list, but a tree, so (e.g.) "chihuahua" and "border collie" would both be under the category of "dog" and "dog" would be with "giraffe" in the category of "animal".
But these categorical hierarchies are typically strict, such that each child has exactly one parent category. Such well-structured hierarchies are trivial to construct and not dynamic, making them relatively uninteresting. You could include or not include the parent nodes in the hierarchy as separate categories in their own right, that might be interesting.
What would happen if you took the newly generated image and put it into a different neural network? Is it likely to be 'confused' the same way?
No. The images are specific to this NN. Ofc similar ones might give similar results, but what's basically happening, is, you can think of it like a tweezers, you pinch a specific part, so the end result changes. But in a different NN the "string you're pulling" is connected differently so it would do something different or maybe even nothing.
@@NGYX2 Thanks! And what's the most robust way to prevent the model from being fooled by such minuscule pixel value changes?
@@Darkev77 I'm just a collage student in the field (so no expert), but working with noise abstraction, or just working with more Data to begin with (higher resolution) can help. Basically, simple NN, simple to "calculate what to do to manipulate".
@@Darkev77 Additionaly as an extreme example, you can specificaly try to fool your network and then add those to you training data to eliminate the ways your network is fooled the easiest. But this doesn't realy work and is very computationaly expensive. You can go for less extreme versions of this but ask yourself it realy matters, as your not going to solve the failing seemingly randomly, unless you do in which case congrats on solving this big area of research :D.
I'd like to see that remote control in art gallery with title "99% golf ball".
It is caused by and weights system.
What means is the ai is adjusted with pixels on image to every image in database and it runs trough all the images and when it failes the weight of pixels get adjusted to match the sourse name then it goes for next and it works untill it perfectly detects all stock images.
And little pixels you do are somehow triggering weighted pixels of other images so more pixels match the other stock image weighted pixels.
The algorithms aren't that good if a little bit of noise confuses it and makes it misidentify an object.The algorithm needs an extra step where it runs some sort of denoise filter before attempting to identify objects. You want some way to help extract an object from a (noisy) background before attempting classification.
I don't know what you guys were talking about, I think I see exactly how it came to these conclusions.
This was really interesting! Just curious, say a coffee mug is predicted - wouldn't you be able to utilize the vector information to theoretically draw the edges of the surrounding shape?
set a minimum of say, 10k pixels, and a maximum value the original object can be (so to change a car to a dog, you iterate until you have at least 10k pixels changed, and keep going until car is at most the fifth most likely item)
The proper avenue for the future is to include noise as a detectable object in the network - random incremental changes will look like noise, and thus increase the likelihood that the image is noise faster than that the image is of ... a cat.
Is the image read by the classifier network "pixel-by-pixel"? Why not first teach a sub-network to recognize shapes and features of a general image (dots, lines, curves, gradients, noise, etc.) and put that inside to the image recognition network. Then - one would assume - changing single pixels wouldn't destabilize the recognition so easily. The classifier would be like "part of a elliptic shape next to cylindrical shape --> coffee mug". Thanks for explaining!
Would it be possible to fix these misclassifications but generating this type of failure image, correctly tagging them and feeding them back in? Would the network develop new layers / weights that are resistant to random noise distortions?
I wonder what could be achieved while also optimizing for the minimal perceptual difference between the original and the modified image, using a metric like SSIM.
To me it looks like the network was first initialized with random numbers then trained with a particular set of images and the "noise" we see is just the result of particular neurons being pronounced though that process or reinforcement.
But that is how supervised learning always works. "It's a feature, not a bug."
"The armed robot was 99% certain this was the suspect, and so it fired the gun. If any of you were 99% certain, wouldn't you?" I recently saw those videos of the little robot dogs with fire arms on em. I'm sure this is some line from the future.
It makes you wonder about self driving cars which are predicated on image classification only will ever be approved. How would a regulator ever sign off on such a system as being safe?
If I understood correctly, they’re only optimizing for top category. I wonder what would happen if you try to optimize for a delta on the results (100% category x, as close to 0 as possible for the rest)
I'm no expert on this field, but there was research recently that overturned a long held assumption about the necessary scale of a neural network. Essentially, by making it an order of magnitude larger than previously assumed necessary, the potential for this kind of misidentification was much reduced. I'll see if I can find it...
Found it. It tackles exactly this issue. Article title is "Computer Scientists Prove Why Bigger Neural Networks Do Better"
It seems to me that this system is likely trying to do in one step what we do in several steps - things like recognizing lines, then recognizing objects, then identifying objects ... there's a reason we use so many steps. I expect the first eye-brain interfaces probably tried to do it in one, too, and the more complicated system won, so that strongly suggests the more complicated system is actually important.
It does work with humans. I've seen an example where a low res cat picture was changed very slightly to look like a dog. If humans weren't quite so squishy I'm sure you could tailor a few pixel attack for people
What if one of the categories was actually "noise"?
Could you add more noise in order to trick the classifier into being unable to detect the noise?
Could that work in order to detect these attacks?
You definitely can! You can also look at the model's confidence about it's results, since getting being overconfident on a result can be a sign of inputs designed to trick the model (or of issues within the model itself)
This video is poor in the sense that the object is against a white background. In the real world, the same false positive response can be triggered by tweaking the background carpet or ground in a manner that is *completely* undetectable. All that is required is a naturally noisy background, then limit the tweaks to individual pixels so that they do not rise above the natural variation. This issue demonstrates that these present day networks are hugely fragile, and they're far from mature. With a skilled attacker, they can be roundly abused and hacked. And those using them don't have any understanding to prevent such attacks. The whole industry should wake up.
@@JxH it is even less noticeable if you change pixels by only a small amount
@@JxH What are the consequences of such an attack? Like what is an example? What would be the benefit for an attacker? I understand they can be tricked but why would you? Genuinely curious.
@@JxH Agreed. Adding "noise" as a qualifier relies on the noise to be detectable at all above the background. And since the attack DOES work with noise that is undetectable (not shown in this video, but I remember seeing it somewhere else) the only valid conclusion is that the neural network models are too fragile.
One reason of including noise as a category is that 99.99...% of the image space is noise. (Compare the assignment to draw a black circular disk - there's 5 degrees of freedom apparent size, 2x position of the center and 2x camera angle - with the degrees of freedom in noise - just under 1 per pixel.)
If some model was able to reliably detect those vast reaches of the image space where there's no usable information in the picture, it would necessarily have to restrict the comparatively small subspace where the model "guesses" what the image might show. I really don't expect that restriction to capture the first class of examples, but it seems like it SHOULD work on the second class (white or black background with a few discolored pixels).
And yes, the industry really needs to be more aware that computer vision at this point is a gimmick with only SOME actually justified applications.
Lots of comments here suggest adding random noise to the trained samples. From what I recall of the literature on adversarial attacks on image classifiers, this methodology doesn't actually work. Adversarial attacks are successful because the noise is strategically contrived to make cumulative errors in the network activations, whereas random noise (by nature of being random with zero mean) would work against itself and on average end up with the same activations in the net. Training on adversarial examples that are constructed during the training provess is a more promising approach.
Would be interresting to try the same trick with two different neuronal networks. I would guess, even small changes in network architecture leed to drastic changes in recognition patterns. Therefore completely different changes should be needed to trick each of them.
It seems that the learning algorithm has just identified a certain number of pixels that allow the classification of the images.
Ideally one pixel could select between 50% of the categories and if you find 10 independent pixels acting like that you could select between 2^10 categories.
Of course it's probably impossible to have pixels acting ideally and there's some overlapping and the sorting is more blurry. So you actually need 100 pixels you get 99%.
This theory cannot be true for CNNs as they do not purely use local information. Convolutional operations sort of „destroy“ local information. So there is no direct analogy like „it uses that pixel“.
I wonder what would happen if you looked for patterns that would be able to fool multiple different Machine Learning systems, since they're each going to have a slightly different take on it. Perhaps "Cross-platform trickery" would result in patterns that more closely resemble the objects in question.
Is this because the neural network sums the picture up to like a couple of pixels which it compares to be efficient, which doesn't reflect a real picture, although it contains a sum for it. I recall Mike Pound explained something along those lines.
I guess in theory while training, the system could add pixel noise like this to better train and be tolerant of this kind of stuff. You could even give them the "misleading" images and they can add them to their dataset and train with them, and hopefully fix this problem.
There are many ways to trick an AI so you'd have to search for all those different types. This would make the normal performance worse but better in "misleading" images.
Does @Computerphile have a merch store? I want to get the image from 6:21 printed on a coffee mug now.
Wow. This can be used for data augmentation.
7:47 you can see the curved shape at the bottom of the picture, like it was the bottom of a coffee mug pretty close, and from slightly up
I think for the envelope one, there was a small thing that looked like the triangle flap of an envelope bottom middle ish.
Suggestion: a GAN-like architecture where the recognition algorithm is trained to recognize the quality of an image and adapt to noise and artifacts
The remote control, centered in the image, does look like a coffee cup. The remote makes up the handle and the white background is the cup!
You, sir, have a distinctive talent.
seems to me that the sensitivity to noise indicates overfitting to the image set it was given and the classifications it was told to make.
Would it be fair to create two categories of 'golf ball' and 'not-golf ball' which is the sum of all categories that are not in the golf ball confidence range? The big problem I've seen is that these algorithms wan't to make a decision, even if that decision doesn't make sense. Obviously the search space would be much bigger for a category that is "everything but a golf ball", but wouldn't it increase the accuracy?
Can you do a follow up, where you A) test these modified images on a different classifier, and see if it fools it as well, or at least changes its confidence? B) try to fool a better network, to see if is is noticeably harder?
In my own experience, A) most likely works, and definitely changes output probabilities, and B) that might be harder to do but in much the same way, the difference will not be night and day.
There are some ways however to defend a network against some attacks (but there's no golden bullet yet)
What happens if you train a neural network on (a lot of) these noisy images? Can you teach it to be resilient to this type of issues?
Sooo if you now do this to a bunch of images then retrain them it will start to look past that noise right? I guess eventually if you kept doing the changes then retraining it would eventually start looking like human changes
The AI in my head recognized this guy as half David Mitchell, half Daniel Radcliffe.
He tricked me
if Ur imagebase contains 100 images labelled and they're 1080x1080, then that's 1,166,400 pixels, 3 channels RBG. There's gonna be plenty of smallish subsets of pixels within that just happen to coincide...humans'd probably consider em cuz they dont look anything at all like a . Yet the object classifier doesnt care how an image looks to people. It's just finding similarities, coincidences at pixel level.
That method works pretty good cuz it coincides with what humans think objects should look like.
But this video demonstrates how it's not very hard to randomly generate sets of pixel trash that happen to coincide very tightly to an object category....
How many different sets of just 2 pixels out of 1,166,400 will coincide with each other a lot more than they coincide with any other category?
Maybe your imagebase can distinguish 1000 categories, but how many images does it contain labelled as ? I'd guess not so many. A few 100 at most.
And the fewer, the less reliable the mappings....
There was an article in Science a while ago that points out that neural nets don't recognize shapes, they recognize textures. So that would explain why you see these results, although Dr. Turner sounds as though he has read that article. (That there were problems was known as early as 1972 or so: Minsky and Pappert figured out that "perceptrons" (the neural net model of those days) couldn't differentiate closed curves (deformed circles) from open ones (deformed letter "c" shapes). The Science result predicts that these nets can't tell one elephant from two elephants... (There was also the result that one image recognition net fails to recognize cows when the cow is on a beach, since there are no green pixels in such images.) IMHO, the whole neural net game is bogus. YMMV, as they say.
Yeah CNNS are basically all textures but I've heard that Vision Transformers are more robust to these attacks due to focusing slightly more on shapes.
11:10 Why are the two pictures not actually identical here? (I mean, beyond the noise, obviously.)
Can't you use gradient descent to update the image in the most optimal way instead of trying random pixels?
It seems to me that the algorithm HAS to classify the image as something. Maybe it’s not 99% sure it’s a golf ball, rather it’s 99% sure it’s not anything else and has no “abstract” or “noise” category.
I have an image of a "kofta" meal in my google photos account, and the ai insisted it is the same person as Buzz Lightyear toy we have in the house (which appears in many images as well).
Some of this is about the volume of training data.
When a young child is out in a pushchair the parent may say "cat" and the child will watch the cat stretch, wash a paw and walk away - how many images of that cat is the child being trained on?
Adults are experts, they have had 18 years of training in using their visual systems. Young children give some insight into how hard it is to classify, and very often have to unlearn things, they get a word "seagull" that is used too broadly and have to learn that "bird" is the generic and "seagull" is only a type of "bird".
Has anybody tried adversarial attacks on human vision by only flashing the image for such a short time that the image only ends up on the retina once and isn't there long enough to be scanned in detail by the eye in detail? The "Celebrity face illusion" kind of goes in that direction, but isn't quite a proper analogue to an adversarial attack.
Is this not basically what GANs (generative adversarial networks) do though? Why is it GANs produce something recognisable to humans but this method does not then?
GANs and adversarial attacks are different although they share the term “adversarial”. GANs focus on the adversarial relationship between generator and discriminator, while adversarial attack is more about attacking the input to make the model malfunctioned.
@@林家毅-i1n I didn't even know this was an adversarial attack! Glad I made the connection between the two though.
@@林家毅-i1n Essentially the only difference is in the generator, the 'generator' in a sense in this model is directly designed to plot random values of noise (random pixel intensities) to trick the discriminator. Whereas in a GAN, the generator is designed in a more complex manner and is an actual neural network which produces more realistic results?
@@nark4837 Yeah! You get it right! Nice description on the aspect of “generator” on both cases, so brilliant!
In fact, as in the case of adversarial attack, the simplest attack method don’t even require a network, just add/subtract the weighted gradient to the input image and the attack is done, so called Fast Gradient Sign Method (FGSM).