I use Zipf's Law to Generate Synthetic Text. Text 'written' by machine (a Python Script) convincing enuf so human readers will consider it 'weird' English...but English. It relies on rank/frequency incidences of words in text. HiFreq words clumped together in short groups ('to', 'the') ('with','her') are observed in HUGE text (100s of MILLIONs of words) to have nearly interchangeable sequiturs.....while InFreqs do not.. However InFreqs support Textual Topicality: when a Text displays Infrequent words more frequently than usual, then the Topic involves those Infreqs. while HiFreqs contribute to Textual Coherency.. Modern Statistical Probabilistic Machine Translation makes even better use of Zipf's Law. An example Synthetic Text: 0 +3x3 ['united*Kingdom', ',', 'we', 'have'] 1 +3x3 ['included', 'it', 'in', 'the'] 2 +3x3 ['places', 'in', 'which', 'it'] 3 +3x3 ['states', 'that', 'on', 'the'] 4 +3x3 ['record', 'now', ',', 'because'] 5 0 +2x3 ['i', 'hope', 'that', 'before', 'the'] 6 +3x3 ['start', 'of', 'this', 'vote'] 7 +2x2 ['of', 'confidence', 'from', 'parliament'] 8 0 +2x3 ['is', 'extremely', 'important', ',', 'and'] 9 0 +2x3 ['intergovernmental', 'solidarity', '.', 'what', 'the'] 10 +3x3 ['court', 'has', 'to', 'say'] 11 0 +2x3 [',', 'mr*Newton*Dunn', ',', 'that', 'the'] 12 +3x3 ['difficulty', 'will', 'now', 'be'] 13 +3x3 ['listened', 'to', '.', 'the'] 14 +3x3 ['conviction', 'that', ',', 'by'] 15 +3x3 ['30*June', 'of', 'this', 'year'] 16 +3x3 [';', 'so', 'we', 'can']
Very interesting. I don't know if Zipf's Law is involved, but I have sometimes looked to see what sentences emerge from predictive text on a phone, when you keep following the next suggested word. Nowadays, quite scarily, it can start reproducing whole passages I have written in previous emails etc.
This is a really interesting topic and well discussed here.... for some reason in front of a construction site and when the garbage truck is picking up everyones trash cans along the street. Just a very odd choice of time and place to film. But great thoughts on Zipf! Lol
I was chased away from the place I originally intended to record by some uptight guy who thought I was planning to park on his land and walk into town. I had to move down the road. But also I just liked that grungy area and wanted to record a video there.
I love that you showed the whole pages so i could gather a tiny little bit of context for myself. I love the "there is THE case" on 3:34. because it points to how little was known about bisexuality at the time. With the context in mind i do have a little bit of ground to stand on, on which to disagree. Perhaps the apartment across the street is a brothel and that goes unmentioned? Probably not, he just knows that people have sex, and if you see an apartment with 100 residences in it, and a happy couple (from my perspective anyways) spends at least a hundredth of their time having sex, than statistically there is SOMEBODY having sex in there. I've read lots of Freud, so maybe i'm trying too hard to find justification. At around 12:50 you say "But a person does not have to be right in everything, in order to have fruitful thoughts." Brilliant. Are you quoting or is that an original, fruitful thought? LOL You would be much more watchable with a higher quality microphone, and more consistent quality. The quality changes through the video, which is very odd. Also, sunglasses? They make you harder to read and less interesting to watch. Last note, i don't know if all of the video was build up to the enthralling final two minutes, but that's where you got really interesting. Maybe all the explanation could be condensed, or maybe the revelation could be expanded upon. Very enjoyable 17minutes even if i have some nitpicks. Will make for great conversation when socializing later. Thanks bra.
Thank you. I hadn't thought of the possibility that it is a brothel. Even so, the idea that this would calm the bride's nerves is somewhat bizarre. I imagine this illustration might be based on something that Zipf had read or experienced, and it all made sense to him at the time. That thing about not having to be right in everything in order to have fruitful thoughts comes from me, and I am glad you found it applied to my own video..! I'm sorry you didn't like the sunglasses. I bought them specially for the video, and since I wear glasses normally and can't see very much without them, it took quite an effort to do the video with the shades. Yes, the sound dropped out quite a lot, I'm not sure why, and there was a lot of background noise. I do need to up my game in the sound department. Regarding the length, the first half isn't particularly relevant to Zipf's theory or necessary to the video, but I thought some people might find it amusing.
To reply to your specific question, by the way, I believe that is my own original point that a person does not have to be right in everything in order to be interesting. It seems a sad aspect of modern science, especially social science, that there is a tendency to dismiss wacky people rather than looking for the good in what they say. And I accept your points about the quality of the video. It is all just a game and an experiment. One day, I plan to make videos more seriously.
Please correct me if I'm wrong but would it be accurate to say that Zipf's law is some sort of representation/measurement of variability? . For example: word distribution in youtube comments with only 2 or 3 words won't look very "Zipfian" at all. But the more words a comment has, the more "Zipfian" it becomes. In other words, the more items in your comparison, the more zipfian it will be, and vice versa. Or no?
Interesting point. Still, there is the question of why things come out as Zipfian rather than say a normal/Gaussian distribution. If you take 2 or 3 people their heights won't look particularly Gaussian, but the more you take, the more Gaussian they will be. So why do these things emerge as Zipfian? What you seem to be saying is that this is a likely outcome of variability, and I think some derivations of Zipf's law show how it is a result of a general class of possible behaviours for systems.
If the line is more horizontal, then that means the groups are all about evenly matched. The hypothesis is that, in such a case, you have a balance of power. No group considers itself strong enough to take on any other. So conflict is most likely when the line is at 45 degrees, and it is less likely if the line is either steeper or shallower than that.
sorry, I was asking about an actual example, an actual country. for example, it seems like all those sub-saharan nations break up violently due to a very large minority ethnic group.
I don't think the U.S. Is determined to maintain its dominance to the point of war. However, it would be a huge problem if that dominance was had by Russia or China as they currently do "business". Like all countries the U.S. Tries to be the best and strongest it can be for its people. But I believe if a peaceful country was to over take the U.S. In power, we wouldn't have a problem with it. It wouldn't lead to conflict. But there is something to be said for ability of those types of "aggressive" authoritarian nations to rise above others to challenge for dominance. While more peaceful nations are content with how things are. So, is conflict due to keeping dominance or someone wanting dominance that rejects the status quo?
Jeez, that guy was crazy. Almost as crazy as all the background noise. Still a good video though. Edit: just got to the tool analogy. Zipf must not have been a tool guy cause I live by the mantra of hoarding them, not shrinking.
![[UA] NAVI проти G2 Esports | Blast Premier Fall Final](http://i.ytimg.com/vi/QWlIm4FGESQ/mqdefault.jpg)
I use Zipf's Law to Generate Synthetic Text. Text 'written' by machine (a Python Script) convincing enuf so human readers will consider it 'weird' English...but English.
It relies on rank/frequency incidences of words in text. HiFreq words clumped together in short groups ('to', 'the') ('with','her') are observed in HUGE text (100s of MILLIONs of words) to have nearly interchangeable sequiturs.....while InFreqs do not..
However InFreqs support Textual Topicality: when a Text displays Infrequent words more frequently than usual, then the Topic involves those Infreqs.
while HiFreqs contribute to Textual Coherency..
Modern Statistical Probabilistic Machine Translation makes even better use of Zipf's Law.
An example Synthetic Text:
0 +3x3 ['united*Kingdom', ',', 'we', 'have']
1 +3x3 ['included', 'it', 'in', 'the']
2 +3x3 ['places', 'in', 'which', 'it']
3 +3x3 ['states', 'that', 'on', 'the']
4 +3x3 ['record', 'now', ',', 'because']
5 0 +2x3 ['i', 'hope', 'that', 'before', 'the']
6 +3x3 ['start', 'of', 'this', 'vote']
7 +2x2 ['of', 'confidence', 'from', 'parliament']
8 0 +2x3 ['is', 'extremely', 'important', ',', 'and']
9 0 +2x3 ['intergovernmental', 'solidarity', '.', 'what', 'the']
10 +3x3 ['court', 'has', 'to', 'say']
11 0 +2x3 [',', 'mr*Newton*Dunn', ',', 'that', 'the']
12 +3x3 ['difficulty', 'will', 'now', 'be']
13 +3x3 ['listened', 'to', '.', 'the']
14 +3x3 ['conviction', 'that', ',', 'by']
15 +3x3 ['30*June', 'of', 'this', 'year']
16 +3x3 [';', 'so', 'we', 'can']
Very interesting. I don't know if Zipf's Law is involved, but I have sometimes looked to see what sentences emerge from predictive text on a phone, when you keep following the next suggested word. Nowadays, quite scarily, it can start reproducing whole passages I have written in previous emails etc.
This is a really interesting topic and well discussed here.... for some reason in front of a construction site and when the garbage truck is picking up everyones trash cans along the street. Just a very odd choice of time and place to film. But great thoughts on Zipf! Lol
Exactly where Zipf belongs
Amazing video. Well done.
Shoot next to an airport next
Lol ok
why would u go and record a video next to a loud building site?
I was chased away from the place I originally intended to record by some uptight guy who thought I was planning to park on his land and walk into town. I had to move down the road. But also I just liked that grungy area and wanted to record a video there.
What about the racial composition of USA? Did you look at it with the idea of Zipf's Law in mind?
I didn't. That would be an interesting exercise.
I love that you showed the whole pages so i could gather a tiny little bit of context for myself. I love the "there is THE case" on 3:34. because it points to how little was known about bisexuality at the time.
With the context in mind i do have a little bit of ground to stand on, on which to disagree. Perhaps the apartment across the street is a brothel and that goes unmentioned? Probably not, he just knows that people have sex, and if you see an apartment with 100 residences in it, and a happy couple (from my perspective anyways) spends at least a hundredth of their time having sex, than statistically there is SOMEBODY having sex in there. I've read lots of Freud, so maybe i'm trying too hard to find justification.
At around 12:50 you say "But a person does not have to be right in everything, in order to have fruitful thoughts." Brilliant. Are you quoting or is that an original, fruitful thought? LOL
You would be much more watchable with a higher quality microphone, and more consistent quality. The quality changes through the video, which is very odd. Also, sunglasses? They make you harder to read and less interesting to watch.
Last note, i don't know if all of the video was build up to the enthralling final two minutes, but that's where you got really interesting. Maybe all the explanation could be condensed, or maybe the revelation could be expanded upon.
Very enjoyable 17minutes even if i have some nitpicks. Will make for great conversation when socializing later.
Thanks bra.
Thank you. I hadn't thought of the possibility that it is a brothel. Even so, the idea that this would calm the bride's nerves is somewhat bizarre. I imagine this illustration might be based on something that Zipf had read or experienced, and it all made sense to him at the time.
That thing about not having to be right in everything in order to have fruitful thoughts comes from me, and I am glad you found it applied to my own video..!
I'm sorry you didn't like the sunglasses. I bought them specially for the video, and since I wear glasses normally and can't see very much without them, it took quite an effort to do the video with the shades. Yes, the sound dropped out quite a lot, I'm not sure why, and there was a lot of background noise. I do need to up my game in the sound department.
Regarding the length, the first half isn't particularly relevant to Zipf's theory or necessary to the video, but I thought some people might find it amusing.
To reply to your specific question, by the way, I believe that is my own original point that a person does not have to be right in everything in order to be interesting. It seems a sad aspect of modern science, especially social science, that there is a tendency to dismiss wacky people rather than looking for the good in what they say.
And I accept your points about the quality of the video. It is all just a game and an experiment. One day, I plan to make videos more seriously.
Interesting.
Please correct me if I'm wrong but would it be accurate to say that Zipf's law is some sort of representation/measurement of variability? . For example: word distribution in youtube comments with only 2 or 3 words won't look very "Zipfian" at all. But the more words a comment has, the more "Zipfian" it becomes. In other words, the more items in your comparison, the more zipfian it will be, and vice versa. Or no?
Interesting point. Still, there is the question of why things come out as Zipfian rather than say a normal/Gaussian distribution. If you take 2 or 3 people their heights won't look particularly Gaussian, but the more you take, the more Gaussian they will be. So why do these things emerge as Zipfian? What you seem to be saying is that this is a likely outcome of variability, and I think some derivations of Zipf's law show how it is a result of a general class of possible behaviours for systems.
what about an ethnic composition where the line goes above the 45 degree line?
If the line is more horizontal, then that means the groups are all about evenly matched. The hypothesis is that, in such a case, you have a balance of power. No group considers itself strong enough to take on any other. So conflict is most likely when the line is at 45 degrees, and it is less likely if the line is either steeper or shallower than that.
sorry, I was asking about an actual example, an actual country.
for example, it seems like all those sub-saharan nations break up violently due to a very large minority ethnic group.
I don't think the U.S. Is determined to maintain its dominance to the point of war. However, it would be a huge problem if that dominance was had by Russia or China as they currently do "business". Like all countries the U.S. Tries to be the best and strongest it can be for its people. But I believe if a peaceful country was to over take the U.S. In power, we wouldn't have a problem with it. It wouldn't lead to conflict. But there is something to be said for ability of those types of "aggressive" authoritarian nations to rise above others to challenge for dominance. While more peaceful nations are content with how things are. So, is conflict due to keeping dominance or someone wanting dominance that rejects the status quo?
Loving the 70s porn glasses.
Jeez, that guy was crazy. Almost as crazy as all the background noise.
Still a good video though.
Edit: just got to the tool analogy. Zipf must not have been a tool guy cause I live by the mantra of hoarding them, not shrinking.