The user should have the neurons. The computer chips just need to network. Unfortunately as Facebook demonstrates, modern web is very good at reversing this.
You're the only person I've seen so far that actually takes advantage of tiling windows + terminal applications, everyone else seems to be wasting time with "I can customize it to do anything" but never get anything done, I don't want waste time customizing for no benefit. Sadly I don't see how I could impove my current workflow with this format, but it's really cool. PS: It would be real magic if we could get a stdout of a pdf.
I had no idea that vim could do this! All I ever knew about vim was how to quit it (after accidentally opening it and then spending a couple of minutes googling how to exit). Thanks :D
I use vim all the time but admittedly I've never put in the effort to become this much of a wizard with it. Next time I have a task like this (could be years) I'll pull this video back up again though!
1:31 pdfinfo (part of the poppler-utils package) will show info about PDF files such as titles. But these don’t always match the text you see on the title page of the document.
Maybe I missed the part where he said it, but you can of course use r instead of read, and the space may be omitted if you use external commands. This might save time if you often do short external commands like these: :r!ls *pdf r!date etc
I use vim since maybe 10 years and I used this editor like a begginer all this time (I used only 2 modes, no buffers, and basic commands like w, q, y, p, b, w). Now I try to use it more like a poweruser because I want to be more productive when I write my thesis and I am surprised how this editor is powerful if you use it correctly. The "visual block" mode is magic. I never seen something like that from an other editor/program.
Thanks for sharing these vim tips! I'll share how I normally generate such files with a BASH loop:- for i in *.pdf; do echo -e "$(pdfinfo "$i"|sed -n '1s/^Title: *//p')" >>links.html;done This will write the HTML tags for each PDF found in the directory, and use the Title extracted with pdfinfo as the linked content. It might save a bit of time. Doing this stuff can be tedious.
Cool, I just watched and read my first lessons on Vim earlier today, and I was going back and forth debating on if I should learn it or not...and this video showed me a good reason why I should bother going down that path.
With a bit of playing around I found the following will pull the title out of about 95% of pdf documents: pdftotext filename.pdf - | head -n 1 Would be fairly easy to get vim to paste that in the appropriate position. OK so you'd still need to go through and check for the odd few where it doesn't find the title properly but it would save a lot of work.
I wrote a opengl loader with all the entry point declaration. They said to use a third party loader, that it would take forever. I was done in a hour. Thanks Vim.
7:55 Emacs has built-in online help, available from CTRL-H. For example, CTRL-H followed by A is “apropos”, where you can find relevant functions by keyword, CTRL-H C lets you get information about key bindings, etc.
Nice example of mapping a key to bring up "mupdf" as an external command; but in split screen. BTW: You could wrap your file names in tags with just one ":vi" command line, rather than 2 steps: :g/pdf/s/\(.*\)// ( which translates to ) global-find all lines with "pdf" then substitute \( line-match \) with pre_text \1=insert-match end_text Note: REgex of ".*" matches whole line \( \) remembers it, and \1 shows where to re-insert string. "vi/vim" editor's :colon: command line has full "ex" editor commands for lots of bulk edit magic. Also: If you are more comfortable with "sed" you could use "!G" to pipe the whole file through cmd sed -e 's/^/
kate's block visual mode actually lets you go further than end of line so you can add prefix and suffix.opening the pdf's and the place holder trick was neat though
I came to this video from a pinned post in r/vim that listed some resources to understand vim better. All I can say is I wish there were more videos like this showing how things are done in real world usage of the editor. (I have been using vim for almost 20 years.)
I just started trying to get used to Vim a couple of weeks ago since everyone says that nano is nooby, hadn't really read any advanced tutorial either, just learned to navigate mostly. This stuff is amazing!
Interesting. As an emacs user I do not know much about vim but I would have chosen a similar approach. I would have either renamed all the files first with a proper name , similar to what op did in the end with the article name and than wrote a macro in wdired mode to change everything automatically or I would have done it basically exactly like op. Open wdired mode with split screen, write macro that opens file in other screen, adds all necessary html code and put the courser at the correct position, so that I only have copy the name by hand. I like that in vim you just redirect bash-command output into the editors buffer. Pretty neat.
How he is opening pdf and doing all these kinds of stuff in the terminal.
5 років тому+2
Awk is perfect for this task. A one-liner like this should be all that's necessary: $ ls *pdf | awk '{ print "ADD TITLE" }' > list.html If your pdf files have proper metadata, you can use the tool pdfinfo to find out the title and modify the awk program above to automatically include it.
First time viewer of your channel. I was a linux (and other unices') sysadmin back in the day, and an infosec analyst back in the day. It's been a while since I've had linux on the desktop, though, the fruity company sucked me in with their wiles about a decade ago. Introductions aside, what window manager/shell is that? It looks amazing! All the efficiency of a CLI, and the ability to do very quick document previews. I gotta check that out.
Really loved the vid Great job. I was able to do everything you did, in Emacs evil-mode, pretty much the same way you did. I had forgotten about :norm command. Again, GREAT JOB.
Woawww!!! You are really using the Computer. And not let the Computer using You. That is insane how powerful VIM is with a few little inocent commands ;)
It might be possible to avoid even manually typing out article titles: - run the PDF through tesseract-ocr, - pipe the text out to 1969_syntactic.pdf.txt etc, - go into each file and - put SomeSpecialCharacterSequence at the beginning of the line with the article title - back in Vim with html file, run shell that takes pdf filename + .txt append and greps it for SomeSpecialCharacterSequence and inserts the result - substitute SomeSpecialCharacterSequence with nothing - clean up remaining .txt files
Hi, Below's one liner will runs xpdf or pdf viewer which will open pdf file. Select the title with mouse. Quit the pdf viewer and it will generate the html tag, and move to the next file.. The script needs the xsel package to b installed , and pdf files need to be in slectable text mode, not raster/image . $ ls *.pdf| while read f; do xpdf $f&> /dev/null; echo "`xsel | tr ' ' ' '`"; done > refs.html Enjoy. Update: If we encounter with image/raster pdf file, open xclipboard and type in the edit box. The content will be output by the xsel command in the script.
There are a million ways to do it. If I had to do this personally, I'd probably go with a python script. import os outputString = "" filePath = "/path/to/folder" for elem in os.listdir(filePath): if elem[-3:] == "pdf": outputString += "++ " f = open("outputFile.html","w+") f.write(outputString) f.close There you go, same thing he did (minus setting up the workflow for copying in the titles) achieved by 9 lines of python. You could probably write that script a million ways, just like you could come up with a million ways to do what OP did. The point of the video, I think though, was to show the potential that Vim has for complex solutions to every day problems. Honestly though, I don't think I'm ever going to use Vim. Sublime text gives me plenty of shortcuts for coding, and if I ever need to do any type of data processing, like OP did, I would probably just write a quick python script.
@@SimonWoodburyForget He's not telling you that this is the best way to do it. He's showcasing the capabilities of vim. It's pretty cool that he was able to do all of this through Vim. Thats the point.
On vs code you can type the same thing on multiple lines at a time. Just hold shift + alt and click and drag down and type. Im sure there is also a way to open the file and read the title but i havent had to do it yet.
Any timesaves would surely be lost from having to type in the paper names (or using vim magic tricks required to paste something from the system clipboard) would they not?
Do you think you could freeze vim while you were with the PDF opened, copied the title and, when closing the document, pasting the title inside de link tags, all with that command? Maybe it would make that easier.
I'm newbie: I wonder what if you handle to open a 2 view with pdftotxt and move and yank manually the Title. I don't know, just to avoid typing. You still will need to see the original pdf view to know where is the title, but then you can search and yank.
That was cool. If you said this in the video I'm sorry for missing it. Just wanted to say that some of the credit should go to i3 for presenting the PDF like that instead of opening a other window on that that you would have to alt+tab a couple of times.
If you know, what vim can do, this is a good way for sure. Last time I had to sort and extract the text from a few thousand PDFs, I used a mix of AppleScript, Python and golang. They were exported to PDF the same way, so they were all the same layout and everything.
After doing it for some years, I came to the conclusion that trying to script GUI apps is a waste of time. The Linux approach is to provide the command-line functionality, then put the GUI as a separate layer on top of that. If you want to automate functions, you skip the GUI and do it at the command line. Much more efficient and reliable, with much less stuffing around.
Wow, this is even better and faster than recording a macro to repeat the process by pressing @@. Also, I liked to see mupdf behavior on it. I think I'm gonna stop using apvlv and zathura. Is mupdf able to invert colors?
2:17 I defined a custom “make-shell-buffer” command in my Emacs prefs github.com/ldo/emacs-prefs which gives me an interactive window where I can execute arbitrary shell commands and get back their output as editable text in the buffer.
5 років тому+1
Fantastic.
Рік тому
okay i would be interested to know how you could do it using php...shoul be doable as I have done something similar, where upon load it shows a list if websites (directories that have a index.html file) and show the name of the folder as the name of the link...so I am guessing that something like what u are doing could be done in a similar fashion
Could you upload that zipfile? those pfs seem to have interesting subjects! :D Thanks! I know I could just google them one by one but a) I still wouldn't have them all since only some files of that zip file are visible in your video b) it would be still boring to do search each pdf one by one and download it (going through all the waiting lists filesharing websites put you through etc :P )
URGENT! Read this:
lukesmith.xyz/deletion
can we get the zip file?
No
not found
Doesn't exist.
Nvm, found an archive of it.
>didn't train a neural network to identify pdf titles automatically
neural networks are bloat
The user should have the neurons. The computer chips just need to network. Unfortunately as Facebook demonstrates, modern web is very good at reversing this.
You're the only person I've seen so far that actually takes advantage of tiling windows + terminal applications, everyone else seems to be wasting time with "I can customize it to do anything" but never get anything done, I don't want waste time customizing for no benefit. Sadly I don't see how I could impove my current workflow with this format, but it's really cool.
PS: It would be real magic if we could get a stdout of a pdf.
I know I'm late but you can do it with the command pdftotext
I had no idea that vim could do this! All I ever knew about vim was how to quit it (after accidentally opening it and then spending a couple of minutes googling how to exit). Thanks :D
Do a flip with vim.
:g/^/m0
I went to vim to test it because I'm basic and I can't read jargon lol
@@LukeSmithxyz amazing
Note to future self: test what this does
@@SimonWoodburyForget great
I use vim all the time but admittedly I've never put in the effort to become this much of a wizard with it. Next time I have a task like this (could be years) I'll pull this video back up again though!
Did you end up doing it?
@@geticz5636 No! Haven't pulled up this video again until just now. But I did learn how to do search and replace in vim since then.
This is awesome, Luke! The exact type of video I was hoping you would make.
1:31 pdfinfo (part of the poppler-utils package) will show info about PDF files such as titles. But these don’t always match the text you see on the title page of the document.
Maybe I missed the part where he said it, but you can of course use r instead of read, and the space may be omitted if you use external commands. This might save time if you often do short external commands like these:
:r!ls *pdf
r!date
etc
The first part was just ls *pdf | sed 's/.*//' > file. But I had no idea about all the other abilities of vim!
I use vim since maybe 10 years and I used this editor like a begginer all this time (I used only 2 modes, no buffers, and basic commands like w, q, y, p, b, w). Now I try to use it more like a poweruser because I want to be more productive when I write my thesis and I am surprised how this editor is powerful if you use it correctly. The "visual block" mode is magic. I never seen something like that from an other editor/program.
Thanks for sharing these vim tips! I'll share how I normally generate such files with a BASH loop:-
for i in *.pdf; do echo -e "$(pdfinfo "$i"|sed -n '1s/^Title: *//p')" >>links.html;done
This will write the HTML tags for each PDF found in the directory, and use the Title extracted with pdfinfo as the linked content. It might save a bit of time. Doing this stuff can be tedious.
I just break out Python if I can't solve something with pipes in 3 minutes :D
so regex basically? I thought that would be way easier than figuring out what he's talking about.
Arrived for the thumbnail, stayed for dank knowledge 👌🔥
Yes, that's how almost all his videos work.
www.urbandictionary.com/define.php?term=Dank - so not cool.
Cool, I just watched and read my first lessons on Vim earlier today, and I was going back and forth debating on if I should learn it or not...and this video showed me a good reason why I should bother going down that path.
Extracting information from PDF files has never been this fast and dank
This is the type of stuff I subscribed for :D
Arrived to see vim potential. Received it. Will still never in my lifetime use it, even tho the mapping was damn neat.
Awesome vid, every time I watch one of these it just reinforces to me that I only use a fraction of vim.
With a bit of playing around I found the following will pull the title out of about 95% of pdf documents:
pdftotext filename.pdf - | head -n 1
Would be fairly easy to get vim to paste that in the appropriate position. OK so you'd still need to go through and check for the odd few where it doesn't find the title properly but it would save a lot of work.
I wrote a opengl loader with all the entry point declaration. They said to use a third party loader, that it would take forever. I was done in a hour. Thanks Vim.
I tried to use vim once, nearly killed myself. Doctor said if I cut my wrists over stress from trying to use a text editor its probably not worth it.
yes you can't hurt yourself with notepad. Not all have what it takes to use vim
XDD
When you use chad vim instead of virgin gatsby to generate static html
7:55 Emacs has built-in online help, available from CTRL-H. For example, CTRL-H followed by A is “apropos”, where you can find relevant functions by keyword, CTRL-H C lets you get information about key bindings, etc.
Any equivalent to “apropos”?
Nice example of mapping a key to bring up "mupdf" as an external command; but in split screen.
BTW: You could wrap your file names in tags with just one ":vi" command line, rather than 2 steps:
:g/pdf/s/\(.*\)// ( which translates to )
global-find all lines with "pdf" then substitute \( line-match \) with pre_text \1=insert-match end_text
Note: REgex of ".*" matches whole line \( \) remembers it, and \1 shows where to re-insert string.
"vi/vim" editor's :colon: command line has full "ex" editor commands for lots of bulk edit magic.
Also: If you are more comfortable with "sed" you could use "!G" to pipe the whole file through cmd
sed -e 's/^/
kate's block visual mode actually lets you go further than end of line so you can add prefix and suffix.opening the pdf's and the place holder trick was neat though
The thumbnail is hilarious.
His thumbnails, combined with what his videos are about, generally scream "/g/ lurker".
Imagine using Sublime, lol
8:50 It worked on the first try.
What kind of sorcery is this?
Nice video. The read ! was new to me, as was the yank in quotes (which I think I'll use all the time now!).
Wow so many uploads..
Wish you all the best, Luke.
bever language and thoughts seems like a fun paper
Thanks for zooming the text! Made table viewing feasible.
I came to this video from a pinned post in r/vim that listed some resources to understand vim better. All I can say is I wish there were more videos like this showing how things are done in real world usage of the editor. (I have been using vim for almost 20 years.)
I just started trying to get used to Vim a couple of weeks ago since everyone says that nano is nooby, hadn't really read any advanced tutorial either, just learned to navigate mostly. This stuff is amazing!
Great to hear! How's your progress been?
summary for me:
1:55 * :read !ls
3:25 * :norm
5:34 jumping to ++, some mapping (how?)
6:22 * scoop install mupdf
7:02 * default leader is backslash
7:17 * yi" : yank in quotes
7:45 * " paste in from default buffer (how?)
8:03 * disown
* CR: carriage return (enter)
Interesting. As an emacs user I do not know much about vim but I would have chosen a similar approach. I would have either renamed all the files first with a proper name , similar to what op did in the end with the article name and than wrote a macro in wdired mode to change everything automatically or I would have done it basically exactly like op. Open wdired mode with split screen, write macro that opens file in other screen, adds all necessary html code and put the courser at the correct position, so that I only have copy the name by hand. I like that in vim you just redirect bash-command output into the editors buffer. Pretty neat.
Luke, you are the vim/shell guru!
How he is opening pdf and doing all these kinds of stuff in the terminal.
Awk is perfect for this task. A one-liner like this should be all that's necessary:
$ ls *pdf | awk '{ print "ADD TITLE" }' > list.html
If your pdf files have proper metadata, you can use the tool pdfinfo to find out the title and modify the awk program above to automatically include it.
it's not working, even after deleting one of the quotes that shouldnt be there, i think you have to use printf
I feel like you can pull the heading text from the PDF document
You could significantly increase the last step's speed by using a mouse and copy/pasting the titles of the articles instead of typing them.
could someone explain what is mapped for space + space or space + tab
Amazing. Vim-fu black belt. Thanks master
First time viewer of your channel. I was a linux (and other unices') sysadmin back in the day, and an infosec analyst back in the day. It's been a while since I've had linux on the desktop, though, the fruity company sucked me in with their wiles about a decade ago.
Introductions aside, what window manager/shell is that? It looks amazing! All the efficiency of a CLI, and the ability to do very quick document previews. I gotta check that out.
Ah, found your FAQ. FYI, the FAQ link from your homepage is broken. Missing the "\.html$"
This is i3wm
Really loved the vid Great job. I was able to do everything you did, in Emacs evil-mode, pretty much the same way you did. I had forgotten about :norm command. Again, GREAT JOB.
Your "blah blah blah" is so hypnotic.
Rythme?
Woawww!!!
You are really using the Computer.
And not let the Computer using You.
That is insane how powerful VIM is with a few little inocent commands ;)
Whats the "file explorer" you use at the beginning
Ranger I think: github.com/ranger/ranger
4 years late but i think it's "lf"
Nice video in my recommended section. I like the thumbnail.
It might be possible to avoid even manually typing out article titles:
- run the PDF through tesseract-ocr,
- pipe the text out to 1969_syntactic.pdf.txt etc,
- go into each file and
- put SomeSpecialCharacterSequence at the beginning of the line with the article title
- back in Vim with html file, run shell that takes pdf filename + .txt append and greps it for SomeSpecialCharacterSequence and inserts the result
- substitute SomeSpecialCharacterSequence with nothing
- clean up remaining .txt files
Nice workflow, man. Keep it up)
every one of these videos feels like a personal attack xd
nice rice you have there. Could you say what's you was using on it?
Hi,
Below's one liner will runs xpdf or pdf viewer which will open pdf file. Select the title with mouse. Quit the pdf viewer and it will generate the html tag, and move to the next file.. The script needs the xsel package to b installed , and pdf files need to be in slectable text mode, not raster/image .
$ ls *.pdf| while read f; do xpdf $f&> /dev/null; echo "`xsel | tr '
' ' '`"; done > refs.html
Enjoy.
Update:
If we encounter with image/raster pdf file, open xclipboard and type in the edit box. The content will be output by the xsel command in the script.
couldnt you just do ls *pdf > filename and then edit that ?
But what's cool in doing it the intuitive way?
There are a million ways to do it. If I had to do this personally, I'd probably go with a python script.
import os
outputString = ""
filePath = "/path/to/folder"
for elem in os.listdir(filePath):
if elem[-3:] == "pdf":
outputString += "++
"
f = open("outputFile.html","w+")
f.write(outputString)
f.close
There you go, same thing he did (minus setting up the workflow for copying in the titles) achieved by 9 lines of python.
You could probably write that script a million ways, just like you could come up with a million ways to do what OP did. The point of the video, I think though, was to show the potential that Vim has for complex solutions to every day problems.
Honestly though, I don't think I'm ever going to use Vim. Sublime text gives me plenty of shortcuts for coding, and if I ever need to do any type of data processing, like OP did, I would probably just write a quick python script.
Can't you just fucking serve a folder with static files using NGINX? Sometimes people waste TOO DAMN MUCH time overengineering edgy bullshit.
@@SimonWoodburyForget He's not telling you that this is the best way to do it. He's showcasing the capabilities of vim. It's pretty cool that he was able to do all of this through Vim. Thats the point.
@@gustavoschrf do you even Arch bro? I'd bang this out in hand assembled machine code on my SGI workstation
Mind-blowing, I am empressed, superb!
or "Ctrl+V" (mark first characters in line) "I" "a href" inserts anything before first character. after that (ESC)
On vs code you can type the same thing on multiple lines at a time. Just hold shift + alt and click and drag down and type.
Im sure there is also a way to open the file and read the title but i havent had to do it yet.
Any timesaves would surely be lost from having to type in the paper names (or using vim magic tricks required to paste something from the system clipboard) would they not?
what is the terminal directory browser that you use?
ranger
*Where did you get that wallpaper... It's legendary...*
Connor RK800 legen ... wait for it
Dary
I personally would've just used search and replace for the part where you turn the filenames into links.
Do you think you could freeze vim while you were with the PDF opened, copied the title and, when closing the document, pasting the title inside de link tags, all with that command? Maybe it would make that easier.
It's 3am, I have no idea what you did or how you did it, but it was awesome! 😀
this is so cool!
thx for the tips
Nice vid man, this is what I want to learn more of - how to do mo clever sh** with Vim
What distro is that?
Also, what commandline filemanager is that? Looks nice :)
ranger.github.io/
poppler-utils has pdf-info that outputs title along with a bunch of other stuff.
But can it barrel rol?
What's the file explorer tool he is using in the terminal?
Cebrail Erdogan ranger I think
@@zackjostar6872 Thanks! that was it
i can't even find the save command in the menu bar of word :(
Amazing tutorial . Just showing the power of vim. Love!
This is crazy. I didn't know vim is this powerful
I'm newbie: I wonder what if you handle to open a 2 view with pdftotxt and move and yank manually the Title. I don't know, just to avoid typing. You still will need to see the original pdf view to know where is the title, but then you can search and yank.
That was cool. If you said this in the video I'm sorry for missing it. Just wanted to say that some of the credit should go to i3 for presenting the PDF like that instead of opening a other window on that that you would have to alt+tab a couple of times.
I guess to use the thumbnails analogy, emacs would be the god Ra bestowing nourishing rays upon the earth
What plugin did you use to preview pdf @1:16
He used the mupdf software
How does he do that thing where he instantly splits the terminal to vertically or horizontally be half the screen?
I think it's *set splitright* in his vimrc
Some Questions:
What OS he's using?
what cli file manger he's using?
how come his window manager is plain without those close min buttons
You could also use a macro that records you changing one line then run it 180 times - 180@[macro]
So helpful!
If you know, what vim can do, this is a good way for sure.
Last time I had to sort and extract the text from a few thousand PDFs, I used a mix of AppleScript, Python and golang.
They were exported to PDF the same way, so they were all the same layout and everything.
AppleScript ... my condolences.
(I used to be a fan of it back in the 1990s.)
It is great for interfacing with apps. I could not think of any easier way to access menu bar items...
After doing it for some years, I came to the conclusion that trying to script GUI apps is a waste of time. The Linux approach is to provide the command-line functionality, then put the GUI as a separate layer on top of that. If you want to automate functions, you skip the GUI and do it at the command line. Much more efficient and reliable, with much less stuffing around.
Sure, if your program is built like that, fine. But not all programs are built with easy access through the CLI...
Most of the ones on Linux are.
Wow, this is even better and faster than recording a macro to repeat the process by pressing @@. Also, I liked to see mupdf behavior on it. I think I'm gonna stop using apvlv and zathura. Is mupdf able to invert colors?
how did you manage to underline your vim text?
And I can't even exit vim feelsbadman
2:17 I defined a custom “make-shell-buffer” command in my Emacs prefs github.com/ldo/emacs-prefs which gives me an interactive window where I can execute arbitrary shell commands and get back their output as editable text in the buffer.
Fantastic.
okay i would be interested to know how you could do it using php...shoul be doable as I have done something similar, where upon load it shows a list if websites (directories that have a index.html file) and show the name of the folder as the name of the link...so I am guessing that something like what u are doing could be done in a similar fashion
Fucking awesome thumbnail, brother!
Cant you combine the two norm commands into one?
Can anybody link me that wallpaper?
What program do you use to preview with tree. Grep style???
That's... impressive!
How do deal with files that have spaces or %20 in their link names?
Anybody knows which file manager Luke uses? Is this the midnight manager?
Probably Ranger, he did a video on it.
Which is that CLI file manager?
What file browser is that, which looks like the console but can preview pdfs for you?
What's that mapping you have with space tab?
What is that running in the term at 0:29
I've the same question
I believe he's using ranger
ForTheReallys hold up, ranger can give you graphical thumbnails of files?
Java the Nutt yes but i don't think all terminals support it
@@JavaIsnom If you have sxiv, w3m, etc, as handlers, yes.
Superb video
Could you upload that zipfile? those pfs seem to have interesting subjects! :D Thanks!
I know I could just google them one by one but a) I still wouldn't have them all since only some files of that zip file are visible in your video b) it would be still boring to do search each pdf one by one and download it (going through all the waiting lists filesharing websites put you through etc :P )