Code here: bit.ly/2MyRDLR Menu: 00:00 - Introduction 02:29 - Spreadsheet conversion 04:19 - Importing using Python 13:41 - What are the advantages of using Jinja2 templates? 16:35 - Skills that Network Engineers should learn? 19:00 - The working template 32:09 - Exporting to a text file 35:22 - Using a script 37:46 - How would I turn this into something that configures a device? 51:18 - Should I use CSV/YAML/JSON? Links: Video 1 in series: ua-cam.com/video/PB9D4RqBaQw/v-deo.html Video 2 in series: ua-cam.com/video/23hPg88pZBo/v-deo.html Start Now Page on Devnet: bit.ly/2WBjb5v NetDevOps Live Episode on Python Libraries including CSV: bit.ly/2VjbGP4 Coding Fundamentals Learning Labs: bit.ly/2vU1BO8 Network programmability Basics Video Series covering CSV: bit.ly/2JzlbY4 Jinja2 template video: ua-cam.com/video/ARFU73tah8A/v-deo.html
I work in telco service provider land and we manage literally a few million devices internally or for customers as a managed service. This is about how we do it but in large scale. The template process we made is in 2 phases to first assemble a complete device template file out of sub-template modules that models a specific piece of equipment or config package we want to deploy from an internal catalog. Next it populates that file with the specific fill in the blank data from whatever data source we want. We have an API service for template generation. I have used netmiko and CSV for small jobs all the time like this and is a great thing for an engineer to learn. Template engines like jinja or a YAML model used in a python app is best for assembling boiler-plate modules but let something else do the fill in the blank stuff. Pythons built-in string replace method is implemented in C in the background and lightning fast already. We have modules for everything we use; DHCP, ACLs, NAT, routing protocols and maps, QOS maps, SMNP, AAA, VPN, EEM or TCL scripts etc. We have used a few template systems over the years and let me caution about jinja and template engine use in general from my experience. You want as little template engine logic mixed in with your device config stuff as possible so it's more readable by other network engineers who are not programmers and lets them contribute modules to the library easier. Your templates also become coupled to that engine so change is hard later. It's a programmers best practice to separate business data from business logic as much as possible. In their example we do not use logic for trunk or access mode anymore in the interface module templates because it can vary by manufacturer, OS series, or firmware so we need a few versions anyway. We pull in the right access version, trunk version, or sub interface version of a template module as needed. Instead of a CSV our equipment inventory database has all the device specific IP data and options our generator program consumes via a fancy DB query. The database field names that get returned in the JSON reply match the template module variable names so string find/replace to fill in the blanks is trivial and tiny intuitive code to read/maintain. We can add new variables without changing code except the DB query which is in a isolated module by itself the program imports. We just add a new DB field name and referencing them in the template modules as variables. As long as they match it just works. Passwords, license keys, and security certificates should be variables called from a separate true-crypt like database when it renders the template. Do not leave those things lying around in plain text template modules. Whatever method you want to use put all your template modules, YAMLs, and program code in an internal GIT server so you have revision control and it facilitates secure collaboration. Our template system pulls all the module pieces out of git on demand and uses databases to fill in the blanks when we want to render a template. We maintain the central repository and every template is rendered in the latest version by default. We can use those to audit/update the network with worker scripts but that is another novel to explain lol.
Hi David, I came here to see if we can achieve Zero touch provisioning for Cisco Devices, like literally out of the box. Can you please recommend this piece if you've already made it. Thanks!
I would suggest to give some more instruction, how to create the .csv template file, you had in first column "Switch", and inside seem only one device is there sbx-n9kv-ao, but what about if we had a 2nd, 3rd switch, for example sw02, sw03...how to make the loop to generate separate config file for each of the switches? Also some more recommandation how to create the structure of inventory file (no matter in csv or yaml) could be helpful, where to include stuff like ospf, bgp, and how to connect with proper interfaces, neighbors etc. Anyway, thanks so much for you guys brining this session
Hi David, I had scheduled Cisco 200-901 exam on 2nd of July 1845 IST however at the scheduled time I was not able to launch the exam and now it's been 5 days I'm chasing Pearson but not getting any response can you help me to reach out someone else who can help me..
came back today just for fun. This video series with hank got me started back then. Turned out i was already perfect skilled for upcoming job requierements! Same as Cybersecurity today. Thank you so much David! How do you archive always being ahead of the time?
I loved Jinja2 until I tried to use a variable I had declared in a for loop. Why the hell would they limit the scope that much, limits Jinjas use cases so much by doing so. I was using Jinja to write 15000 line config files but can no longer as I'm unable to carry variables across multiple blocks.
I'm guessing the answer is "use jinja2 templates" but what if we had to do deeper into the prompt, such as*interface* or *config-bgp* *config-ospf*, is it still possible to configure with just netmiko? I guess we would just have to change what we expect back from the router as the expect value (so not #) or just put in send_config_timing?
Very cool. You can also use spreadsheets to convert them into YAML files, to be further processed by Ansible or similar tools. Pretty much the same approach, in that case you can have the same stuff to configure not only routers but also firewalls, ACI data center, and so on. Good job anyway, very useful explanation. A 'device' column could be added in such a way that the same format could be used for all devices in the network.
Thanks for the video Bombal and Hank! Idea for next video: use cases for network automation? Or what other network engineers are doing to start automating?
Hank & David this is an awesome video thanks for putting this together! I'm going through the Jinja2 templating doc but the method to adapt for IOS configs hadn't clicked just yet. This helps alot! Thanks again keep up the great work
Hi David the videos are a great content and it feels like I am being greedy but if you could upload the scripts used in the videos to github and share it would be great. Specially the ones used for restconf/netconf
thanks david always delivering wat is best for us ....but this new 200-301 ccna is scary to me ...cant we just become network developers with out the certificates us u know u programming is free ......
The new CCNA will only have a small portion focusing on Network Automation. The bulk of the cert is still focused on traditional networking. I don't think you have to worry.
Code here: bit.ly/2MyRDLR
Menu:
00:00 - Introduction
02:29 - Spreadsheet conversion
04:19 - Importing using Python
13:41 - What are the advantages of using Jinja2 templates?
16:35 - Skills that Network Engineers should learn?
19:00 - The working template
32:09 - Exporting to a text file
35:22 - Using a script
37:46 - How would I turn this into something that configures a device?
51:18 - Should I use CSV/YAML/JSON?
Links:
Video 1 in series: ua-cam.com/video/PB9D4RqBaQw/v-deo.html
Video 2 in series: ua-cam.com/video/23hPg88pZBo/v-deo.html
Start Now Page on Devnet: bit.ly/2WBjb5v
NetDevOps Live Episode on Python Libraries including CSV: bit.ly/2VjbGP4
Coding Fundamentals Learning Labs: bit.ly/2vU1BO8
Network programmability Basics Video Series covering CSV: bit.ly/2JzlbY4
Jinja2 template video: ua-cam.com/video/ARFU73tah8A/v-deo.html
I work in telco service provider land and we manage literally a few million devices internally or for customers as a managed service. This is about how we do it but in large scale. The template process we made is in 2 phases to first assemble a complete device template file out of sub-template modules that models a specific piece of equipment or config package we want to deploy from an internal catalog. Next it populates that file with the specific fill in the blank data from whatever data source we want. We have an API service for template generation. I have used netmiko and CSV for small jobs all the time like this and is a great thing for an engineer to learn. Template engines like jinja or a YAML model used in a python app is best for assembling boiler-plate modules but let something else do the fill in the blank stuff. Pythons built-in string replace method is implemented in C in the background and lightning fast already. We have modules for everything we use; DHCP, ACLs, NAT, routing protocols and maps, QOS maps, SMNP, AAA, VPN, EEM or TCL scripts etc.
We have used a few template systems over the years and let me caution about jinja and template engine use in general from my experience. You want as little template engine logic mixed in with your device config stuff as possible so it's more readable by other network engineers who are not programmers and lets them contribute modules to the library easier. Your templates also become coupled to that engine so change is hard later. It's a programmers best practice to separate business data from business logic as much as possible. In their example we do not use logic for trunk or access mode anymore in the interface module templates because it can vary by manufacturer, OS series, or firmware so we need a few versions anyway. We pull in the right access version, trunk version, or sub interface version of a template module as needed. Instead of a CSV our equipment inventory database has all the device specific IP data and options our generator program consumes via a fancy DB query. The database field names that get returned in the JSON reply match the template module variable names so string find/replace to fill in the blanks is trivial and tiny intuitive code to read/maintain. We can add new variables without changing code except the DB query which is in a isolated module by itself the program imports. We just add a new DB field name and referencing them in the template modules as variables. As long as they match it just works.
Passwords, license keys, and security certificates should be variables called from a separate true-crypt like database when it renders the template. Do not leave those things lying around in plain text template modules.
Whatever method you want to use put all your template modules, YAMLs, and program code in an internal GIT server so you have revision control and it facilitates secure collaboration. Our template system pulls all the module pieces out of git on demand and uses databases to fill in the blanks when we want to render a template. We maintain the central repository and every template is rendered in the latest version by default. We can use those to audit/update the network with worker scripts but that is another novel to explain lol.
Glad everyone is enjoying the video and series. You can find the code samples at github.com/hpreston/python_networking/tree/master/csv_config_gen
Hi David, I came here to see if we can achieve Zero touch provisioning for Cisco Devices, like literally out of the box. Can you please recommend this piece if you've already made it. Thanks!
I would suggest to give some more instruction, how to create the .csv template file, you had in first column "Switch", and inside seem only one device is there sbx-n9kv-ao, but what about if we had a 2nd, 3rd switch, for example sw02, sw03...how to make the loop to generate separate config file for each of the switches? Also some more recommandation how to create the structure of inventory file (no matter in csv or yaml) could be helpful, where to include stuff like ospf, bgp, and how to connect with proper interfaces, neighbors etc. Anyway, thanks so much for you guys brining this session
Bombal the Rock of the Network . we're wainting your ccnp enterprise or Service provider after febuary sir .please
Thank you. I will be working on CCNP :)
This is priceless information, thank you for sharing!
Thanks David .. really appreciate your efforts. Can you please make some video on pyang Vs YDK and deep dive on pyang..
Hi David,
I had scheduled Cisco 200-901 exam on 2nd of July 1845 IST however at the scheduled time I was not able to launch the exam and now it's been 5 days I'm chasing Pearson but not getting any response can you help me to reach out someone else who can help me..
Hank & David, David & Hank, an awesome network - automation - couple !!!
Thank you guys !!!
came back today just for fun. This video series with hank got me started back then. Turned out i was already perfect skilled for upcoming job requierements! Same as Cybersecurity today. Thank you so much David! How do you archive always being ahead of the time?
I loved Jinja2 until I tried to use a variable I had declared in a for loop. Why the hell would they limit the scope that much, limits Jinjas use cases so much by doing so. I was using Jinja to write 15000 line config files but can no longer as I'm unable to carry variables across multiple blocks.
I'm guessing the answer is "use jinja2 templates" but what if we had to do deeper into the prompt, such as*interface* or *config-bgp* *config-ospf*, is it still possible to configure with just netmiko? I guess we would just have to change what we expect back from the router as the expect value (so not #) or just put in send_config_timing?
I have a quesiton? What OS and software does he use for simulating in the video?
Hopefully you reply me.
Font is very small, i can bearly see
Awesome! Thanks for sharing! This will definitely help me with Network Automation. Thanks guys!
Very cool. You can also use spreadsheets to convert them into YAML files, to be further processed by Ansible or similar tools. Pretty much the same approach, in that case you can have the same stuff to configure not only routers but also firewalls, ACI data center, and so on. Good job anyway, very useful explanation. A 'device' column could be added in such a way that the same format could be used for all devices in the network.
Thanks for the video David. What's the prerequisite for this tutorial? Seems like good knowledge of Python is important.
This is the best, no nonsense tutorial on this stuff that I’ve seen. Well done gentlemen!
Thank you very much! This is a very useful and detailed tutorial for network professionals looking for automation!
I have a file with extension text, how conversion to spreadsheet? or how can read with python?
Thanks for the video Bombal and Hank! Idea for next video: use cases for network automation? Or what other network engineers are doing to start automating?
Hank & David this is an awesome video thanks for putting this together! I'm going through the Jinja2 templating doc but the method to adapt for IOS configs hadn't clicked just yet. This helps alot! Thanks again keep up the great work
Can u do a video on how to automate ios patch upgrade with Python
great video ☻
Thank you very much for such session
This was one of the best tutorials I've ever seen for network admins with little programming experience.
Glad to have found this channel!
10/10
Glad you liked it!
Thanks for all your videos david, you're one of the main reasons I am studying my CCNP R&S :D,
This is really cool. Thank you for sharing!!
Enjoyed this ,thanks hanks and david
Hi David the videos are a great content and it feels like I am being greedy but if you could upload the scripts used in the videos to github and share it would be great. Specially the ones used for restconf/netconf
Here you go: bit.ly/2MyRDLR
great video David.. thank you
Thumbs up David and Preston for this support 😃
Thank you
thank you for sharing
Amazing work! Great video Hank and David
Glad you enjoyed it!
This is so great
Awesome video
Whoa! 🤯
Such a great video!
Perfect!
Great review of Netmiko, Jinja2 and CSV. Is there a link to the code used in this example?
Here you go: bit.ly/2MyRDLR
what great info and well explained. Thanks for the video David and Hank, powerful stuff.
Thank you! Hank is amazing :)
thanks david always delivering wat is best for us ....but this new 200-301 ccna is scary to me ...cant we just become network developers with out the certificates us u know u programming is free ......
The new CCNA will only have a small portion focusing on Network Automation. The bulk of the cert is still focused on traditional networking. I don't think you have to worry.
@@davidbombal okey david thanks