- 23
- 37 709
K12 Analytics Engineering
United States
Приєднався 17 сер 2020
K12 Analytics Engineering aims to help those working with data at education organizations. This channel covers everything to help analytics engineers learn how to bring data from their various systems together to produce impactful data dashboards and visualizations for administrators, staff, teachers, and students.
Deploying Data Import in Google Cloud
Data Import 1.3 has seen a migration to .NET Core and added support for PostgreSQL. This video walks through how to deploy Data Import in Google Cloud.
github.com/xmarcosx/data-import
techdocs.ed-fi.org/display/EDFITOOLS/What%27s+New
github.com/xmarcosx/data-import
techdocs.ed-fi.org/display/EDFITOOLS/What%27s+New
Переглядів: 339
Відео
May 16th What's new with Ed-Fi
Переглядів 1982 роки тому
A look at what's new in the Ed-Fi community. Data Import v1.3 techdocs.ed-fi.org/display/EDFITOOLS/What's New Ed-Fi Migration Utility EOL medium.com/@marcos_alcozer/ed-fi-migration-utility-eol-a3f38cd64582 Ed-Fi v5.3 Hotfix techdocs.ed-fi.org/display/ETKB/Advisory: API Case Sensitivity Issue
![Creating API credentials in Ed-Fi's Admin App [5 of 12]](http://i.ytimg.com/vi/-UNAIYpCQAw/mqdefault.jpg)
Creating API credentials in Ed-Fi's Admin App [5 of 12]
Переглядів 2472 роки тому
If you are a newbie to the Ed-Fi Admin App, this video is for you. This video will walkthrough how to generate API credentials that you can use to configure an Ed-Fi sync in your SIS.
Hands-on Ed-Fi Change Queries and Deletes
Переглядів 3022 роки тому
If you are regularly pulling data from the Ed-Fi API, change queries and deletes are a great way to speed up that workflow and minimize the number of API calls you need to make. This video is a hands-on demonstration of how those endpoints work as data is changed in the Ed-Fi ODS. Resources: Jupyter Notebook: github.com/xmarcosx/edfi-notebook Postman Collection: techdocs.ed-fi.org/pages/viewpag...
Generating an Ed-Fi client SDK
Переглядів 1932 роки тому
A look at how to generate a client SDK in your preferred programming language. This video also shows a starter Jupyter Notebook that demonstrates how to use the python client SDK to interact with an Ed-Fi API.
Ed-Fi API to Analytics Middle Tier
Переглядів 5322 роки тому
Demo of an ELT workflow to extract data from a target Ed-Fi API to create tables in Google BigQuery that match the Ed-Fi Analytics Middle Tier spec. If you implement this, please do reach out! GitHub repository: github.com/K12-Analytics-Engineering/dagster
Ed-Fi data out: I was wrong!
Переглядів 4662 роки тому
After working on a project to extract data from the Ed-Fi API for analytics, I found myself thinking differently about data out! Previously I recommended hitting the ODS layer, but this video walks through why I think the API is likely a better choice.
Extracting and visualizing Google Forms data
Переглядів 16 тис.2 роки тому
Google Forms generates a Sheet containing responses. This data source, however, only provides some of the data one would want when analyzing and visualizing the Forms data. This video walks through some of the common pitfalls I've seen when working with Forms data and explains how to navigate around them. Google Forms API: developers.google.com/forms/api Jupyter Notebook: github.com/xmarcosx/no...
Pt II: Querying NWEA MAP data in GCS from BigQuery
Переглядів 2512 роки тому
Part I showed how to extract your NWEA MAP data nightly and load it into Google Cloud Storage. This video will look at that next step in the process, querying that CSV (plus historical CSVs) from BigQuery and pulling the results into a nice and tidy table for your analytics work. Resource: github.com/xmarcosx/nwea-map-etl
Should I Use OneRoster if I Use Ed-Fi?
Переглядів 5993 роки тому
A common question I have heard over the years is if a district should look at OneRoster if they are already planning to use Ed-Fi. This video touches briefly on both data standards using a sample architecture to show the usefulness of both.
Pt I: How to Extract and Load NWEA MAP Data
Переглядів 7663 роки тому
Learn how to extract your NWEA MAP data nightly and load it into Google Cloud Storage. Future videos will show how to transform that data for analytics and visualize in Google Data Studio. Resource: github.com/xmarcosx/nwea-map-etl
Adding Learning Standards to Your Ed Fi-ODS
Переглядів 1253 роки тому
This video explains how to load learning standards into your Ed-Fi ODS and how assessment vendors reference those standards when sending data to your ODS.
Right-sizing Your Ed-Fi API and ODS in Google Cloud
Переглядів 2653 роки тому
In this video I walk through how auto-scaling works in Google Cloud Run, and how to configure both Cloud Run and Cloud SQL to use an optimal amount of resources for your Ed-Fi deployment. Resources cloud.google.com/run/docs/about-instance-autoscaling cloud.google.com/sql/docs/quotas#limits
Pulling Canvas LMS Data into BigQuery
Переглядів 3,2 тис.3 роки тому
Pull your Canvas LMS data into Google BigQuery without having to manage any code. This ETL code replicates data out of the Canvas API into BigQuery which sets us up for future work where we will transform the raw Canvas data into tables that set us up for our analytics work. Resources - - github.com/xmarcosx/canvas-etl - cloud.google.com/dataflow/docs/concepts/beam-programming-model - canvas.in...
Connecting Google Sheets to BigQuery
Переглядів 10 тис.3 роки тому
This video covers how to connect Google Sheets to Google BigQuery so data can be transformed via SQL. I connect a Sheet to BigQuery and use dbt to manage the SQL transformation work to connect that data to data brought in from the Ed-Fi ODS. K12 Analytics Engineering dbt repo: github.com/xmarcosx/dbt
Using dbt to Move Ed-Fi ODS Data to BigQuery
Переглядів 6233 роки тому
Using dbt to Move Ed-Fi ODS Data to BigQuery
The Ed-Fi SIS Integrations I Have Seen
Переглядів 5913 роки тому
The Ed-Fi SIS Integrations I Have Seen
Installing Ed-Fi’s Analytics Middle Tier
Переглядів 2383 роки тому
Installing Ed-Fi’s Analytics Middle Tier
Using Ed-Fi Starter Kits to Access a Sandbox Environment
Переглядів 4773 роки тому
Using Ed-Fi Starter Kits to Access a Sandbox Environment
3 Things I Wish I Knew When I Started My Ed-Fi Journey
Переглядів 1,1 тис.3 роки тому
3 Things I Wish I Knew When I Started My Ed-Fi Journey
Is their Market for providing Data Solutions to K-12 Organizations?
Thanks! This was very informative! Btw, would this also auto update or is there a way that the table would update whenever an update is made in the google sheet? thanks!
Interesting, but very complicated for a regular user. Is there a no code possibility?
pivot tables honestly. and using charts in word doc. this is so complex simply to display pie charts
Thanks heaps for this video. Am I correct in assuming you need a google workspace account to use your method with service accounts and impersonation? And if so, do you have a method I can use if I don't have a google workspace account? Thanks a lot!
Great work. It helped me a lot. Thanks once again
Very informal video, great tip about creating the schema to prevent any upload errors
Thank You!!!
Hi, I am starting to learn about LMS Canvas ,I need to pull Canvas data into SQL , can you help me step by step
thank you!
Thanks!!
Hi Marcos, Amazing job. I would like to deep dive in dagster code in your giphub repository (dagster-etl) but is not activate. Is there any posibility to see that repository, specially to see the connection between this tools and Googlep APIs?
The problem was resolved by workaround. It was bult the python logic from the scratch. Thanks!
Great walkthrough! Quick question: Will I be charged (by AWS) if i try the Ed-Fi starter kit (my AWS free tier has expired so I get charged for other practice projects i perform and then forget to delete the instances/services I used).
Thank you! Yes, you will be charged for the VM that runs the Starter Kit. Alternatively, you can open an Ed-Fi Tracker ticket and request that the Ed-Fi Alliance provide you access to one for free for a limited time.
Nice and clear! Thanks!
Hi Marcos- Have you had any success in Azure Containerization of EdFi?
I have not looked at Azure for Ed-Fi, but feel fairly confident Azure App Service is the way to go for the API and Admin App. I'd use PostgreSQL via their managed db service for the ODS due to it not having a license cost.
Very helpful, Thank you.
I totally loved the video, thanks a bunch
Is there a way to do this from Java?? I mean, I can't figure out what would be the equivalente of the google.auth.default() and credentials.with_subject(...).with_scopes(...) on the Java implementation of the API. I checked the documentation but I just can't get it and I don't want to have to make a python app to be able to consume the data from my form to then send it to my Java app...
Hello sir! I have been watching your videos and they seem extremely helpful for something I have to do at work. Is there a complete course that I can take?
Seriously.. I've been looking for how to do this for 2 days !! THANK YOU !!
So glad you showed this! I’ve been working on trying to write a custom python script to post to Ed-Fi, but this seems much easier! I’m going to be trying this out soon! Thanks Marcos!
I'd like to be able to do this in the cloud rather than my machine so a video on that topic would be great!
Well explained Marcos!
Thank you! What if I want to keep the BQ table automatically updated as the data is changed in the connected Google Sheet?
Great question, the BigQuery table is always up to date because the table is not actually storing any data in BigQuery itself. It is an external table which means every time a SQL query is run on the table, BigQuery pulls from the Sheet.
@@k12analyticsengineering So in my own words, the query I see in BigQuery is actually from data within my google sheet?
@@k12analyticsengineering Thanks! how about the PREVIEW? Why can't we see any preview for the table, is it normal?
it'd be nice to link to text docs
Wonderful explanation of how this counterintuitive idea can be the beginning of a giant leap forward in productivity. Side note - you can get the API to move pretty quickly by changing the batch sizes away from default to 1000's. Talk to your administrator about how to do this (it is easy in latest releases).
Great work. How does this handle checkbox type g.form questions that are comma separated? I know that is a pain point in data studio.
Checkbox type responses come through in a way that is easy to work with. The answers come through as an array. For example, this JSON is for a response where the person chose two values... 'questionId': '663f9b20', 'textAnswers': {'answers': [{'value': 'Cool'}, {'value': 'Collected'}]}}
Deep dive DBT!!!
Do you have an instanceyearspecific video yet?
Thanks Marcos!!!
awesome
Excellent video, I managed to connect with "terms" but I can't visualize the table in the bigquery space. Any advice?
Happy you found the video interesting! I recommend you dig into your Dataflow logs to see if that's telling you what went wrong. If you run across something, feel free to open an issue in the GitHub repo. If you're still running into trouble, send me a message and we can try jumping on a quick Zoom to troubleshoot.
Excellent !!!
Alma is missing from your list! :)
Alma, reach out and let's do a video together covering your Ed-Fi integration! I know there have been 8 LEAs in Colorado trying to use it for the past several months, but it hasn't been available. Would love to detail how your integration works and how a LEA can get access to it!
I would be interested in a DBT deep dive as well as a comparison between DBT and Dataform (pros and cons)
Just subscribed and can't wait for more episodes! I love your avatar!
Thank you so much! I'm excited to get this channel going.