@@unikrau4963 Yup, exactly, Spark is implemented in Scala, so with Scala you already have access to all Spark's features at the lowest cost in terms of performance (and maintainability - that's my opinion). Since Scala is a JVM language and there are interfaces to Java in the Scala Core, you can also use Java at the lowest cost in terms of performance, but it is much more verbose language than Scala is. Python API uses py4j to communicate with Scala (Java) API, there is some overhead, and there is some indirection, e.g.. if spark worker will get killed by linux kernel - you'll see strange error saying that py4j client cannot connect to java server.. there are other issues... So as we see, Python is an additional feature of Spark that is highly popular, but it requires a lots of effort to keep up with the native APIs and native performance.
i need to give it a shot!
amazing how rapidly this awesome project is evolving. good job!
I want the koalas t-shirt!
He is showing the Languages in the slide at 5:32 as Python, R, SQL. Why is he not counting Scala or Java?
I think Java and Scala maybe are native
@@unikrau4963 Yup, exactly, Spark is implemented in Scala, so with Scala you already have access to all Spark's features at the lowest cost in terms of performance (and maintainability - that's my opinion). Since Scala is a JVM language and there are interfaces to Java in the Scala Core, you can also use Java at the lowest cost in terms of performance, but it is much more verbose language than Scala is.
Python API uses py4j to communicate with Scala (Java) API, there is some overhead, and there is some indirection, e.g.. if spark worker will get killed by linux kernel - you'll see strange error saying that py4j client cannot connect to java server.. there are other issues... So as we see, Python is an additional feature of Spark that is highly popular, but it requires a lots of effort to keep up with the native APIs and native performance.
good presentation.
Data scientists in these presentations always are like aliens that studied human social behavior but not quite got there ....
hahhahahahahhahah
hahahaha I felt the same but didnt know how to express it.. .this nails it :v