You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I have Enterprise Spark Cluster and remote Zeppelin server, and I am using sparkmagic in my local laptop to submit jobs to remote spark clusters.
The same code runs much faster when using Zeppelin or spark submit, and it is slower when submitted from local laptop by sparkmagic. For example, one piece of code ran about 7-10 minutes in Zeppelin, but at the same time, this code completed in 30-40 minutes when submitted locally using sparkmagic. So wondering if this is related to network conditions between spark clusters and machine where the code was submitted? As there are 10GB bandwidth between spark clusters and Zeppelin server, but I am using DSL (work from home). Are there lots of traffics between kernel and clusters required? Or it could be other reasons?
To Reproduce
Steps to reproduce the behavior.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Add a comparison of Zeppelin and sparkmagic (using vs code jupter extension), sparkmagic used 3 times of time
Code running with Zeppelin:
Code running with sparkmagic
Describe the bug
I have Enterprise Spark Cluster and remote Zeppelin server, and I am using sparkmagic in my local laptop to submit jobs to remote spark clusters.
The same code runs much faster when using Zeppelin or spark submit, and it is slower when submitted from local laptop by sparkmagic. For example, one piece of code ran about 7-10 minutes in Zeppelin, but at the same time, this code completed in 30-40 minutes when submitted locally using sparkmagic. So wondering if this is related to network conditions between spark clusters and machine where the code was submitted? As there are 10GB bandwidth between spark clusters and Zeppelin server, but I am using DSL (work from home). Are there lots of traffics between kernel and clusters required? Or it could be other reasons?
To Reproduce
Steps to reproduce the behavior.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions:
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: