Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] job runs much slower than Zeppelin (remote server) or spark submit #743

Open
GaryLiuGTA opened this issue Dec 30, 2021 · 1 comment

Comments

@GaryLiuGTA
Copy link

GaryLiuGTA commented Dec 30, 2021

Describe the bug
I have Enterprise Spark Cluster and remote Zeppelin server, and I am using sparkmagic in my local laptop to submit jobs to remote spark clusters.

The same code runs much faster when using Zeppelin or spark submit, and it is slower when submitted from local laptop by sparkmagic. For example, one piece of code ran about 7-10 minutes in Zeppelin, but at the same time, this code completed in 30-40 minutes when submitted locally using sparkmagic. So wondering if this is related to network conditions between spark clusters and machine where the code was submitted? As there are 10GB bandwidth between spark clusters and Zeppelin server, but I am using DSL (work from home). Are there lots of traffics between kernel and clusters required? Or it could be other reasons?

To Reproduce
Steps to reproduce the behavior.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions:

SparkMagic 0.19.1
Livy (if you know it): company's on-premise livy service, internal server.
Spark 2.2.3

Additional context
Add any other context about the problem here.

@GaryLiuGTA GaryLiuGTA changed the title [BUG] [BUG] job runs much slower than Zeppelin (remote server) or spark submit Dec 31, 2021
@GaryLiuGTA
Copy link
Author

Add a comparison of Zeppelin and sparkmagic (using vs code jupter extension), sparkmagic used 3 times of time
Code running with Zeppelin:
image
Code running with sparkmagic
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant