{"id":24616,"library":"sparkmagic","title":"SparkMagic: Spark execution via Livy","description":"SparkMagic is a Jupyter/IPython magic command extension for interactively working with Spark clusters via Livy (REST API). It supports multiple sessions, language interpreters (PySpark, Spark, Scala, SQL), and automatic visualization of DataFrames. Current version is 0.23.0 (released 2025-02-14) with active but low-frequency releases; the project is in maintenance mode.","status":"active","version":"0.23.0","language":"python","source_language":"en","source_url":"https://github.com/jupyter-incubator/sparkmagic","tags":["spark","jupyter","livy","pyspark","magic"],"install":[{"cmd":"pip install sparkmagic","lang":"bash","label":"Install from PyPI"}],"dependencies":[{"reason":"Core dependency for magic command support; version >=7 and <9 required.","package":"ipython","optional":false},{"reason":"Spark sessions are created via Livy REST API.","package":"livy","optional":false},{"reason":"Used for DataFrame conversion and display; pinned <2.0.0 in some versions.","package":"pandas","optional":true},{"reason":"Used for asynchronous HTTP communication with Livy.","package":"tornado","optional":false}],"imports":[{"note":"Direct import is rarely needed; load via %load_ext sparkmagic","symbol":"sparkmagic","correct":"import sparkmagic"}],"quickstart":{"code":"# In a Jupyter notebook cell:\n%load_ext sparkmagic\n%manage_spark\n# Then start a session:\n%%spark -s mysession\n# Create a simple DataFrame:\ndf = spark.range(10)\nprint(df.count())","lang":"python","description":"Load the extension, manage Spark sessions, and execute code against a remote Spark cluster via Livy."},"warnings":[{"fix":"Update any custom code to handle async execution; test with simple cells first.","message":"In version 0.20.0, the extension changed to support async kernel execution in ipykernel>=6. Custom magic commands or code that assumes synchronous execution may break.","severity":"breaking","affected_versions":">=0.20.0"},{"fix":"Upgrade Python to 3.7+.","message":"Python 3.6 support was dropped in v0.20.0. Users on Python 3.6 must stay on v0.19.2 or lower.","severity":"deprecated","affected_versions":">=0.20.0"},{"fix":"Ensure Livy is installed and running, and configure the endpoint via %manage_spark or config file.","message":"You must have a running Livy server reachable from the notebook server. The extension does not start Livy.","severity":"gotcha","affected_versions":"all"},{"fix":"Install the JupyterLab extension: `jupyter labextension install jupyterlab-sparkmagic`","message":"The %manage_spark widget may not display correctly in JupyterLab without the appropriate JupyterLab extensions (jupyterlab-sparkmagic).","severity":"gotcha","affected_versions":"all"}],"env_vars":null,"last_verified":"2026-05-01T00:00:00.000Z","next_check":"2026-07-30T00:00:00.000Z","problems":[{"fix":"Run `pip install sparkmagic` in the terminal and restart the kernel.","cause":"sparkmagic is not installed in the current Python environment.","error":"ModuleNotFoundError: No module named 'sparkmagic'"},{"fix":"Use `%load_ext sparkmagic` in a notebook cell instead of a Python import.","cause":"Incorrect import path; sparkmagic provides a magic extension, not a direct importable class.","error":"ImportError: cannot import name 'SparkMagic' from 'sparkmagic'"},{"fix":"Check your Livy URL and ensure the Livy server is running. Use `%manage_spark` to set the endpoint.","cause":"Livy endpoint not configured or not responding. The extension returns None for session info.","error":"TypeError: 'NoneType' object is not subscriptable"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"install_tag":null,"quickstart_score":null,"quickstart_tag":null}