Any chance to get a support for jupyter lab 3.4?

[AlekseiSaff]

The package is downgrading jupyterlab to 3.1.19

any chance to making it work with the newer version?

[jovanni-hernandez]

This should be pretty easy to implement if you'd like to contribute a PR. Assuming Jupyter didn't change their plugin API's, it should just be a matter of bumping the versions in the package.json up

[surfaceowl]

I've been working on a fork at https://github.com/surfaceowl/jupyterlab_scheduler get this working for jupyterlab 3.6.1 and python 3.11 which is already in place for an existing project (built with pip in a virtualenv).

Since the package on pypi won't install with these version of jupyterlab and python - by tweaking setup.py and temporarily including git+https://github.com/surfaceowl/jupyterlab_scheduler.git in requirements.txt -- jupyter lab and "run now" jobs work but, scheduled jobs are failing , with this error that implies jupyter_scheduler is searching specifically for a conda environment - which does not exist since my project was pip installed.
Two relevant sections of code I see are:

jupyter_scheduler/environments.py", line 35, in list_environments
envs = subprocess.check_output(["conda", "env", "list", "--json"])

...and

..scheduler.py", line 677, in get_staging_paths


which triggers an unhandled python error. The entire error message is:

[E 2023-02-17 13:38:02.535 ServerApp] Uncaught exception GET /scheduler/runtime_environments?1676669882358 (127.0.0.1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/scheduler/runtime_environments?1676669882358', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "/home/chris/dev/venv/lib/python3.11/site-packages/tornado/web.py", line 1713, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/home/chris/dev/venv/lib/python3.11/site-packages/jupyter_scheduler/handlers.py", line 361, in get
        environments = await ensure_async(self.environments_manager.list_environments())
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/chris/dev/venv/lib/python3.11/site-packages/jupyter_scheduler/environments.py", line 35, in list_environments
        envs = subprocess.check_output(["conda", "env", "list", "--json"])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/subprocess.py", line 466, in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/subprocess.py", line 548, in run
        with Popen(*popenargs, **kwargs) as process:
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/subprocess.py", line 1024, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/usr/local/lib/python3.11/subprocess.py", line 1901, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'conda'
[W 2023-02-17 13:38:02.536 SchedulerApp] wrote error: 'Unhandled error'
[E 2023-02-17 13:38:02.537 SchedulerApp] {
      "Host": "localhost:8888",
      "Accept": "*/*",
      "Referer": "http://localhost:8888/lab",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
    }
[E 2023-02-17 13:38:02.537 SchedulerApp] 500 GET /scheduler/runtime_environments?1676669882358 (127.0.0.1) 174.82ms referer=http://localhost:8888/lab
[E 2023-02-17 13:38:06.185 SchedulerApp] 'NoneType' object is not iterable
    Traceback (most recent call last):
      File "/home/chris/dev/venv/lib/python3.11/site-packages/jupyter_scheduler/handlers.py", line 305, in post
        self.scheduler.create_job_from_definition(job_definition_id, model=model)
      File "/home/chris/dev/venv/lib/python3.11/site-packages/jupyter_scheduler/scheduler.py", line 663, in create_job_from_definition
        input_uri = self.get_staging_paths(definition)["input"]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/chris/dev/venv/lib/python3.11/site-packages/jupyter_scheduler/scheduler.py", line 677, in get_staging_paths
        for output_format in model.output_formats:
    TypeError: 'NoneType' object is not iterable

Any pointers on how this might be resolved? I'm not finding the problematic line 361 in handlers.py or the environments.py in the original repo or my clone... where does that come from? Hopefully by changing the

[surfaceowl]

More people are reporting this problem over in the jupyterlab package - so this is a breaking issue. Seems like current code in this package assumes a conda environment, so need to write to work in both conda and pip installed environmetns..

Please see jupyter-server/jupyter-scheduler#337

[jovanni-hernandez]

Thanks for doing some digging on the issue! Interesting to see that Jupyter released an official schedule plug-in. Does their release make this project obsolete?

[surfaceowl]

@jovanni-hernandez -- you are most welcome, glad to help.

I'm not sure if the recent release of jupyter-server makes this plugin obsolete or not... but hopefullly @dlqqq can comment - has have been very active recently on the scheduler in the juypter-server project

[dlqqq]

Hey folks! So, Jupyter Scheduler is an official extension that is under active development and maintenance, and works out-of-the-box with the latest JupyterLab versions (currently 3.6.1). We would be happy to welcome new users to Jupyter Scheduler. 🤗

We do also currently require a Conda environment, but that is very likely to change in the near future (thanks to @surfaceowl for reminding me).

[JasonWeill]

We introduced Jupyter Scheduler late last year, and we are still improving it. Please feel free to file issues on our project if you find bugs or if you have enhancement suggestions. Thanks for your interest!

[jovanni-hernandez]

@dlqqq @JasonWeill it's great to see an official scheduler be released, it's a feature I've used in tons of my projects, and I was surprised by how many other folks found it useful. I'm more than happy to retire my project and start pointing folks who stumble upon it over to the official plugin.

[dlqqq]

@jovanni-hernandez If you're comfortable with that, could you create a separate issue so we can discuss next steps in this process?

[mouatez25]

We introduced Jupyter Scheduler late last year, and we are still improving it. Please feel free to file issues on our project if you find bugs or if you have enhancement suggestions. Thanks for your interest!

is it possible to use cron as scheduling engine with jupyter Scheduler?

[JasonWeill]

@mouatez25 Jupyter Scheduler supports custom schedulers, passed as the name of a Python class, as documented here: https://jupyter-scheduler.readthedocs.io/en/latest/operators/index.html#scheduler-class

Jupyter Scheduler does not specifically use cron as a scheduler. If you run crond separately from Jupyter Scheduler, and you specify your jobs as papermill commands, for example, then you can run your notebooks on a schedule even if Jupyter Scheduler is not running. The main downside of this approach is that you would need to maintain your crontab independently, in text or using a different tool; Jupyter Scheduler does not write to your system's crontab.

If you would like to request an enhancement to Jupyter Scheduler, you can open an issue here: https://github.com/jupyter-server/jupyter-scheduler/issues . Thanks for your interest!

[mouatez25]

@JasonWeill thanks for your quick reply !!!

we are used to use https://pypi.org/project/jupyterlab-scheduler/ and it writes to our user's crontab which makes things more easier for our users.

thanks

[mouatez25]

@jovanni-hernandez is there any chance we can make it work with jupyterlab 4 ?

https://pypi.org/project/jupyterlab-scheduler/