FYI I had a similar problem to yours in terms of having jobs I wanted to run on a schedule. I also had an extra layer where I wanted to let users to define jobs with their own special parameters etc. Maybe what I did is helpful for you:
- Form submission in front-end admin panel for users, for "new scheduled job"
- Form allows defining a job name, job type (while I let users define jobs, I limit it to a subset of python functions that I trust but are still general enough), job parameters (just a json blob for kwargs for the python func), frequency, and timeout.
- For the whitelisting of functions it's easiest to have a directory in your src which has the different python functions that are usable, and when you do a "get" for the form you populate the dropdown with the scripts from that directory / some metadata.
- The backend (fastAPI) saves the details in a DB and creates a CRON file for the job, to add to the crontab. I basically have a template bash statement with the timeout built in and logging hooked up to it. And when python functions are called I have a helper which grabs the JSON kwargs from the DB or a filecache (if remote DB and I don't want to hit it every minute or something) to avoid cmd line injection.
- Also have an "edit scheduled job" which opens the same form view but with items pre-populated, and the submit going to a patch endpoint instead of a post.
It's stupid simple but lets users define a range of jobs. Things like having daily backups, where to send the backups, pulling things from / pushing things to an API frequently, etc.
Really appreciate you sharing this setup. The cron file generation for persistence is clever, and the whitelisting approach for user-defined jobs makes a lot of sense. Might borrow some of these ideas if I add a UI for job creation. Thanks!
Interesting idea. Right now jobs are code-only which keeps it simple, but a UI for defining basic jobs could work. I'll think about it.
FYI I had a similar problem to yours in terms of having jobs I wanted to run on a schedule. I also had an extra layer where I wanted to let users to define jobs with their own special parameters etc. Maybe what I did is helpful for you:
- Form submission in front-end admin panel for users, for "new scheduled job"
- Form allows defining a job name, job type (while I let users define jobs, I limit it to a subset of python functions that I trust but are still general enough), job parameters (just a json blob for kwargs for the python func), frequency, and timeout.
- For the whitelisting of functions it's easiest to have a directory in your src which has the different python functions that are usable, and when you do a "get" for the form you populate the dropdown with the scripts from that directory / some metadata.
- The backend (fastAPI) saves the details in a DB and creates a CRON file for the job, to add to the crontab. I basically have a template bash statement with the timeout built in and logging hooked up to it. And when python functions are called I have a helper which grabs the JSON kwargs from the DB or a filecache (if remote DB and I don't want to hit it every minute or something) to avoid cmd line injection.
- Also have an "edit scheduled job" which opens the same form view but with items pre-populated, and the submit going to a patch endpoint instead of a post.
It's stupid simple but lets users define a range of jobs. Things like having daily backups, where to send the backups, pulling things from / pushing things to an API frequently, etc.
Really appreciate you sharing this setup. The cron file generation for persistence is clever, and the whitelisting approach for user-defined jobs makes a lot of sense. Might borrow some of these ideas if I add a UI for job creation. Thanks!