Updating prefect requires rethink flow execution
User Story
updating prefect to the latest version (2.16.3) requires a change in how the code deploys the flows.
https://docs.prefect.io/latest/guides/upgrade-guide-agents-to-workers/
details can be found here. The new version doesn't use an agent anymore, instead you can choose between
- 'Serving' the flow, where the flow can run on a schedule as usual, and the flow is found in the local environment, but the difference is that it runs in the same window as you use for the cli
- 'Deploying' the flow, where the flow code must be available in a git repo or a docker image. It can then be run with a worker pool which can be a subprocess or other kind of pool.
There are advantages and disadvantages to both. It would be nice to have an option to execute a local flow to test local changes, however it is not so clean as you will have to pull up another window every time, or have this execute inside a manually made subprocess.
Deploying the flow with git or docker makes operational sense, and in this case maybe it also makes sense to separate the cli code and the flow code to different repos.
Suggestion:
Have two repos, rt-ramsis-flow and rt-ramsis-operational (renamed for hermes or whatever)
rt-ramsis-flow has the cli option to run the flow with 'serve'
rt-ramsis-operational has the cli options to deploy with a prefect server and work-pool and gets code from gitlab.
Possible plan:
just update the current repo to use serve for now, and when there is a need, create the new repo with work-pool method.