hanging on commit

User Story

Especially in real time mode, and occasionally in lab mode, there will often be behavior in ramsis where everything hangs on a commit. This used to happen very occasionally and re-running the forecast would work, but now seems to happen a lot. There is no memory or ram being used while this transaction is in progress so I think it is some sort of lock or race condition which should be dealt with by sqlalchemy/postgresql but isn't.

There have been a few different points in the code where the hanging started. An update to the forecast.status has been seen, but the most common one is on the commit of hydws data to the database after a merge has happened.

Acceptance Criteria

should be able to execute forecasts in both modes with no hanging issues.