Deal with chunk (re)reading errors
The CH/Sargans scan (CH.PLONS..HH*) encountered the following error (with --debug flag):
Processing chunk 19633 template 0
~/.local/lib/python3.6/site-packages/obspy/io/mseed/headers.py:828: InternalMSEEDWarning: readMSEEDBuffer(): Record starting at offset 8388608 is not valid SEED. The rest of the file will not be read.
warnings.warn(_w, InternalMSEEDWarning)
Traceback (most recent call last):
...
File "~/code_TM_app/TM/core.py", line 228, in do_xcorr_onechannel
xCorrTrace /= stdWindowed * stdT * lenT
ValueError at line operands could not be broadcast together with shapes (1834408,) (2593201,) (1834408,)
It could be solved by manually deleting the preloaded chunk 19633 (stdwin, proc, raw + symlinks), which indicates a (re)reading error of the _proc.mseed file that has been preloaded and saved originally by TM.io.preload_chunk().
Two options to fix it:
- in TM.io.preload_chunk(): immediately after saving the chunk to _proc.mseed, assert that it is identical to the actual stream object (in the memory), otherwise save it again.
- Downside: more overhead due to always rechecking the file.
- in mpi.TemplateMatchingTasks.xcorr_chunk(): after loading the chunk (proc + stdWind), assert that they are compatible (len(stdWin) == len(stream) - len(template) + 1); otherwise, reload chunk from archive
- Downside: need to somehow mark the chunk to be read again by a reader slave. Can that be done easily (@lucasca)?
Could potentially fix the following error as well:
The CH/Chatel-St-Denis scan (CH.AIGLE..HH*) encountered:
Cannot read chunk 9481, ()
Error: couldn't process chunk 9481 with template 0. This should never happens, please check what failed
(triggered by if chunk is None or stdWind is None or template is None:
in mpi.TemplateMatchingTasks.xcorr_chunk() --> both the chunk and its stdWind must have been 0; could not be reproduced again --> a rare re-reading occurrence(?))