ramsis.sfm.worker not parsing JSON file datetimes properly

fake_json_post.json

Making the call:

curl -d "@/path/fake_json_post.json" -X POST http://localhost:5000/v1/EM1/runs

Most of the JSON was accepted at the validation stage, apart from the datetimes:

marshmallow.exceptions.ValidationError: {'model_parameters': {'modelparam1': ['Unknown field.']}, 'scenario': {'well': {'sections': {0: {'endtime': ['Not a valid datetime.'], 'starttime': ['Not a valid datetime.'], 'hydraulics': {0: {'datetime': {'value': ['Not a valid datetime.']}}}}}}}, 'well': {'sections': {0: {'endtime': ['Not a valid datetime.'], 'starttime': ['Not a valid datetime.'], 'hydraulics': {0: {'datetime': {'value': ['Not a valid datetime.']}}, 1: {'datetime': {'value': ['Not a valid datetime.']}}}}}}}

If you could shed any light on this Daniel, that would be great!

changed the description

changed the description

Thanks, I'm having a look.

@sarsonl,

I tried it with

curl -v -H "Content-Type: application/json" --data @fake_json_post.json -X POST http://localhost:5000/v1/EM1/runs
{"warning": "", "data": {"8de4badf-ed1f-490f-805a-a311e131d271": {}}, "length": 1, "status": "TaskAccepted", "status_code": 202}

Seems to work fine. What version/branch of

• ramsis.sfm.worker

are you using?

After modifying my curl call, there was no change to the errors I was getting. It could be an issue to do with the differences in library versions, so here is my pip freeze:

• amqp==2.2.2
• aniso8601==7.0.0
• astropy==2.0.3
• attrs==17.4.0
• billiard==3.5.0.3
• celery==4.1.0
• certifi==2018.1.18
• chardet==3.0.4
• Click==7.0
• cycler==0.10.0
• decorator==4.2.1
• Django==2.0.4
• docutils==0.14
• Flask==1.1.1
• Flask-RESTful==0.3.7
• Flask-SQLAlchemy==2.4.0
• future==0.16.0
• GDAL==2.2.3
• GeoAlchemy2==0.6.3
• h5py==2.9.0
• idna==2.6
• itsdangerous==1.1.0
• Jinja2==2.10.1
• kombu==4.1.0
• lxml==3.3.3
• MarkupSafe==1.1.1
• marshmallow==3.0.0rc5
• marshmallow-jsonschema==0.6.0
• matplotlib==2.1.2
• nose==1.3.7
• numpy==1.15.0
• obspy==1.1.0
• -e git+https://github.com/gem/oq-engine.git@1fab2cb92442100b03c67666327e0fbc10458ad9#egg=openquake.engine
• pbr==4.0.0
• pkg-resources==0.0.0
• pluggy==0.6.0
• psutil==5.4.3
• psycopg2==2.8.3
• py==1.5.2
• pymap3d==1.3.0
• PyOpenGL==3.1.1a1
• pyparsing==2.2.0
• pyproj==1.9.5.1
• PyQt5==5.12
• PyQt5-sip==4.19.18
• pyqtgraph==0.10.0
• pyshp==1.2.3
• pytest==3.4.0
• python-dateutil==2.6.1
• python-prctl==1.6.1
• pytz==2017.3
• PyYAML==3.12
• pyzmq==17.0.0
• -e git+https://gitlab.seismo.ethz.ch/indu/rt-ramsis.git@56a45fdcec072cefc3fd7b0c15b5a11de5c8a7b9#egg=RAMSIS
• -e git+https://gitlab.seismo.ethz.ch/indu/ramsis.datamodel.git@fabe9e1832c5d218a3401baf384f0b868075956b#egg=ramsis.datamodel
• -e git+https://gitlab.seismo.ethz.ch/indu/ramsis.sfm.em1.git@cb0fbc909cef233ad835adf4ab7fd0ab37a9366c#egg=ramsis.sfm.em1
• -e git+https://gitlab.seismo.ethz.ch/indu/ramsis.sfm.worker.git@5d73e553d9d51219b295cf93f010821b7b0c8a85#egg=ramsis.sfm.worker
• -e git+https://gitlab.seismo.ethz.ch/indu/ramsis.utils.git@a03800b8c4834d7c44ec04c6765ed5d2c52e48f0#egg=ramsis.utils
• requests==2.18.4
• scipy==1.0.0
• setproctitle==1.1.10
• Shapely==1.6.4.post1
• sip==4.19.7
• six==1.11.0
• SQLAlchemy==1.3.5
• toml==0.10.0
• transitions==0.6.9
• urllib3==1.22
• vine==1.1.4
• webargs==5.3.2
• Werkzeug==0.15.4

Ok, the problem seems to be related to the version of marshmallow you're using. I updated to

• webargs==5.3.2
• marshmallow==3.0.0rc5
• python-dateutil==2.8.0

> Host: localhost:5000
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 8180
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* HTTP 1.0, assume close after body
< HTTP/1.0 422 UNPROCESSABLE ENTITY
< Content-Type: application/json
< Content-Length: 1975
< Server: Werkzeug/0.14.1 Python/3.6.5
< Date: Mon, 15 Jul 2019 13:35:24 GMT
< 
{
    "errors": {
        "well": {
            "sections": {
                "0": {
                    "endtime": [
                        "Not a valid datetime."
                    ],
                    "starttime": [
                        "Not a valid datetime."
                    ],
                    "hydraulics": {
                        "0": {
                            "datetime": {
                                "value": [
                                    "Not a valid datetime."
                                ]
                            }
                        },
                        "1": {
                            "datetime": {
                                "value": [
                                    "Not a valid datetime."
                                ]
                            }
                        }
                    }
                }
            }
        },
        "scenario": {
            "well": {
                "sections": {
                    "0": {
                        "endtime": [
                            "Not a valid datetime."
                        ],
                        "starttime": [
                            "Not a valid datetime."
                        ],
                        "hydraulics": {
                            "0": {
                                "datetime": {
                                    "value": [
                                        "Not a valid datetime."
                                    ]
                                }
                            }
                        }
                    }
                }
            }
        },
        "model_parameters": {
            "starttime": [
                "Not a valid datetime."
            ],
            "endtime": [
                "Not a valid datetime."
            ],
            "modelparam1": [
                "Unknown field."
            ]
        }
    }
}
* Closing connection 0

However, I'd recommend updating to the most recent marshmallow==3.0.0rc8 version, immediately.

mentioned in merge request !3 (merged)

Closed with !3 (merged).

closed