Proposal for a modification in the SOAP event-webservice

Hallo community-members,
the OpenClinica SOAP web-services (WS) do not provide a simple way to retrieve event-information in a study. For (automated) upload scenarios this is a problem because clients using WS have to know which events and repeats have to be created/scheduled.
Workarounds exist but are problematic : the required information is available in a complete ODM-extract of the entire study but it has to be available to the client in a timely manner. REST-call possibilities have the disadvantage that the entire clinicalData is also present which potentially contains information not meant to be disclosed. The Event REST-API seems buggy as far as I can establish. For example a call to the URL: http://localhost:8080/OpenClinica/pages/odmk/study/S_EVENTFUL/studysubject/SS_EVS0001/events results in a response containing only <ns5:ODM/>. Moreover to use REST in unattended situations you have to setup an environment with OAuth 2.0.
Krikor, please correct me if my analysis is incomplete or if there are other solutions for the problem.
For these reasons we would like to propose an extension to the SOAP WS-calls which will specifically return the required event-information. We’ve developed a SOAP proof-of-concept which returns the event information. An example response is:

<SOAP-ENV:Body>
<eventInformationResponse xmlns="http://openclinica.org/ws/event/v1">
<result>Success</result>
<events>
<eventStatus eventDefinitionOID="SE_BASELINEREPEATING" repeatNumber="1" sampleOrdinal="2" stage="not started" status="data entry started" studySubjectOID="SS_EV0001"/>
<eventStatus eventDefinitionOID="SE_EVENTFULNONREPEATING" repeatNumber="1" sampleOrdinal="1" stage="not started" status="completed" studySubjectOID="SS_EV0001"/>
<eventStatus eventDefinitionOID="SE_BASELINEREPEATING" repeatNumber="1" sampleOrdinal="1" stage="not started" status="completed" studySubjectOID="SS_EV0001"/>
<eventStatus eventDefinitionOID="SE_BASELINEREPEATING" repeatNumber="1" sampleOrdinal="3" stage="not started" status="scheduled" studySubjectOID="SS_EV0001"/>
<eventStatus eventDefinitionOID="SE_EVENTFULNONREPEATING" repeatNumber="1" sampleOrdinal="1" stage="not started" status="data entry started" studySubjectOID="SS_EV0002WI"/>
<eventStatus eventDefinitionOID="SE_EVENTFULNONREPEATING" repeatNumber="1" sampleOrdinal="1" stage="not started" status="scheduled" studySubjectOID="SS_EVS0001"/>
<eventStatus eventDefinitionOID="SE_BASELINEREPEATING" repeatNumber="1" sampleOrdinal="1" stage="not started" status="scheduled" studySubjectOID="SS_EVS0001"/>
<eventStatus eventDefinitionOID="SE_BASELINEREPEATING" repeatNumber="1" sampleOrdinal="1" stage="not started" status="data entry started" studySubjectOID="SS_EVS0002"/>
</events>
</eventInformationResponse>
</SOAP-ENV:Body>
Could you, as community help, with the following questions:
  1. What level of granularity/atomicity do you consider best: a single request to return all the event information in a study or a request which only returns the information for a specific subject and/or event?
  2. The class StudyEventBean contains both the 'repeatNumber' and 'sampleOrdinal'. Do you know what the semantic difference is?
  3. In the response I've put the information in XML-attributes. This deviates from the current OC convention to place the information in XML-elements. Do you think this structure is OK?
  4. Do you consider this a good enough addition to the SOAP web-services API of OpenClinica to add it to the main-source tree?
Cheers and eager to hear your comments
Jacob
«1

Comments

  • lindsay.stevenslindsay.stevens Posts: 392 ✭✭
    via Email
    Sounds great! If the CRF status and version oid could be included it would
    be amazing.
  • toskriptoskrip Posts: 244 ✭✭
    edited April 2016
    Hi Jacob,

    > REST API and authentication
    actually right now the situation there is very confusing. Some API endpoints are ment to work with OAuth (e.g. Designer)

    RESTfull URLs uses POST request to create a conventional user session in spring security (it will log out user from another session which is not very comfortable behaviour)

    Some API like the one that you mention (odmk) have been created only for Participate and have no authentication what so ever (need to be dealt with on network layer infrastructure e.g. firewall).

    BTW: the REST call you mention is ment to report only scheduled Participate event forms. That is why it is returning empty ODM element - this endpoint was not ment for usage outside of participate scenario.

    And finally some API endpoints switched to the concept of API key, that that is unique secret generated for every new OC user during user creation. This API key can be retrieved programatically with POST REST call to /pages/accounts/login

    The last approach is reasonable, and it shows some light to where the APIs of OC should go, but what is still missing is real concept for new APIs.... these are interfaces that need to be clean, and consistent. So to speak I totally understand why you have decided to continue and extend the SOAP version of APIs.

    > extending SOAP services
    this is the road that some users went (I think I have seen it in solutions from Aachen...), but in long run there probably will not be any support from OC directly.

    now to the questions:
    1. I think it depend on the use case where you want to use it. I think that both would be useful. But if all events are requested, they should be grouped into subjects elements (otherwise the response is a little bit flat). Check the RESTfull URLs that uses SubjectData and StudyEventData elements. In general one would expect valid ODM elements as result from web service call.
    2. Can't really say... have a feeling that there will not be a difference, although repeatNumber sound to me more close to repeatKey semantics used in ODM.
    3. I think that this is more the question on how libraries are handling marshalling/unmarshalling operations on domain model. Generally speaking the attributes are used as a parameters for constructor when processing XML data element (the elements are mapped to setters). If your domain model does not provide constructors with parameters some libraries may have a trouble to unmarshall the XML data back to domain model. But good libraries should be able to handle it without any troubles.
    4. As an addition it would be very nice (kudos to you!), but it is unlikely to get any official support for this, which also means that it would be not possible to use these services in validated OpenClinica installations.

    Tomas
  • Csaba.HalmagyiCsaba.Halmagyi Posts: 51
    It would be great to have a webservice like that in the main source tree. ODIN and the OCSYNCHRONISER currently reads that information directly from the database.

    Thanks Jacob for working this!

    Csaba
  • toskriptoskrip Posts: 244 ✭✭
    Just to follow up.... I would personally prefer to "fix" the StudySubject SOAP endpoint (listAllByStudy) which already returns some event details (but not complete enough), there it could be also possible to extend StudySubject to return the OID (which is very useful) and potentialy add CRFs details into the Event element. I think there is a better chance to get such fix merged to OC master branch in timely manner as this is service that is there for years, and only needs some improvements.

    Jacob have you though about that, rather than creating a brand new one?

    Tomas
  • toskriptoskrip Posts: 244 ✭✭
    My current approach of getting all the information needed for data integration scenarios is to mix SOAP with RESTfull URLs. This works but it has disadvantages: more complicated data processing and kind of slowish performance (RESTfull URLs are not handling well the level of details for data retrieval - as Jacob already mentioned). But it is still possible, so if somebody is struggling with such task... feel free to contact me.

    T
  • lindsay.stevenslindsay.stevens Posts: 392 ✭✭
    via Email
    Somewhat relevant recent post [1] in case you come across it in development
    of this. Basically, I tried to speed up SOAP by using concurrent client
    requests. High peak rates of random request errors ensued. I didn't find
    the root cause but concluded that oc-ws doesn't like async, though oc-web
    is fine with it. Connection reuse gives a good boost to sync requests
    without this problem.

    [1]
    https://forums.openclinica.com/discussion/15942/webservices-concurrency-problem
  • kkrumliankkrumlian Posts: 25
    Hey guys,

    Great discussion, What do you think of returning more of an ODM data model minus the data that we don't need.

    Take a look at this: http://104.131.126.24:8080/OpenClinica/pages/odmk/study/S_THEJUNOD/studysubject/SS_0506/events

    Obviously as @toskrip mentioned this API was designed for participate so we would need to do some modifications there. Going forward also we need to come up with a strategy on returning snippets of the data rather than full ODM because we all know that could be extremely heavy.

    Krikor
  • toskriptoskrip Posts: 244 ✭✭
    Hey Krikor

    this would do but...

    - the closer to ODM the better
    - it would be good to differentiate between request for study metadata and clinicaldata

    e.g.

    /odm/clinicaldata/S_THEJUNOD/subjectdata/SS_0506/studyeventdata/

    IMHO I would not include FormData details into such response directly, I would include only reference to FormData REST resource (because the URL resource says that the client want to retrieve studyeventdata not the formdata).

    The link can be made like this:

    FormData Url="http://104.131.126.24:8080/OpenClinica/pages/odm/clinicaldata/S_THEJUNOD/subjectdata/SS_0506/studyeventdata/SE_TEST_8378/studyeventrepeatkey/1/formdata

    which if requested will give the odm that your current service is returning.

    I would be happy if somebody could explain me why we need new namespaces for ODM scheme:
    xmlns="http://www.cdisc.org/ns/odm/v1.3"
    xmlns:ns5="http://www.cdisc.org/ns/odm/v1.3-api"

    and also for OC ODM extensions scheme:
    xmlns:ns2="http://www.openclinica.org/ns/odm_ext_v130/v3.1"
    xmlns:ns6="http://www.openclinica.org/ns/odm_ext_v130/v3.1-api"

    best

    Tomas
  • toskriptoskrip Posts: 244 ✭✭
    I agree with what @kkrumlian says about returning snippets from ODM (otherwise it would be to heavy), however what I think is wise to add is that there should be a navigation pattern included in these snippets (like to URL link for different resources from ODM hierarchy).

    Basically you create a RESTfull way of navigating through full ODM graph. I was already calling for it in different thread (CDISC API):
    https://forums.openclinica.com/discussion/15800/cdisc-api-is-there-any-development

    With this sort of resource referencing you could create very interesting scenarios where from clinical data elements you refer to ODM study metadata definition that can be actually stored on different servers.

    P.S.: sorry @jacob_rousseau that the discussion now went out of the original topic with was SOAP extensions.... this was not my intention

    T
  • kkrumliankkrumlian Posts: 25
    Hey @toskrip,

    That's a clever idea. Building on your suggestion:
    # Get a specific study subject's event data - includes downstream FormData navigational URLs
    /rest/clinicaldata/xml/view/{studyOID}/{studySubjectOID/Id}/studyeventdata
    # Get every study subject's event data only no downstream data - includes downstream FormData navigational URLs
    /rest/clinicaldata/xml/view/{studyOID}/*/studyeventdata
    # Get every study subject's event data only no downstream data - includes downstream FormData navigational URLs - includes event based DNs & audits
    /rest/clinicaldata/xml/view/{studyOID}/*/studyeventdata?includeDNs=y&includeAudits=y
    • I've modeled the URLs based on our casebook URLs. (Will create a separate discussion on API naming conventions soon)
    • In the above call would we need to also specify a specific event ?

    Using this pattern would we go downstream - so for formdata we have something like:
    # Get a specific study subject's form data no downstream data - includes downstream Itemdata navigational URLs
    /rest/clinicaldata/xml/view/{studyOID}/{studySubjectOID/Id}/formdata
    or 
    /rest/clinicaldata/xml/view/{studyOID}/{studySubjectOID/Id}/{studyEventOID}/formdata

    @jacob_rousseau rousseau For your immediate need are you married to doing it in SOAP ?

    Thank you
    Krikor
Sign In or Register to comment.