'Hidden' CRFs and XML REST requests

Hi,

I am using XML REST requests to retrieve clinical data and audit log information for groups of subjects in studies. Because of the size of some of our studies, a study-wide request (using the /Study_OID/*/*/* pattern) is too slow and times out, so I am forced to use site-level REST requests (using the /Site_ID/*/*/* pattern), iterating over all sites in a study.

I have noticed, when using site-level requests, that no clinical data (i.e. anything below the StudyEventData node) is included in the XML, I believe, for CRFs that are marked as 'Hide CRF'.

My questions are:
  1. Am I correct that this is the case?
  2. Is this by design?
  3. Is there any way that I can retrieve clinical data in CRFs with 'Hide CRF' selected using site-level REST requests?
Many thanks,
Richard Welsh
Tagged:
«1

Comments

  • haenselhaensel Posts: 567 ✭✭
    Hi Richard

    I didn't tested this on my own but it sounds reasonable that crfs that can not be seen by the user (hidden crfs) are excluded from the data extract.
    But maybe you meant crfs that are hidden in some sites but not in all and these crfs don't show up in the data extract for sites where they are visible?

    Regards,
    Christian
  • Richard WelshRichard Welsh Posts: 13
    Hi Christian,

    These 'hidden' CRFs are visible, but only to users who have study-wide access. I f a user is only assigned to one site, they are prevented from seeing CRF data in the web interface. We are using the 'Hide CRF' flag from on-site staff, but still allowing staff in the central study-coordinating centre (who have study-wide access) to do data entry on this form.

    What I find strange is that the user I am authenticating as when doing the REST request is granted study-wide access. So this user does have the rights to see this data even from the context of one site... so shouldn't the data appear in the XML response?

    Richard

  • toskriptoskrip Posts: 249 ✭✭
    As Christian said, this is reasonable behaviour. I find it only logical that when you are requesting site level clinical data (SiteOID) you will only get eCRFs that are available for the site and not the whole study. If the user have also study level access that you can use StudyOID to obtain everything. Changing such behaviour to respond study eCRF when requesting Site dat would be, at least in my opinion, very confusing.

    Tomas
  • Richard WelshRichard Welsh Posts: 13
    Hi Tomas,

    Actually, I agree with you, it is reasonable behaviour! Now that I better understand that site-level REST requests are made in the same context as if you were accessing the web interface for just that site, it makes perfect sense. I misunderstood; at first thought it was just a site filter on overall study data.

    Our user does have study-level access and I would prefer to access everything using the StudyOID, but unfortunately the REST response is too slow and times out after 20 minutes.

    Can you think of any other way I could filter all the site data so that I can retrieve it in manageable chunks?

    Richard
  • toskriptoskrip Posts: 249 ✭✭
    Hi,

    what I normally do is that first I get the list of subject to collect the SSIDs via SOAP and then I lazy load clinical data one by one: /StudyOID/SSID/*/*

    the speed of response in this case naturally depends on number of subjects in a study.

    Tomas
  • haenselhaensel Posts: 567 ✭✭
    Hi Richard and Tomas

    Unless it is possible to retrieve all data at once you can try it with smaller chunks (as Tomas suggested).
    As alternative you can move the * around to create chunks in different levels (e.g. study event).

    Regards,
    Christian
  • Richard WelshRichard Welsh Posts: 13
    Hi Tomas & Christian,

    I have considered retrieving data subject at a time, but this is not really a very satisfactory or scalable solution, with each subject REST request taking around 8 seconds and a study that is due to grow to over 3,000 subjects. I am worried about the extra load that this would put on the OC server, and it would also needlessly multiply the post-processing overhead (I am processing the result of each REST request using queries that are optimised to work atomically across large sets of subjects).

    It would be nice to be able to subdivide the data in other ways, as you suggest Christian, for instance /StudyOID/*/StudyEventOID/* to retrieve all data for a given event, but this doesn't work (the above example results in an 'Oops!' error. Are queries like this meant to work?

    Richard
  • haenselhaensel Posts: 567 ✭✭
    edited February 7
    Hi Richard

    There are some examples in the OC documentation (that I never tested).

    1.3 RestFul URL access to OpenClinica metadata and print Resources

    One is
    STUDY_OID/*/FORM_VERSION_OID


    I don't know if your extra '*' is valid because the normal format is like this
    {STUDY_OID}/{STUDY_EVENT_OID}/{FORM_VERSION_OID}


    Regards,
    Christian
  • toskriptoskrip Posts: 249 ✭✭
    I don't think that bulk for all subjects filtered on event or crf lvl is officially supported.

    T
  • Richard WelshRichard Welsh Posts: 13
    Hi Christian,

    The page you reference mentions 4 levels to the ODM_XML_PATH, not 3:
    For Clinical Data the ODM_XML_PATH: Consists of typically 4 variables. {STUDY_OID}/{Study_Subject_OID}/{STUDY_EVENT_OID}/{FORM_VERSION_OID} plus parameters for additional options.

    I think the section of that page that you quote is an outdated proposed form (it says "This filtering at XML level will be worked on in near future.") In any case, that form of query doesn't work in my experience.

    Richard
Sign In or Register to comment.