31 Dec 2018: Thank you to our knowledgeable and friendly forums users for another great year. We are aware of the uptick in spam accounts and are doing our best to ban these at first sight. Thank you for your patience!

Bulk Reassign CRF to a New Version with Selenium

Hi all,

OpenClinica has the "Reassign CRF to a New Version" button. Unfortunately you have to do this per patient, which is not very practical if your study contains more than a handful of patients.
After reading Lindsay’s “delete many rules at once” post in which he uses Selenium, I figured reassigning CRFs automatically should be possible as well. Turns out it is, but it’s quite complicated. The script was developed for OC 3.3 and it is quite likely it will not work for older versions. I’ve successfully tested the script on several of my studies. As the script apparently is too long to paste in this post, I've attached it as txt file (I wasn't allowed to upload an html file).

If you feel like giving it a go, definitely try it in a test environment first! Here’s what you’ll need to provide:
eventCol: the column number of the event that contains CRF
crfName: the name of the CRF
toVersion: the new version of the CRF
instanceURL: the url of your OpenClinica server. Looks something like this: https://yourserver/OpenClinica/

Currently it changes the version of CRFs that have a status of Initial Data Entry or Data Entry Complete, since that is what I needed. You can easily change that to only update when the status is Initial Data Entry by changing this bit:

<tr>
    <td>storeEval</td>
    <td>(storedVars['dataEntryStatus'] ==&quot;Initial Data Entry&quot;) || (storedVars['dataEntryStatus']==&quot;Data Entry Complete&quot;)</td>
    <td>changeDataEntry</td>
</tr>

To

<tr>
    <td>storeEval</td>
    <td>(storedVars['dataEntryStatus'] ==&quot;Initial Data Entry&quot;) </td>
    <td>changeDataEntry</td>
</tr>

If you run into errors, I might be able to help you out, but no guarantees as debugging is pretty time-consuming.

Cheers,
Sander


«1

Comments

  • agoodwinagoodwin Posts: 131 admin
    Hello Sander,
    It looks like you're updating the version so the existing will render in the new version but it does not appear to touch the audit table. Is there an audit record with this approach?

    Best,
    Alicia
  • sderiddersderidder Posts: 57
    via Email
    Hi Alicia,

    Yes there is. This is from the audit log from one of the patients for a change I did earlier today with the script:
    Change CRF version 31-Jul-2014 12:25:04 sderidder CRF version (0) 1.0 1.04

    Actually, if it hadn't appeared in the audit log, it probably would have been an OC bug, as the selenium script basically just clicks OC's Change Version button. Glad it's there though.

    Cheers,
    Sander
  • lindsay.stevenslindsay.stevens Posts: 404 ✭✭✭
    via Email
    Hi Sander,

    Nice one, it can be quite tedious otherwise. If you like I can add this to
    the wiki page and either put the script on there or add it to my github
    repo.

    Couple of things to watch out for:

    1. CRF migration action audit logs the CRF instance owner (the person
    starting the CRF data entry) rather than the person migrating versions. So
    you might get some incorrect audit entries, e.g. where site users started a
    CRF that you migrated.

    2. Show/Hide status isnt preserved in migration so Simple Conditional
    Displays, ShowAction and HideAction rules may need to be re-triggered
    afterwards.

    Best regards,
    Lindsay
  • sderiddersderidder Posts: 57
    via Email
    Hi Lindsay,

    Sounds great. If desired I could add some more comments / explanation?

    1. Oh really? I didn’t know that. Weird...
    2. Am familiar with this one. Very annoying...

    Thanks!
    Sander
  • lindsay.stevenslindsay.stevens Posts: 404 ✭✭✭
    Hi Sander

    I moved the old DeleteManyRules page with a new one called SeleniumTasks [1], and rewrote it so that it is more general. The wikibook is for all so feel free to add or modify. I assumed eventCol is supposed to be the column number in the Subject Matrix, so the first event in the study is eventCol=2. Is that right?

    My old scripts and your new one are now in my github repo [2]. Hope you don't mind but I changed the format from .txt to .xml so I could apply indentation rules. Selenium IDE still reads it in the same.

    Lastly, I have a minor suggestion. Rather of specifying the eventCol as an integer, the event could be specified by name. The required column integer could then be determined by passing the storedVar into the following javascript:

    window.document.evaluate( "//td[div='My event name' and ancestor::table[@id='findSubjects']]" ,document, null, XPathResult.ANY_TYPE, null ).iterateNext().cellIndex-4;

    Which is saying, select the table cell (td) who has a child div with the text 'My event name' and has an ancestor (any parent/parent's parent/etc) which is a table with the id 'findSubjects'. Look for that td in the current document and don't worry about namespaces or the return type. In the result, iterate to the first object, which should be the target td. Then return the cell index of that td (column number). The minus 4 is because cell index is zero-based, the first column (0) is StudySubjectID, and there are 5 hidden columns after that but before the event columns start, which are revealed when 'Show More' is clicked. So if 'My event name' is the second event in the study, this would return 3.

    Best regards,
    Lindsay

    [1] https://en.wikibooks.org/wiki/OpenClinica_User_Manual/SeleniumTasks
    [2] https://github.com/lindsay-stevens-kirby/openclinica_scripts/tree/master/selenium_tasks
  • sderiddersderidder Posts: 57
    via Email
    Hi Lindsay,

    Looks great!
    Actually, no, the first event is in eventCol=1. So you're right, it's less confusing if we change it to a String version. I included the slightly modified version of your statement:
    window.document.evaluate( "//td[div='${eventName}' and ancestor::table[@id='findSubjects']]" ,window.document, null, XPathResult.ANY_TYPE, null ).iterateNext().cellIndex-5;

    Not very familiar with xpath; nice that you can search for the eventName. Don’t know whether that's possible with javascript? Couldn’t find an equivalent querySelector-ish statement. Come to think of it, it should be possible to rewrite some other things as well, such as the part where I try to find the CRF. Currently it's a loop, but I should be able to simply select it using xpath:
    window.document.evaluate( "//text()[.='TRACER Comor' and ancestor::td[@class='content']]" ,window.document, null, XPathResult.ANY_TYPE, null ).iterateNext()

    Well, that works, but only if the CRF's status is Not Started.
    The source code in that case is:
    TRACER Comor

    If the status is Data Entry Started, the source code is:
    TRACER Intoxicatie ESRA0 

    So OC decides to add a space...? I figured removing removing the space shouldn't be too hard using normalize-space. A test case without the ancestor:
    //*[normalize-space(text())="TRACER Intoxicatie ESRA0"] That doesn't seem to do the trick. Tried replacing   with "", doesn't seem to work either. Any ideas?

    I've attached the new version which uses the eventName.

    Cheers,
    Sander
  • lindsay.stevenslindsay.stevens Posts: 404 ✭✭✭
    via Email
    Hi Sander,

    Thanks, I'll upload the new script and update the wiki.

    My impression is that 'evaluate' is a way for javascript to do xpath, while
    the querySelector functions are for css-based selection.

    For the CRF name, it might be that there is a real space there? But anyway
    the xpath matching string needs to use single quotes because it is inside a
    double quoted string for the evaluate function, so like:

    "//*[normalize-space(text())='TRACER Intoxicatie ESRA0']"

    I was playing around with the Mozilla documentation page [1], by adding
    spaces to the end of the 'Parameters' heading. There is also a 'Parameters'
    link on the right, so there are 2 candidates for matching.

    The following correctly trimmed the text for the xpath selection, and with
    a replace added on, returned the space-normalized string 'Parameters' from
    the heading (selected by 'h3', to eliminate the 'a' match).

    window.document.evaluate("//h3[normalize-space(text())='Parameters']",window.document,
    null, XPathResult.ANY_TYPE, null
    ).iterateNext().textContent.replace(/\s+/g, '')

    The examples on that page are really good, like this one will give you a
    count of matched nodes:

    window.document.evaluate("count(//*[normalize-space(text())='Parameters'])",window.document,
    null, XPathResult.ANY_TYPE, null ).numberValue

    And this will alert the content of all the matched nodes:

    var iterator =
    document.evaluate("//*[normalize-space(text())='Parameters']",
    window.document, null, XPathResult.ANY_TYPE, null ); try { var thisNode =
    iterator.iterateNext(); while (thisNode) { alert( thisNode.textContent );
    thisNode = iterator.iterateNext(); } } catch (e) { dump( 'Error: Document
    tree modified during iteration ' + e );}

    [1]
    https://developer.mozilla.org/en/docs/Introduction_to_using_XPath_in_JavaScript

    Best regards,
    Lindsay
  • sderiddersderidder Posts: 57
    via Email
    Hi Lindsay,

    Nah, it really has something to do with the CRF status and/or location in the table, as the space for the Intoxication CRF is not there when its status is Not Started. I double-checked with the Excel CRF; definitely no space.

    The quotes are not the problem as I'm using the firepath plugin and therefore I'm not calling the evaluate function yet:) I guess this demonstrates the problem:
    concat(normalize-space((//*[contains(text(), 'TRACER Intoxicatie ESRA0')])[5]), 'X')
    String: TRACER Intoxicatie ESRA0 X

    So basically I retrieved any text that contains 'TRACER Intoxicatie ESRA0', took the fifth one (I checked in firebug; the fifth one is the one I need in this case), call the normalize-space function and attach an "X" to the String to see whether the white-space is removed. As you can see it isn't, which is why it doesn't work.

    So I switched to a translate to see what happens when I simply remove all normal spaces:
    concat(translate((//*[contains(text(), 'TRACER Intoxicatie ESRA0')])[5], ' ', ''), 'X')
    String: TRACERIntoxicatieESRA0 x

    And after some more googling, this seems to do the trick:
    concat(translate((//*[contains(text(), 'TRACER Intoxicatie ESRA0')])[5], ' ', ''), 'X')
    String: TRACER Intoxicatie ESRA0x

    Looks identical, except that the white-space I replace is now Alt+0160
    I think I should be able to change the code now. Should have some time this week.

    Really interesting stuff by the way. I'll definitely read the Mozilla page.

    Thanks for your help!
    Sander
  • sderiddersderidder Posts: 57
    via Email
    Hi Lindsay,

    I succeeded via translate(., '\u00A0', '')
    Removed the loop and rewrote the Handle the Change section.
    Here's the final version for now.

    Thanks!
    Sander
    ________________________________________
    From: Ridder, Sander de
    Sent: 06 August 2014 09:39
    To: 'lindsay.stevens'
    Subject: RE: [OpenClinica] Bulk Reassign CRF to a New Version with Selenium

    Hi Lindsay,

    Nah, it really has something to do with the CRF status and/or location in the table, as the space for the Intoxication CRF is not there when its status is Not Started. I double-checked with the Excel CRF; definitely no space.

    The quotes are not the problem as I'm using the firepath plugin and therefore I'm not calling the evaluate function yet:) I guess this demonstrates the problem:
    concat(normalize-space((//*[contains(text(), 'TRACER Intoxicatie ESRA0')])[5]), 'X')
    String: TRACER Intoxicatie ESRA0 X

    So basically I retrieved any text that contains 'TRACER Intoxicatie ESRA0', took the fifth one (I checked in firebug; the fifth one is the one I need in this case), call the normalize-space function and attach an "X" to the String to see whether the white-space is removed. As you can see it isn't, which is why it doesn't work.

    So I switched to a translate to see what happens when I simply remove all normal spaces:
    concat(translate((//*[contains(text(), 'TRACER Intoxicatie ESRA0')])[5], ' ', ''), 'X')
    String: TRACERIntoxicatieESRA0 x

    And after some more googling, this seems to do the trick:
    concat(translate((//*[contains(text(), 'TRACER Intoxicatie ESRA0')])[5], ' ', ''), 'X')
    String: TRACER Intoxicatie ESRA0x

    Looks identical, except that the white-space I replace is now Alt+0160
    I think I should be able to change the code now. Should have some time this week.

    Really interesting stuff by the way. I'll definitely read the Mozilla page.

    Thanks for your help!
    Sander
  • lindsay.stevenslindsay.stevens Posts: 404 ✭✭✭
    Hi Sander

    That's a genius find, I certainly wouldn't expect invisible characters, and especially when they appear only in one context.

    The script didn't come through by email - would you mind uploading it to the forums? Or as a pull request on the file at [1].

    Also, the following kind of thing can remove all non-ASCII characters [2]

    window.document.evaluate("//h3[starts-with(.,'Parameters')]",window.document, null, XPathResult.ANY_TYPE, null ).iterateNext().textContent.replace(/[^\x00-\x80]/g, '')

    [1] https://github.com/lindsay-stevens-kirby/openclinica_scripts/blob/master/selenium_tasks/migrateVersion.xml
    [2] http://stackoverflow.com/questions/14613080/replace-unicode-matches-in-javascript
This discussion has been closed.