CRIS Natural Language Processing

Brief Description

Application to identify instances of aggressive behaviour in patients, including verbal, physical and sexual aggression.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions include:

“reported to be quite aggressive towards…”,

“violence and aggression, requires continued management and continues to reduce in terms of incidents etc”.

Also include verbal aggression and physical aggression.

Excludes negative and irrelevant mentions, e.g.:

“no aggression”,

“no evidence of aggression”

“aggression won’t be tolerated”.

etc. Definitions: Search term(case insensitive): *aggress*

Evaluated Performance

Cohen's k = 85% (50 un-annotated documents - 25 events/25 attachments, search term ‘aggress*’). Instance level (testing done on 100 random documents): Precision (specificity / accuracy) = 91%

Recall (sensitivity / coverage) = 75% Patient level (testing done on 50 random documents): Precision (specificity / accuracy) = 76%

Additional Notes

Run schedule– Monthly

Other Specifications

Version 1.0, Last updated: xx

DOI

Agitation

Brief Description

Application to identify instances of agitation

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions include:

“Very agitated at present, he was agitated”,

“He was initially calm but then became agitated and started staring and pointing at me towards”,

“Should also include no longer agitated. “

Excludes negative and irrelevant mentions, e.g.:

“He did not seem distracted or agitated”,

“Not agitated”,

“No evidence of agitation”.

“a common symptom of psychomotor agitation” Definitions: Search term(case insensitive): *agitat*

Evaluated Performance

Cohen's k = 85% (50 un-annotated documents - 25 events/25 attachments, search term ‘agitat*’). Instance level (testing done on 100 random documents): Precision (specificity / accuracy) = 85%

Recall (sensitivity / coverage) = 79% Patient level: All patients with primary diagnosis code F32* or F33* (testing done on 30 random document, one document per patient) Precision (specificity / accuracy) = 82%

Additional Notes

Run schedule– Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Anergia

Brief Description

Application to identify instances of anergia

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

“feelings of anergia…”

Excludes negative mentions, e.g:

“no anergia”,

“no evidence of anergia”,

“no feeling of anergia”. Definitions: Search term(case insensitive): *anergia*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘anergia*’). Instance level i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 95% Recall (sensitivity / coverage) = 89% Patient level – All patients with primary diagnosis code F32* or F33* (testing done on 30 random document, one document per patient): Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule– Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Anhedonia

Brief Description

Application to identify instances of anhedonia (inability to experience pleasure from activities usually found enjoyable).

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions include:

“ X had been anhedonic”,

“ X has anhedonia”.

Excludes Negative mentions, e.g

“no anhedonia”,

“no evidence of anhedonia”,

“not anhedonic”,

Exclude ‘Unknown’ mentions e.g:

i) used in a list, not applying to patient (e.g. typical symptoms include …);

ii) uncertain (might have anhedonia, ?anhedonia, possible anhedonia);

iii) not clearly present (monitor for anhedonia, anhedonia has improved);

iv) listed as potential treatment side-effect;

v) vague (‘she is not completely anhedonic’, ‘appears almost anhedonic’) Definitions: Search term(s): *anhedon*

Evaluated Performance

Cohen's k=85% (50 un-annotated documents - 25 events/25 attachments, search term 'anhedon*’). Instance level, (testing done on 100 random documents):

Precision (specificity / accuracy) = 93% Recall (sensitivity / coverage) = 86% Patient level – All patients with primary diagnosis code F32* or F33* (testing done on 30 random document, one document per patient) Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule– Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Anosmia

Brief Description

Application to extract and classify mentions related to anosmia.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

“Loss of enjoyment of food due to anosmia”,

“COVID symptoms such as anosmia”

Excludes Negative and Unknown mentions, e.g.:

“Nil anosmia”,

“Doctor mentioned they had anosmia so could not smell patient”,

“Anosmia related to people other than the patient”,

Unknown mentions: Annotations are coded as unknown when it is not clear if the patient has symptoms/experiences of anosmia

E.g.

“Mentions of medications for it”,

“Don’t come to the practice if you have any covid symptoms such as anosmia” etc. Definitions: Search term(case insensitive): Anosmia*

Evaluated Performance

Cohen's k = 83% (100 Random Documents). Instance level, (testing done on 100 random documents):Precision (specificity / accuracy) = 83% Recall (sensitivity / coverage) = 93%

Additional Notes

Run schedule– On demand

Other Specifications

Version 1.0, Last updated:xx

Anxiety

Brief Description

Application to extract and classify mentions related to (any kind of) anxiety.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Affirmed, Negated, and Irrelevant.

Output and Definitions

The output includes-

Affirmed examples of anxiety includes:

“ZZZZZ shows anxiety problems”

Exclude Negative mentions of anxiety includes:

“ZZZZ does not show anxiety problems”

Excludes Irrelevant examples of anxiety includes:

“If ZZZZ was anxious he would not take his medication”

Examples related to the patient or someone else (“experiencer”)

i. patient: “ZZZ shows anxiety problems”

ii. other: “nurse is worried about the patient”

iii. unknown: “he showed clear signs of anxiety” Search Terms (case insensitive)-Available on request

Evaluated Performance

Cohen’s k = 94% ( 3000 random documents) Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 87% Recall (sensitivity / coverage) = 97%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Apathy

Brief Description

Application to extract the presence of apathy

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions includes:

any indication that apathy was being reported as a symptom:

“ continues to demonstrate apathy”

“ some degree of apathy noted”

“presentation with apathy”

“his report of apathy given”.

Exclude Negative mentions of apathy:

“denied apathy”

“no evidence of apathy”

Exclude ‘Unknown’ annotations of apathy:

“may develop apathy or as a possible side effect of medication”

“*apathy* found in quite a few names” Definitions: Search term(s): *apath*

Evaluated Performance

Cohen's k=86% (50 un-annotated documents - 25 events/25 attachments, search term ‘apath*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 93% Recall (sensitivity / coverage) = 86% Patient level – All patients with primary diagnosis code F32* or F33* (testing done on 30 random document, one document per patient): Precision (specificity / accuracy) = 73%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Arousal

Brief Description

Application to identify instances of arousal excluding sexual arousal.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes:

Positive mentions include:

“...the decisions she makes when emotionally aroused”,

“...during hyperaroused state”,

“following an incidence of physiological arousal”

Exclude Negative mentions include:

“mentions of sexual arousal”,

“no arousal”,

“not aroused”,

“denies being aroused”

Unknown mentions include:

“annotations include unclear statements and hypotheticals” Definitions: Search term(s): *arous*

Evaluated Performance

Cohen's k = 95% (50 un-annotated documents - 25 events/25 attachments, search term ‘*arous*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 91%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Bad Dreams

Brief Description

Application to identify instances of experiencing a bad dream

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations include:

“ZZZZZ had a bad dream last night”,

“she frequently has bad dreams”,

“ZZZZZ has suffered from bad dreams in the past”,

“ZZZZZ had a bad dream that she was underwater”,

“ he’s been having fewer bad dreams”

Exclude Negative mentions:

“she denied any bad dreams”,

“does not suffer from bad dreams”,

“no other PTSD symptoms such as bad dreams”,

“he said the experience was like a bad dream”,

“ZZZZZ compared his time in hospital to a bad dream”

Exclude Unknown mentions:

“she said it might have been a bad dream”,

“he woke up in a start, as if waking from a bad dream”,

“ZZZZZ couldn’t remember whether the conversation was just a bad dream”,

“doesn’t want to have bad dreams” Definitions: Search term(s): bad dream*

Evaluated Performance

Cohen's k = 100% (100 unannotated documents- 50 events/50 attachments, search terms ‘dream’ and ‘dreams’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 100%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Blunted Affect

Brief Description

Application to identify instances of blunted affect

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations include:

“his affect remains very blunted”,

“objectively flattened affect”,

“states that ZZZZZ continues to appear flat in affect”

Exclude Negative annotations:

“incongruent affect”,

“stable affect”,

“no blunted affect”

Exclude Unknown annotations:

“typical symptoms include blunted affect”,

“slightly flat affect”,

“relative shows flat affect” Definitions: Search term(s):

*affect*

blunt* [0 to 2 words in between] *affect*

flat [0 to 2 words in between] *affect*

restrict [ 0 to 2 words in between *affect*

*affect* [0 to 2 words in between] blunt

*affect* [0 to 2 words in between] flat

*Affect* [0 to 2 words in between] restrict

Evaluated Performance

Cohen's k = 100% (50 annotated documents - 25 events/24 attachments/1 mental health care plan). Instance level, i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 100% Recall (sensitivity / coverage) = 80% Patient level – Testing done on 30 random document, one document per patient. Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Circumstantiality

Brief Description

Application to identify instances of circumstantiality

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

“loose associations and circumstantiality”,

“circumstantial in nature”,

“some circumstantiality at points”,

“speech is less circumstantial”

Exclude Negative mentions:

“no signs of circumstantiality”,

“no evidence of circumstantial”

Exclude Unknown mentions:

“Such as a hypothetical cause of something else” Definition: Search term(s): *circumstan*

Evaluated Performance

Cohen's k = 100% (50 annotated documents - 25 events/25 attachments). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 94% Recall (sensitivity / coverage) = 92% Patient level – Testing done on 30 random document, one document per patient. Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Cognitive Impairment

10.3389/fdgth.2021.711941

Brief Description

Application to identify instances of cognitive impairment. The application allows to detect cognitive impairments related to attention, memory, executive functions, and emotion, as well as a generic cognition domain. This application has been developed for patients diagnosed with schizophrenia.

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Affirmed and relating to the patient and Negated/Irrelevant.

Output and Definitions

The output includes-

Affirmed mentions:

“patient shows attention problems (positive) “

“ZZZ does not show good concentration “

“ZZZ shows poor concentration “

“patient scored 10/18 for attention “

“ZZZ seems hyper-vigilant “

Exclude Negated and Irrelevant mentions:

“patient uses distraction technique to ignore hallucinations “

“attention seeking”

“patient needs (medical) attention”

“draw your attention to…” Definition- Search term(s): attention, concentration, focus, distracted, adhd, hypervigilance, attend to

Evaluated Performance

Cohen’s k

Cognition – 66% (Done on 3000 random documents)

Emotion – 84% (Done on 3000 random documents)

Executive function – 40% (Done on 3000 random documents)

Memory – 68% (Done on 3000 random documents)

Attention – 99% (Done on 2616 random documents). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 96% Recall (sensitivity / coverage) = 92% Instance level – Patient with an F20 diagnosis (testing done on 100 random documents). Precision (specificity / accuracy) = 78% Recall (sensitivity / coverage) = 70%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

DOI

Concrete Thinking

Brief Description

Application to identify instances of concrete thinking.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations include:

“text referring to ‘concrete thinking’”,

“speech or answers to questions being ‘concrete’”,

“the patient being described as ‘concrete’ without elaboration”,

“answers being described as concrete in cognitive assessments”,

“‘understanding’ or ‘manner’ or ‘interpretations’ of circumstances being described as concrete”

Exclude Negative annotations:

“no evidence of concrete thinking”

Exclude Unknown annotations:

“reference to concrete as a material (concrete floor, concrete house etc.)”

“no concrete plans”,

“delusions being concrete”,

“achieving concrete goals using concrete learning activities” Definitions: Search term(s): Concrete [word][word]think* think [word] [word] concret*

Evaluated Performance

Cohen's k = 83% (50 un-annotated documents - 25 events/25 attachments, search term ‘concrete*’) Instance level, i.e. for all specific mentions (testing done on 146 random documents):

Precision (specificity / accuracy) = 84% Recall (sensitivity / coverage) = 41% Patient level – Testing done on 30 random document, one document per patient. Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Delusions

Brief Description

Application to identify instances of delusions

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions include:

“paranoid delusions”,

“continued to express delusional ideas of the nature”

“no longer delusional- indicates past”

Exclude Negative mentions:

“no delusions”,

“denied delusions”

Exclude Unknown mentions:

“delusions are common” Definitions: Search term(s): *delusion*

Evaluated Performance

Cohen's k = 92% (50 un-annotated documents - 25 events/25 attachments, search term ‘delusion*’) Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 93% Recall (sensitivity / coverage) = 85% Patient level – Testing done on 30 random document, one document per patient. Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Derailment

Brief Description

Application to identify instances of derailment.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“he derailed frequently”,

“there was evidence of flight of ideas”,

“thought derailment in his language”‘speech no longer derailed’.

Exclude Negative annotations:

“no derailment”,

“erratic compliance can further derail her stability”

“no evidence of derailment”,

“without derailment”,

“without derailing”,

“no evidence of loosening of association, derailment or tangential thoughts”

Exclude Unknown annotations:

“train was derailed” Definitions: Search term(s): *derail*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘derail*’) Instance level, i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 88% Recall (sensitivity / coverage) = 95% Patient level – Testing done on 30 random document, one document per patient :Precision (specificity / accuracy) = 74%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Disturbed Sleep

Brief Description

Application to identify instances of disturbed sleep.

Development Approach

Development approach: Rule-Based.

Classes produced: Positive

Output and Definitions

The output includes-

Instances of disturbed sleep:

“complains of poor sleep”,

“poor sleep”,

“sleep disturbed”,

“sleep difficulty”,

“sleeping poorly”,

“not sleeping very well”,

“cannot sleep”,

“sleep pattern poor”,

“difficulties with sleep”,

“slept badly last couple of nights” Definitions: Search term(s): Not poor*, interrupt*, disturb*, inadequat*, disorder*, prevent*, stop*, problem* , difficult*, reduc*, less*, impair*, erratic*, unable*, worse*, depriv* [0-2 token] sleep* or slep* , little sleep, sleepless night, broken sleep, sleep intermittently, sleep* or slep* [0-2 token] not, poor*, interrupt*, disturb*, inadequat*, disorder*, prevent*, stop*, problem*, difficult*, reduc*, less*, impair*, erratic* ,unable*, worse*, depriv* .

Evaluated Performance

Cohen's k = 75% (50 un-annotated documents - 25 events/25 attachments, search term ‘*sleep*’ or ‘slept’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 88% Recall (sensitivity / coverage) = 68%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 2.0, Last updated:xx

Diurnal Variation

Brief Description

Application to identify instances of diurnal variation of mood

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

• ‘patient complaints of diurnal variation’

• ‘he reported diurnal variation in his mood’

• ‘Diurnal variation present’

• ‘some diurnal variation of mood present’

Exclude Negative examples:

• ‘no diurnal variation’

• ‘diurnal variation absent’

• ‘patient complaints of ongoing depression but no diurnal variation’

• ‘depressive symptoms present without diurnal variation’

Exclude Unknown examples:

• ‘diurnal variation could be a symptom of more severe depression’

• ‘we spoke about possible diurnal variation in his mood’

• ‘it was not certain if there were diurnal variation’ Definitions: Search term(s): diurnal variation

Evaluated Performance

Cohen's k = xx Instance level, i.e. for all specific mentions (Testing done on 100 Random Documents): Precision (specificity / accuracy) = 94% Recall (sensitivity / coverage) = 100%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Drowsiness

Brief Description

Application to identify instances of drowsiness

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive examples:

“ZZZZZ appeared to be drowsy”,

“She has complained of feeling drowsy”

Exclude Negative examples:

“He is not drowsy in the mornings”,

“She was quite happy and did not appear drowsy”,

“ZZZZZ denied any symptoms of drowsiness”,

“Negative annotations should be when the patient denies drowsiness, or is described as not drowsy etc”

Exclude Unknown examples:

“In reading the label (of the medication), ZZZZZ expressed concern in the indication that it might make him drowsy”,

“Monitor for increased drowsiness and inform for change in presentation”,

“risk of drowsiness, instructions to reduce medication if the patient becomes drowsy” Definitions: Search term(s): drows*

Evaluated Performance

Cohen's k = 83% 1000 un-annotated documents, search term ‘drows*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 80% Recall (sensitivity / coverage) =100%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Early Morning wakening

Brief Description

Application to identify instances of early morning wakening.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“ patient complaints of early morning awakening”,

“he reported early morning wakening”,

“Early morning awakening present”,

“there is still some early morning wakening”

Exclude Negative annotations:

“no early morning wakening”,

“early morning wakening absent”,

“patient complaints of disturbed sleep but no early morning awakening”,

“sleeps badly but without early morning wakening”

Exclude Unknown annotations:

“early morning awakening could be a symptom of more severe depression”,

“we spoke about how to deal possible early morning wakening”,

“it was not certain if there were occasions of early morning awakening” Definitions: Search term(s): early morning wakening

Evaluated Performance

Cohen's k = xx Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 95% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Echolalia

Brief Description

Application to extract occurrences where echolalia is present.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations include:

“no neologisms, but repeated what I said almost like echolalia”,

“intermittent echolalia”,

“some or less echolalia”

Exclude Negative annotations:

“no echolalia”,

“no evidence of echolalia”,

“cannot remember any echolalia or stereotyped utterances”

Exclude Unknown annotations:

“Echolalia is not a common symptom”,

“Include hypotheticals such as he may have some echolalia, evidence of possible echolalia” Definitions: Search term(s): *echola*

Evaluated Performance

Cohen's k = 88% (50 un-annotated documents - 25 events/25 attachments, search term ‘echola*’) Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 86% Patient level – Testing done on 30 random document, one document per patient, Precision (specificity / accuracy) = 74%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

10.1017/S0033291721004402

Elation

Brief Description

Application to identify instances of elation.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“mildly elated in mood”,

“elated in mood on return from leave”,

“she appeared elated and aroused”

Exclude Negative annotations:

“ZZZZZ was coherent and more optimistic/aspirational than elated throughout the conversation”,

“no elated behaviour" etc.

Exclude Unknown annotations:

“In his elated state there is a risk of accidental harm”,

“monitor for elation”,

“elation is a known side effect” and

“Statements were term is used in a list, not applying to patients (e.g. Typical symptoms include...)” Definitions: Search term(s): *elat*

Evaluated Performance

Cohen's k = 100% (50 annotated documents - 25 events/25 attachments) 1) Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 94% Recall (sensitivity / coverage) = 97% Patient level – Testing done on 30 random document, one document per patient. Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Emotional Withdrawal

Brief Description

Application to identify instances of emotional withdrawal

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

To any description of the patient being described as withdrawn or showing withdrawal but with the following exceptions (which are annotated as unknown):

• Alcohol, substance, medication withdrawal

• Withdrawal symptoms, fits, seizures etc.

• Social withdrawal (i.e. a patient described as becoming withdrawn would be positive but a patient described as showing ‘social withdrawal’ would be unknown – because social withdrawal is covered in another application).

• Thought withdrawal (e.g. ‘no thought insertion, withdrawal or broadcast’)

• Withdrawing money, benefits being withdrawn etc.

Negative and unknown annotations are restricted to instances where the patient is being described as not withdrawn and categorised as unknown. Definition: Search term(s): withdrawn

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘withdrawn’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 85% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Eye Contact (Catergorisation)

Brief Description

Application to identify instances of eye contact and determine the type of eye contact.

Development Approach

Development approach:Rule-Based

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions: the application successfully identifies the type of eye contact (as denoted by the keyword) in the context (as denoted by the contextstring)

e.g., keyword: ‘good’; contextstring: ‘There was good eye contact’

Negative mentions: the application does not successfully identifies the type of contact (as denoted by the keyword) in the context (as denoted by the contextstring). The keyword does not related to the eye contact

e.g., keyword: ‘showed’; contextstring: ‘showed little eye-contact’. Definitions: Search term(s): Eye *contact*

Keyword: the term describing the type of eye contact

ContextString: the context containing the keyword in its relation to eye-contact

Evaluated Performance

Cohen's k = xx Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 91%

Recall (sensitivity / coverage) = 80%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Fatigue

Brief Description

Application to identify symptoms of fatigue.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“ZZZZ has been experiencing fatigue”,

“fatigue interfering with daily activities”

Exclude Negative annotations:

“No mentions of fatigue”,

“her high levels of anxiety impact on fatigue”,

“main symptoms of dissociation leading to fatigue”

Exclude Unknown annotations:

“ZZZZ is undertaking CBT for fatigue”. Definitions: Search term(s): Fatigue, exclude ‘chronic fatigue syndrome’

Evaluated Performance

Cohen's k = xx Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 78% Recall (sensitivity / coverage) = 95%

Additional Notes

Run schedule – Monthly

Other Specifications

Version: xx, Last updated:xx

Flight of Ideas

Brief Description

Application to extract instances of flight of ideas.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“Mrs ZZZZZ was very elated with by marked flights of ideas”,

“marked pressure of speech associated with flights of ideas”,

“Some flight of ideas”.

Exclude Negative annotations:

“no evidence of flight of ideas”,

“no flight of ideas”

Exclude Unknown annotations:

“bordering on flight of ideas”

“relative shows FOI” Definitions: Search term(s): *flight* *of* *idea*

Evaluated Performance

Cohen's k = 96% (50 un-annotated documents - 25 events/25 attachments, search term ‘flight of’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 91% Recall (sensitivity / coverage) = 94% Patient level – Testing done on 30 random document, one document per patient.Precision (specificity / accuracy) = 72%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Fluctuation

10.1017/S0033291724001922

Brief Description

The purpose of this application is to determine if a mention of fluctuation within the text is relevant

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-Positive annotations:

“Mrs ZZZZZ’s mood has been fluctuating a lot”,

“suicidal thoughts appear to fluctuate”

Exclude Negative annotations:

“no evidence of mood fluctuation”,

“does not appear to have significant fluctuations in mental state”

Exclude Unknown annotations:

“unsure whether fluctuation has a mood component”,

“monitoring to see if fluctuations deteriorate”,

“his mother’s responsibility fluctuated”,

“is the person’s risk likely to fluctuate.. yes/no…”

Evaluated Performance

Cohen's k = xx Instance level, i.e. for all specific mentions (testing done on 100 random documents):

Precision (specificity / accuracy) = 87% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

DOI

Formal Thoughts Disorder

Brief Description

Application to extract occurrences where formal thought disorder is present.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“deteriorating into a more thought disordered state with outbursts of aggression”,

“there was always a degree thought disorder”,

“Include some formal thought disorder”

Exclude Negative annotations:

“no FTD”,

“no signs of FTD”,

“NFTD”

Exclude Unknown annotations:

“?FTD”,

“relative shows FTD”,

“check if FTD has improved”,

“used in a list, not applying to patient ‘typical symptoms include...” Definitions: Search term(s): *ftd*,*formal* *thought* *disorder*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘*ftd*, *formal**thought* *disorder*) Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 83% Recall (sensitivity / coverage) = 83% Patient level – Testing done on 50 random document, one document per patient: Precision (specificity / accuracy) = 72%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Grandiosity

Brief Description

Application to extract occurrences where grandiosity is apparent.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“ZZZZZ was wearing slippers and was animated elated and grandiose”,

“reduction in grandiosity”,

”No longer grandiose”

Exclude Negative annotations:

“No evidence of grandiose of delusions in the content of his speech”,

“no evidence of grandiose ideas”

Exclude Unknown annotations:

“his experience could lead to grandiose ideas” Definitions: Search term(s): *grandios*

Evaluated Performance

Cohen's k = 89% (50 un-annotated documents - 25 events/25 attachments, search term ‘grandio*’). Instance level: (testing done on 100 random documents)

Precision (specificity / accuracy) = 95% Recall (sensitivity / coverage) = 91% Patient level: Testing done on 30 random document, one document per patient,

Precision (specificity / accuracy) = 97%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Guilt

Brief Description

Application to identify instances of guilt.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“she then feels guilty/angry towards mum”,

“Being angry is easier to deal with than feeling guilty”,

“Include feelings of guilt with a reasonable cause and mentions stating”,

“no longer feels guilty”

Exclude Negative annotations:

“No feeling of guilt”,

“denies feeling hopeless or guilty”

Exclude Unknown annotations:

“he might be feeling guilty”,

“some guilt”,

“sometimes feeling guilty” Definitions: Search term(s): *guil*

Evaluated Performance

Cohen's k = 92% (50 un-annotated documents - 25 events/25 attachments, search term ‘guil*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 83% Recall (sensitivity / coverage) = 83% Patient level – All patients with primary diagnosis code of F32* and F33*.Testing done on 90 random document, one document per patient: Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Hallucinations (All)

Brief Description

Application to identify instances of hallucinations.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“her husband was minimising her hallucinations”,

“continues to experience auditory hallucinations”,

“doesn’t appear distressed by his hallucinations”,

“he reported auditory and visual hallucinations”,

“this will likely worsen her hallucinations”,

“his hallucinations subsided”,

“Neuroleptics were prescribed for her hallucinations”.

Exclude Negative annotations:

“denied any hallucinations”,

“no evidence of auditory hallucinations”,

“he reports it is a dream rather than hallucinations”,

“hears voices but denies command hallucinations”,

“did not report any further auditory hallucinations”,

“hallucinations have not recurred”,

“no longer appeared to have hallucinations”,

“has not had hallucinations for the last 4 months”,

“the hallucinations stopped”

Exclude Unknown annotations:

Statements contains probably/possibly/maybe/likely/unclear/unable to ascertain/unconfirmed reports of hallucinations/ experiencing hallucinations,

“pseudo(-) hallucinations”,

“hallucinations present?”,

“?hallucinations”,

“this is not a psychiatric symptom such as hallucinations”,

“abnormalities including hallucinations, derealisation etc”,

“rating scale including delusions, hallucinations, clinical domains”

“hallucinations is a sign of relapse”,

“it is unusual for hallucinations to present in this way”,

“CBT is effective for hallucinations” Definitions: Search term(s): hallucinat*

Evaluated Performance

Cohen's k = 83% (100 un-annotated documents - 50 events/50 attachments, search term ‘hallucinat*’). Instance level, i.e. for all specific mentions (testing done on 100 dandom documents): Precision (specificity / accuracy) = 84% Recall (sensitivity / coverage) = 98%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 2.0, Last updated:xx

DOI

Hallucinations - Auditory

Brief Description

Application to identify instances of auditory hallucinations non-specific to diagnosis.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“Seems to be having olfactory hallucination”,

“in relation to her tactile hallucinations”

Exclude Negative annotations:

“denies auditory, visual, gustatory, olfactory and tactile hallucinations at the time of the assessment”,

“denied tactile/olfactory hallucination”

Exclude Unknown annotations:

“possibly olfactory hallucinations” Definitions: Search term(s): auditory hallucinat*

Evaluated Performance

Cohen's k = 96% (50 un-annotated documents - 25 events/25 attachments, search term ‘auditory’ or ‘halluc*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 80% Recall (sensitivity / coverage) = 84%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Hallucinations - Olfactory Tactile Gustatory (OTG)

Brief Description

Application to extract occurrences where auditory hallucination is present. Auditory hallucinations may be due to a diagnosis of psychosis/schizophrenia or may be due to other causes, e.g. due to substance abuse.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“seems to be having olfactory hallucinations”,

“in relation to her tactile hallucinations”

Exclude Negative annotations:

“denies auditory, visual, gustatory, olfactory and tactile hallucinations at the time of the assessment”,

“denied tactile/olfactory hallucinations”

Exclude Unknown annotations

“possibly olfactory hallucinations” Definitions: Search term(s):

*olfactory* [0-10 words in between] *hallucin*

*hallucin* [0-10 words in between] *olfactory*

*gustat* [0-10 words in between] *hallucin*

*hallucin* [01-10 words in between] *gustat*

*tactile* [0-10 words in between] *hallucin*

*hallucin* [0-10 words in between] *tactile*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘olfact*’ or ‘gustat*’ or ‘tactile’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 78% Recall (sensitivity / coverage) = 68% Patient level – Testing done on 50 random document, one document per patient, Precision (specificity / accuracy) = 86%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Hallucinations - Visual

Brief Description

Application to extract occurrences where visual hallucination is present. Visual hallucinations may be due to a diagnosis of psychosis/schizophrenia or may be due to other causes, e.g. due to substance abuse.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“Responding to visual hallucination”,

“Experiencing visual hallucination”,

“history of visual hallucination”,

“distressed by visual hallucination”

Exclude Negative annotations:

“denied any visual hallucination”,

“not responding to visual hallucination”,

“no visual hallucination”,

“no current visual hallucination (with no reference to past)”

Exclude Unknown annotations:

“if/may/possible/possibly/might have visual hallucinations”,

“monitor for possible visual hallucination” Definitions: Search term(s): visual hallucinat*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘visual’ and ‘halluc*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 91% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Helplessness

Brief Description

Application to identify instances of helplessness.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“Ideas of helplessness secondary to her physical symptoms present”,

“ideation compounded by anxiety and a sense of helplessness”

Exclude Negative annotations:

“denies uselessness or helplessness”,

“no thoughts of hopelessness or helplessness”.

Exclude Unknown annotations:

“there a sense of helplessness”,

“helplessness is a common symptom” Definitions: Search term(s):*helpless*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘helpless*’).Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 93% Recall (sensitivity / coverage) = 86% Patient level – All patients with primary diagnosis of F32* or F33* in a structured field (random sample of 30 patients, one document per patient): Precision (specificity / accuracy) = 90%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Hopelessness

Brief Description

Application to identify instances of hopelessness

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“feeling very low and hopeless”,

“says feels hopeless”

Exclude Negative annotations:

“denies hopelessness”,

“no thoughts of hopelessness or helplessness”

Exclude Unknown annotations:

“there a sense of hopelessness”,

“hopelessness is a common symptom” Definitions: Search term(s):*hopeles*

Evaluated Performance

Cohen's k = 90% (50 un-annotated documents - 25 events/25 attachments, search term ‘hopeless*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 90% Recall (sensitivity / coverage) = 95% Patient level – All Patients with a primary diagnosis of F32* or F33* in a structured field (one document per patient) Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Hostility

Brief Description

Application to identify instances of hostility.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“increased hostility and paranoia”,

“she presented as hostile to the nurses”

Exclude Negative annotations:

“not hostile”,

“denied any feelings of hostility”

Exclude Unknown annotations:

“he may become hostile”,

“hostility is something to look out for” Definitions: Search term(s): *hostil*

Evaluated Performance

Cohen's k = 94% (50 un-annotated documents - 25 events/25 attachments, search term ‘hostil*’). Instance level, i.e. for all specific mentions (testing done on 100 random documents): Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 94% Patient level – Random sample of 30 patients (one document per patient): Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Insomnia

Brief Description

Application to identify instances of insomnia.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“initial insomnia”,

“contributes to her insomnia”,

“problems with insomnia”,

“this has resulted in insomnia”,

“this will address his insomnia”

Exclude Negative annotations:

“no insomnia”,

“no evidence of insomnia”,

“not insomniac2

Exclude Unknown annotations:

“Typical symptoms include insomnia”,

“might have insomnia”,

“?insomnia”,

“possible insomnia”,

“monitor for insomnia”,

“insomnia has improved” Definitions: Search term(s): *insom*

Evaluated Performance

Cohen's k = 94% (50 un-annotated documents - 25 events/25 attachments, search term ‘insomn*’). Instance level, Random sample of 100 random documents:Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 94% Patient level – All patients with primary diagnosis of F32* or F33* in a structured field, random sample of 50 patients (one document per patient): Precision (specificity / accuracy) = 94%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

10.1002/gps.6097

Irritability

Brief Description

Application to identify instances of irritability.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“can be irritable”,

“became irritable”,

“appeared irritable”,

“complained of feeling irritable”

Exclude Negative mentions:

“no evidence of irritability”,

“no longer irritable”,

“no sign of irritability”

Exclude Unknown annotations:

“irritable bowel syndrome”,

“becomes irritable when unwell”,

“can be irritable if …[NB some ambiguity with positive ‘can be’ mentions, although linked here with the ‘if’ qualifier]”,

“less irritable” Definitions: Search term(s): *irritabl*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘irritabil*’ or ‘irritabl*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 100% Recall (sensitivity / coverage) = 83%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Loss of Coherence

Brief Description

Application to identify instances of incoherence or loss of coherence in speech or thinking.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“patient was incoherent”,

“his speech is characterised by a loss of coherence”

Exclude Negative annotations:

“patient is coherent”,

“coherence in his thinking”

Exclude Unknown annotations:

“coherent discharge plan”,

“could not give me a coherent account”,

“more coherent”,

“mood was coherent with speech”

“a few instances where coherence/incoherence was part of a heading or question” Definitions: Search term(s): coheren*, incoheren*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘incoheren*’). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 98%Recall (sensitivity / coverage) = 95% Patient level – All patients with primary diagnosis code F32* or F33* in a structured field, random sample of 50 (one document per patient). Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Low energy

Brief Description

Application to identify instances of low energy.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“low energy”,

“decreased energy”,

“not much energy”,

“no energy”

Exclude Negative annotations:

“no indications of low energy”,

“increased energy”

Exclude Unclear annotations:

“..., might be caused by low energy”,

“monitor for low energy”,

“energy levels have improved”,

“fluoxetine reduces her energy”,

“some energy bars” Definitions: Search term(s): *energy*

Evaluated Performance

Cohen's k = 95% (50 un-annotated documents - 25 events/25 attachments, search term ‘energ*’). IInstance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 82% Recall (sensitivity / coverage) = 85% Patient level – All patients with primary diagnosis code F32* or F33* in a structured field, random sample of 50 (one document per patient). Precision (specificity / accuracy) = 76%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Mood instability

Brief Description

This application identifies instances of mood instability.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“she continues to have frequent mood swings”,

“expressed fluctuating mood”

Exclude Negative annotations:

“no mood fluctuation”

“no mood unpredictability”,

“denied diurnal mood variations”

Exclude Unknown annotations:

“she had harmed others in the past when her mood changed”,

“tried antidepressants in the past but they led to fluctuations in mood”,

“no change in mood”,

“her mood has not changed and she is still depressed” Definitions: Search term(s):

Chang* [0-2 words in between] *mood*

Extremes [0-2 words in between] *mood*

Fluctuat* [0-2 words in between] *mood*

Instability [0-2 words in between] *mood*

*labil* [0-2 words in between] mood

Rapid cycling [0-2 words in between] mood

*swings* [0-2 words in between] mood

*unpredictable* [0-2 words in between] mood

unsettled [0-2 words in between] mood

unstable [0-2 words in between] mood

*variable* [0-2 words in between] mood

*variation* [0-2 words in between] mood

*volatile* [0-2 words in between] mood

mood [0-2 words in between] chang*

mood [0-2 words in between] Extremes

mood [0-2 words in between] fluctuat*

mood [0-2 words in between] Instability

mood [0-2 words in between] *labil*

mood [0-2 words in between] Rapid cycling

mood [0-2 words in between] *swings*

mood [0-2 words in between] *unpredictable*

mood [0-2 words in between] Unsettled

mood [0-2 words in between] Unstable

mood [0-2 words in between] *variable*

Evaluated Performance

Cohen's k = 91% (50 un-annotated documents - 25 events/25 attachments, search term ‘mood’). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 100% Recall (sensitivity / coverage) = 70% Patient level – All patients with random sample of 50 (one document per patient). Precision (specificity / accuracy) = 72%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Mutism

Brief Description

Application to identify instances of mutism.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“she has periods of 'mutism”,

“he did not respond any further and remained mute”

Exclude Unknown annotations:

“her mother is mute”,

“muted body language” Definitions: Search term(s): *mute* , *mutism*

Evaluated Performance

Cohen's k = 100% (50 un-annotated documents - 25 events/25 attachments, search term ‘mut*’). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 91% Recall (sensitivity / coverage) = 75% Patient level – All patients with random sample of 30 (one document per patient).

Precision (specificity / accuracy) = 93%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Negative Symptoms

Brief Description

Application to identify instances of negative symptoms.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“she was having negative symptoms”,

“diagnosis of schizophrenia with prominent negative symptoms”

Exclude Negative annotations:

“no negative symptom”,

“no evidence of negative symptoms”

Exclude Unknown annotations:

“symptoms present?”,

“negative symptoms can be debilitating” Definitions: Search term(s): *negative* *symptom*

Evaluated Performance

Cohen's k = 85% (50 annotated documents - 25 events/25 attachments). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 86% Recall (sensitivity / coverage) = 95% Patient level – All patients with random sample of 30 (one document per patient).

Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Nightmares

Brief Description

Application to identify instances of nightmares.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“she was having nightmares”,

“unsettled sleep with vivid nightmares”

Exclude Negative annotations:

“no nightmares”,

“no complains of having nightmares”

Exclude Unknown annotations:

“it’s been a nightmare to get this arranged”,

“a nightmare scenario would be….” Definitions: Search term(s): nightmare*

Evaluated Performance

Cohen's k = 95% (50 un-annotated documents - events, search term 'nightmare*’). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 100%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Obsessive Compulsive Symptoms

Brief Description

Application to identify obsessive-compulsive symptoms (OCS) in patients with schizophrenia, schizoaffective disorder or bipolar disorder

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations of OCS include

• Text states that patient has OCD features/symptoms

• Text states that patient has OCS

• Text including hoarding, which is considered part of OCS, regardless of presence or absence of specific examples

• Text states that patient has either obsessive or compulsive or rituals or Yale-Brown Obsessive Compulsive Scale (YBOCS) [see keywords below] and one of the following:

o Obsessions or compulsions are described as egodystonic

o Intrusive, cause patient distress or excessive worrying/anxiety

o Patient feels unable to stop obsessions or compulsions

o Patient recognises symptoms are irrational or senseless

• Clinician provides specific YBOCS symptoms

• Text reports that patient has been diagnosed with OCD by clinician

Negative annotations of OCS include

• Text makes no mention of OCS

• Text states that patient does not have OCS • Text states that patient has either compulsions or obsessions, not both, and there is no information about any of the following:

o Patient distress

o Obsessive or compulsive symptoms described as egodystonic

o Inability to stop obsessions or compulsions

o Description of specific compulsions or specific obsessions

o Patient insight

• Text states that non-clinician observers (e.g., patient or family/friends) believe patient has obsessions or compulsions without describing YBOCS symptoms.

• Text includes hedge words (i.e., possibly, apparently, seems) that specifically refers to OCS keywords

• Text includes risky, risk-taking or self-harming behaviours

• Text includes romantic or weight-related (food-related) words that modify OCS keywords

Evaluated Performance

Cohen's k = 80% (600 annotated documents for interrater reliability). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 72%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Paranoia

Brief Description

Application to identify instances of paranoia. Paranoia may be due to a diagnosis of paranoid schizophrenia or may be due to other causes, e.g. substance abuse.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“vague paranoid ideation”,

“caused him to feel paranoid”

Exclude Negative annotations:

“denied any paranoia”,

“no paranoid feelings”

Exclude Unknown annotations:

“relative is paranoid about me”,

“paranoia can cause distress” Definitions: Search term(s): *paranoi*

Evaluated Performance

Cohen's k = 92% (100 annotated documents - 25 events/69 attachments/1 mental state formulation/3npresenting circumstances/2 progress notes). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 86% Recall (sensitivity / coverage) = 94% Patient level – All patients, random sample of 50 (one document per patient), Precision (specificity / accuracy) = 82%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Passivity

Brief Description

Application to identify instances of passivity.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive annotations:

“patient describes experiencing passivity”,

“patient has experienced passivity in the past but not on current admission”

Exclude Negative annotations:

"denies passivity",

"no passivity".

Exclude Unknown annotations:

“passivity could not be discussed”,

“possible passivity requiring further exploration”,

“unclear whether this is passivity or another symptom” Definitions: Search term(s):passivity

Evaluated Performance

Cohen's k = 83% (438 unannotated documents – search term ‘passivity’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 89% Recall (sensitivity / coverage) = 100%

Additional Notes

Run schedule – On request

Other Specifications

Version 1.0, Last updated:xx

Presecutory Ideation

Brief Description

Application to identify instances of ideas of persecution.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“she was having delusions of persecution”,

“she suffered persecutory delusions”,

“marked persecutory delusions”,

“paranoid persecutory ideations”,

“persecutory ideas present”

Exclude Negative annotations:

“denies persecutory delusions”,

“he denied any worries of persecution”,

“no persecutory delusions”,

“no delusions of persecution”,

“did not report persecutory ideas”,

“no persecutory ideation present”

Exclude Unknown annotation:

“this might not be a persecutory belief”,

“no longer experiencing persecutory delusions” Definitions: Search term(s): [Pp]ersecu*

Evaluated Performance

Cohen's k = 91% (50 un-annotated documents - 25 events/25 attachments, search term ‘persecut*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 80% Recall (sensitivity / coverage) = 96%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poor Appetite

Brief Description

Application to identify instances of poor appetite (negative annotations).

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations( applied to adjectives implying a good or normal appetite):

“Appetite fine”

“Appetite and sleep OK”,

“Appetite reasonable”,

“appetite alright”,

“sleep and appetite both preserved”

Exclude Negative annotations:

“loss of appetite”,

“reduced appetite”,s

“decrease in appetite”,

“not so good appetite”,

“diminished appetite”,

“lack of appetite”

Exclude Unknown annotations:

“Loss of appetite as a potential side effect”,

“as an early warning sign, as a description of a diagnosis (rather than patient experience)”, “describing a relative rather than the patient, ‘appetite suppressants’” Definitions: Search term(s): *appetite* within the same sentence of *eat* *well*, *alright*, excellent*, fine*, fair*, good*, healthy, intact*, not too bad*, no problem, not a concern*.

Evaluated Performance

Cohen’s k = 91% (Done on 50 random documents). Instance level, Random sample of 100 random documents: Precision (specificity / accuracy) = 83%

Recall (sensitivity / coverage) = 71% Patient level – All patients with primary diagnosis code F32* or F33* in a structured field, random sample of 30 (one document per patient), Precision (specificity / accuracy) = 97%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poor Concentration

Brief Description

Application to identify instances of poor concentration.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“my concentration is still poor”,

“she found it difficult to concentrate”,

“he finds it hard to concentrate”

Exclude Negative annotations:

“good attention and concentration”,

“participating well and able to concentrate on activities”

“concentrate is adequate or reasonable”

Exclude Unknown annotations:

“‘gave her a concentration solution”,

“talk concentrated on her difficulties”,

“urine is concentrated”, “he is able to distract himself by concentrating on telly”. Definitions: Search term(s): *concentrat*

Evaluated Performance

Cohen's k = 95% (100 annotated documents – 45 attachments/3 CAMHS events/1 CCS correspondence/35

mental state formulation/1POSProforma/10 ward progress note). Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 84% Recall (sensitivity / coverage) = 60% Patient level – All patients with primary diagnosis code F32* or F33* in a structured field, random sample of 50 (one document per patient), Precision (specificity / accuracy) = 76%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poor Eye Contact

Brief Description

Application to identify instances of poor eye contact.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes-

Positive mentions:

“looked unkempt, quiet voice, poor eye contact”,

“eye contact was poo”,

“she refused eye contact”,

“throughout the conversation she failed to maintain eye contact”,

“unable to engage in eye contact”,

“eye contact was very limited”,

“no eye contact and constantly looking at floor”

Exclude Negative mentions:

“good eye contact”,

“he was comfortable with eye contact”,

“she made eye contact whilst talking”,

“excessive eye contact was made throughout our conversation”,

“ZZZZZ made occasional eye contact with me”,

“eye contact was inconsistent”,

“Mr ZZZZZ made reasonable eye contact”,

“low voice, average eye contact”

Exclude Unknown mentions:

“she showed increased eye contact”,

“I noticed reduced eye contact today” Definitions: Search term(s): Available on request

Evaluated Performance

Cohen’s k = 92% (100 annotated documents). Patient level – All patients, random sample of 100 Random Documents (one document per patient) Precision (specificity / accuracy) = 81% Recall (sensitivity / coverage) = 65%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Poor Insight

10.1136/bmjopen-2019-028929

Brief Description

Applications to identify instances of poor insight.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotation:

(An instance is classed as positive if the patient’s insight is minimal or absent)

“Lacking/ Lack of insight”

“Doesn’t have insight”

“No/ None insight”

“Poor insight”

“Limited insight”

“Insightless”

“Little insight”

Exclude Negative annotations (An instance is classed as negative if the patient displays a moderate or high degree of insight into their illness).

“Clear insight”

“Had/ Has insight”

“Improving insight”

“Partial insight”

“Some insight”

“Good insight”

“Insightful”

“Present insight” Exclude Unknown annotation:

“There is a lengthy and unclear description of the patient’s insight, without a final, specific verdict”

“Insight was not assessed”

“The word ‘insight’ is not used in a psychiatry context, rendering it irrelevant”

“The record does not refer to the patient’s current level of insight, perhaps mentioning predicted/ previous levels instead”

“It doesn’t contain the above keywords, despite the general conclusion that can be drawn from it, as this would decrease the overall accuracy of the app”

“Lack of insight not suggestive of psychotic illness, e.g. ‘lack of insight into how his drinking affects his son’ or ‘lack of insight into how she repeats the same cycles with romantic partners” Definitions: Search term(s): insight

Evaluated Performance

Cohen’s k = 88% (50 un-annotated documents - 25 events/25 attachments, search term ‘insight*’). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 87% Recall (sensitivity / coverage) = 70% Patient level – Random sample of 50 (one document per patient)

Precision (specificity / accuracy) = 83%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Poor Motivation

Brief Description

This application aims to identify instances of poor motivation.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive’ annotations:

“poor motivation”,

“unable to motivate’ self”,

“difficult to motivate’ self”,

“struggling with motivation”

Exclude Negative annotations:

“patient has good general motivation”,

“participate in alcohol rehabilitation”,

“improving motivation”

Exclude Unknown annotations:

“tasks/groups designed for motivation”,

“comments about motivation but not clearly indicating whether this was high or low”,

“plans to ascertain motivation levels”,

“other use of the word (e.g. ‘racially motivated’)” Definitions: Search term(s): Motivate* with in the same sentence of lack*, poor, struggle*, no

Evaluated Performance

Cohen's k = 88% (50 un-annotated documents - 25 events/25 attachments, search term ‘motiv*’). Instance level, Random sample of 100 Random Documents:

Precision (specificity / accuracy) = 85% Recall (sensitivity / coverage) = 45% Patient level – Random sample of 30 (one document per patient) Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Poverty Of Speech

Brief Description

Application to identify poverty of speech.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

“he continues to display negative symptoms including blunting of affect, poverty of speech”,

“he does have negative symptoms in the form of poverty of speech”

“less poverty of speech”

Exclude Negative annotations:

“no poverty of speech”,

“poverty of speech not observed”

Exclude Unknown annotations:

“poverty of speech is a common symptom of…, “

“?poverty of speech” Definitions: Search term(s): speech within the same sentence of poverty, impoverish.

Evaluated Performance

Cohen's k = 100% (50 annotated documents - 12 events/32 attachments/5 CCS_correspondence, 1 discharge notification summary). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 87% Recall (sensitivity / coverage) = 85% Patient level – Random sample of 30 (one document per patient), Precision (specificity / accuracy) = 87%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

DOI

Poverty Of Thought

Brief Description

Application to identify instances of poverty of thought.

Development Approach

Development approach: Machine-learning.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive mentions:

“poverty of thought was very striking”,

“evidence of poverty of thought”,

“some poverty of thought”

Exclude Negative mentions:

“no poverty of thought”,

“no evidence of poverty of thought”

Exclude Unknown mentions:

“poverty of thought needs to be assessed”,

“poverty of thought among other symptoms” Definitions: Search term(s): *poverty* *of* *thought*

Evaluated Performance

Cohen's k = 90% (50 annotated documents). Instance level, Random sample of 100 Random Documents: Precision (specificity / accuracy) = 95%

Recall (sensitivity / coverage) = 93% Patient level – Random sample of 30 (one document per patient), Precision (specificity / accuracy) = 73%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Psychomotor Activity (Catergorisation)

Brief Description

Application to identify instances of psychomotor activity and determine the level of activity

Development Approach

Development approach: Rule-Based.

Classification of past or present symptom: Both.

Classes produced: Positive

Output and Definitions

The output includes- Positive annotations:

Positive/correct mentions identifies the level of psychomotor activity (as denoted by the keyword) in the context (as denoted by the contextstring). In addition, psychomotor_activity column correctly states whether the reference to abnormal levels of psychomotor activity in the contextstring.

For example: Keyword: ‘psychomotor agitation’; Contextstring: ‘patient showed psychomotor agitation’;

Negativity: ‘No’; psychomotor_activity: ‘psychomotor agitation’

Negative/incorrect/irrelevant mentions do not successfully identify the level of activity (as denoted by the keyword) in the context (as denoted by the contextstring). Or an instance of psychomotor activity is noted as negated.

For example: Keyword: ‘psychomotor activity’; Contextstring: ‘normal psychomotor activity’; Negativity: ‘yes’;psychomotor_activity’: ‘psychomotor activity’

Keyword: ‘psychomotor activity’; Contextstring: ‘change in psychomotor activity’; Negativity: ‘yes’; psychomotor_activity: ‘psychomotor activity’

Evaluated Performance

Cohen's k = xx Instance level, Random sample of 100 Random Documents:Precision (specificity / accuracy) = 92% Recall (sensitivity / coverage) = 92%

Additional Notes

Run schedule – Monthly

Other Specifications

Version 1.0, Last updated:xx

Smell