Peeking into the Mind’s Eye: Detecting Drowsiness With Intelligent Eyeglasses

You have waked me too soon; I must slumber again;

as the door on its hinges, so he on his bed,

turns his sides, and his shoulders and his heavy head.

—Issac Watts¹

Early electronic recordings of horizontal and vertical eye movements (EMs) or saccades associated with the conscious activities of visual imagination and event recollection were conducted 90 years ago.^2,3 However, decades elapsed before electrooculographic (EOG) features that characterized diminished consciousness associated with sleep onset were identified.⁴ Cyclic variations observed in simultaneous electroencephalographic (EEG) and electromyographic (EMG) activity eventually led to the creation of formal rules for scoring human sleep.⁵ Various features used to quantify EOG activity appear in Table 1.

Table 1. Common EOG Features
Feature	Operational Definition	Measurement Units
Blinking Activity	Rate of short-duration, high amplitude pulses primarily in vertical channel	Blinks/min, microvolts, respectively
Saccadic Activity	Rate, amplitude and velocity of left-right or up-down EMs in horizontal or vertical channels	EMs/sec, microvolts, degrees/sec, respectively
Saccadic Duration	Reciprocal of EMs/sec in horizontal channel	Milliseconds
Fixation Duration	Time during which eye position remains at baseline level in horizontal channel	Milliseconds

Defining Drowsiness

“A Technologist’s Handbook” for scoring human sleep relies on the bioelectric measures of EEG, EMG and EOG to discern drowsiness, defined as the transition from waking behaviors to sleep onset.⁶ Common behavioral correlates of drowsiness include reduced psychomotor activity, reading attempts interrupted by vacant gazes indicative of prolonged fixations or watching television while in “relaxed, couch potato” mode.^6-8 Specific EOG features defining wakefulness are frequent blinking, horizontal reading EMs and irregular, fast EMs coupled with increased muscle tone in the EMG. Specific EOG features defining sleep onset are the replacement of waking EM activity by slow EMs (SEMs) coupled with decreased muscle tone relative to epochs with full electrographic arousals (see Fig 1). A noteworthy point: drowsiness is not interchangeable with fatigue, which presents as weariness, weakness or depleted energy.⁹

Fig 1.  A 30-second epoch of sleep shows frontal, central and occipital brain activity in the EEG, left and right EMs in the EOG, and chin muscle tone in the EMG. Conjugate SEMs (appearing as out-of-phase deflections) indicate drowsiness. Image from “A Technologist’s Handbook.” ⁶

In contrast to bioelectric measures, subjective inventories such as the Stanford Sleepiness Scale assess sleepiness based on a respondent’s agreement at the time of assessment with descriptive statements ranging from “feeling active, vital, alert or wide awake” to “almost in reverie, sleep onset soon [or] lost struggle to remain awake.”⁹ Other assessments of sleepiness include the Epworth Sleepiness Scale, the Sleep-Wake Activity Inventory and the Visual Analog Scale; details of which are available elsewhere in the clinical sleep literature.⁹

EOG Acquisition and Feature Extraction

Exponential progress in microelectronic engineering has occurred since individuals designed and assembled their own EOG/EEG monitoring devices before publishing their projects in journals and magazines.^10-12 Modern EOG acquisition systems filter, amplify and subsequently digitize analog data. This processing is functionally the same as what occurs in digital polysomnography (PSG).¹³ However, wearable EOG relies on dry conductive pads for signal monitoring in place of conductive, gel-filled electrodes used in sleep studies. These electrodes are designed for long-term and unobtrusive monitoring during activities of daily living.¹⁴

Computational scientists have advanced feature-extracting algorithms beyond the basic eye position and EM velocity analyses reported almost half a century ago to identify blinks, saccades and fixations. The analysis of position identified these features by the amplitude of signal deflections using biocalibrations requiring eye closures, left to right or up and down EMs and straight-ahead gazes.¹⁵ Feature extractions based on velocity used algorithms to calculate instantaneous changes in position over time. Biocalibration procedures permitted the identification of high velocity eyelid closures while fixations were identified by a near-zero velocity with respect to higher saccadic velocities.¹⁶ The mathematical physicist Ingrid Daubechies is an award-winning innovator in the signal processing community. Her work, which involves extracting patterns from digital data, was developed at New York University’s Courant Institute of Mathematical Sciences and can be applied to physiologic signals.¹⁷

Product Development

A patent application for intelligent glasses filed by two inventors at The Research Foundation of the State University at Binghamton, New York, provides some information on the functionality of this concept.¹³ A frame-embedded acquisition system relies on the detection of EOG features, and by following the extraction of these features using Daubechies’ algorithms, values for specific EOG features are calculated depending on the needs of the wearer.

The Belgian research and development organization Imec International has pursued the development of intelligent eyewear by integrating EOG and EMG data acquisition into fashionable reading glasses (see Fig 2). Imec designed their eyewear to improve human-computer interactions and understand the effects of degenerative neurologic disorders such as Alzheimer’s and Parkinson’s diseases on the oculomotor system.

 

Fig 2.  (Left) This proof-of-concept prototype senses horizontal and vertical EMs using  dry contact electrodes attached to the left and right temples and each nose pad. A bridge-mounted electrode between the rims serves as a common reference. The temple tip  houses a battery to power signal amplifiers and a Bluetooth transmitter in the other tip  streams signals to a smart phone for analysis.¹⁸ (Right) A 1950s-era, dual channel eye  position recorder is pictured for size comparison.¹⁰ Images from  www.Imec.com and www.WorldRadioHistory.com, respectively.

In 2015, the Japanese firm Jins Meme began marketing intelligent glasses. The firm’s alertness program for vehicle drivers receives data from an embedded acquisition system through electrodes on the frame’s nose and bridge. Alertness is determined by changes in saccadic and blinking responses (see Fig 3).

Fig 3. EOG biocalibration of binocular saccadic amplitudes includes seven, single-eye winks appearing in red ink. Data points are streamed to a digital device for automated feature extraction and scoring. Image adapted from www.jins-meme.com.

Quality Assurance

Technical Challenges

Intelligent glasses that automatically detect sleep onset depend on the accurate and reliable identification of EOG features — accuracy being the ability to identify these features using standardized rules for sleep scoring while reliability is the consistent repetition of this ability.

Reports of low frequency baseline drifts have plagued EM recordings, at least since the era of direct coupled (DC), vacuum tube amplifiers.^19,20 These artifacts, later attributed to sweating and electrode polarization, can sabotage SEM detection and confound calculations of fixation duration.²¹ However, the use of capacitance-coupled amplifiers — now termed AC amplifiers — with low frequency filters can preserve the channel baseline.^{6, 20} Secondly, physical movements can saturate amplifiers and obscure the EOG, which makes reliable record scoring nearly impossible. Consequently, an eyeglass frame should maintain electrode placements securely on the wearer’s face.

Accuracy can be improved by supplementing EOG data with data from other non-invasive physiologic measures. For example, Imec’s intelligent glasses include EMG monitoring obtained by selective bandpass filtering of signals from EOG electrodes. This approach produces a record of muscle tone from the wearer’s temples. Another example is the manufacturer’s use of an embedded, triaxial accelerometer to detect head tilts in any direction that identify postural changes associated with drowsiness.

Medical and Environmental Issues

Oculomotor dysfunction secondary to neuro-degenerative diseases can distort saccadic amplitudes because of limited movements of one or both eyes.²² Furthermore, changes in ambient lighting affect corneo-retinal potentials. The resulting change in saccadic amplitudes are due to differences in the rate of metabolic reactions that occur in light-adapted versus dark-adapted eyes.²² Intelligent eyewear needs to accommodate these situations.

Conclusion

Wearable EM monitoring equipment relying on EOG signals is more than a trendy fashion statement on a social networking website. On-the-job performance deficits resulting from drowsiness due to night and shift work are a public health concern. Increasing numbers of employees working nontraditional hours have contributed to the need for intelligent eyewear that alerts the wearer of drowsiness. Product development has benefited from nearly a century of advancements in biodata acquisition, wireless signal transmission and pattern recognition algorithms.

Obvious benefits of intelligent glasses include their compactness and comfort level compared to EEG and EOG-EEG prototypes requiring multiple electrode placements on the wearer’s head.^23,24 Furthermore, the use of dry contact sensors embedded in eyeglass frames, which typically require minimal skin prepping, are an improvement over adhesive electrodes that are hard-wired to signal amplifiers. Lastly, biodata streamed to digital devices for real-time analysis permit automatic software updates that enhance artifact rejection and improve feature extraction. Considering these benefits, intelligent glasses promise to be a high-demand item in the sleep technology devices market. However, further testing at job sites and during activities of daily living is recommended.

References

1. Watts I. The sluggard. In: De la Mare W. Behold the Dreamer! London: Faber and Faber. 1984: 198-9. 
2. Jacobson E. Electrical measurements of neuromuscular states during mental activities: I. Imagination of movement involving skeletal muscle. American J Physiology. 1930;91:567-608.
3. Jacobson E. Electrical measurements of neuromuscular states during mental activities: II. Visual imagination and Recollection. 1930;94:22-34.
4. Loomis AL, Harvey EN, & Hobart GA III. Cerebral states during sleep, as studied by human brain potentials. J Experimental Psychology. 1937;21(2):127-44.
5. Rechschaffen A, Kales A, eds. A Manual of Standardized Terminology, Techniques and Scoring Systems for Sleep Stages of Human Subjects. Los Angeles, CA: UCLA, 1968.
6. A Technologist’s Handbook for Understanding and Implementing the AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technological Specifications. Westchester, IL: AASM, 2009:13. For recent scoring rule updates see: Whitmore H, Brooks R. Adult sleep scoring. In: Mattice C, Brooks R, Leo-Chiong T. Fundamentals of Sleep Technology, 3^rd ed. Philadelphia, PA: Wolters Kluwer. 2020:457-77.
7. Stern JA, Bremer DA, & McClure J. Analysis of eye movements and blinks during reading: Effects of Valium. Psychopharmacologia. 1974;40:171-5.
8. Pigeon WR, Sateia MJ, & Ferguson RJ. Distinguishing between excessive daytime somnolence and fatigue toward improved detection and treatment. J Psycho-somatic Res. 2003;54:61-9.
9. Dolan DC & Rosenthal L. Subjective evaluation of sleepiness. In: Butkov N and Lee-Chiong T, eds. Fundamentals of Sleep Technology, 2^nd edition. Philadelphia, PA: Wolters Kluwer, 2007:421.
10. Powsner ER & Lion KS. Testing eye muscles. Electronics. 1950:23(March):96-99.
11. Shackel B, Sloan RC, & Warr HJJ. Detector plots eye movements. Electronics: Engineering edition. 1958;31(5):36-9.
12. Waite M. Build an alpha wave brain feedback monitor. Popular Electronics. 1973;3(1):40-5.
13. Jin Z & Laslo S. Wearable head-mounted, glass-style computing devices with EOG acquisition and analysis for Human-computer interfaces. US Patent Application. 2015. Accessed on 07/22/2020 from: http://www.freepatentsonline.com/9955895.html
14. Gargiulo G et al. A new EEG recording system for passive dry electrodes. Clinical Neuro-physiology. 2010;121(5):686-93.
15. Troelstra A & Garcia CA. Computer automated measurement of eye movement parameters with applications to electrooculography and nystagmus movements. Computer Programs in Biomedicine. 1974;3:231-36.
16. Cheng M & Outerbridge JS. Inter-saccadic interval analysis of vestibular nystagmus. Acta Otolaryngology. 1974;77:348-53. Also: Yue C.  EOG Signals in Drowsiness Research (Unpublished masters thesis). Linköping University, Sweden. 2011. Accessed on 07/15/2020 from: https://www.diva-portal.org/smash/get/diva2:555912/fulltext01.pdf
17. Duong Y. Making wavelets: A profile of Ingrid Daubechies. Simons Foundation. Accessed on 07/09/2020 from: https://www.simmonsfoundation.org
18. Wearable Eye-tracking (video). Leuven, Belgium: Imec: Accessed on: 07/10/2020 from: https://www.imec-int.com/en/eog
19. Carmichael L & Dearborn WF. Reading and Visual Fatigue. Boston, MA: Houghton Mifflin, 1947.
20. Tursky B & O’Connell DN. A comparison of AC and DC eye movement recording. Psychophysiology. 1966;3(2):157-63.
21. Hulce VD. Skin surface electrodes and the recording of neural events. J Polysomnographic Tech. 1993;12:21-4.
22. Patrick R. Recording the biopotentials of sleep. In: Mattice C, Brooks R, & Leo-Chiong T. Fundamentals of Sleep Technology, 3^rd ed. Philadelphia, PA: Wolters Kluwer. 2020:399-424.
23. Tsai PY, Hu W, Kuo TBJ, & Shyu LY. A Portable Device for Real Time Drowsiness Detection Using Novel Active Dry Electrode System. 31st Annual International Conference of the IEEE EMBS. Minneapolis, MN. Sept 2-6, 2009.
24. Arnin J et al. Wireless-based portable EEG-EOG monitoring for real time drowsiness detection. Conf Proc IEEE Eng Med Biol Soc. 2013;4977-80.