The Passive Smart Home: Unobtrusive At-Home Behavior Monitoring

class: center, middle, title-slide

# The Passive Smart Home: Unobtrusive At-Home Behavior Monitoring
## Garrick Aden-Buie
### Utilizing Technology for Data Collection and Intervention
### October 29, 2018

---

class: inverse middle center
## Aging in the United States

???
Thank you: Drs. **Brandon** and **Jim**, Dr. Businelle

- Recently joined Moffitt
- Cancer new, learning everyday
- Doctoral candidate USF IE
- Wondering what is IE?
  - Decision Engineering
  - Data and math. models to improve decision making
  - Inventory allocation
  - vehicle routing
  - optimizing patient flows in ER
  - choosing the best medical option under uncertainty

- Last 4 years: how we age, aging in US

---
layout: true
### US Population by Age
---

![](images/video/pop-anim-abs_first.png)

???
The first reason is summarized in this population pyramid.

Population of US by 5-year age range, by percentage, non-gendered.

---

![](images/video/pop-anim-abs.gif)

???
- Baby-boomers: born mid-40s through late 60s

- Millennials, Gen X, _echo boomers_ in the 80s-90s.

- __Life expectancy has increased by 3 mon/year since 1840*__ 
 - and while 75% of seniors report being in __good to excellent health__, 
 - __a majority are managing chronic health conditions__.

---

![](images/video/pop-anim-abs_last.png)

---

![](images/video/pop-anim-abs_first-last.png)

---

![](images/video/pop-anim-abs_dependency-ratio.png)

???
Interesting: dependency ratio

Number of working force aged adults to number of dependents.

Work force classically defined as 15-64.

---
layout: true
### US Elder Dependency Ratio
---

![](images/video/pop-dependency_first.png)

???
Look at how this ratio changes over time

Ratio of older adults to work-force age adults

---

![](images/video/pop-dependency.gif)

---

![](images/video/pop-dependency_last.png)

???
So our population growth is occurring along two dimensions:

- Increase in absolute number of older adults
- Increase in the relative number of older adults

---
layout: true
### US Dependency Ratio
---

![](images/video/pop-dependency-both.gif)

???
Include children in the dependency ratio

- Giant Baby Boomer bump
- Older adults driving trends in dep. ratio

Doesn't take into account:

- More older adults are divorced or live alone than ever before, which in conjunction with demogrph. changes leads to decreased availability of traditional family caregivers
- # of 75+ without a spouse will x2 between 2010 and 2030 and without an adult child within 10 miles x6

---

![](images/video/pop-dependency-both_last.png)

---
layout: false
class: no-slide-number
background-image: url('images/ruth-individual.JPG')
background-size: cover

???

Early in 2012, just after her 80th birthday my grandmother was diagnosed with Alzheimer’s.

Same time: grandfather hospitalized for heart failure. Also COPD.

They had just re-retired to Florida, 1200 miles from Chicago where my parents were living.

My mom left a full-time position and for a while would fly back and forth.

My grandparents were highly independent, very much wanted to live on their own, but it was very difficult for my mom to monitor them from a distance.

As Alz progressed, tougher to know if they ate, showered, left the house, slept normally, etc.

Two highlights from their story

1. Not uncommon
2. Highlighting the need and value for passive technologies that link caregivers and seniors

---
class: bottom center no-slide-number
background-image: url(images/participant-pointing-sensor.jpg)
background-size: cover
background-position: 0px -20px

???
My parents would have benefited from a system like the one we installed in "Dorothy's" house

Uses passive, wireless sensors installed throughout the home

Collects basic information about her movements and interactions with objects

---
class: no-slide-number
background-image: url(images/participant-computer-pointing.jpg)
background-size: cover

???
Summarized and reported on a web interface that she shared with her daughter

---
background-image: url(images/createhealthlogo.png)
background-size: 400px
background-position: 80% 5%
## HomeSense

.muted[
CREATE Health, USF  
Dr. Carla VandeWeerd and Dr. Ali Yalcin  
usf.edu/engineering/create-health/
]

.pull-left[
### Goals

- Leverage commerical low-cost home monitoring technology

- Support older adults to **age in place**

- Improve quality of life throught the use of technology
]

???
**Aging in Place**: Overwhelming majority of OA want to stay in current homes, live independently

Increase QoL by mitigating the impact of inconsistent monitoring and delayed health assessments.

.pull-right[
### Lifestyle Reassurance

- Monitor health and daily activities

- Provide safety and security around changes in health and routine

- Alleviate burden of chronic disease
]

---
layout: true
name: villages
class: bottom, center, bg-full, no-pad, no-slide-number

<video loop muted autoplay poster="images/villages-flyover.png" class="bg-video">
 <source src="images/video/villages-flyover2.mp4" type="video/mp4">
</video>
---

.box-center-bottom.box-light[
<img src="images/thevillages-logo.svg" width="60%">

<div style="text-align: left; padding-left: 60px">
<ul>
<li>The largest retirement community in US</li>
<li>75 miles north of Tampa, FL</li>
<li>115,000 residents in 50,000 homes</li>
<li>Average age: 62 (M) and 60 (F)</li>
<ul>
</div>
]

???
A major community partner in CREATE Health's work is The Villages...

.notes-list[
- 32 sq miles
- 3 town squares
- 63 rec centers
- legend has it a Villager can play 18 holes of gold for 30 straight days without repeating
]

---
.box-center-bottom.box-light[
<h3>Study Participants</h3>
<div style="text-align: left; padding-left: 60px">
<ul>
<li>Residents 55+ from The Villages</li>
<li>Living alone in pet-free home with internet access</li>
<li>Not exhibiting signs of cognitive impairment</li>
<ul>
</div>
]

---
layout: false
## Overview of Study Participation

- 14 total participants (7 ongoing) with 5,944 total days of observation

--
    
- Participant gender and age at study start

- Days of participation in study

???

- Minimum days is 48, newest has 75 days
- Minimum (included) days is just over 181
- Max days is 786.7916667
- Median (> 180) is 446.4166667

Reasons for leaving:

- 2 participants moved (002, and 010)
- 1 participant died
- 1 participant faced medical challenges
- 2 participants did not continue after 6 months
- 1 participant did not like the sensors

---

## Timeline

???

Most at once: 11 participants

---
layout: true
## Sensor System
---
background-image: url(images/sensors.png)
background-position: center
background-size: 500px
---
background-image: url(images/example-house-floorplan/simple-legend.png)
background-position: 50% 45%
background-size: 80%
---
background-image: url(images/example-house-floorplan/simple-exp-covered.png)
background-position: 50% 45%
background-size: 80%
---
background-image: url(images/SystemArchitecture2.png)
background-position: 50% 90%
background-size: 800px
---
layout: true
## Challenges
---

.left-column[
### Sensors in the home
]

.right-column[
- Real occupants in unscripted situations performing unknown actions

- Remote sensing in low-visibility locations

- Balance between high density of sensors and cost-efficiency and acceptance

- Cameras and wearable devices perceived as too invasive or too difficult to use
]

---

.left-column[
### Sensors in the home
### Reliability issues
]

.right-column[
- Low-power wireless network — communication issues

- Battery powered sensors

- Human interaction with sensors

- Various devices, device types and manufacturers

- Non-failure related messiness
]

???

“Homes [. . . ] can be hazardous for sensors, particularly when hundreds of sensors are deployed over long time durations” Hnat et al. (2011)

Hnat et al. (2011). The hitchhiker’s guide to successful residential sensing deployments. doi:10.1145/2070942.2070966.

---

.left-column[
### Sensors in the home
### Reliability issues
### Record Keeping
]

.right-column[
- Inherent heirarchy of sensor data as a function of floor plan

- Physical location of sensors critical to interpretation

- System needs to be resilient to changes
]

???

Very important to keep track of where sensors are located

This is not something that's carried around with the data

Systems need to be resilient agains changes in devices, locations and human interactions

---
layout: true
## Inventory and System Management

.pull-left.third[
- Inventory Management

- Installation Preparation

- System Configuration

- Planning and Review

- Maintenance Visits

- Data Export
]
---
.pull-right.two-third[<img src="images/admin-pages/admin-01.png" class="mh500"></img>]
---
.pull-right.two-third[<img src="images/admin-pages/admin-02.png" class="mh500"></img>]
---
.pull-right.two-third[<img src="images/admin-pages/admin-09.png" class="mh500"></img>]
---
.pull-right.two-third[<img src="images/admin-pages/admin-08.png" class="mh500"></img>]
---
.pull-right.two-third[<img src="images/admin-pages/admin-10.png" class="mh500"></img>]

---
exclude: true
layout:false

![](images/chdash-sensor-activitiy.png)
---
layout:false
background-image: url(images/chdash/chdash-02-activity-out-of-home.png)
background-size: cover
background-position: top
---
background-image: url(images/chdash/chdash-01-polar-activity.png)
background-size: cover
background-position: top
---
background-image: url(images/chdash/chdash-05-overview-by-hour.png)
background-size: cover
background-position: top
---

.pull-left.center[
## Notifications

<img src="images/notification-email.png" class="mh500">
]

.pull-right.center[
## Activity Validation App

<img src="images/ava.png" class="mh500">
]

---
class: center middle inverse

## Activity Profiles

---

.pull-left[
### Activity Recognition

- **Generative**
    - Hidden Markov Models
    - Naive Bayes
- **Discriminative**
    - Machine learning methods
    - NN, ANN, SVM, etc.
- **Ontological Models**

- Require labelled training data

- Accurate annotations very difficult
]

.pull-right[
### Activity Profiles

- Summarize occupant's activities

- Do not require labelled training data

- Maintaining lifestyle (and smart home system)

- Enable detection of changes or anomalies in routine, behavior
]

---
layout: true
## Example Day of Sensor Activity
---

.pull-left[
- Use only **active** sensor firings

- Day begins at first 5 active
]

.pull-right[
- Insert _pause event_: no activity in 15 min

- Summarize as **bag-of-event _n_-grams**
]

---

.ngram-example[
<code>
iIbEAaDdDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPpPpgDdPpPpEPpPpDdPpDdDdPpIiEE1EPaA1AaEaEIiIDdDdDd1aEIiaAIia1AaEDdDdpPDdDdDdDdDdDdDdEDdDd1EIpaAGgaAP1EaAAa1EaA1AaEEIipDdPp1EPpaA1AE1aEIiDdGgGEgEIP1EIDdDdDdDdEEEaAEaApPpGgIPaA1AaEpDdIBiiI1ii1
</code>
]

---

.ngram-example[
<code>
iIbEAaDdDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]

---

.ngram-example[
<code>
.hl-text-overline[i]IbEAaDdDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]
 In Bathroom Master

---

.ngram-example[
<code>
.hl-text-overline[iI]bEAaDdDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]
 In Bathroom Master ➡ In Bedroom Master

---

.ngram-example[
<code>
.hl-text-overline[iIb]EAaDdDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]
 In Bathroom Master ➡ In Bedroom Master ➡ Opened Bedroom Door

---

.ngram-example[
<code>
.hl-text-overline[iIbE]AaDdDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]
 In Bathroom Master ➡ In Bedroom Master ➡ Opened Bedroom Door ➡ In Kitchen

---

.ngram-example[
<code>
.hl-text-overline[iIbEA]aDdDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]
 In Bathroom Master ➡ In Bedroom Master ➡ Opened Bedroom Door ➡ In Kitchen ➡ In Living Room

---

.ngram-example[
<code>
.hl-text-overline[iIbEAa]DdDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]
 In Bathroom Master ➡ In Bedroom Master ➡ Opened Bedroom Door ➡ In Kitchen ➡ In Living Room ➡ In Front Door Area

---

.ngram-example[
<code>
.hl-text-overline[iIbEAaD]dDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]
 In Bathroom Master ➡ In Bedroom Master ➡ Opened Bedroom Door ➡ In Kitchen ➡ In Living Room ➡ In Front Door Area ➡ Opened Fridge

---

.ngram-example[
<code>
.hl-text-overline[iIbEAaDd]DdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]
 In Bathroom Master ➡ In Bedroom Master ➡ Opened Bedroom Door ➡ In Kitchen ➡ In Living Room ➡ In Front Door Area ➡ Opened Fridge ➡ Closed Fridge

---

.ngram-example[
<code>
.hl-text-overline[iIbEAaDdD]dDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]
 In Bathroom Master ➡ In Bedroom Master ➡ Opened Bedroom Door ➡ In Kitchen ➡ In Living Room ➡ In Front Door Area ➡ Opened Fridge ➡ Closed Fridge ➡ Opened Fridge

---

.ngram-example[
<code>
.hl-text-overline[iIbEAaDdDd]DdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp
</code>
]
 In Bathroom Master ➡ In Bedroom Master ➡ Opened Bedroom Door ➡ In Kitchen ➡ In Living Room ➡ In Front Door Area ➡ Opened Fridge ➡ Closed Fridge ➡ Opened Fridge ➡ Closed Fridge

---

.ngram-example[
<code>.word.hl-text.hl[iIbEA]aDdDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp</code>
]

&nbsp;&nbsp;&nbsp;&nbsp;.big[⬇]

#### Bag of event n-grams

.word[iIbEA]

---

.ngram-example[
<code>i.word.hl-text.hl[IbEAa]DdDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp</code>
]&nbsp;&nbsp;&nbsp;&nbsp;.big[⬇]

#### Bag of event n-grams

.word[iIbEA] .word[IbEAa]

---

.ngram-example[
<code>iI.word.hl-text.hl[bEAaD]dDdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp</code>
]&nbsp;&nbsp;&nbsp;&nbsp;.big[⬇]

#### Bag of event n-grams

.word[iIbEA] .word[IbEAa] .word[bEAaD]

---

.ngram-example[
<code>iIb.word.hl-text.hl[EAaDd]DdDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp</code>
]&nbsp;&nbsp;&nbsp;&nbsp;.big[⬇]

#### Bag of event n-grams

.word[iIbEA] .word[IbEAa] .word[bEAaD] .word[EAaDd]

---

.ngram-example[
<code>iIbE.word.hl-text.hl[AaDdD]dDdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp</code>
]&nbsp;&nbsp;&nbsp;&nbsp;.big[⬇]

#### Bag of event n-grams

.word[iIbEA] .word[IbEAa] .word[bEAaD] .word[EAaDd] .word[AaDdD]

---

.ngram-example[
<code>iIbEA.word.hl-text.hl[aDdDd]DdPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp</code>
]&nbsp;&nbsp;&nbsp;&nbsp;.big[⬇]

#### Bag of event n-grams

.word[iIbEA] .word[IbEAa] .word[bEAaD] .word[EAaDd] .word[AaDdD] .word[aDdDd]

---

.ngram-example[
<code>iIbEAa.word.hl-text.hl[DdDdD]dPpPpPIiEIiIIIEpDdDdiPpIiIEPpIiEPDpdIiiIEDdDdiIPpEPpIiPpDdIiiIE1EGgAaIiDdPpGg1GEgGDdDdPp</code>
]&nbsp;&nbsp;&nbsp;&nbsp;.big[⬇]

#### Bag of event n-grams

.word[iIbEA] .word[IbEAa] .word[bEAaD] .word[EAaDd] .word[AaDdD] .word[aDdDd] .word[DdDdD]

---

.pull-left[
**Bag-of-event n-grams**

- Repeat process for *n* from 1 to 1

- Calculate relative frequency *p* of each *n*-gram

- Can be used to summarize *m* days

**Compare two profiles**

- Using Kullback-Leibler Symmetric Distance

- DKL (P || Q) = 0.331
]

.pull-right[
<table>
 <thead>
 <tr>
 <th style="text-align:center;"> n-gram </th>
 <th style="text-align:center;"> Count </th>
 <th style="text-align:center;"> Rank </th>
 <th style="text-align:right;"> Freq. </th>
 </tr>
 </thead>
<tbody>
 <tr>
 <td style="text-align:center;"> I </td>
 <td style="text-align:center;"> 16 </td>
 <td style="text-align:center;"> 1 </td>
 <td style="text-align:right;"> 0.0239 </td>
 </tr>
 <tr>
 <td style="text-align:center;"> D </td>
 <td style="text-align:center;"> 12 </td>
 <td style="text-align:center;"> 2 </td>
 <td style="text-align:right;"> 0.0179 </td>
 </tr>
 <tr>
 <td style="text-align:center;"> d </td>
 <td style="text-align:center;"> 12 </td>
 <td style="text-align:center;"> 3 </td>
 <td style="text-align:right;"> 0.0179 </td>
 </tr>
 <tr>
 <td style="text-align:center;"> i </td>
 <td style="text-align:center;"> 12 </td>
 <td style="text-align:center;"> 4 </td>
 <td style="text-align:right;"> 0.0179 </td>
 </tr>
 <tr>
 <td style="text-align:center;"> Dd </td>
 <td style="text-align:center;"> 11 </td>
 <td style="text-align:center;"> 5 </td>
 <td style="text-align:right;"> 0.0164 </td>
 </tr>
 <tr>
 <td style="text-align:center;"> P </td>
 <td style="text-align:center;"> 11 </td>
 <td style="text-align:center;"> 6 </td>
 <td style="text-align:right;"> 0.0164 </td>
 </tr>
 <tr>
 <td style="text-align:center;"> E </td>
 <td style="text-align:center;"> 10 </td>
 <td style="text-align:center;"> 7 </td>
 <td style="text-align:right;"> 0.0149 </td>
 </tr>
 <tr>
 <td style="text-align:center;"> p </td>
 <td style="text-align:center;"> 10 </td>
 <td style="text-align:center;"> 8 </td>
 <td style="text-align:right;"> 0.0149 </td>
 </tr>
 <tr>
 <td style="text-align:center;"> Ii </td>
 <td style="text-align:center;"> 8 </td>
 <td style="text-align:center;"> 9 </td>
 <td style="text-align:right;"> 0.0119 </td>
 </tr>
 <tr>
 <td style="text-align:center;"> Pp </td>
 <td style="text-align:center;"> 8 </td>
 <td style="text-align:center;"> 10 </td>
 <td style="text-align:right;"> 0.0119 </td>
 </tr>
</tbody>
</table>
]

???

***

KL Distance provides a distance measure between probability distributions.

We use KL dist to compare frequencies across b-o-e n-grams (activity profiles)

`\(D_{KL} (P \Vert Q) = \sum_{i} \left( p_i - q_i \right) \log \frac{p_i}{q_i}\)`

---
layout: true
## Baseline Profile Comparison
---

.center[
<img src="index_files/figure-html/baseline-single-day-calendar-1.png" width="720" />
]

---

.center[
<img src="index_files/figure-html/baseline-dist-points-1.png" width="864" />
]

---

.center[
<img src="index_files/figure-html/unnamed-chunk-4-1.png" width="864" />
]

---

.center[
<img src="index_files/figure-html/baseline-inlier-range-1.png" width="864" />
]

.center[
<img src='index_files/figure-html/baseline-single-day-calendar-1.png' width="70%">
]

---

.center[
<img src="index_files/figure-html/baseline-profile-tau-1.png" width="432" />
]

---

.center[
<img src="index_files/figure-html/baseline-all-range-1.png" width="864" />
]

---

---
layout: true
### Behavior Change Detection

- Complementary method to find changes in behavior

- Permutation-based methods for change detection1

.footnote.small[
1: Sprint, Cook, and Schmitter-Edgecombe (2016)
]

???

- Having validated that the activity profiles are capable 
 of summarizing the activity patterns of an occupant, 
 we now turn to the task of behavior change detection

- compare occupant's recent activity to previous behavior 
 previous behavior is either a baseline period of normalcy 
 or sliding windows

- Given that KL-dist is a scale-less distance metric, 
 need to establish bounds guidelines for expected similarity
 *What is normal variation vs. anomaly/change?*

- Overall, our goal is to bring abnormalities to the attention of caregiver 
 human review and intervention is expected 
 and system issues are equally requiring of attention

---

---

---
layout: true
### Permutation-Based Comparison
---
exclude: true

.center[
<img src="images/permutations4.gif" height="450px">
]

???

Each shuffling of

---

.center[
<img src="images/permutations4-boxplot.png" height="450px">
]

---

.center[
<img src="images/behavior-change-house013-1.png" width="750px">
]

---
layout: false
background-image: url(images/coen-staal-454813-unsplash.jpg)
background-size: cover
class: center top

# Conclusions

---

## Conclusions

Passive sensor systems can address challenges faced by aging population

System, inventory, and data management infrastructure are critical

Local leaders at the University of South Florida:

- CREATE Health (VandeWeerd, Yalcin)
- Dr. Bill Kearns

CASAS "Smart Home in a Box"

- Washington State University
- casas.wsu.edu/smart-homes