Smartphone Systems as Sensing Systems PowerPoint Presentation

Slide 1 -

Design and Implementation of Smartphone-based Systems and Networking Dong Xuan Department of Computer Science and Engineering The Ohio State University, USA

Slide 2 -

2 Outline Smartphones Basics Mobile Social Networks E-Commerce E-Health Safety Monitoring Future Research Directions

Slide 3 -

A smartphone is a mobile phone offering advanced capabilities, often with PC-like functionality Hardware (Apple iPhone 3GS as an example) CPU at 600MHz, 256MB of RAM 16GB or 32GB of flash ROM Wireless: 3G/2G, WiFi, Bluetooth Sensors: camera, acceleration, proximity, light Functionalities Communication News & Information Socializing Gaming Schedule Management etc. Smartphone Basics 3

Slide 4 -

Smartphones are popular and will become more popular Smartphone Popularity 4

Slide 5 -

Smartphone Accessories 5

Slide 6 -

6 Smartphone Features Communication/Sensing/Computation Inseparable from our human life

Slide 7 -

Our Smartphone Systems E-SmallTalker [IEEE ICDCS10]: senses information published by Bluetooth to help potential friends find each other (written in Java) E-Shadow [IEEE ICDCS11]: enables rich local social interactions with local profiles and mobile phone based local social networking tools P3-Coupon [IEEE Percom11]: automatically distributes electronic coupons based on an probabilistic forwarding algorithm 7

Slide 8 -

Our Smartphone Systems Drunk Driving Detection [Per-Health10]: uses smartphone (Google G1) accelerometer and orientation sensor to detect Stealthy Video Capturer [ACM WiSec09]: secretly senses its environment and records video via smartphone camera and sends it to a third party (Windows Mobile application) Download & Run Video sent by Email Captured Video 8

Slide 9 -

Exemplary System I:E-SmallTaker Small Talk A Naïve Approach Challenges System Design Implementation and Experiments Remarks 9

Slide 10 -

Small Talk People come into contact opportunistically Face-to-face interaction Crucial to people's social networking Immediate non-verbal communication Helps people get to know each other Provides the best opportunity to expand social network Small talk is an important social lubricant Difficult to identify significant topics Superficial 10

Slide 11 -

A Naive Approach of Smartphone-based Small Talk Store all user’s information, including each user’s full contact list User report either his own geo-location or a collection of phone IDs in his physical proximity to the server using internet connection or SMS Server performs profile matching, finds out small talk topics (mutual contact, common interests, etc.) Results are pushed to or retrieved by users 11

Slide 12 -

However…… Require costly data services (phone’s internet connection, SMS) Require report and store sensitive personal information in 3rd party Trusted server may not exist Server is a bottleneck, single point of failure, target of attack 12

Slide 13 -

E-SmallTalker – A Fully Distributed Approach No Internet connection required No trusted 3rd party No centralized server Information stored locally on mobile phones Original personal data never leaves a user’s phone Communication only happens in physical proximity 13

Slide 14 -

Two Challenges How to exchange information without establishing a Bluetooth connection Available data communication channels on mobile phones Cellular network (internet, SMS, MMS), Bluetooth, WiFi, IrDA Bluetooth is a natural choice Bluetooth connection needs user’s interaction due to security reasons How to find out common topics while preserving users privacy No pre-shared secret for strangers Bluetooth Service Discovery Protocol can only transfer limited service information 14

Slide 15 -

System Architecture Context exchange Context encoding and matching Context data store User Interface 15

Slide 16 -

Context Exchange Exploit Bluetooth service discovery protocol No Bluetooth connection needed Publish encoded contact data (non-service related) as (virtual) service attributes Limited size and number( e.g. 128 bytes max each attribute) 16

Slide 17 -

Context Encoding Example of Alice’s Bloom filter Alice has multiple contacts, such as Bob, Tom, etc. Encode contact strings, Firstname.lastname@phone_number, such as “Bob.Johnson@5555555555” and “Tom.Mattix@6141234567” 17

Slide 18 -

Implementation J2ME about 40 java classes, 127Kb jar file On real phones Sony Ericsson (W810i), Nokia (5610xm, 6650, N70, N75, N82) 18

Slide 19 -

Experiments Settings 6 phones, n=150, k=7, m=1024 bits, default distance=4m, average of 10 runs Performance Metrics Discovery time: the period from the time of starting a search to the time of finding someone with common interest, if there is any Discovery rate: percentage of successful discoveries among all attempts Power consumption Factors Bluetooth search interval Number of users Distance 19

Slide 20 -

Experiment Results Minimum, average and maximum discovery time are 13.39, 20.04 and 59.11 seconds respectively Always success if repeat searching, 90% overall if only search once Nokia N82 last 29 hours when discovery interval is 60 seconds 20

Slide 21 -

Related Work Social network applications on mobile phones Social Serendipity Centralized, Bluetooth MAC and profile matching, SMS, strangers PeopleTones, Hummingbird, Just-for-Us, MobiLuck, P3 Systems, Micro-Blog, and Loopt Centralized, GPS location matching, Internet, existing friends Nokia Sensor and PeopleNet Distributed, profile, Bluetooth / Wifi connection, existing friends Private matching and set intersection protocols Homomorphic encryption based Too much computation and message overhead for mobile phone Limitations Require costly data services (phone’s internet connection, SMS) Require report and store sensitive personal information Bottleneck, single point of failure, target of attack 21

Slide 22 -

Remarks Propose, design, implement and evaluate the E-SmallTalker system which helps strangers initialize a conversation Leveraged Bluetooth SDP to exchange these topics without establishing a connection Customized service attributes to publish non-service related information. Proposed a new iterative commonality discovery protocol based on Bloom filters that encodes topics to fit in SDP attributes to achieve a low false positive rate 22

Slide 23 -

Exemplary System II:E-Shadow Concept Application Scenario Goals and Challenges System Design Implementation and Experiments Remarks 23

Slide 24 -

Concept Motivation Importance of Face-to-Face Interaction Prevalence of mobile phones Distributed mobile phone-based local social networking system Local profiles Mobile phone based local social interaction tools

Slide 25 -

Application Scenario: Conference

Slide 26 -

Goals and Challenges Design Goals Far-reaching and Unobtrusive Privacy and Security Auxiliary Support for Further Interactions Broad Adoption Challenges Lack of Communication Support Power and Computation Limitation Non-pervasive Localization Service

Slide 27 -

Layered Publishing Spatial Layering WiFi SSID at least 40-50 meters, 32 Bytes Bluetooth Device (BTD) Name 20 meters, 2k Bytes Bluetooth Service (BTS) Name 10 meters, 1k Bytes Temporal Layering For people being together long or repeatedly Erasure Code

Slide 28 -

E-Shadow Publishing Procedure

Slide 29 -

Matching E-Shadow with its Owner Intuitive Approach: Localization However, imprecision beyond 20-25 meters

Slide 30 -

Human Direction-driven Localization Direction more important than distance Human observation A new range-free localization technique RSSI comparison: Less prone to errors Space partitioning: Tailored for direction decision

Slide 31 -

Walking Route and Localization We allow users to walk a distance Triangular route: A->B->C in (a), for illustration purposes Semi-octogonal route: A->B->C->D->E in (c), more natural Take measurements on turning points Calculate the direction through RSSI comparison and space partitioning

Slide 32 -

Implementation Information Publishing Module Database Generator Buffers Control Valve Broadcasting Interfaces Retrieval & Matching Module Receivers Localization Decoding & Storage Sensing Module User Interface

Slide 33 -

Evaluations (1)-Time & Energy E-Shadow Collection Time WiFi SSID: 2 seconds BTD: 12-18 seconds BTS: 25-35 seconds E-Shadow Power Consumption 3 hours in full performance operation >12 hours in typical situation

Slide 34 -

Evaluations (2)-Localization 3 Outdoor Experiments: Open field campus 2 Indoor Experiments: Large classroom

Slide 35 -

Evaluation (3)-Simulations Large-Scale Simulations: Angle deviation CDFs 12 times of exemplary direction decisions

Slide 36 -

Related Work Centralized mobile phones applications Social Serendipity Centralized, Bluetooth MAC and profile matching, SMS, strangers Decentralized mobile phone applications Nokia Sensor Distributed, profile, Bluetooth / Wifi connection, existing friends E-Smalltalker Distributed, no Bluetooth / Wifi connection, strangers Localization techniques for mobile phones applications GPS Virtual Compass peer-based relative positioning system using Wi-Fi and Bluetooth radios Limitations Privacy compromise Unable to capture the dynamics of surroundings No mapping between electronic ID and human face Localization techniques either not pervasive or not accurate for long range 36

Slide 37 -

Remarks Propose, design, implement and evaluate the E-Shadow system which lubricates local social interactions E-Shadow concept Layered publishing to capture the dynamics of surroundings Human-assisted matching that works for mapping E-Shadow with its owner in a fairly large distance Implementing and evaluating E-Shadow on real world mobile phones 37

Slide 38 -

Exemplary System III:P3-Coupon Coupon Distribution A Naïve Approach Challenges System Design Implementation and Experiments Remarks 38

Slide 39 -

Electronic Coupon Distribution Electronic coupons Similar to paper coupons Can be stored on mobile phones Two distribution methods Downloading from Internet websites Need to define target group Limited coverage Hard to maintain dynamic preferences lists on central databases Peer to Peer Distribution No special destination/target group More coverage More flexible user-maintained preferences list 39

Slide 40 -

A Naive Approach of Peer-to-Peer Coupon Distribution A store periodically broadcast the coupon Users within broadcast range receive the coupon User can decide whether to use, forward or discard the coupon Users forward the coupon to others in physical proximity Forwarder’s IDs are recorded in a dynamically expanding list The coupon is used by some user The store reward all users who have forwarded the coupon 40

Slide 41 -

However…… Require manually establishing wireless connections Cumbersome Not prompt Not possible for coupon forwarding among strangers Require recording the entire forwarding path Potential privacy leakage Discourage user’s forwarding incentives 41

Slide 42 -

Challenge How to design a prompt coupon distribution mechanism that Incentivize coupon forwarder appropriately for keeping the coupons circulating Preserve the privacy of coupon forwarders 42

Slide 43 -

P3-Coupon – A Probabilistic Coupon Forwarding Approach Probabilistic sampling on forwarding path Keep only one forwarder for each coupon: NO privacy leakage Probabilistically flip ownership at each hop Accurate approximation of coupon rewards plenty of chances of interpersonal encounters Accurate bonus distribution with 50 coupons and 5000 people Adaptive to different promotion strategies Flip-once model Always-flip model No manual connection establishment Connectionless information exchange via Bluetooth SDP 43

Slide 44 -

System Architecture Store Side A central server for broadcasting and redeeming coupons Client side Coupon forwarding manager, coupon exchange, coupon data store, user interface 44

Slide 45 -

Probabilistic Forwarding Algorithm Always-Flip Model The coupon ownership keeps flipping with certain probability at each hop. Good at assigning relative bonuses affected by the whole path lengths E.g. the parent forwarder receives k times the bonus given to children forwarders The flip probability can be calculated in advance by the store, once k is fixed, using the following formula 45

Slide 46 -

Probabilistic Forwarding Algorithm Extension: Flip-Once Model Once flipped, a coupon’s ownership remain the same in a forwarding path. Good at assigning absolute bonuses irrelevant of the number of following forwarders E.g. hop 1 user gets 10%, hop 2 user gets 5%, etc. The flip probability can be calculated in advance by the store using the following formula 46

Slide 47 -

Coupon Format Coupon description Product description Discounts Coupon issuer Coupon code Start/end date Coupon forwarder information The current owner Digital signature Prevent forging fraud coupons 47

Slide 48 -

Implementation J2ME about 17 java classes, 1390Kb jar file On real phones Samsung (SGH-i550), Nokia (N82, 6650, N71x) 48

Slide 49 -

Experiments Experimental evaluations Coupon forwarding time Power consumption Simulation evaluation Number of Coupon holders vs. Time Distribution saturation time vs. Number of Seeds Coupon ownership distribution for probabilistic sampling Deviation between theoretical and actual bonus (Always-Flip, Flip-Once) Factors Number of coupons Number of users Number of initial coupon holders 49

Slide 50 -

Experiment Results Average coupon forwarding time is 33.52 seconds Nokia N82 last 25 hours with P3-Coupon running in background One coupon could be delivered to 5000 people within 32 hours Very small deviation between theoretical and actual bonus distribution with 50 coupons circulating among 5000 people 50

Slide 51 -

Remarks Propose, design, implement and evaluate the P3-Coupon system which helps prompt and privacy preserving coupon distribution Probabilistic one-ownership coupon forwarding algorithm Implement the system on various types of mobile phones Extensive experiments and evaluations show that our approach accurately approximate the theoretical coupon distribution in which the whole forwarding path needs to be recorded Practical for real-world deployment 51

Slide 52 -

Exemplary System IV – Drunk Driving Detection Motivation Our Contributions Detection Criteria Our System Related Work Implementation and Evaluation Remarks 52 52

Slide 53 -

Motivation Crashes caused by alcohol-impaired driving pose a serious danger to the general public safety and health 13,041 and 11,773 driving fatalities happened in 2007 and 2008* 32% of the total fatalities in these two years* Drunk driving also imposes a heavy financial burden on the whole society Annual cost of alcohol-related crashes totals more than $51 billion** * Data from U.S. NHTSA (National Highway Traffic Safety Administration) ** Data from U.S. CDC (Central of Disease Control) 53 53

Slide 54 -

Motivation Detection of drunk driving so far still relies on visual observation by patrol officers Drunk drivers usually make certain types of dangerous maneuvers NHTSA researchers identify cues of typical drunk driving behavior Visual observation is insufficient to prevent drunk driving The number of patrol officers is far from enough The guidelines are only descriptive and qualitative Usually, it is too late when drunk drivers are stopped by officers It is essential to develop systems actively monitoring drunk driving and to prevent accidents 54 54

Slide 55 -

Our Contributions Propose utilizing mobile phones as a platform for active drunk driving detection system Design a real-time algorithm for drunk driving detection system using mobile phones Simple sensors required only i.e., accelerometers and orientation sensors Design and implement a mobile phone-based active drunk driving detection system Reliable, Non-intrusive, Lightweight and power efficient, and No extra hardware and service cost 55 55

Slide 56 -

Cues for Drunk Driving Detection Cues related to lane position maintenance problems E.g., weaving, drifting, swerving and turning with a wide radius Cues related to speed control problems E.g., accelerating or decelerating suddenly, and braking erratically Cues related to judgment and vigilance problems E.g., driving with tires on lane marker, slow response to traffic signals 56 56

Slide 57 -

Drunk Driving Detection Criteria Extract fundamental detection criteria from these cues Capture the acceleration features E.g., for the lane position maintenance problems 57 57

Slide 58 -

Drunk Driving Detection Criteria Focus on the first two categories of cues They correspond to higher probabilities of drunk driving Map them into patterns of acceleration Probability of drunk driving detection goes higher while the number of observed cues increases 58 58

Slide 59 -

Our System 59 59

Slide 60 -

Implementation Develop the prototype system on Android G1 phone with accelerometer and orientation sensor Implement the prototype in Java, with Eclipse and Android 1.6 SDK The whole prototype system can be divided into five major components ☆ User interface ☆ System configuration ☆ Monitoring daemon ☆ Data processing ☆ Alert notification 60 60

Slide 61 -

Evaluation - Testing Data Collection Test data 72 sets of data with simulated drunk driving related behaviors Weaving, swerving, turning with a wide radius Changing speed erratically (accelerating or decelerating) 22 sets of data for regular driving Each one for 5 to 10 minutes Mobile phone positions in the vehicle 61 61

Slide 62 -

Evaluation - Detection Performance Study the accuracy of detecting drunk driving related behaviors In terms of false negative and false positive Study performance in the special case, such as the phone slides in the vehicle during driving Slides has obvious impacts on detection accuracy May add additional calibration procedure to solve it (future work) 62 62

Slide 63 -

Evaluation – Energy Efficiency Curves of battery level states during mobile phone running Phone runs without drunk driving detection system Monitoring daemon of system keeps running, sensing and doing the pattern matching on the monitoring results 63 63

Slide 64 -

Related Work Driver vigilance monitoring and driver fatigue prevention Monitoring the visual cues of drivers to detect fatigue in driving Installed cameras just in front of drivers are potential safety hazard Monitoring through vehicle-human interface Capture fatigued or drunk driving through monitoring interactions Low compatibility, vehicles need to couple with auxiliary add-ons Detect abnormal driving through GPS and acceleration data Pattern matching with GPS and acceleration data However, GPS data are not always available 64 64

Slide 65 -

Remarks First to propose utilizing mobile phones as a platform for developing active drunk driving detection system Design and implement an efficient detection system based on mobile phone platforms Experimental results show our system achieves good detection performance and power efficiency In the future work, to improve the system with additional calibration procedure and by integrating all available sensing data on a mobile phone such as camera image 65 65

Slide 66 -

Exemplary System V: Stealthy Video Capturer Background SVC Overview Challenges Our Approaches Experimental Evaluations Remarks 66

Slide 67 -

Background More and more private information is entrusted to our friend, the 3G Smartphone, which is getting more and more powerful in performance and diversified in functionality. 67

Slide 68 -

SVC Overview Almost every 3G Smartphone is equipped with a camera and the wireless options, such as 3G networks, BlueTooth, WiFi or IrDA. These wireless connections are good enough to handle certain types of video transmission. We turn 3G Smartphones into an online stealthy video-recorder. 68

Slide 69 -

System Architecture 69

Slide 70 -

Challenges Stealthily install SVC into 3G Smartphones Windows Hiding Infection Method Collect the video information from 3G Smartphones DirectShow Controls Data Compressing Send the video file to the SVC intender File Sending 70

Slide 71 -

Infection Method To embed SVC in a 3G Smartphone is called a infection process. We employ Trojan horse for downloads as the infection approach. Our experimental SVC is hidden in the game of ”tic-tac-toe” that we develop in Windows Mobile environment. 71

Slide 72 -

The Scenario of Tic-Tac-Toe 72

Slide 73 -

Triggering Schemes Triggering Algorithm is designed to determine when to turn on the video capture process and send the captured video to make SVC stealthier and get more useful information. Three scenarios are under consideration. The first scenario is tracking. The second scenario is related with political or business espionage. The third scenario is a hybrid one, where SVC is used for much diversified everyday purposes. 73

Slide 74 -

Applications Suspects tracking Kids care 74

Slide 75 -

Kids tracking 75

Slide 76 -

Implementation 76

Slide 77 -

Experimental Evaluations: Power Consumption Power curve 77

Slide 78 -

Experimental Evaluations: CPU and Memory Usage CPU and Memory 78

Slide 79 -

Remarks The initial study exploited from SVC will draw wide attentions on 3G Smartphone’s privacy protection and open a new horizon on 3G Smartphones security research and applications. We are currently investigating the modeling of smart spyware from the study of ”spear and shield”. 79

Slide 80 -

A Summary 80

Slide 81 -

Future Research Directions Smartphone-based Systems and Networking Mobile social networking, e-commerce, e-health, safety monitoring etc. Easy to start and exciting but too many competitors, lack of scientific depth Smartphone Core Improvement Multitasking, power management, efficient local communication protocol, accurate localization, security/privacy protection Deep but hard to start 81

Slide 82 -

Final Remarks Smartphones have brought significant impacts to our daily life. We present five exemplary systems on mobile social networking, e-commerce, e-health and safety. Research and development on smartphones will be hot. 82