An IoT-Based Multimodal Real-Time Home Control System for the Physically Challenged: Design and Implementation

Aligned with the Envision 2030 movement and the Sustainable Development Goals, this investigation takes a particular interest in Goal 10, which seeks to decrease inequality within and among countries. The promotion and empowerment of the social, economic, and political inclusion of all, including individuals with disabilities, is a key aim [1].

The ongoing evolution of the Internet of Things (IoT) carries significant relevance in today’s technologically advanced society. It is observed that a new era of computing and communication has been ushered in by the IoT [1]. The rapid development of IoT technology brings the need for energy conservation and remote home security to the fore. This is particularly pertinent for individuals with physical impairments, who often face challenges navigating their homes unassisted. The focus of this research pertains to these individuals, including those with visual impairments.

It is recognized that many homes were not originally designed with accessibility in mind, thus retrofitting them can be a costly endeavor. Nevertheless, as technology improves and becomes more cost-effective, intelligent home systems are becoming an increasingly viable investment [2]. It has been established that the benefits of home automation include controlling all home appliances from a single location, compatibility with new gadgets and devices, increased home security, and improved appliance functionality [2].

Individuals with physical impairments have distinct needs. Granting such individuals the ability to control home appliances without third-party assistance could bring considerable value and efficiency to the disabled community. Current computer systems offer remote lighting control via intricate microcontroller or computer-based networks, with varying degrees of intelligence and automation. Given the security, energy efficiency, and convenience afforded by home automation, it holds potential as a solution for the challenges faced by visually impaired and other disabled individuals in controlling home devices.

Despite the extensive documentation on IoT technology related to various forms of disabilities, there remains a lack of consolidated and foolproof support for automation in aiding physically challenged individuals. This study aims to bridge this gap by developing a comprehensive system primarily intended for physically challenged individuals, but not limited to these groups.

This study sets out to implement a real-time multimodal (five different techniques) e-switching system on a single model for the physically challenged, utilizing the Internet of Things. One limitation identified is the reliance on internet connectivity, hence a manual switch has been integrated into the model as a backup measure.

In the past decade, advancements in home automation systems have made significant strides, notably in the creation of user-friendly, cost-effective, and efficient systems suitable for a wide range of users, including the elderly and individuals with physical disabilities. However, there is a noted gap in the provision of a unified system that accommodates a broad spectrum of physical challenges. The research herein aims to address this need.

Rizvi et al. [3] pioneered a system designed to furnish a safe, secure, and manageable environment for the elderly. This system enabled the remote operation of household appliances through an Android application or SMS, with Bluetooth technology further facilitating remote monitoring and local administration. Initial trials evidenced the efficacy of the system, showing successful appliance operation through various inputs including voice, Bluetooth, and SMS.

Leveraging Bluetooth technology, Mahith et al. [4] introduced a home automation system for the elderly and physically disabled individuals, who could control home appliances through voice commands. Their implementation markedly improved the system's efficiency.

In contrast, Abidi et al. [5] proposed a system employing a speech reader Android application to recognize voices and control the activation of household appliances. They incorporated an ultrasonic sensor to detect motion, converting the signal into a message to the mobile user via a GSM module. To achieve the system's operation, an Arduino Mega is required, powered by a 12V DC adapter, with all AC power loads connected to the power supply. Their system exhibited remarkable accuracy ($>$95%) and effective regulation of electrical loads through voice commands.

Focusing on visually and hearing-impaired individuals, Ciabattoni et al. [6] developed a user interface for commercial Smart Home Systems (SHS). This interface, via a mobile application, translates visual information and alerts into aural signals and vice versa. The goal was to enhance the acceptability and usability of commercial SHS through inclusive technology, specifically smartphones.

Bhatnagar et al. [7] sought a cost-effective alternative to commercial home automation systems. They aimed for simplicity and ease of use to accommodate users including the elderly and disabled individuals. An Android application or Google Assistant could be used to operate the system, which communicates with the Firebase database to manage various sensors and electrical appliances.

Presenting an economical method to upgrade existing home appliances, Nath and Pati [8] introduced a system allowing voice commands for almost any traditional home device. Particularly beneficial to the elderly and disabled, the system uses a Bluetooth-connected Android phone to interpret voice instructions and communicate them to a home automation node, which in turn controls the connected devices based on the command's keywords.

In a novel approach, Mohanaprasad [9] developed an audio-based software monitoring system accepting inputs from voice, claps, snaps, or a combination thereof. This voice control system allowed individuals with any physical limitation to operate home appliances with ease.

Sravanthi et al. [10] discussed the use of a wireless system for home automation. This system included a speech recognition program for ease of installation and reduced costs. It automatically switched lights based on detected footstep counts when entering a room.

Mtshali and Khubisa [11] designed a system that employed smart plugs, cameras, power strips, and digital assistants like Amazon Alexa or Google Home. These components could record spoken commands from physically impaired individuals, controlling home appliances with minimal effort.

Lastly, Husain et al. [12] demonstrated a method for remote control of household equipment, which responded with an ON/OFF voice response via a speaker. The system was bifurcated into a transmitter or Bluetooth remote and a receiver or relay module, with the former having multiple buttons for appliance control.

While previous research has indeed made significant strides in home automation for those with physical challenges, none have yet achieved the sophistication of integrating mobile application soft buttons, speech recognition applications, recognition devices, clap control, and manual push buttons into a single system. This is the void our research aims to fill. The current models of home automation systems are unable to fully cater to the diverse range of physical challenges faced by individuals, whether they are blind, bedridden, deaf, or mute.

Our research endeavors to develop a system that can address these varied needs effectively. A comparative analysis of related works and our developed system, as shown in Table 1, reveals the distinctive advantages of our design using key performance indices. With the integration of these diverse control mechanisms, our system is primed to deliver a more inclusive and responsive home automation experience for individuals facing a broad spectrum of physical challenges.

In the following section, the methodology employed in controlling an automated appliance switching system was delineated, with particular attention paid to the various methods and their intricacies. Home automation has also been linked with an increased sense of safety and security, convenience, control, and energy conservation [23], [24], [25].

As shown in Figure 1, the management of appliance loads is attained through interfacing two 8-module relays. This control is achieved utilizing several distinct methods, each with its unique strengths and capabilities.

The first method involves using the Amazon Alexa application, wherein the application's soft buttons can toggle ON/OFF states of the appliances. Another method of control exploits the speech recognition capability of the Amazon Alexa application. Notably, the Amazon Alexa Echo Dot speaker is also actuated through voice commands. In scenarios where other systems fail, such as during an internet outage, a safety mechanism in the form of push buttons serves as a manual control for the appliances.

Central to all these methods is the ESP32 microcontroller board, which operates over the internet. Equipped with a suite of input and output GPIO pins, the ESP32 microcontroller board's output pins are interfaced with the normally open contacts of the 8-module relay. Furthermore, a unique incremental clap pattern, recognized by a sound detector connected to an Arduino Uno microcontroller board, is utilized to operate the appliances. This board, similarly connected to the normally open contacts of an 8-module relay, ensures that these methods operate independently and without mutual interference.

The methodology presented here hinges on several key hardware components. Details about these components are elaborated below.

The Arduino UNO, a high-performance microcontroller development board centered around the ATmega328P microcontroller, operates at 5 volts. With its 8-bit AVR RISC-based microcontroller, 14 digital input/output pins, a 16MHz ceramic 27 resonator, and six analog input pins, it forms a crucial part of the control system.

Amazon's Alexa Echo Dot, a popular device embodying Alexa's virtual assistant technology, responds to voice commands, streams music and podcasts, sets alarms, provides real-time information, and performs numerous other functions.

The 8-module relay, an electromagnetic switch driven by a small current, is employed to toggle higher currents. It interfaces the Arduino UNO microcontroller with the appliances and can function either as a switch or an amplifier. Its maximum switching current and voltage stand at 10A and 30VDC/ 240VAC, respectively.

The ESP32 Devkit Module, designed specifically for low-power IoT applications, is characterized by high processing power, built-in WiFi/Bluetooth, and deep sleep operation capabilities. With the official release of the ESP32 board manager by the Arduino IDE, programming these devices has become straightforward.

The sound sensor module, comprising a simple microphone, can detect sounds surpassing a set threshold, courtesy of the LM386 power amplifier and electret microphone. The built-in potentiometer adjusts this threshold, outputting high/low signals contingent on the sound's relative intensity.

A router, critical for internet connectivity, links two or more LANs or the Internet. The wireless or Wi-Fi router combines the routing capabilities of a standard router with the networking capabilities of a wireless access point, managing internet traffic effectively.

The Amazon application platform is an integral part of the online distribution triad consisting of the Apple App Store, Google Play, and Amazon App-store. These platforms provide compatible companion applications, permitting the owners of Alexa-enabled devices to engage in a variety of functions such as adding skills, managing alarms, controlling music, and maintaining shopping lists. The app interface allows users to review detected text and provide feedback to Amazon on the accuracy of its recognition capabilities. A web-based interface further supplements this system, providing the ability to configure compatible devices, including but not limited to Amazon Echo, Amazon Dot, and Amazon Echo Show.

Arduino IDE serves as an open-source software framework predominantly utilized for composing and compiling code for Arduino boards. This framework encompasses a text editor designed for coding principles, a message area for displaying feedback and error messages, a text console, a toolbar facilitating standard tasks, and a selection of menus. It is designed to aid in uploading programs to Arduino and facilitate its communication. The text editor within this software framework hosts the programming language, termed as a 'sketch', with an 'a.ino' file extension. The programming language employed for sketch creation for Arduino is C.

The Proteus Design Suite is a proprietary software toolset primarily employed in electrical design automation. This software package is widely used by electronic design professionals to develop schematics and electronic prints for the production of printed circuit boards. The software includes a vast library of components.

As depicted in Figure 2, the developed system operates based on a distinct principle. The power supply originates from an AC socket, channeled via a USB cable to provide a 5V supply to the system. This power is allocated to the Arduino Uno and ESP32 Devkit. The sound detector acquires analog signals (such as claps) and transforms them into digital signals to formulate a clap switch pattern. The frequency at which the detector receives signals influences the programming that regulates the on/off mechanism of home appliances. The Arduino UNO microcontroller functions as the central processing unit of the system, executing necessary arithmetic operations to compare measured and preset values. It transforms signals from the sound detector into digital signals that it can manipulate.

The system utilizes two 8 – module relays to interface between the Arduino UNO microcontroller, the ESP-32 Devkit microcontroller, and the appliances. Upon receiving a voice command from Alexa or realizing clap control, the corresponding microcontroller emits a control signal to the relay, which initiates contact and supplies power to various appliances.

An ESP32 Devkit serves as the interface between Amazon Alexa and the connected home appliances. When a voice command is issued, the microcontroller board relays a control signal to the relay, which in turn powers the appliances. The virtual pins communicate with the application via WIFI, acting as a system control mechanism.

The Amazon application, with configurations established through the virtual pins, transmits digital signals to the ESP32 Wi-Fi module. This interaction triggers the relay to supply necessary voltages to the appliances. This is facilitated through the activation of soft buttons on a phone screen and voice-controlled Alexa on the application.

The system integrates push buttons for manual switching, connected to GPIO pins on the ESP 32, which are also employed for toggling loads on and off. An Amazon Alexa Echo dot, a smart speaker interfaced with the ESP32 microcontroller, facilitates voice control of appliances and information access when connected to the internet.

The circuit diagram depicted in Figure 3 was designed and fabricated on a printed circuit board (PCB) featuring two microcontrollers: the ESP32 breakout board and Arduino Uno board, both powered by a 5-volt input. The ESP32, a low-cost and low-power microcontroller with Wi-Fi and Bluetooth capabilities, is well-suited for controlling home appliances utilizing the Internet of Things (IoT). This microcontroller contains 38 pins, including 32 general-purpose input/output (GPIO) pins, 1 enable (EN) pin, and 5 power pins. GPIO pins 26, 25, 23, 21, 22, 21, 19, 18, and 5 are connected to the U2 relay module inputs IN1, IN2, IN3, IN4, IN5, IN6, IN7, and IN8, respectively, to transmit instructive signals originating from manual buttons, soft control buttons on the Alexa App installed on an Android phone, and voice assistant from the Alexa Echo Dot. This is achieved by connecting the ESP32 module to a local router with internet access using the same SSID and password, while the Alexa App on the Android phone is connected to the Alexa Echo Dot via Bluetooth to access the Amazon App store. GPIO pin 2 serves as an indicator for Wi-Fi availability and connectivity; when the LED is on, the system is connected to Wi-Fi, and when it is off, Wi-Fi is unavailable.

Virtual switching is facilitated by incorporating the $<$WiFi.h$>$ and $<$Espalexa.h$>$ files into the sketch and defining the various GPIO pins as output pins. Device names were assigned from devices 1 to 8 and designated as Main Lamp, Socket, Bed Lamp, Study Lamp, Security Lamp, Kitchen Lamp, and Pool Lamp. When any of these names are mentioned, the Alexa voice assistant initiates the corresponding instruction. GPIO pins 33, 32, 27, 14, 15, 13, 12, and 4 are connected to manual buttons for manual load control.

Arduino Uno digital pins 13, 12, 11, 10, 9, 8, 7, and 6 are linked to U5 relay module inputs IN1, IN2, IN3, IN4, IN5, IN6, IN7, and IN8. The U5 relay module output from the various normally open (NO) connections is further interconnected with the output of the U2 relay module normally open (NO) in a cascaded format, allowing both microcontrollers to access and control the same load. A sound detection sensor is employed to detect clap sounds, with its output pin connected to Arduino Uno's analog pin 2. When a clap sound is detected, the sensor converts it into an analog electrical signal, which is fed into the microcontroller's analog pin 2. The signal is then converted into a digital signal using an analog-to-digital converter (ADC) mechanism. As claps are detected repeatedly, loads are controlled sequentially. The schematic design is shown in Figure 3.

This analysis shows how load is selected and distributed across the home system.

Power supply = 240Vac (converted to 5V dc)

Relay output maximum contact is AC 240V/10A

The working voltage of Relay = 5V dc

Relay power (PR) = I * V = 10 * 240 = 2400W or 2.4KW

Therefore, any load to be controlled by this relay should not exceed 2.4KW of power or else the system would trip off.

This prototype makes use of an 8-channel relay, with total power = 2.4KW * 8 = 19.2KW.

Power consumption of Lamps (PL) = 25W,40W and 60W

This prototype makes use of 7 Lamps,total power = (`4*25W) + (2*40W) + 60W = 240W.

A 13A socket was selected to be used in this research work.

Maximum power consumption of socket (PS) = I * V = 13A * 240V = 3120W or 3.12KW

From the above analysis, it is observed that to keep the relay functional, the external load connected to the socket should be less than the power of the relay controlling that socket (2.4KW) otherwise the relay would trip off.

Analysis of current limiting resistance of load LEDs

Voltage, V = 3.5V

Current, I = 25mA

Value of resistance = V/I = 3.5/25Ma = 140 ohms.

A critical aspect of the system's functionality lies in the proper operation of the relay, which serves as an intermediary between the control systems and the appliances being controlled. The relay requires a 5-volt supply to be triggered and function as a switch. It was confirmed using a multimeter that the actual 5-volt supply was provided to the opto-coupler, triggering the relay. The Wi-Fi module's ability to connect to an open Wi-Fi network is crucial, as the app control heavily relies on it. The control's strength and speed must be sufficient to prevent delays and lags when operating appliances from the Android device. In this case, a router was utilized to connect to the internet, and the strength of the connection was tested, demonstrating a wide range. An LED was designed to indicate to users whether the system is online or offline. A 330 Ohm resistor was connected to limit the current flowing through the LED, preventing it from burning out. Upon connecting to a Wi-Fi network, the LED automatically illuminates.

As long as a wireless connection links the ESP32 Devkit microcontroller board, the Android device, and Amazon Alexa, voice control of the programmed appliances can be achieved by invoking Alexa with the wake word and initiating the command. The clap-switching system can be employed to control appliances in the event of a faulty internet connection. In rare instances of system failure, manual switching has been incorporated into the system, allowing appliances to be turned ON or OFF manually using push buttons. The Amazon Alexa app can also control the appliances. The testing of the system is depicted in Figure 4, Figure 5, and Figure 6.

The success of the system in both voice-controlled and clap-switching scenarios illustrates its versatility in various environments, including those with limited internet connectivity. Furthermore, the incorporation of manual switching provides a reliable fallback option, ensuring uninterrupted control of the connected appliances. Future research could explore the integration of additional sensors, such as motion or light sensors, to enhance the system's adaptability to different user preferences and environmental conditions. Additionally, energy consumption monitoring and optimization features could be investigated to promote energy efficiency and conservation.

This study has presented the hardware implementation of an IoT-based smart home system, primarily designed to assist elderly and physically challenged individuals, including those who may be visually impaired, hearing impaired, or have limited mobility. It was found that the system could also be beneficial for older adults, individuals recovering from illness, and even healthy individuals seeking enhanced home automation. The system integrates three autonomous control methods, enabling users to manage their home appliances without requiring external assistance.

A fail-safe mechanism, consisting of physical control buttons or switches, was implemented to address potential mishaps that may occur during system usage. The proposed design offers an elegant, secure, low-energy consumption solution that enhances accessibility for disabled individuals.

In summary, the IoT-based smart home system demonstrates the potential to improve the quality of life for a wide range of users, particularly those with physical limitations. By combining voice control, clap-switching, and manual control options, the system provides a versatile and inclusive solution for managing home appliances.

Future research could focus on refining the system's capabilities and expanding its range of applications. For instance, integrating additional sensors, such as motion or light sensors, could further tailor the system to individual user preferences and environmental conditions. Moreover, incorporating energy consumption monitoring and optimization features may contribute to promoting energy efficiency and conservation in the residential sector. Ultimately, continued advancements in IoT-based smart home technologies can lead to more accessible and sustainable living environments for all individuals, regardless of their physical abilities.