Floor Segmentation Approach Using FCM and CNN

Floor segmentation has a wide range of applications in engineering field and is considered as a complex task in image processing [1], [2]. Certain patterns in the set of image pixels need to be recognized to segment the floor. Texture and color are both important characteristics. However, due to illumination conditions, finding specular reflections and shadows in floors is commonplace, which is one of the main difficulties faced by most segmentation algorithms. Nowadays there are lot of mathematical calculations done by computer vision techniques, which can be used for floor recognition. The floor recognition process can be considered as the preliminary approach for autonomous robot navigation and several other applications. In industrial environments the use of autonomous robots have large impact and varied applications. Because working in toxic and other hazardous environments is very difficult for human being, the work is preferred to be done by unmanned vehicles and robots. Proper navigation scheme and floor maps are required to assure smooth working of the system. More intelligent and automatic devices are capable of doing several tasks. Navigation requires not only path planning but also current location tracking.

A lot of studies were conducted on navigation of living beings in neuroscience history showed that the information captured from environment was stored as special information in human brain, including location data. When a human being walks through an environment, the previously stored spatial information creates a proximity sense, which decides how much the person moves forward and backward in the environment. Taking car driving as another example. The person diving the car normally has an idea of car dimensions and road area. There are some blind spots which the driver cannot see while sitting inside the car. The side mirrors and the camera were activated when the car moves backward, which enables the driver to see the free empty space and helps to calculate the path to move in any directions easily [3], [4], [5]. The navigation of robots and autonomous vehicles uses visual information captured by camera. These approaches replaced traditional schemes because of their robustness and simplicity. The system developed in these scenario needs to detect static and moving objects to guide robots within the given area, which may be indoor or outdoor [6], [7]. Compared with other sensor-based navigation system, the visual based system has more advantages, particularly, lesser costs and higher effectiveness. The information acquired from those images are higher than the sensor based systems.

Floor segmentation is a difficult task in real-time scenario. The pixel content recognized from the captured scene is used for recognition. Due to illumination in different scenarios like indoor and outdoor, some features important to detect edges and corners are missing in the image captured. In some cases the shadows cover some corners, whose segmentation is more complex. Reflections are one major issue affecting the segmentation. These issues cannot be avoided in real-world scenario but can be removed little bit by image pre-processing, such as brightness adjustment etc. This paper proposed a novel approach for floor segmentation, which overcomes all the limitations in previous research and mainly focuses on image pre-processing and segmentation in multiple steps.

The remainder of this paper was organised as follows: Section 2 began with a brief introduction of related research; Section 3 proposed floor region detection algorithm; Section 4 discussed the performance analysis of the proposed algorithm; Sections 5 and 6 showed the experimental results.

Incorporation of image processing framework into navigation opens up a new era for navigation technology, which is considered as one of the most important advancement in the field. Vision based navigation also includes moving robots, which have some features, such as segmenting floors, appliances, walls etc. Recent works have discussed different types of techniques used for floor segmentation and path planning. In this section, we discussed about those technologies and found there relevance. Adachi et al. [8] replaced costly three-dimensional image sensor with high definition webcam to acquire surrounding images of robot movement. They developed a road following robot with high accuracy and robustness and used semantic segmentation to identify boundaries of the movable area. The system initially detected the boundaries and estimated lines. Their study developed the target point through intersection between two lines and considered a centroid for target path. The centroid point was updated according to changes in boundary lines. They tested the system in an indoor lab, whose performance was good in obstacle detection and corner detection. But the system needed to be tested in a complex outdoor and indoor scenario with more corners and obstacles.

Barcel´o et al. [9] presented a single grey scale image sequence based indoor navigation, which combined previously detected vertical and horizontal lines together for segmentation. The system detected several types of indoor scenes and its performance was not affected by camera movement. The system paid more attention to moving features. Features of moving objects in the scene were discarded in order to estimate the moving features of segmented area. Orientation of the camera was kept relative to ground to improve the result. The system was tested in an indoor environment and its performance was around 60%. One important issue in mobile navigation is the detection of free space. The study of Geovani Rodr´ıguez-Telles et al. [10] showed that the robot navigation was possible using a low resolution image. The system focused on free-space segmentation in front of camera for robot navigation. For capturing the environment scenes, monocular camera was used and placed downward to face the floor. The system captured low resolution images from real-time environment, required neither camera calibration nor optical flow ground plane constraints, and used simple linear iterative clustering super pixel algorithm for segmentation. Free spaces and objects were easily segmented using the system. Then the system was tested in several indoor environments. In some cases, such as objects with shadow, smaller size objects, dark areas, etc. were not segmented well, which affected the system performance. The proposed system could be applied to indoor environments, but was not suitable for outdoor real-time navigation.

Ma et al. [11] introduced a fixed camera based floor segmentation for robot navigation. Shaking initially caused the problems of video capturing procedure. When camera was placed on a moving robot, the chances of getting noisy image were high, which may reduce the performance. In their proposed system, the camera was fixed on the ceiling to capture video of the floor and moving robot. The segmentation process was implemented in the detection system of floor area and moving robot position. The system had some major limitations. For example, the number of fixed cameras depended on the area covered; obstacles or objects with the same colour as the floor were not identified. Wang et al. [12] proposed the floor segmentation framework based on gravity vector estimation and developed semantic segmentation for indoor environment analysis. In this proposed system, RGB-D image sensor was used for surface segmentation. The system detected floor, wall, and small objects using surface normal clustering and captured geometry information of the object from the depth information. Then the information was used to objects and surface accordingly. The system was only capable to detect indoor environments and tested according to indoor scenarios. The effect of shadow also affected the system performance in real-time experiment.

De Jesús Osuna-Coutiño and Martinez-Carranza [13] proposed a combined approach for floor segmentation made two types of analysis. The first one was floor connection, used for increasing recognition robustness. The second one was the recognition framework avoiding misrecognition. Experiments showed that the system performed better than previous work. Three different datasets were used for analysis using binary features. The system ran without Graphic Processing Unit (GPU), which made its application in low budget. Chen et al. [14] proposed a deep learning approach for image segmentation. They applied atrous convolution for feature extraction and used the upsampling filters for increasing the features. The system provided accurate segmentation with proper boundaries for objects. The work was done through experiments with several complex datasets, which achieved sufficient result. One of the major limitation was the efficiency of the system. Some objects, such as bicycles, chairs etc. were not detected properly and their boundaries were not accurate. It was suggested to improve encoder and decoder with high resolution features.

Skoryk et al. [15] proposed computer vision floor segmentation methods, which showed multiple algorithms and segmentation quality. Considering classical computer vision approach and deep learning approach in a fusion format, they used two different datasets for analysis, NYUDv2 and SUN-RGB-D. They initially analysed using classical approach based on super pixels and then used deep learning techniques like CNN and Fast CNN. They finally fused two approaches for better performance because both approaches had limitations. But the system was effected due to lack of segmentation quality. Kida et al. [16] proposed an algorithm to detect objects and surface in real-time environment, which focused on detecting human bodies and properties. But surfaces and objects were also segmented using sampling methods. The free space was detected using the position vectors distributed in the area. When being tested in an indoor environment with one human standing and one moving, the system segmented human body and surface properly.

Du and Arslan [17] proposed a magnetic field based indoor positioning scheme. The segmentation based K nearest neighbour (KNN) algorithm was implemented in the proposed system for estimating location. This approach decreased the computational complexity by partitioning and selecting desired area for calculation. This partitioned target area was fed to KNN which increased the accuracy of the entire system. The experiment results showed 9.24% performance improvement than the standard KNN model. Bormann et al. [18] conducted a survey on multiple room segmentation techniques to determine the most effective one. The comparison showed the advantages and disadvantages of each algorithm. Four algorithms were selected for implementation with help of locally available ROC package. Their study helped new developers to find the best segmentation method. Honto et al. [19] developed a floor map segmentation system for indoor navigation, which captured the map of indoor area and detected the passage ways. A semi-automatic way was proposed to segment floor from the map. Information retrieved from the map was used for simple operation for users. Both grabcut method and snake method were used to segment passageway and other surface.

Ambruş et al. [20] introduced 3D point information based room segmentation approach, which reconstructed room information from raw points and created a floor plan with minimized energy consumption. The system initially detected primitive points from the 3D cloud data and then detected ceiling and wall area. The study detected openings using 2D projection and calculated last view point for reconstruction. Lu et al. [21] proposed floor planning approach for rural housing, which was a new method for room segmentation. Deep learning neural network algorithm was utilized to decode information for floor plan. 1D loop was used to detect the wall, window and door openings, which was considered as room. Then the room region was identified using text information. The system converted the floor plan to room layout which had all room attributes. The major limitation of the system was lack of semantic information of rooms. Fleer [22] created a room cleaning robot with panoramic camera and laser diodes. The idea was to develop a robot which considered the room as human being. The robot compared the image input and sensor data for floor detection and path planning for navigation. The surrounding video was captured by the panoramic camera and distance between objects was detected using lasers. The system was tested in real and simulated environments, which segmented the room environments. But the system failed in some room border detection, which effected the performance.

The above recent segmentation research was based on deep neural network and other models. However, due to storage limitations, it was not workable to implement segmentation neural network architecture on an embedded platform, such as a field-programmable gate array (FPGA). Therefore, apart from machine learning technology, this paper proposed floor region segmentation algorithm, which combined boundary identifying methods with classification topology to differentiate the floor region from non-floor region. The proposed algorithm first identified the boundaries of the object using FCM method. Then CNN classification topology distinguished the floor region from non-floor region. By far most of the vision-based floor regions detection algorithms are still dependent on the depth sensors or complex models with relatively more power consumption. This paper proposed a new floor estimating algorithm based on image segmentation using a single image, which combined extracting surface texture characteristics with a specific geometric area to discover object boundaries and distinguish between the floor and non-floor regions using CNN classification.

The proposed method consisted of two stages: FCM and CNN segmentation. In FCM segmentation, the given input image was partitioned into homogeneous regions based on similarity for merging. In CNN stage, the interactive information was introduced as markers of the object area and background area, which were input by users to roughly indicate the position and main features of the object and background. The markers were in the form of simple strokes. The proposed method calculated the similarity between different regions and merged them based on the proposed maximal similarity rule with the help of these markers. Then the objects were extracted from the background when the merging process ended. Figure 1 showed the proposed block diagram.

Clustering is the method of forming homogenous data by separating the data and considering the object relationship, which allocates v feature vectors into N clusters. Every n-th cluster has Cn as its center. Fuzzy clustering was employed in numerous areas, such as pattern recognition and fuzzy detection. FCM was extensively used among various kinds of fuzzy clustering processes. FCM utilized reciprocal distance to determine fuzzy weights. The input of this process was known amount of clusters N. The mean spot of every member of a cluster was identified. The output was segregating N clusters in the object class. The FCM clustering was performed to reduce the total weighted mean square error (MSE).

The FCM allowed every feature vector to match several clusters with different fuzzy membership values. The final segmentation was based on the feature vector’s optimum weight in all clusters. The steps involved in FCM algorithm were given below.

Algorithm-1: FCM process

By implementing the clustering process FCM, the lung of the input chest computed tomography (CT) image was segmented into left and right parts.

Input: Feature vectors (image voxels) v = {v1,v2,……,vn}, N = count of clusters.

Output: N, number of cluster groups with less sum of distance error.

Steps:

1. Fuzzy weighting was used to set random weight for every pixel, with positive weights{W^v_n} ranging from 0 to 1.

2. The starting weights for each v-th voxel on all N clusters were normalized.

3. The weights on n= 1,…,N for each v were normalized to get W^v_n.

4. Estimated new centroids Cn, n = 1,...,N

5. Updated the weights {W^v_n}

6. If the input was altered, repeat from step 3. Or the process should be stopped.

7. Every pixel was matched its corresponding cluster. The clustering process FCM segmented the image into left and right parts. Figure 2 shows the initial segmentation using FCM.

Among the available CNN, U-Net had its U-shaped encoder-decoder structure. It segmented images by down-sampling and up-sampling using the original images and took out the feature map similar to its actual type of image. Composed of contracting and expanding paths, U-Net was used in delineations needed for radiation treatment planning because it integrated local features with general location information of the object. Even with a large number of samples, Unet still produced relatively acceptable results. Both 2D and 3D format was used to construct it, with respective benefits and drawbacks. In the 2D approach, the U-net planning was practiced with input-output pair of part of 2D; a fractional architecture was fed by any 3D sub volume by replacing all 2D operations with equivalent 2D operations for 3D approach.

An accurate 3D U-net extension was accomplished by U-net only. The 3D U-net authorized to proceed in 3D sub volume and received output of each graphical information of 3D space in the volume specifying tumour expectation. 2D convolutions produced a single image by applying the same weights throughout the entire depth of the frame stack (many channels). In order to preserve temporal information of the frame stack, 3D convolutions employed 3D filters and resulted in a 3D volume. 2D U-net reduced the 3D direction of information because each image was treated separately. However, this system was capable of learning from several samples. The 3D direction of information was enriched while the number of samples was reduced in 3D U-net, thus increasing the quantity of information per sample. Semantic segmentation involved labeling each pixel in an image or voxel in a 3D volume. Because 3D U-net offered the best segmentation for brain tumour sub-regions in Magnetic Resonance Imaging (MRI) modalities, 3D U-net was used for semantic segmentation of MRI image. The 3D U-net, made of a contractive (encoder path) and expanding (decoder path), used convolution and pooling to build a bottleneck in the middle of the path (Figure 3).

Convolutions and up-sampling were used to recreate the image after this bottleneck. As part of the deep convolution neural network, the 3D U-net segmentation presented a network and test approach based on application of data enlargement to given images, which was more effective. A contracting path was used to capture circumstances, and a balancing growing path was used for exact localization. A CNN-based architecture was frequently utilized to categories labels. But the goal in medical imaging should be more than just classification. Apart from localization, the goal was set to foretell the class label of each pixel using the context of its immediate surroundings as input. The image context, captured via the encoder path, was merely a stack of maximum pooling and convolutional layers. Depth of the context increased as the image size was gradually decreased by the encoder. This basically meant the network forgot to learn the "WHAT" information while learning the "WHERE" information in the image. The encoder network learned a conceptual representation of the input image using a sequence of encoder blocks, thus performing the function of a feature extractor.

Set of two 3x3 convolutions formed one block, which followed a rectified linear unit (ReLU) activation function. ReLU added nonlinearity to the network, assisting in the generalization of training data. The associated decoder block was the skip connection for the ReLU output. The ReLU activation function was followed by the dropout function. By deleting (ignoring) a few randomly selected neurons, the network was compelled to learn a new representation. Neurons became independent with the help of network. In turn, this promoted generalisation and prevented overfitting in the network. Then a 2x2 max-pooling was used to cut the structural dimensions (height and width) of the feature maps in half. The computational cost was decreased by lowering the number of trainable parameters. The bridge completed the information flow by connecting the encoder and decoder networks. The flow consisted of two 33-convolutional layers, with a ReLU activation function placed after each. The size steadily grew while the depth gradually lowered, with the decoder path providing precise localisation. A semantic segmentation mask was produced from the abstract representation using the decoder network. Starting with a 2x2 transpose convolution, the decoder block was activated. Then the block joined the relevant skip connection feature map for the encoder block. These skip connections offered functionality that occasionally lost owing to network depth. Then two 3x3 convolutions were utilised, with each followed by a ReLU activation function. The final decoded output was subjected to 1x1 convolutions with sigmoid activation. The pixel wise classification was represented by masked segmentation which was created by sigmoid activation function.

As a result, this network was entirely convolutional from beginning to end. To retrieve the "WHERE" information, the decoder gradually used up-sampling (precise localization). Output of the transposed convolution layers was concatenated with the feature maps from these encoders in order to leverage skip connections at each stage of the decoder to achieve more precise positioning. These skip connections gave the decoder additional data, enabling it to produce more precise semantic characteristics.

In addition, the skip connections acted as a shortcut connection, enabling gradients to pass to lower layers not in a degraded way. Skip connections, to put it simply, increased the gradient flow during backpropagation, enabling the network to learn better representation. After each concatenation, two consecutive regular convolutions were applied so that the model learned to put together a more precise result. Due to the size, complexity, and memory requirements, a full image could not be offered for training. As a result, the data was standardized, which generated random sub-volumes. The image we utilized had the size 128x128x128x3, but the dimensions of the input image were 256x256x256x3. As a result, the size varied from that of the original at various spots, but the essential element stayed the same.

Evaluation results of different evaluators varied significantly, because each evaluator may have distinct standards for measuring the segmentation quality. S1 and S2 were two images.

Rand index [23]

where, I is the identity function, and the denominator is the number of possible unique pairs among N data points. This gave a measure of similarity ranging from 0 to 1.

Variation of information [24]

The first term in the above equation measured the amount of information about S_test that we lost, while the second term measured the amount of information about S_K that we had to gain, when going from segmentation S_test to ground truth S_K. Where, H (.|.) is the conditional entropy.

Global consistency error [25]

Global consistency error measured the extent to which the regions in one segmentation were subsets of the regions in second segmentation (i.e., the refinement). Let R(S, p_i ) be the set of pixels in segmentation S containing pixel P_i, then the local refinement error was defined as:

Boundary displacement error (BDE) [25]

$B D E=\min \left(\left|x-y_i\right|\right)$, where $\mathrm{x} \in \mathrm{B} 1$ and $y_i \in B 2$, i=1,2,...,n, n is the number of boundary points in B2.

To evaluate the segmentation results produced by different algorithms, a database was compiled, which contained 300 corridor images along with ground truth segmentations. The corridor images were taken in twenty different buildings, exhibiting a wide variety of different visual characteristics. Segmentation method was evaluated by assessing its consistency with the ground truth segmentation given by the human expert. Any evaluation metric desired should take into account the effects of over-segmentation and under segmentation. The former refered to the reference region represented by two or more regions in the examined segmentation. The latter refered to two or more reference regions represented by a single region in the examined segmentation. In accurate boundary localization, the ground truth was usually produced by humans that segmented at different granularities. Finally, when there were different numbers of segments, two segmentations needed to be compared. Table 1 shows the parameter values of different segmentation methods. An image should have higher PRI value and lower VOI, GCE and BDE values. Each parameter was described by ground truth and proposed method. Each row was represented by average of each class of totally about 100 images. According to the Table 1, the proposed method achieved 0.9725 PRI, 2.23 VI, 2.14 GCE and 1.24 BDE. Y-axis represented number of samples. The proposed method achieved good results. From this evaluation, it was found that region merging segmentation was well suited for the corridor images. Figure 4 showed examples of segmentation results obtained by the proposed floor segmentation method. Further, to evaluate the performance of the proposed method, we selected images from MIT Scene dataset [26] and implemented the proposed method (Figure 5). In addition, this paper analyzed the measures of PRI, VI, GCE and BDE in Table 1 in graphical representation for floor segmentation shown in Figure 6, Figure 7, Figure 8 and Figure 9. Finally, our method was compared with other well-known methods, such as SegNet [27], Deep Lab and FCN [28], shown in Figure 10.

This paper proposed an approach for floor segmentation using FCM and CNN. In FCM segmentation, the given input image was partitioned into homogeneous regions based on similarity for merging. In CNN stage, the interactive information was introduced as markers of the object area and background area, which were input by the users to roughly indicate the position and main features of the object and background. The segmentation performance was measured using probabilistic rand index, variation of information, global consistency error, and boundary displacement error. Experiments were conducted on a relatively large database of floors. It was observed that the proposed floor segmentation achieved relatively good segmentation results, compared with other existing well-known methods. Further, we worked on large datasets for mobile navigation of robots.