Fuzzy-Based Multi-Crop Classiﬁcation Using High Resolution UAV Imagery

Accurate information regarding crop classiﬁcation is important to estimation crop yield. It is used to depict the relationship between exponentially growing world population and food demand. The purpose of this research is to recognize multiple crops in a single UAV-based image. The task itself is chaotic as every crop exhibits similar hue, color and other plant characteristics. In this paper, an Adaptive Neuro-Fuzzy Inference System (ANFIS) is proposed to classify 17 diﬀerent crops based on their high spatial and temporal signatures of normalized diﬀerence vegetation index (NDVI) values acquired through multispectral sensor onboard a quadrotor. The multispectral images were classiﬁed into two classes (soil and crop) and NDVI signatures for each crop were extracted from images. Detailed dataset was prepared as a timeline through sampling, covering almost all phenological phases of the crops. The NDVI dataset was passed through ANFIS to classify NDVI vectors. ANFIS had only one output variable: the crop type that was formulated from 8 input variables. ANFIS used 2 membership functions for one input variable and formulated 256 fuzzy rules for the classiﬁcation. The results show a high level of classiﬁcation accuracy.


Introduction
C rop classification is important to monitor agriculture farming operations that are used to increase crop yield, variable rate fertility, and helps farmers to understand the effects of climate change on crops. Accuracy in crop classification systems is critical to precisely address the stress in the plants triggered by different biotic and abiotic factors that consequently helps in taking appropriate decisions at right time [1]. Most of the countries are using different precision agriculture techniques to enhance the economy by producing high quality crop yields. Crop classification may produce incorrect labels as many crops exhibit the same color, texture, and other visual characteristics at early stage leading to produce similar spectral signatures for different crops [2]. Satellitebased remote sensing techniques are in use since decades for large area estimations quite efficiently, but are not so effective for small scale farming with complex crop patterns. Nevertheless, the provision is achievable through UAV (Unmanned Aerial Vehicle) based high resolution image data as proved in many applications for site specific precision agriculture. In contrast to the satellite image data, UAVs capture the ISSN: 2523-0379 (Online), ISSN: 1605-8607 (Print) sample information from the crop fields at much lower height to deliver high quality imagery.
In this work, we used high spatial and temporal signatures of NDVI values to classify 17 different crops using a fuzzy inference system (FIS). Fuzzy logic is a generalization of multi-valued logic that assigns a degree of membership to objects rather than assigning boundaries and it returns values between 0 and 1. It is widely accepted that fuzzy logic is used for the classification of objects for which we have uncertain data or less amount of data [3]. Detailed dataset was prepared and NDVI signature of each crop was extracted. In this study, we trained the sugeno-type FIS by using adaptive neuro fuzzy inference system (ANFIS) to map the input to the output.
In ANFIS, each rule is linearly dependent on input variable and this property of sugeno method makes it ideal to work with dynamic nonlinear system. ANFIS works with a combination of fuzzy inference and neural network models. The membership functions used in fuzzy inference system (FIS) are adjusted in a way that it appropriately maps the input values to the expected output using the decision rules, just like it works in Convolutional Neural Network (CNN) that fine tunes the parameters in order to classify and recognize the object accurately. The proposed proposed system has 8 input variables (NDVI1January, NDVI15January, NDVI1February, NDVI15February, NDVI1March, NDVI15 March, NDVI1April, NDVI15 April) and a single output variable of crop type.
ANFIS is a supervised method for the selection of the membership functions with the training on input images to get the best results from the fuzzy inference model [4][5]. Conclusively, pixel-based classification of multispectral images, fuzzy logic and UAVs are leading ways for accessing accuracy in multi-crop classification problems.

Related Work
Researchers are widely using multiple techniques for crop classification and recognition that are providing new ways in precision agriculture. Shackelford and Davis [6] combined the object-based and pixel-based fuzzy approach to classify the urban land. They used multispectral IKONOS images and enhanced the resolution of those images with the help of the pansharpening method. Musande et. al. [7] used five different spectral indices: NDVI, TNDVI, SAVI, TVI and SR to recognize the cotton crop based on multispectral images. They studied the impact of various indices on the process of classification of cotton. They used a fuzzy classification approach and handled the issue of mixed-pixel to get a single discrete class. Ji et. al. [8] applied the support vector machine (SVM) on HJ satellite multiple combinations of data to recognize three classes of tree, cotton and wheat crop. They acquired the multispectral images during wheat growth stages in 2009 and used a different combination of multi-temporal Hj data for classification. This research indicated that multi-temporal data is more effective to identify wheat fields with 91.7% rather than single temporal HJ data.
In crop classification, problems existing remote sensing methods use high resolution image data to extract crop phonological information. Liu et. al. [9] proposed a method to classify the crop type using high temporal frequency satellite images and low resolution satellite images. They used a low temporal frequency of satellite images to extract the phenological characteristic of crops. They achieved 20% more accuracy by combining the low spatial resolution with high frequency data in the training process. To investigate the effect of length of time series in multiple crops classification, Hao et al. [10] used random forest algorithm on MODIS time series data. They used NDVIs and NDWI as decision parameters. They measured the class distinction for each time series and concluded that the most significant feature for the classification of the crop is NDVI. Moreover, five months were enough for the best performance of the classification process. Taufik and Ahmad [11] proposed a method to use three spectral ranges of Landsat-8 satellite, NDWI, NDBI and NDVI as input variables for fuzzy inference system (FIS). They used an adaptive neurofuzzy inference system (ANFIS) for the classification of three land types, i.e., vegetation, water and populated land.
Bargiel [12] proposed a classification method that considered information about the growth cycle of crops. He used phenological sequence patterns (PSP) of different crops using a dataset coupled with knowledge about the phenological stages of the specific crop. Different crops including rye, potatoes, sugar beets, grasslands, canola, maize, barley were used for classification purposes. PSP with RF (Random Forest) and ML (Maximum Likelihood) classification techniques were compared in this research. PSP approach outperformed other techniques in recognizing different crops.
Traditional crop classification and identification techniques take much computational and processing time but have a low accuracy rate. The authors in [13] used a deep learning neural network to recognize the rice area by using UAV based image data. Four classes, i.e., rice, weed, wasteland, and rice and wasteland were selected. They changed the image size to fit the dataset according to the network requirements and normalized the images for the training phase. They used a double Convolutional Neural Network (CNN) and ResNet50 to train the classifier. According to time complexity, double CNN was better than ResNet50 because it took a short training time.
Crop cover maps could be generated by using remote sensing data that is used to calculate the vegetation indices from different bands by spectral transformation. The authors in [14] used a multispectral sensor to cover near infrared, visible and short wave infrared by using 13 bands. The random forest classifier was used to generate land cover maps with four bands and eight spectral indices. Normalized difference vegetation index (NDVI) is widely used with time series as it gives information about the crop growth cycle. The imagery dataset accumulated by Landsat and Sentinel-2 is probably incomplete or with missing values. NDVI is affected by the revisit growth cycle and the weather has significant effects on plant growth. This problem is addressed in the system proposed by Tian et. al. [15] using multi-temporal NDVI with Landsat-7 and -8 and Sentinel-2, optical images. Zhao et. al. [16] proposed a method to classify crops by combining sentinel 2-A (satellite data) and UAV imagery data that was collected from Heilongjiang Academy Park of Agriculture Area, China. They used random forest algorithm and classified the crops into 10 classes in which 4 crops were rice, buckwheat, corn, and soybean, other vegetation's like, buildings, roads, greenhouse, general bare lands and water.

Methodology
Following sections describe the methodology used in this reserach.

Study Area
The field images were taken at the agronomy research farms of the University of Agriculture, Faisalabad, Pun-jab, Pakistan using an unmanned aerial vehicle. A plot of size 35m × 18m (Latitude: N 31 • 26'29.75", Longi-tude: E 73 • 4'23.33") in the research farm containing multiple crops is captured by using UAV and the aerial data was collected for a specified time. This plot was designed with 17 sub plots, each of size 18m × 1m considering 1m gap between adjacent subplots. Subplots were further designed with 4 rows having an approximate distance between rows as 0.3m. The detailed aerial view of the plot is shown in the Figure 1.  Table 1 contains the list of crops with their labeled value and seasonal time. All these crops are important from an economical view of the country and researchers are more interested in their classification with maximum accuracy.

Phantom-4 DJI with payload
A quadrotor (Phantom-4, DJI) platform was used along with an integrated customized payload that included a multispectral camera (ADC Micro, Tetra-cam  and TM4 as it consists of three defined bands (Red, Green and NIR). Aptina CMOS sensor and a Bayer RGB filter array along with a checker board pattern were involved inside the camera. These lightweight sensors operate within the wave length range of 520nm to 920nm that matches the requirement of a suitable remote sensing agriculture application. In an uncompressed raw-10 format at nadir position, the multispectral images were captured via ADC-Micro sensor. The calibration for IMU (Inertial Measurement Unit) and compass were first flight operations only and was not further required to perform in later flight operations the whole season. The right association between the two measures to exhibit the true vegetation pixel value is very crucial. To accomplish this, a Teflon calibration target was used, it gives 100% reflection in the range of wave length about 400nm to 1000nm. Spectral balance of sunlight in-situ by using this target and is then used to render corrections in the R / G and NIR bands. This data was further used in the calculation of NDVI values in multispectral image post processing using the following equations, where RED is the value of reflection in the red spectrum, NIR is the value of reflection in the near infrared range of the spectrum. NDVI values were used to recognize crops. The classification was totally based upon RED and NIR Fig. 2: Phantom-4 DJI with payload Fig. 3: Climate-variables observed during Experiment reflectance exposed by the canopy pixels [17]. The portion of vegetation shows a higher NIR reflectance. Moreover, higher red values are exposed by the soil sections (also backdrop shades, etc.). A specific part of area (1m 2 ) was identified at the center of each sub plot to extract NDVI values for these crops. The total data collection period was of 4 months. The flight operations were performed with a difference of 15 days at an altitude of 50m above the ground level. The ground sampling distance observed was of 2cm spatial resolution. Observation of climate variables during the experimental phase (2019-2020) is shown in Figure 3.

Pre-Processing and Pixel Classification
The multispectral images were segmented into two classes, i.e., crop and soil before extracting the NDVI values from a particular crop class. The crop part showed higher values in near infrared bands and the soil class showed greater values in red ranges. The purpose of this classification was to discard soil pixels so that only the canopy pixels participate in the extraction of the NDVI values and the whole classification process. NDVI shows greater positive values when subjected to the vegetation part of the image and negative values when subjected to cloud and snow. It shows positive low values for water bare soil and it has a range from -1 to +1 [18].

Data Sampling & Spectral Signature Extraction
High spatial and temporal NDVIs were calculated from multispectral images for each sub plot [19]. As mentioned earlier, a fixed area of 1m 2 dimension that contained crop pixels for the subplots marked in the middle for data acquisition (NDVI value) for each crop during the decided period of monitoring the crops. Canopy pixels (≈ 3000) inside each fixed area were averaged to calculate the NDVI value of a particular crop against a specific date. In this way, NDVI values were extracted for 17 different crops each day. This process is repeated for 8 different samples taken with an interval of 15 days during the months January to April to cover almost all phenological stages of each crop. In this way, this data helped in designing the detailed dataset as a time series of NDVI values of the 17 crops. Thus, each crop was represented as a distinct vector of 8 scaler NDVI values. This is an NDVI time series table in which each vector for each crop is represented as a spectral signature of that crop with 8 input NDVI values. Classification of these 17 crops was challenging because all crops follow almost the same pattern of NDVI time series having less variation, i.e., from rising until mid-season and then falling at the end. Line graph in Figure 5 is used to see the pattern followed by these 17 different crops. Most of the lines are very close to each other and exhibit minor changes in their pattern of growth in the season.

Classification
To classify the model based on NDVI time series data, there were 500 sample vectors of 8 distinct NDVI for the testing of the fuzzy model. An adaptive neuro fuzzy inference system (ANFIS) was used to classify 17 crops by using the Sugeno or Takagi-Sugeno-Kang fuzzy model. Sugeno-type inference systems use singleton output membership functions that can only have two possible values, i.e., linear or constant. The de-fuzzification process of the Mamdani-type fuzzy inference system is computationally more expensive than a Sugeno-type model. In the sugeno-type fuzzy model [20], a rule can be written as follows, where m and n are the input values combined with the fuzzy OR operation, z is the weight assigned to this rule; whereas a, b and c are the numbers that indicate the output level of the rule. The output level of a rule is assigned according to the strength of its firing (w i ). For a zero-order Sugeno model, the output level is constant (i.e., a = b = 0). For example, for an OR rule with Input 1 = m and Input 2 = n, the firing strength is calculated as follows, were F 1 (m) and F 2 (n) are the membership functions for the inputs m and n. The aggregated output of the system is calculated as follows, where c is the number of rules, z i is the weight assigned to each rule, and w i is its firing strength. The membership function parameters of an adaptive neuro fuzzy inference system (ANFIS) are tuned using neuro adaptive learning methods similar to what we do in the training of the conventional neural networks.
In the proposed method, we generated a Sugenotype FIS with singleton outcome result and then using the specified input and output training data, tuned the device parameters. The generated FIS structure used the grid partitioning method for the formation of the FIS. The proposed algorithm used a combination of the least squares and gradient descent methods for backpropagation to interpret the training dataset. A grid partitioning method of the input variables was used to construct the FIS structure along with two membership functions (MFs) for each input variable. The use of two membership functions is important, as the use of greater number of membership function may lead to higher accuracy but that may also increase the computation time of the model as well. Therefore, the choice of number of membership functions for the fuzzy model is critical because we cannot compromise on accuracy as well. Figure 6 shows the proposed FIS formation diagram with 8 input variables (NDVI-Jan01, NDVI-Jan15, NDVI-Feb01, NDVI-Feb15, NDVI-Mar01, NDVI-Mar15, NDVI-Apr01, NDVI-Apr15) and a single output variable: the CropType. The labels of the crop A, B, C. . . Q were converted into numbers as 1, 2, 3. . . 17 respectively for the model to work on numerical data and map the numeric input NDVIs to the output number. Moreover, in the proposed FIS formation, 2 membership functions were used for each input variable; consequently, having 16 MFs for 8 distinct input variables. In the proposed study, the type of membership function used for each input variable was Gaussian Bell Member-ship Function. Through these 16 MFs, 256 decision rules (i.e., 16 x 16 = 256) were formulated to classify the input values as a single output. These decision rules are also known as fuzzy rules and being the second parameter of the ANFIS structure, they play a vital role in mapping the output as a single value [21].
With these 256 fuzzy rules, 256 fuzzy MFs are used for the single output variable and the type of MF used for the output variable is linear. The reason behind this linear type is that Sugeno-type FIS works with multiple input variables, but maps those inputs to one single output and that output can be linear or constant. As we use multiple MFs, the probability of accurate classification increases as the input variable becomes more robust in identifying the discerned patterns of the data.
After training the ANFIS, the proposed method defined soft boundaries for the input variables in terms

Results & Discussion
The actual and predicted outcomes were compared to calculate Mean Square Error (MSE) for each row of the training set and, then an overall MSE for the model was computed using Equation 5, where P re i represents the predicted outcome of the input, Obs i shows the observed or actual outcome of that input, and n represents the total number of observations. MSE is frequently used to measure the performance of the proposed method. Its value ranges from 0 to ∞ such that nearer it is to zero, greater the accuracy of the classifier [22]. In the training process, we calculated the Root Mean Squared Error

Rule256
If (NDVI Jan01 is NDVI Jan01mf2) AND (NDVI Jan15 is NDVI Jan15mf2) .. . . AND (NDVI Apr15 is NDVI Apr15mf2) Then CropType is CropTypemf256 (1) In this study, NDVI signatures were extracted as a time series of crops using multispectral images that were used to recognize different crops. The findings of this study are mainly focused on the accuracy evaluation and the application of the fuzzy decision rules. The 17 output classes were identified by training the ANFIS to evaluate parameter selection of fuzzy inference system, such as the parameters of membership functions (MFs). The predicted and actual values were compared to check the accuracy of the proposed system. To test the fuzzy inference system for multicrop classification, a subset of the prepared dataset was used to examine the results of trained FIS on the unseen data which has delivered impressive results. The classification was based on the NDVI values for 8 different samples taken with a gap of 15 days. Based on different evaluations, the overall accuracy of the model was highly satisfactory to implement it in real conditions.
The proposed ANFIS considered to be a significantly efficient classifier because of its fast learning rate and high accuracy with limited number of samples. It can also be used in agriculture applications and has potential to become a primary interest for the agriculture researchers to classify crops using only eight distinct NDVI values extracted at different days to cover maximum phenological stages of each crop. Although, different models can be developed for crop classification problems such as artificial neural network (ANN) that works on RGB images as well as multispectral images, but these networks take more time in computation and require a large amount of data to train the model [24]. According to More et. al. [25], comparative study of ANN has limitations such as less transparency in reasoning process, long model-learning time, and tendency to overfit training data. As the main limitation to the proposed work, the operating principle in the data acquisition seems difficult as the flight operation must be completed in 4 months with 8 distinct NDVI values obtained at 8 different days to classify the crop at multiple stages. However, the results suggest that high spatial and temporal NDVI values will turn out to be a vital source for site specific crop classification.

Conclusion
The proposed study presented a robust approach to classify the 17 different crops. We used a quadrotor to capture high-resolution aerial images of the plot with a more detailed view. We used a FIS for multi-crop classification using high spatial and temporal NDVI signatures. multispectral images were captured using a Micro ADC multispectral sensor onboard a quadrotor. A plot with 17 different crops was selected and with a regular gap of 2 weeks, we performed a total of eight flight operations on eight different days. In this way, mean NDVI values for each crop were calculated and NDVI vectors for each crop were formulated. These high-resolution temporal signatures proved to be a primary requirement to train the multi crop classification. The proposed fuzzy inference system formulated 256 fuzzy membership functions for the classification using 8-input NDVI values. The trained ANFIS showed the ranges of each input variable according to the dataset of 17 crops. Taking a closer look at the ranges tuned for each input variable, it is observed that the range gets longer from early to mid-season of the crops and then the range gets shorter again. It simply means that the crops look more distinct from early to mid-season. This also confirms that NDVI values calculated in this season are important to classify the crops accurately. We have used the fuzzy inference system in which each rule has equal weight, i.e., each input variable equally participates in the classification process.
For future work, we aim to include some other important vegetation indices like SAVI in the classification process, which cuts down the effect of soil background and can increase the accuracy of the classifier. Moreover, we also target to add other plant traits, such as plant height, plant shape, and hue in the classification process to improve the accuracy of multi-crop classification system.