Pattern recognition on FPGA for aerospace applications

This paper presents a low power near real-time pattern recognition technique based on Mathematical Morphology-MM implemented on FPGA (Field Programmable Gate Array). The key to the success of this approach concerns the advantages of machine learning paradigm applied to the translation invariant template-matching operators from MM. The paper shows that compositions of simple elementary operators from Mathematical Morphology based on ELUTs (Elementary Look-Up Tables) are very suitable to embed in FPGA hardware. The paper also shows the development techniques regarding all mathematical modeling for computer simulation and system generating models applied for hardware implementation using FPGA chip. In general, image processing on FPGAs requires low-level description of desired operations through Hardware Description Language-HDL, which uses high complexity to describe image operations at pixel level. However, this work presents a reconfiguring pattern recognition device implemented directly in FPGA from mathematical modeling simulation under Matlab/Simulink/System Generator environment. This strategy has reduced the hardware development complexity. The device will be useful mainly when applied on remote sensing tasks for aerospace missions using passive or active sensors.

One of the most promising proposals to overcome these difficulties is the development of new low power embedded intelligent devices that can provide smart compression method for images and/or that can recognize patterns near real-time using low power consumption on the spacecraft. These intelligent devices also are great promise for more efficiently use regards high-resolution sensors with the restrictions of size, energy consumption and communication bandwidth in space links (Rapuano et al., 2021;Furano et al., 2020;Giuffrida et al., 2020;Felipe et al., 2006;Dawood et al., 2002;Chien et al., 2004;Silva & Lucena, 2005a;Silva & Lucena, 2005b).
In the specialized literature there are also other important papers with contributions from Mathematical Morphology-MM and machine learning hybridization techniques (Nogueira et al., 2021;Franchi et al., 2020;Jouni et al., 2020;Shen et al., 2019;Mellouli et al., 2019;Hao et al., 2019). However, none of these works has implemented in hardware any morphological operators based on Elementary Look-Up Tables-ELUTs paradigm as proposed in Silva (1998) and summarized at Section 2.
Besides most of these devices implemented by FPGA hardware have used RTL schematics (Xilinx-RTL, 2011), Verilog or VHDL programing methodology to generate HDL codes (D'AMORE, 2005) as entry method to build configuration files (bitstream) for hardware definition. The classical methodology used in these previous works requires hard effort to build scripts for hardware description in place of the use of functional models for description of tasks.
On the other hand, a signal-processing paradigm based on Look-Up Tables-LUTs is advantageous over FPGA architecture, considering LUTs are common blocks found in all modern FPGA chips. Due to their simplicity in terms of mathematical operations, in this work morphological operators for pattern recognition tasks as proposed by Silva (1998) have resulted in a first FPGA hardware device based in elementary operators from MM using LUTs paradigm applied to near real time patterns recognition tasks.
These operators are the mathematical basis to build adaptive controlled threshold template matching (Silva, 1998) without traditional convolution operations. These operators allowed the development of several different applications as described in previous works (Silva & Banon, 1999;Rempel & Silva, 2001;Silva & Silva, 2004;Souza et al., 2012;Souza et al., 2013;Filho et al., 2014;Filho et al., 2015). The present work proposes a new standalone device for near real time pattern recognition based on Mathematical Morphology-MM operators defined by ELUTs, taking advantages from an adaptive engine applied to morphological operators adapted from previous works (Silva, 1998;Silva, 2006;Filho et al., 2014). In this work, the device performs near real-time signal processing in pattern recognition on satellite images.
These morphological operators are part of the formalism of gray levels MM developed using ELUTs (Heijmans, 1991;Banon, 1995;Khosravi & Schafer, 1996;Banon & Faria, 1997;Banon, 2000;Banon, 1995;Silva, 1998). In this work, the hybridization of the MM and machine learning is a combination of the scheme proposed by Silva (1998) and Silva (2006) added to a learning engine adapted from Filho et al. (2014) presented at Sections 2 and 3.
Before the implementation in hardware, the mathematical model has been exhaustively tested on Matlab/Simulink (Matlab-Simulink, 2015) environment in different applications. The main results were show up in previous pattern recognition works (Silva, 1998;Silva & Banon, 1999;Silva & Silva, 2004;Filho et al., 2014;Filho et al., 2015) and in image/video compression tasks (Souza et al., 2012;Souza et al., 2013). However the present work shows the first hardware implementation of the morphological operators based on ELUTs using templates generated using machine learning scheme adapted from Silva (1998) and Silva (2006) and implemented on simulated form in previous works (Filho et al., 2014;Filho et al., 2015).
The main idea of this new approach is to use a training set to obtain a representative pattern for template matching with digital images and this representative pattern, implemented in FPGA hardware. In this work, the training set consisted of elements of the same target with small imaging variations like different remote sensing satellites imaging from revisiting of the same ROI (Region of interest). This strategy enables the morphological operators to use previously trained templates to perform exact or inexact and controlled template matching tasks, which are relevant for the hit or miss detection of patterns in real cases on satellite images as developed in Section 3.
The major contributions of this work are a simple and low power FPGA embedded pattern recognition scheme from simulated mathematical modeling for hardware design. The work shows a new hardware implementation of morphological operators based on ELUTs and the first results. The implementation methodology, presented at Section 2, is a very successful adaptation of the techniques developed in Silva et al. (2015). The device can do near real time pattern recognition for exact matching or inexact matching defined by similarity levels parameters. The processing scheme shows details for each image transformation and respective partial results of output images on each of the main FPGA blocks. The work also presents hardware specifications of the device, power consumption and performance during a pattern recognition task in a satellite image. Therefore, it can also be useful as reference to define different FPGA chips applied to similar tasks.
Section 2 presents the functional design of fixed-point codification of the patterns recognition morphological operators in hardware. It shows the functional blocks details to build MM elementary operators based on ELUTs embedded in FPGA using System Generator (Xilinx-SG, 2014). Details of the mathematical modeling methodology on FPGA using System Research, Society andDevelopment, v. 10, n. 12, e83101219181, 2021 (CC BY 4.0) | ISSN 2525-3409 | DOI: http://dx.doi.org/10.33448/rsd-v10i12.19181 4 Generator (Xilinx-SG, 2014) and Matlab/Simulink environment (Matlab-Simulink, 2015) are in Silva et al. (2015). Section 3 shows the results for the device processing a Landsat satellite image with resources and performance from Xilinx Kintex 7 FPGA (XILINX-KC705, 2014). Final considerations are at Section 4.

Methodology
Before developing the device presented in this article, the authors followed steps: a) Analysis of performance and energy consumption for the near real time pattern recognition tasks applied to aerospace missions, using the results of the previous studies and the references quoted at Introduction Section; b) Study of low-power devices with potential to build near real-time pattern recognition artifacts, using the results of the references quoted at Introduction Section; c) Research on robust mathematical modeling for pattern recognition hardware device implementations, using the results of the references quoted at Introduction Section; d) Research on hardware devices suitable to meet the near real-time image processing requirements from (a) and (b), using the results of the references quoted at Introduction Section; e) Research on techniques for implementing of the mathematical formalism (Silva, 1998) in FPGA hardware ; and f) Design of a low power device for near real time pattern recognition tasks in digital images, subject of this work.
However, the focus of the current work are items (e) (Silva, 1998), from equations (6) and (7), on System Generator environment.
The operator (equation (6)) executes the summation of the intersections between erosion and anti-dilation operations, as shown in Figure 2.
Research, Society and Development, v. 10, n. 12, e83101219181, 2021 (CC BY 4.0) | ISSN 2525-3409 | DOI: http://dx.doi.org/10.33448/rsd-v10i12.19181 In this work mathematical representations are used with gray levels images, for domain (pixels positions set) and scale gray levels. The set of mappings from to is denoted as . Therefore, the digital images in domain are represented by . In the specific case denotes the set of mappings from a window to (Banon, 1995;Silva, 1998).
If ∈ , and is equal to 1, so is a binary image, otherwise, is a gray level image.
In Figure 2, the blocks "fw1", "fw2" and "image" are memory buffers that store the "g" image, and the − and + patterns are obtained using equations (1) and (2) presented below.
where ∈ , ∈ (representative patterns), 1 , 2 ∈ (the integer numbers set), 1 ≤ 2 , and − , + ∈ define two slack images. These two slack images define the intervals around the pattern to perform an inexact matching. Of course if 1 = 2 = 0 it will perform an exact matching condition. The blocks "erosion" and "anti-dilation", Figures 3 and 4, control access to buffers in which the images are stored and used in erosion and anti-dilation operations (equations (3) and (4)).
The operator from to 1 is a Template-Matching Operator-TMO (Banon & Faria, 1997) as summarized below: where the are operators from to 1 .
The result of the sup-generation operation for each pixel accumulated in "Accumulator" block executes the sum of sup-generating morphological operators and executes a kind of correlation measure. Figure 5 shows the "Accumulator" block implementation. Finally, Figure 6 shows how the • operator (equation (7)) finds the predefined similarity "l" greater than or equal to matching condition.
where ∈ and f is the output result from operator. The operator • from to 1 defines an especial erosion (Silva, 1998) that can identifies values greater than or equal to (matching condition).  (Silva, 1998) identifies the equal or predefined similarity values for matched condition.

Source: Authors.
A machine-learning engine called Thresholded Template-Matching Operator (TTMO) adapted from Silva (1998), Silva (2006) and Filho et al. (2014) generates the operator templates. Figure 7 shows the TTMO block diagram similar to the one shown in Filho et al. (2014), where in Adaptive Machine-AM: ∈ and = 1, . . . , ; in : ∈ , − and + ∈ ; in • : ' ∈ and ℎ ∈ 1 . The AM processes the N variants of a pattern contained in the training set, processes the g image and the representative − and + pattern from AM, and h is the output result from • presented on Detection block to indicate if the pattern is present (on) or not (off) on image g (Figure 7).

Results and Discussion
This section shows some results of preliminary experiments using the implemented FPGA device applied to pattern recognition in a Landsat satellite image. Landsat TM images, acquired free from Instituto Nacional de Pesquisas Espaciais -INPE (www.dgi.inpe.br).
The goal is to find in the 116 x 131 pixels image (Figure 8), a 48 x 26 pixels representative pattern (Figure 9), represented by the set − and + target patterns, previously trained by the adaptive engine with the help of the training set. In this case, the training set consisted of models of pattern images selected from other passages of the satellite, different from those acquired in the passage shown in the Figure 8.
Source: Adapted from Landsat satellite image. Source: Authors (from Algorithm 1). Figure 10 shows the signals generated in and • Blocks. The first graph, named "Phi activities", shows the serialized values of the sum of sup-generating morphological operator. When one of these values is greater than the predefined matching conditions, it means that the pattern image is included in the search image, as verified on the second graph, "Pattern detection". The third graph "Led on/off" shows a signal that turn on/off one Led terminal from KC705 board (XILINX-KC705, 2014) to indicate to the user if the pattern is present or not. The level of similarity between the ROI, blue polygon in Figure 8, and the representative pattern in Figure 9, used in this case for definition of matching condition, was equal to 86%.  For all processing, it is necessary to perform a scan in the input image to search the target pattern. The amount of clock cycles (cycles in equation (8)) required for the entire process to be completed is: where , , e are the number of rows and columns of the image to be processed and of the pattern to be identified.
The time required (inference time in equation (9)) for the proposed image processing applied on input image (Figure 8) to detect the pattern in Figure 9, assuming 200 MHz FPGA clock, was approximately 45.68 ms, using equations (8) and (9): Figure 11 shows the processing details step by step for each image transformation and respective partial result with the output images in each of the main processing blocks. Figure 11. The complete scheme of the device tasks with the respective outputs in each block.

Final Considerations
This work has shown a new hardware implementation for ELUTs based morphological operators developed for pattern recognition in future space missions. The final bitstream file generated directly from simulated mathematical modeling running on Matlab/Simulink environment (Matlab-Simulink, 2015) reduces the abstraction gap between the mathematical modeling and the finished hardware. The Matlab/Simulink environment to simulate the mathematical model and generates FPGA bitstreams using the System Generator toolbox (Xilinx-SG, 2014) worked efficiently, similar to the presented in .
As shown in Section 3, the device took only 6% of the resources of the FPGA Kintex7 without any sophisticated feature as DSP blocks, i.e. as seen in Table 1; the device only demanded LUTs and registers, so the whole scheme also can run in an on-orbit reconfigurable radiation tolerant FPGA chip as XQRKU060 Kintex (XILINX-RTK, 2021).
It may be especially useful when applied on detection of environmental changes using small satellites like remote sensing nanosatellites, because this new high performance device can overcome the major limitation of size and low power restrictions from small satellite projects.
In addition to the tasks of pattern recognition proposed in this paper, we expect that the same operators implemented in FPGA can also be adapted for image compression tasks from previous works (Souza et al., 2012;Souza et al., 2013)

Led(on)
Research, Society andDevelopment, v. 10, n. 12, e83101219181, 2021 (CC BY 4.0) | ISSN 2525-3409 | DOI: http://dx.doi.org/10.33448/rsd-v10i12.19181 In this first hardware implementation, the work presented only a simple set of morphological operators as a single canonical artificial neuron model (Silva, 1998) when compared to a full deep learning paradigm as the morphological artificial neural networks shown in previous works. (Nogueira et al., 2021;Franchi et al., 2020;Jouni et al., 2020;Shen et al., 2019;Mellouli et al., 2019;Hao et al., 2019). Currently the authors are working on the parallel graph analysis to find main bottlenecks for each operator to increase image access and multiplicity of operators used in order to explore both spatial and temporal parallelism forms (Downton & Crookes, 1998;Johnston et al., 2004). The authors intend to build this new one approach toward a full parallel model suitable for deep learning paradigm. However differently from the previous works (Nogueira et al., 2021;Gianni et al., 2020;Shen et al., 2019;Mellouli et al., 2019;Hao et al., 2019;Jouni et al., 2020) the authors intend to build a new morphological neural networks based on ELUTs.
The full parallelism approach, implemented in hardware, may also be useful when embedded in the new generation of smart satellites for earth observation. However before trying a satellite embedded version, the authors intend it to perform efficient near-real time pattern recognition tasks in wildfire monitoring taking as reference the strategy presented in (Ban et al., 2020).
The device permits exact or inexact matching so it can be also useful for pattern recognition in digital signals from different kinds of sensors such as for example hyperspectral imaging sensors or active sensors as LIDAR, Radar, SAR or InSAR.
For future applications using this device, the authors suggest testing the device to detect changes on earth surface such as for example due to forests under wildfire or deforestation, soil and water contamination, land degradation, soil erosion and landslides.
For future research work, the authors suggest increasing the parallelism of the device to build new artificial neural networks models suitable for deep learning techniques or new paradigms of computer vision with a strong biological inspiration .