Pattern recognition on FPGA for aerospace applications
DOI:
https://doi.org/10.33448/rsd-v10i12.19181Keywords:
Intelligent satellites; Nanosatellites; Artificial intelligence in hardware; Computer vision; Machine learning; Mathematical morphology; Pattern recognition; Real time systems; Aerospace applications; Remote sensing.Abstract
This paper presents a low power near real-time pattern recognition technique based on Mathematical Morphology-MM implemented on FPGA (Field Programmable Gate Array). The key to the success of this approach concerns the advantages of machine learning paradigm applied to the translation invariant template-matching operators from MM. The paper shows that compositions of simple elementary operators from Mathematical Morphology based on ELUTs (Elementary Look-Up Tables) are very suitable to embed in FPGA hardware. The paper also shows the development techniques regarding all mathematical modeling for computer simulation and system generating models applied for hardware implementation using FPGA chip. In general, image processing on FPGAs requires low-level description of desired operations through Hardware Description Language-HDL, which uses high complexity to describe image operations at pixel level. However, this work presents a reconfiguring pattern recognition device implemented directly in FPGA from mathematical modeling simulation under Matlab/Simulink/System Generator environment. This strategy has reduced the hardware development complexity. The device will be useful mainly when applied on remote sensing tasks for aerospace missions using passive or active sensors.
References
Akil, M., & Zahirazami, S. (1998). Multi-FPGA Processor for Gray Scale Morphology, European Association for Signal Processing, Eusipco-98, IX European Signal Processing Conference, Island of Rhodes, Greece, 8-11 September.
Arechiga, A. P., Michaels, A. J., & Black, J. T. (2018). "Onboard Image Processing for Small Satellites," NAECON 2018 - IEEE National Aerospace and Electronics Conference, 234-240, 10.1109/NAECON.2018.8556744.
Astua, C., Crespo, J., & Barber, R. (2014). Detecting Objects for Indoor Monitoring and Surveillance for Mobile Robots, Emerging Security Technologies (EST), in Fifth International Conference, 124-129.
Ban, Y., Zhang, P., Nascetti, A., Bevington, A. R., & Wulder M. A. (2020). Near Real-Time Wildfire Progression Monitoring with Sentinel-1 SAR Time Series and Deep Learning, Nature Scientific. 10(1322).
Banon, G. J. F., (1995). Characterization of translation invariant elementary morphological operators between gray-level images, Research report, INPE-5616-RPQ/671, INPE.
Banon, G. J. F., & Barrera, J. (1998). Bases da morfologia matemática para análise de imagens binárias, (2nd ed.), MCT/INPE, INPE-6779-RPQ/682.
Banon, G. J. F. (2000). Formal introduction to digital image processing. Technical Report: INPE-7682-PUD/43, São José dos Campos, Brasil., 2000. dpi.inpe.br/ banon/2001/01.11.16.04.
Banon, G. J. F., & Faria, S. D. (1997). Morphological approach for template matching, SIBGRAPI'97, IEEE Computer Society, 171-178.
Bekker, D. L., Werne, T. A., Wilson, T. O., Pingree, P.J., Dontchev, K., Heywood, M., Ramos, R., Freyberg, B., Saca, F., Gilchrist, B., Gallimore, A., & Cutler, J. (2010). A CubeSat design to validate the Virtex5 FPGA for spaceborne image processing, Aerospace Conference, 2010 IEEE, pp.1-9.
Belmonte L. M, Morales R., & Fernández-Caballero A. (2019). Computer Vision in Autonomous Unmanned Aerial Vehicles—A Systematic Mapping Study. Applied Sciences. 2019, 9(15):3196. https://doi.org/10.3390/app9153196
Beul, M., Krombach, N., Zhong, Y., Droeschel, D., Nieuwenhuisen, M., & Behnke, S. (2015). A high-performance MAV for autonomous navigation in complex 3D environments. In Unmanned Aircraft Systems (ICUAS), International Conference on, Denver, USA, 1241-1250. IEEE.
Blake, G., Deslinski, R.G., & Mudge, T. (2009). A survey of multicore processors: A review of their common attributes, IEEE Signal Processing Magazine, vol. 26, no. 6, pp. 26–37.
Brugger, C., Dal'Aqua, L., Varela, J., De Schryver, C., When, N., Klein, M., & Siegrist, M. (2015). A Quantitative Cross-Architecture Study of Morphological Image Processing on CPUs, GPUs, and FPGAs. In Proceedings of the IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE), April 2015.
Castelli, T., Sharghi, A., Harper, D., Tremeau, A., & Shah, M. (2016). Autonomous navigation for low-altitude UAVs in urban areas. arXiv preprint arXiv:1602.08141.
Chien, S., Sherwood, R., Tran, D., Cichy, B., Rabideau, G., Castano, R., Davies, A., Lee, R., M, D., Frye, S. Trout, B. Hengemihle, J., Shulman, S., Ungar, S., & Brakke, T. (2004). The EO-1 autonomous science agent.
Chien, S., Bue, B., Castillo-Rogez, J., Gharibian, D., Knight, R., Schaffer, S., Thompson, D., & Wagstaff, K. (2014). Agile Science: Using Onboard Autonomy for Primitive Bodies and Deep Space Exploration.
Chowdhary, G., Johnson, E. N., Magree, D., Wu, A., & Shein, A. (2013). GPS‐denied Indoor and Outdoor Monocular Vision Aided Navigation and Control of Unmanned Aircraft. Journal of Field Robotics, 30(3), 415-438.
D'AMORE, R. (2005). VHDL: Descrição e Síntese de Circuitos Digitais. Editora LTC, 276 p.
Dawood, A., Visser, S., & Williams, J. (2002). Reconfigurable FPGAs for real time image processing in space. In Digital Signal Processing. IEEE 14th International Conference on, 2. 845–848.
Dinelli, G., Meoni, G., Rapuano, E., Pacini, T., & Fanucci, L. (2020). MEM-OPT: A Scheduling and Data Re-use System to Optimize On-chip Memory Usage for CNNs On-board FPGAs. IEEE J. Emerg. Sel. Top. Circuits Syst. 2020, 10, 335–347.
Downton, A., & Crookes, D. (1998). Parallel Architectures for Image Processing. Electronics & Communication Engineering Journal, 10, 139-151.
Felipe, I., Dohm, J. M., Baker, V. R., Doggett, T., Davies, A. G., Castano, R., Chien, S., Cichy, B., Greeley, R., Sherwood, R., Tran, D., & Rabideau, G. (2006). Flood detection and monitoring with the Autonomous Sciencecraft Experiment onboard EO-1: Remote Sensing of Environment, 101(4), 463-481.
Filho, A., M. P., Silva, F. A. T. F., & Braga, A. P. S. (2014). Machine learning and adaptive morphological operators. In: XI Encontro Nacional de Inteligência Artificial e Computacional (ENIAC 2014). São Carlos, Brasil. Annals of XI Encontro Nacional de Inteligência Artificial e Computacional.
Filho, A., M. P., Silva, F. A. T. F., & Braga, A. P. S. (2015). Operador morfológico adaptativo de casamento de padrões – proposta e aplicação na análise de imagens de satélite. In: Simpósio Brasileiro de Automação Inteligente (SBAI 2015). Natal, Brasil.
Franchi, G., Fehri, A., & Yao, A. (2020). Deep morphological networks, Pattern Recognition, Volume 102.
Fuchs, T., Bue, B., Rogez, J. C., Wagstaff, K., & Thompson, D. (2014). Autonomous Onboard Surface Feature Detection for Flyby Missions, International Symposium on Artificial Intelligence, Robotics and Automation in Space (iSAIRAS), June 2014.
Furano, G., Meoni, G., Dunne, A., Moloney, D., Ferlet-Cavrois, V., Tavoularis, A., Byrne, J., Buckley, L., Psarakis, M., Voss, K. O., et al. (2020). Towards the Use of Artificial Intelligence on the Edge in Space Systems: Challenges and Opportunities. IEEE Aerosp. Electron. Syst. Mag. 2020, 35, 44–56.
Giuffrida, G., Diana, L., de Gioia, F., Benelli, G., Meoni, G., Donati, M., & Fanucci, L. (2020). CloudScout: A Deep Neural Network for On-Board Cloud Detection on Hyperspectral Images. Remote. Sens. 2020, 12, 2205.
Guo, K., Sui, L., Qiu, J., Yu, J., Wang, J., Yao, S., Han, S., Wang, Y., & Yang, H. (2018). Angel-Eye: A Complete Design Flow for Mapping CNN Onto Embedded FPGA. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 2018, 37, 35–47
Hagiwara, H., Asami, K., & Komori, M. (2011). FPGA Implementation of Image Processing for Real-time Robot Vision System. Proc. of International Conference on Convergence and Hybrid Information Technology, pp.134-141, Daejeon, Korea, September.
Hao, R., Wang, X., Zhang, Liu, J. J., Du, X., & Liu, L. (2019). Automatic Detection of Fungi in Microscopic Leucorrhea Images Based on Convolutional Neural Network and Morphological Method. IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC) (2019): 2491-2494.
Heijmans, H. J. A. M. (1991). Theoretical Aspects of Gray-Level Morphology. IEEE Transactions on Pattern Analysis and Machine Intelligence, v.13, pp. 568-582.
Hentati, R., Hentati, M., Aoudni, Y., & Abid, M. (2014). The implementation of basic morphological operations on FPGA using partial reconfiguration. IEEE IPAS'14: INTERNATIONAL IMAGE PROCESSING APPLICATIONS AND SYSTEMS CONFERENCE 2014.
Johnston, C. T., Gribbon, K. T., & Bailey, D. G. (2004). Implementing Image Processing Algorithms on FPGAs. In: Proc. Electronics New Zealand Conference, Palmeston North, New Zealand, pp. 118-123.
Jouni, M., Mura, M. D., & Comon, P. (2020). Hyperspectral Image Classification Based on Mathematical Morphology and Tensor Decomposition, Mathematical Morphology - Theory and Applications, 4(1), 1-30.
Kalomiros, J. A., & Lygouras, J. (2008). Design and evaluation of a hardware/software FPGA based system for fast image processing, Microprocessors and Microsystems, pp. 95-106.
Khosravi, M., & Schafer, R. W. (1996). Template matching based on a grayscale hit-or-miss transform. Image Processing, IEEE Transactions on, 5(6), pp. 1060-1066.
Lange, S., Sünderhauf, N., & Protzel, P. (2008). Autonomous Landing for a Multirotor UAV Using Vision. In Workshop Proceedings of SIMPAR 2008 Intl. Conf. on Simulation, Modeling and Programming for Autonomous Robts, Venice, Italy, 3-4, 482-491.
Matlab-Simulink, (2015). http://www.mathworks.com.
Mellouli, D., Hamdani, T. M., Sanchez-Medina, J. J., Ayed, M. B., & Alimi, A. (2019). Morphological Convolutional Neural Network Architecture for Digit Recognition. IEEE Transactions on Neural Networks and Learning Systems, 30, 2876-2885.
Nagel, G. W., de Novo, E. M. L. M., & Kampel, M. (2020). Nanosatellites applied to optical Earth observation: A review.Rev. Ambient. Água 2020, 15.
Nogueira, K. J. Chanussot, Mura, M. D., & Santos, J. A. D. (2021). "An Introduction to Deep Morphological Networks," in IEEE Access, 9, 114308-114324, 2021, 10.1109/ACCESS.2021.3104405.
Pell, O., Mencer, O., Tsoi, K.H., & Luk, W. (2013). Maximum performance computing with dataflow engines, in High-Performance Computing Using FPGAs, ed: Springer, pp. 747-774.
Ramos, R., A., Sampedro, C., Carrio, A., Bavle, H., Fernández, R. A. S., Miloševic, Z., & Campoy, P. (2016). A monocular pose estimation strategy for uav autonomous navigation in gnss-denied environments. In International micro air vechicle competition and conference 2016, Beijing, PR of China, pp. 22-27.
Rapuano, E., Meoni, G., Pacini, T., Dinelli, G., Furano, G., Giuffrida, G., & Fanucci, L. (2021). An FPGA-Based Hardware Accelerator for CNNs Inference on Board Satellites: Benchmarking with Myriad 2-Based Solution for the CloudScout Case Study. Remote Sens. 2021, 13, 1518. https://doi.org/10.3390/rs13081518.
Rempel, E. L., & Silva, F. A. T. F. (2001). Reconhecimento de Padrões Invariante a Rotação Utilizando uma Rede Neural Morfológica Não Supervisionada. In: V Brazilian Conference on Neural Networks, 2001, Rio de Janeiro. Proceedings of V Brazilian Conference on Neural Networks, 2001. 109-112.
Shen, Y., Zhong, X., & Shih, F. (2019). Deep Morphological Neural Networks. ArXiv, abs/1909.01532.
Silva, F. A.T. F. (1998). Rede Morfológica Não Supervisionada-RMNS. Thesis (Graduate Program in Applied Computing-CAP of the National Institute for Space Research-INPE), INPE-8759-TDI/800, Instituto Nacional de Pesquisas Espaciais-INPE, São José dos Campos, 1998. http://gjfb0520.sid.inpe.br/c ol/dpi.inpe.br/banon/1998/07.30.19.40/doc/tese.pdf
Silva, F. A. T. F. & Banon, G. J. F. (1999). Rede Morfológica Não Supervisionada., Proceedings of the IV Brazilian Conference on neural Networks. ITA- São José dos Campos, pp. 400-405.
Silva, F. A. T. F., & Silva, L. A. (2004). Detecção de múltiplos padrões usando redes neurais morfológicas. Anais do Congresso Brasileiro de Automática, A8-Sistemas Inteligentes, S0801, 1-6, RS, Brasil.
Silva, F. A. T. F., & Lucena, A. M. P. (2005a). Satélites inteligentes aplicados ao monitoramento ambiental em tempo real. 1-6. 10.21528/CBRN2005-068.
Silva, F. A. T. F., & Lucena, A. M. P. (2005b). Processamento Inteligente de Sinais aplicado ao Monitoramento Ambiental em Tempo Real. Anais XII Simpósio Brasileiro de Sensoriamento Remoto, Goiânia, Brasil, 16-21 abril 2005, INPE,. 3341-3348.
Silva, F. A. T. F. (2005). Neurônios morfológicos: Uma introdução às células on-off artificiais. Anais do VII CONGRESSO BRASILEIRO DE REDES NEURAIS – CBRN, São José dos Campos, Brasil, 2005, 1-6. https://sbic.org.br/eventos/cbrn_2005/CBRN2005_041/
Silva, F. A. T. F. (2006). Operadores morfológicos adaptativos. In: Congresso de Matemática e suas Aplicações, Foz do Iguaçu, Anais do Congresso de Matemática e suas Aplicações, pp.1-2.
Silva, F. A. T. F., Filho, P. M., Moreira, N. A., Rios, C. S. N., Oliveira, P. D. L., Camurca, P. J., & Lucena, A. M. P. (2015). Modelagem matemátia em microeletrônica reconfigurável: Estudo de caso sobre modulares BPSK. Relatório de Pesquisa INPE, sid.inpe.br/mtc-m21b/2015/05.28.17.26-RPQ, 2015. http://mtc-m21b.sid.inpe.br/col/sid.inpe.br/mtc-m21b/2015/05.28.17.26/doc/publicacao.pdf?languagebutton=pt-BR
Souza, O., Cortez, P. C., & Silva, F. A. T. F. (2012). Grayscale images and RGB video: compression by morphological neural network. In: Artificial Neural Networks in Pattern Recognition. Springer Berlin Heidelberg, 213-224.
Souza, O., Cortez, P. C., & Silva, F. A. T. F. (2013). Artificial Neural Networks for Compression of Gray Scale Images: A Benchmark. In: X Encontro Nacional de Inteligência Artificial e Computacional (ENIAC 2013). Fortaleza, Brasil. Anais of X Encontro Nacional de Inteligência Artificial e Computacional.
Thompson, D. R, Rogez, J. C., Chien, S., Doyle, R., Estlin, T., & McLaren, D. (2012). Agile science operations: A new approach for primitive bodies exploration. In Proceedings of SpaceOps, Stockholm, Sweden, June 2012.
Zakerhaghighi, M. R., & Naji, H. R. (2013). Whale algorithm for image processing, a hardware implementation, Machine Vision and Image Processing (MVIP), in 8th Iranian Conference, 355-359.
Zhou, G., & Kafatos, M. (2002). Future intelligent Earth observing satellites. Pecora 15/Land Satellite Information IV/ISPRS Commission I/FIEOS 2002 Conference Proceedings.
Xilinx-RTL, (2011). Technology Schematic Viewers Tutorial, UG685, v13.1 (v2011).
Xilinx-SG, (2014). Vivado Design Suite: Model-Based DSP Design using System Generator, UG897 (v2014.1).
XILINX-KC705, (2014). Hardware User Guide. v.1.6. USA: XILINX, 2014. 110 p. <http://www.Xilinx.com/support/documentation/boards_a nd_ki ts/kc705/ug810_KC705_Eval_Bd.pdf>.
Xilinx-RTK, (2021). Radiation Tolerant Kintex UltraScale XQRKU060 FPGA. https://www.xilinx.com/support/documentation/data_sheets/ds8 82-xqr-kintex-ultrascale.pdf.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Francisco de Assis Tavares Ferreira da Silva; Magno Prudêncio de Almeida Filho; Antonio Macilio Pereira de Lucena; Alexandre Guirland Nowosad
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
1) Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
2) Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
3) Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
