Synthesizing an image dataset for text detection and recognition in images
Subject Areas : ICT
Fatemeh Alimoradi
1
(researcher at Iran Telecommunication Research Center)
Farzaneh Rahmani
2
(پژوهشگر پژوهشگاه ارتباطات و فناوری اطلاعات)
Leila Rabiei
3
(researcher at Iran Telecommunication Research Center)
Mohammad Khansari
4
(University of Tehran)
Mojtaba Mazoochi
5
(ICT Research Institute)
Keywords: Text detection, Text recognition, Scene text images, Persian scene text dataset, Deep learning,
Abstract :
Text detection in images is one of the most important sources for image recognition. Although many researches have been conducted on text detection and recognition and end-to-end models (models that provide detection and recognition in a single model) based on deep learning for languages such as English and Chinese, the main obstacle for developing such models for Persian language is the lack of a large training data set. In this paper, we design and build required tools for synthesizing a data set of scene text images with parameters such as color, size, font, and text rotation for Persian. These tools are used to generate a large still varied data set for training deep learning models. Due to considerations in synthesizing tools and resulted variety of texts, models do not depend on synthesis parameters and can be generalized. 7603 scene text images and 39660 cropped word images are synthesized as sample data set. The advantage of our method over real images is to synthesize any arbitrary number of images, without the need for manual annotations. As far as we know, this is the first open-source and large data set of scene text images for Persian language.
S. Long, X. He and C. Yao, "Scene Text Detection and Recognition: The Deep Learning Era," International Journal of Computer Vision, vol. 129, p. 161–184, 2021.
X. Chen, L. Jin, Y. Zhu, C. Luo and T. Wang, "Text Recognition in the Wild: A Survey," ACM Computing Surveys, vol. 54, no. 2, pp. 1-35, 2021.
C. Zhang, W. Ding, G. Peng, F. Fu and W. Wang, "Street View Text Recognition With Deep Learning for Urban Scene Understanding in Intelligent Transportation Systems," IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 7, pp. 4727-4743, 2021.
A. Shinde and M. Patil, "Street View Text Detection Methods: Review Paper," International Conference on Artificial Intelligence and Smart Systems (ICAIS), March 25-27, 2021, Coimbatore, India, pp. 961-965, 2021.
F. Borisyuk, A. Gordo and V. Sivakumar, "Rosetta: Large Scale System for Text Detection and Recognition in Images," KDD '18: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, July, 2018, London, United Kingdom, pp. 71-79, 2018.
W. Huang, Z. Lin, J. Yang and J. Wang, "Text Localization in Natural Images Using Stroke Feature Transform and Text Covariance Descriptors," IEEE International Conference on Computer Vision, Dec. 1-8, 2013, Sydney, NSW, Australia, pp. 1241-1248, 2013.
X. Zhou, C. Yao, H. Wen, Y. Wang, S. Zhou, W. He and J. Liang, "EAST: An Efficient and Accurate Scene Text Detector," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA, pp. 2642-2651, 2017.
B. Shi, X. Bai and C. Yao, "An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 11, pp. 2298-2304, 2017.
Z. Liu, Y. Li, F. Ren, W. L. Goh and H. Yu, "SqueezedText: A real-time scene text recognition by binary convolutional encoder-decoder network," 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, February 2-7, 2018, New Orleans, Lousiana, USA, pp. 7194-7201, 2018.
M. Liao, B. Shi, X. Bai, X. Wang and W. Liu, "TextBoxes: A Fast Text Detector with a Single Deep Neural Network," AAAI, February 4 – 9, 2017, San Francisco, California, USA, pp. 4161-4167, 2017.
Y. Liu, C. Shen, L. Jin, T. He, P. Chen, C. Liu and H. Chen, "ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting," IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021.
M. Bušta, Y. Patel and J. Matas, "E2E-MLT - An Unconstrained End-to-End Method for Multi-language Scene Text," Computer Vision – ACCV 2018 Workshops, December 2–6, 2018, Perth, Australia, pp. 127-143, 2019.
L. Xing, Z. Tian, W. Huang and M. R. Scott, "Convolutional Character Networks," 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 27 - Nov. 2, 2019, Seoul, Korea (South), pp. 9125-9135 2019.
M. Busta, L. Neumann and J. Matas, "Deep TextSpotter: An End-To-End Trainable Scene Text Localization and Recognition Framework," IEEE International Conference on Computer Vision (ICCV), Oct. 22-29, 2017, Venice, Italy, pp. 2204-2212, 2017.
V. Khare, P. Shivakumara, P. Raveendran and M. Blumenstein, "A blind deconvolution model for scene text detection and recognition in video," Pattern Recognition, vol. 54, pp. 128-148, 2016.
A. Gupta, A. Vedaldi and A. Zisserman, "Synthetic Data for Text Localisation in Natural Images," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27-30, 2016, Las Vegas, NV, USA, pp. 2315-2324, 2016.
Z. Zhong, L. Jin and S. Huang, "DeepText: A new approach for text proposal generation and text detection in natural images," IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March 5-9, 2017, New Orleans, LA, USA, pp. 1208-1212, 2017.
W. Liu, C. Chen, K. Y. K. Wong, Z. Su and J. Han, "STAR-Net: A SpaTial Attention Residue Network for Scene Text Recognition," BMVC, September 19-22, 2016, York, UK, pp. 43.1-43.13, 2016.
P. He, W. Huang, Y. Qiao, C. C. Loy and X. Tang, "Reading scene text in deep convolutional sequences," AAAI'16: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12–17, 2016, Phoenix, Arizona USA, pp. 3501–3508, 2016.
C. Y. Lee and S. Osindero, "Recursive Recurrent Nets with Attention Modeling for OCR in the Wild," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27-30, 2016, Las Vegas, NV, USA, pp. 2231-2239, 2016.
M. Jaderberg, K. Simonyan, A. Vedaldi and A. Zisserman, "Reading Text in the Wild with Convolutional Neural Networks," International Journal of Computer Vision, vol. 116, pp. 1-20, 2016.
Y. Dai, Z. Huang, Y. Gao and K. Chen, "Fused Text Segmentation Networks for Multi-oriented Scene Text Detection," 2018 24th International Conference on Pattern Recognition (ICPR), Aug. 20-24, 2018, Beijing, China, pp. 3604-3609, 2018.
D. He, X. Yang, C. Liang, Z. Zhou, A. G. Ororbia, D. Kifer and C. L. Giles, "Multi-scale FCN with Cascaded Instance Aware Segmentation for Arbitrary Oriented Word Spotting in the Wild," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA, pp. 474-483, 2017.
P. He, W. Huang, T. He, Q. Zhu, . Y. Qiao and X. Li, "Single Shot Text Detector with Regional Attention," IEEE International Conference on Computer Vision (ICCV), Oct. 22-29, 2017, Venice, Italy, pp. 3066-3074, 2017.
Y. Liu and L. Jin, "Deep Matching Prior Network: Toward Tighter Multi-oriented Text Detection," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA, pp. 3454-3461, 2017.
M. Samaee and H. Tavakoli, "Farsi Text Localization in Natural Scene Images," International Journal of Computer Science and Information Security (IJCSIS), vol. 15, no. 2, pp. 22-30, 2017.
B. Shi, X. Bai and S. Belongie, "Detecting Oriented Text in Natural Images by Linking Segments," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA, pp. 3482-3490, 2017.
Y. Wu and P. Natarajan, "Self-Organized Text Detection with Minimal Post-processing via Border Learning," IEEE International Conference on Computer Vision (ICCV), Oct. 22-29, 2017, Venice, Italy, pp. 5010-5019, 2017.
Y. Gao, Y. Chen, J. Wang and H. Lu, "Reading Scene Text with Attention Convolutional Sequence Modeling," arXiv preprint arXiv:1709.04303v1, 2017.
S. Bin Ahmed, S. Naz, M. I. Razzak and R. Yousaf, "Deep learning based isolated Arabic scene character recognition," 1st International Workshop on Arabic Script Analysis and Recognition (ASAR), April 3-5, 2017, Nancy, France, pp. 46-51, 2017.
Z. Cheng, F. Bai, Y. Xu, G. Zheng, S. Pu and S. Zhou, "Focusing Attention: Towards Accurate Text Recognition in Natural Images," IEEE International Conference on Computer Vision (ICCV), Oct. 22-29, 2017, Venice, Italy, pp. 5086-5094, 2017.
F. Yin, Y. C. Wu, X. Y. Zhang and C. L. Liu, "Scene Text Recognition with Sliding Convolutional Character Models," arXiv preprint arXiv:1709.01727v1, 2017.
H. Li, P. Wang and C. Shen, "Towards End-to-End Text Spotting with Convolutional Recurrent Neural Networks," IEEE International Conference on Computer Vision (ICCV), Oct. 22-29, 2017, Venice, Italy, pp. 5248-5256, 2017.
S. Zhang, Y. Liu, L. Jin and C. Luo, "Feature Enhancement Network: A Refined Scene Text Detector," Proceedings of the AAAI Conference on Artificial Intelligence, February 2-7, 2018, New Orleans, Lousiana, USA, vol. 32, no. 1, pp. 2612-2619, 2018.
D. Deng, H. Liu, X. Li and D. Cai, "PixelLink: Detecting Scene Text via Instance Segmentation," The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), February 2-7, 2018, New Orleans, Lousiana, USA, pp. 6773-6780, 2018.
M. Liao, Z. Zhu, B. Shi, G. S. Xia and X. Bai, "Rotation-Sensitive Regression for Oriented Scene Text Detection," IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA, pp. 5909-5918, 2018.
P. Lyu, C. Yao, W. Wu, S. Yan and X. Bai, "Multi-oriented Scene Text Detection via Corner Localization and Region Segmentation," IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA, pp. 7553-7563, 2018.
J. Ma, W. Shao, H. Ye, L. Wang, H. Wang, Y. Zheng and X. Xue, "Arbitrary-Oriented Scene Text Detection via Rotation Proposals," IEEE Transactions on Multimedia, vol. 20, no. 11, p. 3111–3122, 2018.
F. Bai, Z. Cheng, Y. Niu, S. Pu and S. Zhou, "Edit Probability for Scene Text Recognition," IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA, pp. 1508-1516, 2018.
Z. Cheng, Y. Xu, F. Bai, Y. Niu, S. Pu and S. Zhou, "AON: Towards Arbitrarily-Oriented Text Recognition," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 18-23, 2018, Salt Lake City, UT, USA, pp. 5571-5579, 2018.
X. Liu, D. Liang, S. Yan, D. Chen, Y. Qiao and J. Yan, "FOTS: Fast Oriented Text Spotting with a Unified Network," IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA, pp. 5676-5685, 2018.
C. Bartz, H. Yang and C. Meinel, "SEE: Towards Semi-Supervised End-to-End Scene Text Recognition," AAAI Conference on Artificial Intelligence, February 2-7, 2018, New Orleans, Louisiana, USA, pp. 6674-6681, 2018.
T. He, Z. Tian, W. Huang, C. Shen, Y. Qiao and C. Sun, "An End-to-End TextSpotter with Explicit Alignment and Attention," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 18-23, 2018, Salt Lake City, UT, USA, pp. 5020-5029, 2018.
J. Ghavidel, A. Ahmadyfard and M. Zahedi, "Natural scene text localization using edge color signature," International Journal of Nonlinear Analysis and Applications, vol. 10, no. 1, pp. 229-237, 2019.
Y. Baek, B. Lee, D. Han, S. Yun and H. Lee, "Character Region Awareness for Text Detection," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 15-20, 2019, Long Beach, CA, USA, pp. 9357-9366, 2019.
Y. Liu, L. Jin, . S. Zhang, C. Luo and S. Zhang, "Curved scene text detection via transverse and longitudinal sequence connection," Pattern Recognition, vol. 90, pp. 337-345, 2019.
Z. Tian, M. Shu, P. Lyu, R. Li, C. Zhou, X. Shen and J. Jia, "Learning Shape-Aware Embedding for Scene Text Detection," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 15-20, 2019, Long Beach, CA, USA, pp. 4229-4238, 2019.
W. Wang, E. Xie, X. Li, W. Hou, T. Lu, G. Yu and S. Shao, "Shape Robust Text Detection With Progressive Scale Expansion Network," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 15-20, 2019, Long Beach, CA, USA, pp. 9328-9337, 2019.
C. Zhang, B. Liang, Z. Huang, M. En, J. Han, E. Ding and X. Ding, "Look More Than Once: An Accurate Detector for Text of Arbitrary Shapes," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 15-20, 2019, Long Beach, CA, USA, pp. 10544-10553, 2019.
Z. Zhong, L. Sun and Q. Huo, "Improved localization accuracy by LocNet for Faster R-CNN based text detection in natural scene images," Pattern Recognition, vol. 96, 2019.
C. Luoa, L. Jin and Z. Sun, "MORAN: A Multi-Object Rectified Attention Network for scene text," Pattern Recognition, vol. 90, pp. 109-118, 2019.
Y. Zhu, S. Wang, Z. Huang and K. Chen, "Text Recognition in Images Based on Transformer with Hierarchical Attention," in IEEE International Conference on Image Processing (ICIP), Sept. 22-25, 2019, Taipei, Taiwan, pp. 1945-1949, 2019.
M. Liao, J. Zhang, Z. Wan, F. Xie, J. Liang, P. Lyu, C. Yao and X. Bai, "Scene Text Recognition from Two-Dimensional Perspective," Proceedings of the AAAI Conference on Artificial Intelligence, January 27 – February 1, 2019, Honolulu, Hawaii, USA, vol. 33, no. 01, pp. 8714-8721, 2019.
W. Feng, W. He, F. Yin, X. Y. Zhang and C. L. Liu, "TextDragon: An End-to-End Framework for Arbitrary Shaped Text Spotting," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 27 - Nov. 2, 2019, Seoul, Korea (South), pp. 9076-9085, 2019.
S. X. Zhang, X. Zhu, J. B. Hou, C. Liu, C. Yang, H. Wang and X. C. Yin, "Deep Relational Reasoning Graph Network for Arbitrary Shape Text Detection," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 13-19, 2020, Seattle, WA, USA, pp. 9696-9705, 2020.
S. Saha, N. Chakraborty, S. Kundu, S. Paula, A. F. Mollah, S. Basu and R. Sarkar, "Multi-lingual scene text detection and language identification," Pattern Recognition Letters, vol. 138, pp. 16-22, 2020.
H. Liu, A. Guo, D. Jiang, Y. Hu and B. Ren, "PuzzleNet: Scene Text Detection by Segment Context Graph Learning," arXiv preprint arXiv:2002.11371, 2020.
M. Fasha, B. Hammo, N. Obeid and J. Alwidian, "A Hybrid Deep Learning Model for Arabic Text Recognition," (IJACSA) International Journal of Advanced Computer Science and Applications, vol. 11, no. 8, pp. 122-130, 2020.
Z. Qiao, Y. Zhou, D. Yang, Y. Zhou and W. Wang, "SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition," in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 13-19, 2020, Seattle, WA, USA, pp. 13525-13534, 2020.
X. Chen, T. Wang, Y. Zhu, L. Jin and C. Luo, "Adaptive embedding gate for attention-based scene text recognition," Neurocomputing, vol. 381, pp. 261-271, 2020.
Y. Liu, H. Chen, C. Shen, T. He, L. Jin and L. Wang, "ABCNet: Real-Time Scene Text Spotting With Adaptive Bezier-Curve Network," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 13-19, 2020, Seattle, WA, USA, pp. 9806-9815, 2020.
L. Qiao, S. Tang, Z. Cheng, Y. Xu, Y. Niu, S. Pu and F. Wu, "Text Perceptron: Towards End-to-End Arbitrary-Shaped Text Spotting," Proceedings of the AAAI Conference on Artificial Intelligence, February 7–12, 2020, New York Hilton Midtown, New York, New York, USA, vol. 34, no. 7, pp. 11899-11907, 2020.
H. Wang, P. Lu, H. Zhang, M. Yang, X. Bai, Y. Xu, M. He, Y. Wang and W. Liu, "All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting," Proceedings of the AAAI Conference on Artificial Intelligence, February 7–12, 2020, New York Hilton Midtown, New York, New York, USA, vol. 34, no. 07, pp. 12160-12167, 2020.
X. Qin, Y. Zhou, Y. Guo, D. Wu, Z. Tian, N. Jiang, H. Wang and W. Wang, "Mask is All You Need: Rethinking Mask R-CNN for Dense and Arbitrary-Shaped Scene Text Detection," ACM MULTIMEDIA, October 20-24, 2021, Chengdu, China, 2021.
Y. Zhu and J. Du, "TextMountain: Accurate scene text detection via instance segmentation," Pattern Recognition, vol. 110, 2021.
C. Ma, L. Sun, Z. Zhong and Q. Huo, "ReLaText: Exploiting visual relationships for arbitrary-shaped scene text detection with graph convolutional networks," Pattern Recognition, vol. 111, 2021.
N. Lu, W. Yu, X. Qi, Y. Chen, P. Gong, R. Xiao and X. Bai, "MASTER: Multi-aspect non-local network for scene text recognition," Pattern Recognition, vol. 117, 2021.
Q. Lin, C. Luo, L. Jin and S. Lai, "STAN: A sequential transformation attention-based network for scene text recognition," Pattern Recognition, vol. 111, 2021.
H. Hassan, A. El-Mahdy and M. E. Hussein, "Arabic Scene Text Recognition in the Deep Learning Era: Analysis on a Novel Dataset," IEEE Access, vol. 9, pp. 107046-107058, 2021.
B. Esfahbod and R. Pournader, "FarsiTEX and the Iranian TEX Community," TUGboat, vol. 23, no. 1, pp. 41-45, 2002.
M. Darab and M. Rahmati, "A Hybrid Approach to Localize Farsi Text in Natural Scene Images," Procedia Computer Science, vol. 13, pp. 171-184, 2012.
P. Arbeláez, M. Maire, C. Fowlkes and J. Malik, "Contour Detection and Hierarchical Image Segmentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 898-916, 2011.
F. Liu, C. Shen and G. Lin, "Deep convolutional neural fields for depth estimation from a single image," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 7-12, 2015, Boston, MA, USA, pp. 5162-5170, 2015.
P. Pérez, M. Gangnet and A. Blake, "Poisson image editing," ACM Transactions on Graphics, vol. 22, no. 3, p. 313–318, 2003.
F. Zhan , . S. Lu and C. Xue, "Verisimilar Image Synthesis for Accurate Detection and Recognition of Texts in Scenes," in Computer Vision – ECCV 2018, September 8-14, 2018, Munich, Germany, pp 257-273, 2018.
M. Liao, B. Song, S. Long, . M. He, C. Yao and X. Bai, "SynthText3D: synthesizing scene text images from 3D virtual worlds," in Science China Information Sciences, vol. 63, no. 2, pp. 120105:1-120105:14, 2020.
W. Qiu and A. Yuille, "UnrealCV: Connecting Computer Vision to Unreal Engine," in Computer Vision – ECCV 2016 Workshops, October 8-10 and 15-16, 2016, Amsterdam, The Netherlands, Springer, Cham, pp. 909-916, 2016.
S. Long and C. Yao, "UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World," arXiv preprint arXiv:2003.10608, 2020.
J. Pont-Tuset, P. Arbeláez, J. T. Barron, F. Marques and J. Malik, "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 1, pp. 128-140, 2017.
I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari and N. Navab, "Deeper Depth Prediction with Fully Convolutional Residual Networks," Fourth International Conference on 3D Vision (3DV), Oct. 25-28, 2016, Stanford, CA, USA, pp. 239-248, 2016.
S. M. Lucas, A. Panaretos, L. Sosa, A. Tang, S. Wong and R. Young, "ICDAR 2003 robust reading competitions," Seventh International Conference on Document Analysis and Recognition, Aug. 6-6, 2003, Edinburgh, UK, pp. 682-687, 2003.
K. Wang, B. Babenko and . S. Belongie, "End-to-end scene text recognition," International Conference on Computer Vision, Nov. 6-13, 2011, Barcelona, Spain, pp. 1457-1464, 2011.
A. Mishra, K. Alahari and C. V. Jawahar, "Scene Text Recognition using Higher Order Language Priors," Proceedings of British Machine Vision Conference, September 3-7, 2012, Guildford, UK, pp. 127.1-127.11, 2012.
D. Karatzas, . F. Shafait, S. Uchida, M. Iwamura, L. G. i. Bi, S. R. Mestre, J. Mas, D. F. Mota, J. A. Almazàn and L. P. d. l. Heras, "ICDAR 2013 Robust Reading Competition," 12th International Conference on Document Analysis and Recognition, Aug. 25-28, 2013, Washington, DC, USA, pp. 1484-1493, 2013.
A. Davoudi, "This is a modified version of Ankush's code for generating synthetic text images which support right-to-left languages such as Persian and Arabic.," [Online]. Available: https://github.com/adavoudi/SynthText. [Accessed 22 06 2021].
S. T. Piantadosi, "Zipf’s word frequency law in natural language: A critical review and future directions," Psychonomic bulletin & review, vol. 21, no. 5, p. 1112–1130, 2014.