1. DSpace KNU
  2. Наукові публікації
  3. 5. Факультет інформаційних технологій
  4. Кафедра професійної та соціально-гуманітарної освіти
Будь ласка, використовуйте цей ідентифікатор, щоб цитувати або посилатися на цей матеріал: http://ds.knu.edu.ua/jspui/handle/123456789/9263
Назва: Advances in neural text generation: A systematic review (2022-2024)
Автори: Slobodianiuk, Artem V.
Semerikov, Serhiy O.
Ключові слова: neural text generation
deep learning
systematic review
natural language processing
evaluation metrics
datasets
applications
low-resource languages
Дата публікації: 7-лют-2025
Видавництво: CEUR Workshop Proceedings
Бібліографічний опис: Advances in neural text generation: A systematic review (2022-2024) / Artem V. Slobodianiuk, Serhiy O. Semerikov // Proceedings of the 7th Workshop for Young Scientists in Computer Science & Software Engineering (CS&SE@SW 2024). Virtual Event, Kryvyi Rih, Ukraine, December 27, 2024 / Edited by: Serhiy O. Semerikov, Andrii M. Striuk // CEUR Workshop Proceedings. – 2025. – Vol. 3917. – P. 332-361. – Access mode : https://ceur-ws.org/Vol-3917/paper59.pdf DOI: https://doi.org/10.55056/ceur-ws.org/Vol-3917/paper59.pdf
Короткий огляд (реферат): Recent years have witnessed significant advancements in neural text generation driven by the emergence of large language models and growing interest in this field. This systematic review aims to identify and summarize current trends, approaches, and methods in neural text generation from 2022 to 2024, complementing the findings of a previous review covering 2015-2021. Following the PRISMA methodology, 43 articles were selected from the Scopus database for analysis. The review reveals a shift towards innovative model architectures like Transformer- based models (GPT-2, GPT-3, BERT), attention mechanisms, and controllable text generation. While BLEU, ROUGE, and human evaluation remain the most popular evaluation metrics, new metrics like BERTScore have emerged. Datasets span diverse domains and data types, with growing interest in unlabeled data. Applications have expanded to areas such as table-to-text generation, knowledge graph-based generation, and medical text generation. Although English dominates, there is increasing research on low-resource languages. The findings highlight the rapid evolution of neural text generation methods, the broadening of application areas, and promising avenues for future research.
Опис: [1] T. Ganegedara, Natural Language Processing with TensorFlow: Teach language to machines using Python’s deep learning library, Packt Publishing, Birmingham – Mumbai, 2018. URL: https://tinyurl.com/3xps3c5u. [2] N. Fatima, A. S. Imran, Z. Kastrati, S. M. Daudpota, A. Soomro, A systematic literature review on text generation using deep neural network models, IEEE Access 10 (2022) 53490 – 53503. doi:10.1109/ACCESS.2022.3174108. [3] OpenAI, Introducing ChatGPT, 2022. URL: https://openai.com/blog/chatgpt. [4] large language models - Google Trends, 2023. URL: https://trends.google.com/trends/explore?date=2022-01-01%202023-12-21&q=large%20language%20models&hl=en. [5] M. J. Page, J. E. McKenzie, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow, L. Shamseer, J. M. Tetzlaff, E. A. Akl, S. E. Brennan, R. Chou, J. Glanville, J. M. Grimshaw, A. Hróbjartsson, M. M. Lalu, T. Li, E. W. Loder, E. Mayo-Wilson, S. McDonald, L. A. McGuinness, L. A. Stewart, J. Thomas, A. C. Tricco, V. A. Welch, P. Whiting, D. Moher, The PRISMA 2020 statement: an updated guideline for reporting systematic reviews, BMJ 372 (2021) n71. doi:10.1136/bmj.n71. [6] A. Bas, M. O. Topal, Ç. Duman, I. Van Heerden, A Brief History of Deep Learning-Based Text Generation, in: J. M. Alja’Am, S. AlMaadeed, S. A. Elseoud, O. Karam (Eds.), Proceedings of the International Conference on Computer and Applications, ICCA 2022 - Proceedings, Institute of Electrical and Electronics Engineers Inc., 2022, pp. 1–4. doi:10.1109/ICCA56443.2022.10039545. [7] J. Zhu, X. Ma, Z. Lin, P. De Meo, A quantum-like approach for text generation from knowledge graphs, CAAI Transactions on Intelligence Technology (2023). doi:10.1049/cit2.12178. [8] H. Zhang, H. Song, S. Li, M. Zhou, D. Song, A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models, ACM Computing Surveys 56 (2023) 64. doi:10.1145/3617680. [9] W. Yu, C. Zhu, Z. Li, Z. Hu, Q. Wang, H. Ji, M. Jiang, A Survey of Knowledge-enhanced Text Generation, ACM Computing Surveys 54 (2022) 227. doi:10.1145/3512467. [10] J. Wu, Y. Guo, C. Gao, J. Sun, An automatic text generation algorithm of technical disclosure for catenary construction based on knowledge element model, Advanced Engineering Informatics 56 (2023) 101913. doi:10.1016/j.aei.2023.101913. [11] H. Du, W. Xing, B. Pei, Automatic text generation using deep learning: providing large-scale support for online learning communities, Interactive Learning Environments 31 (2023) 5021–5036. doi:10.1080/10494820.2021.1993932. [12] Q. Chen, H. Sun, H. Liu, Y. Jiang, T. Ran, X. Jin, X. Xiao, Z. Lin, H. Chen, Z. Niu, An extensive benchmark study on biomedical text generation and mining with ChatGPT, Bioinformatics 39 (2023) btad557. doi:10.1093/bioinformatics/btad557. [13] I. Alonso, E. Agirre, Automatic logical forms improve fidelity in table-to-text generation, Expert Systems with Applications 238 (2024). doi:10.1016/j.eswa.2023.121869. [14] E. Kreiss, F. Fang, N. D. Goodman, C. Potts, Concadia: Towards Image-Based Text Generation with a Purpose, in: Y. Goldberg, Z. Kozareva, Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Association for Computational Linguistics (ACL), 2022, pp. 4667–4684. doi:10.18653/v1/2022.emnlp-main.308. [15] K. Y. Rao, K. S. Rao, S. V. S. Narayana, Conditional-Aware Sequential Text Generation In Knowledge-Enhanced Conversational Recommendation System, Journal of Theoretical and Applied Information Technology 101 (2023) 2820–2836. URL: http://www.jatit.org/volumes/Vol101No7/30Vol101No7.pdf. [16] T. Tazalli, Z. A. Aunshu, S. S. Liya, M. Hossain, Z. Mehjabeen, M. S. Ahmed, M. I. Hossain, Computer Vision-Based Bengali Sign Language To Text Generation, in: 5th IEEE International Image Processing, Applications and Systems Conference, IPAS 2022, Institute of Electrical and Electronics Engineers Inc., 2022, pp. 1–6. doi:10.1109/IPAS55744.2022.10052928. [17] Z. Teng, C. Chen, Y. Zhang, Y. Zhang, Contrastive Latent Variable Models for Neural Text Generation, in: J. Cussens, K. Zhang (Eds.), Proceedings of Machine Learning Research, volume 180, ML Research Press, 2022, pp. 1928–1938. URL: https://proceedings.mlr.press/v180/teng22a.html. [18] C. An, J. Feng, K. Lv, L. Kong, X. Qiu, X. Huang, CONT: contrastive neural text generation, in: Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Curran Associates Inc., Red Hook, NY, USA, 2022, p. 160. URL: https://dl.acm.org/doi/10.5555/3600270.3600430. [19] H. Seo, S. Jung, J. Jung, T. Hwang, H. Namgoong, Y.-H. Roh, Controllable Text Generation Using Semantic Control Grammar, IEEE Access 11 (2023) 26329–26343. doi:10.1109/ACCESS.2023.3252017. [20] W. Zhou, Y. E. Jiang, E. Wilcox, R. Cotterell, M. Sachan, Controlled Text Generation with Natural Language Instructions, in: A. Krause, E. Brunskill, C. K., B. Engelhardt, S. Sabato, J. Scarlett (Eds.), Proceedings of Machine Learning Research, volume 202, ML Research Press, 2023, pp. 42602–42613. [21] M. Bayer, M.-A. Kaufhold, B. Buchhold, M. Keller, J. Dallmeyer, C. Reuter, Data augmentation in natural language processing: a novel text generation approach for long and short text classifiers, International Journal of Machine Learning and Cybernetics 14 (2023) 135–150. doi:10.1007/s13042-022-01553-3. [22] S. Hong, S. Moon, J. Kim, S. Lee, M. Kim, D. Lee, J.-Y. Kim, DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation, in: Proceedings of the Annual International Symposium on Microarchitecture, MICRO, volume 2022-October, IEEE Computer Society, 2022, pp. 616–630. doi:10.1109/MICRO56248.2022.00051. [23] M. Ghazvininejad, V. Karpukhin, V. Gor, A. Celikyilmaz, Discourse-Aware Soft Prompting for Text Generation, in: Y. Goldberg, Z. Kozareva, Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Association for Computational Linguistics (ACL), 2022, pp. 4570–4589. doi:10.18653/v1/2022.emnlp-main.303. [24] J. J. Koplin, Dual-use implications of AI text generation, Ethics and Information Technology 25 (2023) 32. doi:10.1007/s10676-023-09703-z. [25] A. Pautrat-Lertora, R. Perez-Lozano, W. Ugarte, EGAN: Generatives Adversarial Networks for Text Generation with Sentiments, in: F. Coenen, A. Fred, J. Filipe (Eds.), International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, IC3K - Proceedings, volume 1, Science and Technology Publications, Lda, 2022, pp. 249–256. doi:10.5220/0011548100003335. [26] T. Wu, H. Wang, Z. Zeng, W. Wang, H.-T. Zheng, J. Zhang, Enhancing Text Generation with Cooperative Training, Frontiers in Artificial Intelligence and Applications 372 (2023) 2704–2711. doi:10.3233/FAIA230579. [27] Y. Li, L. Cui, J. Yan, Y. Yin, W. Bi, S. Shi, Y. Zhang, Explicit Syntactic Guidance for Neural Text Generation, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 1, Association for Computational Linguistics (ACL), 2023, pp. 14095–14112. doi:10.18653/v1/2023.acl-long.788. [28] X. Chu, Feature extraction and intelligent text generation of digital music, Computational Intelligence and Neuroscience 2022 (2022). doi:10.1155/2022/7952259. [29] S. Shahriar, GAN computers generate arts? A survey on visual arts, music, and literary text generation using generative adversarial network, Displays 73 (2022) 102237. doi:10.1016/j.displa.2022.102237. [30] H. Strobelt, J. Kinley, R. Krueger, J. Beyer, H. Pfister, A. M. Rush, GenNI: Human-AI Collaboration for Data-Backed Text Generation, IEEE Transactions on Visualization and Computer Graphics 28 (2022) 1106–1116. doi:10.1109/TVCG.2021.3114845. [31] X. Yin, X. Wan, How Do Seq2Seq Models Perform on End-to-End Data-to-Text Generation?, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 1, Association for Computational Linguistics (ACL), 2022, pp. 7701–7710. doi:10.18653/v1/2022.acl-long.531. [32] S. Montella, A. Nasr, J. Heinecke, F. Bechet, L. M. Rojas-Barahona, Investigating the Effect of Relative Positional Embeddings on AMR-to-Text Generation with Structural Adapters, in: EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, Association for Computational Linguistics (ACL), 2023, pp. 727–736. doi:10.18653/v1/2023.eacl-main.51. [33] N. Fatima, S. M. Daudpota, Z. Kastrati, A. S. Imran, S. Hassan, N. S. Elmitwally, Improving news headline text generation quality through frequent POS-Tag patterns analysis, Engineering Applications of Artificial Intelligence 125 (2023) 106718. doi:10.1016/j.engappai.2023.106718. [34] E. Seifossadat, H. Sameti, Improving semantic coverage of data-to-text generation model using dynamic memory networks, Natural Language Engineering 30 (2024) 454–479. doi:10.1017/S1351324923000207. [35] C. Meyer, D. Adkins, K. Pal, R. Galici, A. Garcia-Agundez, C. Eickhoff, Neural text generation in regulatory medical writing, Frontiers in Pharmacology 14 (2023). doi:10.3389/fphar.2023.1086913. [36] X. Lu, S. Welleck, P. West, L. Jiang, J. Kasai, D. Khashabi, R. Le Bras, L. Qin, Y. Yu, R. Zellers, N. A. Smith, Y. Choi, NEUROLOGIC AFesque Decoding: Constrained Text Generation with Lookahead Heuristics, in: NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, Association for Computational Linguistics (ACL), 2022, pp. 780–799. doi:10.18653/v1/2022.naacl-main.57. [37] W. Xu, Y. Tuan, Y. Lu, M. Saxon, L. Li, W. Y. Wang, Not All Errors Are Equal: Learning Text Generation Metrics using Stratified Error Synthesis, in: Y. Goldberg, Z. Kozareva, Y. Zhang (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Association for Computational Linguistics (ACL), 2022, pp. 6588–6603. doi:10.18653/v1/2022.findings-emnlp.489. [38] A. Hanafi, M. Bouhorma, L. Elaachak, Machine Learning-Based Augmented Reality For Improved Text Generation Through Recurrent Neural Networks, Journal of Theoretical and Applied Information Technology 100 (2022) 518–530. URL: http://www.jatit.org/volumes/Vol100No2/18Vol100No2.pdf. [39] H. Le, D.-T. Le, V. Weber, C. Church, K. Rottmann, M. Bradford, P. Chin, Semi-supervised Adversarial Text Generation based on Seq2Seq models, in: EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, Association for Computational Linguistics (ACL), 2022, pp. 264–272. doi:10.18653/v1/2022.emnlp-industry.26. [40] X. Yue, H. A. Inan, X. Li, G. Kumar, J. McAnallen, H. Shajari, H. Sun, D. Levitan, R. Sim, Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 1, Association for Computational Linguistics (ACL), 2023, pp. 1321–1342. doi:10.18653/v1/2023.acl-long.74. [41] Z. Lin, Y. Gong, Y. Shen, T. Wu, Z. Fan, C. Lin, N. Duan, W. Chen, Text generation with diffusion language models: a pre-training approach with continuous paragraph denoise, in: Proceedings of the 40th International Conference on Machine Learning, ICML’23, JMLR.org, 2023. URL: https://dl.acm.org/doi/abs/10.5555/3618408.3619275. [42] M. S. Amin, A. Mazzei, L. Anselma, Towards Data Augmentation for DRS-to-Text Generation, CEUR Workshop Proceedings 3287 (2022) 141–152. URL: https://ceur-ws.org/Vol-3287/paper14.pdf. [43] M. Chen, X. Lu, T. Xu, Y. Li, J. Zhou, D. Dou, H. Xiong, Towards Table-to-Text Generation with Pretrained Language Model: A Table Structure Understanding and Text Deliberating Approach, in: Y. Goldberg, Z. Kozareva, Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Association for Computational Linguistics (ACL), 2022, pp. 8199–8210. doi:10.18653/v1/2022.emnlp-main.562. [44] V. Agarwal, S. Ghosh, H. BSS, H. Arora, B. R. K. Raja, TrICy: Trigger-Guided Data-to-Text Generation With Intent Aware Attention-Copy, IEEE/ACM Transactions on Audio, Speech, and Language Processing 32 (2024) 1173–1184. doi:10.1109/TASLP.2024.3353574. [45] W. M. Si, M. Backes, Y. Zhang, A. Salem, Two-in-One: A Model Hijacking Attack Against Text Generation Models, in: 32nd USENIX Security Symposium, USENIX Security 2023, volume 3, USENIX Association, 2023, pp. 2223–2240. URL: https://www.usenix.org/system/files/usenixsecurity23-si.pdf. [46] H. Gong, X. Feng, B. Qin, Quality Control for Distantly-Supervised Data-to-Text Generation via Meta Learning, Applied Sciences 13 (2023) 5573. doi:10.3390/app13095573. [47] L. Mou, Search and learning for unsupervised text generation, AI Magazine 43 (2022) 344–352. doi:10.1002/aaai.12068. [48] D. Taunk, S. Sagare, A. Patil, S. Subramanian, M. Gupta, V. Varma, XWikiGen: Cross-lingual Summarization for Encyclopedic Text Generation in Low Resource Languages, in: ACM Web Conference 2023 - Proceedings of the World Wide Web Conference, WWW 2023, Association for Computing Machinery, Inc, 2023, pp. 1703–1713. doi:10.1145/3543507.3583405. [49] Introducing the next generation of Claude, 2024. URL: https://www.anthropic.com/news/claude-3-family. [50] awesomegpts.ai, Scholar GPT, 2024. URL: https://chatgpt.com/g/g-kZ0eYXlJe-scholar-gpt?oai-dm=1. [51] A. V. Slobodianiuk, Ohliad statei [Papers’ review], 2024. URL: https://docs.google.com/spreadsheets/d/e/2PACX-1vR6ZUaeeBjVgVl-do6QXm-Pua-HdztOxjC4DUqunrSDZ_-YSRz-Ng9xktYH9b0LDT502SiVy3YePx9F/pubhtml. [52] Hugging Face, Languages, 2024. URL: https://huggingface.co/languages.
URI (Уніфікований ідентифікатор ресурсу): https://ceur-ws.org/Vol-3917/paper59.pdf
http://ds.knu.edu.ua/jspui/handle/123456789/9263
ISSN: 1613-0073
Розташовується у зібраннях:Кафедра професійної та соціально-гуманітарної освіти

Файли цього матеріалу:
Файл Опис РозмірФормат 
paper59.pdf1.95 MBAdobe PDFПереглянути/Відкрити
Показати повний опис матеріалу Перегляд статистики


Усі матеріали в архіві електронних ресурсів захищені авторським правом, всі права збережені.