More Web Proxy on the site http://driver.im/

research-article

Seq2Seq dynamic planning network for progressive text generation

Authors:

Yuying ZhengAuthors Info & Claims

Volume 89, Issue C

https://doi.org/10.1016/j.csl.2024.101687

Published: 01 January 2025 Publication History

Abstract

Long text generation is a hot topic in natural language processing. To address the problem of insufficient semantic representation and incoherent text generation in existing long text models, the Seq2Seq dynamic planning network progressive text generation model (DPPG-BART) is proposed. In the data pre-processing stage, the lexical division sorting algorithm is used. To obtain hierarchical sequences of keywords with clear information content, word weight values are calculated and ranked by TF-IDF of word embedding. To enhance the input representation, the dynamic planning progressive generation network is constructed. Positional features and word embedding vector features are integrated at the input side of the model. At the same time, to enrich the semantic information and expand the content of the text, the relevant concept words are generated by the concept expansion module. The scoring network and feedback mechanism are used to adjust the concept expansion module. Experimental results show that the DPPG-BART model is optimized over GPT2-S, GPT2-L, BART and ProGen-2 model approaches in terms of metric values of MSJ, B-BLEU and FBD on long text datasets from two different domains, CNN and Writing Prompts.

Highlights

•

Existing long text models suffer from inadequate semantic representation.

•

The Seq2Seq Dynamic Planning Network Progressive Text Generation Model is proposed.

•

Constructing dynamically planned progressive generative networks.

•

Enrichment of semantic information through conceptual extension modules.

•

The DPPG-BART model has better long text generation capability.

References

[1]

Chen J., Research and Application of Controlled Text Generation Based on Pretrained Language Models, (Master’s thesis) University of Electronic Science and Technology, 2022.

[2]

Dou Y., Forbes M., Koncel-Kedziorski R., et al., Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 7250–7274.

[3]

Duong Q., Hmlinen M., Alnajjar K., Tfw2v: An enhanced document similarity method for the morphologically rich finnish language, 2021, CoRR abs/2112.12489.

[4]

Feng, Y., Yi, X., Wang, X., et al., 2023. DuNST: Dual Noisy Self Training for Semi-Supervised Controllable Text Generation. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8760–8785.

[5]

Goldfarb-Tarrant S., Chakrabarty T., Weischedel R., et al., Content planning for neural story generation with aristotelian rescoring, Comput. Sci. 2020.emnlp-main (2020) 4319–4338.

[6]

Gong Y., Guo Y., Lian X., et al., Data-to-text generation methods based on hierarchical structural representation, Comput. Appl. Res. (2023).

[7]

Gong N., Yao N., A generalized decoding method for neural text generation, Comput. Speech Lang. 81 (2023).

[8]

Guan, J., Mao, X., Fan, C., et al., 2021. Long text generation by modeling sentence-level and discourse-level coherence. In: Annual Meeting of the Association for Computational Linguistics. pp. 6379–6393.

[9]

Hang, Y., Junqi, D., Tuo, J., et al., 2021. A unified generative frame-work for aspect-based sentiment analysis. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 2416–2429.

[10]

Holtzman A., Buys J., Du L., et al., The curious case of neural text degeneration, in: ICLR, 2020.

[11]

Hu, Z., Chan, H.P., Liu, J., et al., 2022. Planet: Dynamic content planning in autoregressive transformers for long-form text generation. In: Annual Meeting of the Association for Computational Linguistics. pp. 2288–2305.

[12]

Hua, X., Sreevatsa, A., Wang, L., 2021. Dyploc: Dynamic planning of content using mixed language models for text generation. In: Annual Meeting of the Association for Computational Linguistics. pp. 6408–6423.

[13]

Hua X., Wang L., Pair: Planning and iterative refinement in pre-trained transformers for long text generation, Comput. Sci. 2020.emnlp-main (2020) 781–793.

[14]

Huang Y., Sun H., Xu K., A chapter-level text generation method based on topic constraints, J. Peking Univ. (Nat. Sci. Ed.) 56 (9–15) (2020).

[15]

Koloski, B., Pollak, S., krlj, B., et al., 2021. Extending neural keyword extraction with tf-idf tagset matching. In: Conference of the European Chapter of the Association for Computational Linguistics. pp. 22–29.

[16]

Le, H., Le, D.-T., Weber, V., et al., 2022. Semi-supervised adversarial text generation based on seq2seq models. In: Conference on Empirical Methods in Natural Language Processing.

[17]

Lee, J.-R., Lee, Y.-J., Moon, Y.-H., 2021. Block-wise word embedding compression revisited - better weighting and structuring. In: Conference on Empirical Methods in Natural Language Processing. pp. 4379–4388.

[18]

Li, Y., Cui, L., Yan, J., et al., 2023a. Explicit Syntactic Guidance for Neural Text Generation. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 14095–14112.

[19]

Li, X., Holtzman, A., Fried, D., et al., 2023b. Contrastive Decoding: Open-ended Text Generation as Optimization. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12286–12312.

[20]

Liang, X., Tang, Z., Li, J., et al., 2023. Open-ended Long Text Generation via Masked Language Modeling. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 223–241.

[21]

Mike, L., Liu, Y., Naman, G., et al., 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Annual Meeting of the Association for Computational Linguistics. pp. 7871–7880.

[22]

Narayan S., Zhao Y., Maynez J., et al., Planning with learned entity prompts for abstractive summarization, Trans. Assoc. Comput. Linguist. 9 (2021) 1475–1492.

[23]

Puduppully R., Fu Y., Lapata M., Data-to-text generation with variational sequential planning, Trans. Assoc. Comput. Linguist. 10 (2022) 697–715.

[24]

Puduppully R., Lapata M., Data-to-text generation with macro planning, Trans. Assoc. Comput. Linguist. 9 (2021) 510–527.

[25]

See A., Pappu A., Saxena R., et al., Do massively pretrained language models make better storytellers?, in: CoNLL, 2019, pp. 843–861.

[26]

Stephen, R., Emily, D., Naman, G., et al., 2021. Recipes for building an open-domain chatbot. In: Conference of the European Chapter of the Association for Computational Linguistics. pp. 300–325.

[27]

Su, Y., Vandyke, D., Wang, S., et al., 2021. Plan-then-generate - controlled data-to-text generation via planning. In: Conference on Empirical Methods in Natural Language Processing. pp. 895–909.

[28]

Tan B., Yang Z., Al-Shedivat M., et al., Progressive generation of long text with pretrained language models, in: North American Chapter of the Association for Computational Linguistics, 2021, pp. 4313–4324.

[29]

Tang C., Loakman T., Lin C., et al., A cross-attention augmented model for event-triggered context-aware story generation, 2023, arXiv, abs/2311.11271.

[30]

Wu, W., Li, W., Liu, J., et al., 2022. Precisely the point: Adversarial augmentations for faithful and informative text generation. In: Conference on Empirical Methods in Natural Language Processing. pp. 7160–7176.

[31]

Xia H., Liu Y., Xiao Y., A long text generation adversarial network model incorporating self-attentive mechanism, Comput. Sci. Explor. 16 (1603–1610) (2022).

[32]

Yang, P., Cong, X., Sun, Z., et al., 2021. Enhanced language representation with label knowledge for span extraction. In: Conference on Empirical Methods in Natural Language Processing. pp. 4623–4635.

[33]

Ye W., Research on Controlled Generation Learning of Ancient Poems with Format Constraints, (Master’s thesis) Hangzhou University of Electronic Science and Technolog, 2022.

[34]

Yin, X., Wan, X., 2022. How do seq2seq models perform on end-to-end data-to-text generation?. In: Annual Meeting of the Association for Computational Linguistics. pp. 7701–7710.

[35]

Zhao Q., Niu J., Liu X., et al., Generation of coherent multi-sentence texts with a coherence mechanism, Comput. Speech Lang. 78 (2023).

Index Terms

Seq2Seq dynamic planning network for progressive text generation

Index terms have been assigned to the content through auto-classification.

Recommendations

Staged Long Text Generation with Progressive Task-Oriented Prompts
Neural Information Processing
Abstract
Generating coherent and consistent long text remains a challenge for artificial intelligence. The state-of-the-art paradigm partitions the whole generating process into successive stages, however, the content plan applied in each stage may be ...
Open-Domain Table-to-Text Generation based on Seq2seq
ACAI '18: Proceedings of the 2018 International Conference on Algorithms, Computing and Artificial Intelligence

Table-to-text generation involves using natural language to describe a table which has formal structure and valuable information. Open-domain table-to-text especially refers to table-to-text generation for open domain. This paper introduces a theme ...
GGP: A Graph-based Grouping Planner for Explicit Control of Long Text Generation
CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Existing data-driven methods can well handle short text generation. However, when applied to the long-text generation scenarios such as story generation or advertising text generation in the commercial scenario, these methods may generate illogical and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computer Speech and Language

Computer Speech and Language Volume 89, Issue C

Jan 2025

618 pages

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Academic Press Ltd.

United Kingdom

Publication History

Published: 01 January 2025

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents