Python实现文章自动生成

发布时间:2022-03-04 09:44:04 人气:85 作者:多测师

  下面的Python程序实现了通过从网页抓取一篇文章,然后根据这篇文章来生成新的文章,这其中的原理就是基于概率统计的文本分析。

  过程大概就是网页抓取数据->统计分析->生成新文章。网页抓取数据是通过BeautifulSoup库来抓取网页上的文本内容。统计分析这个首先需要使用ngram模型来把文章进行分词并统计频率。因为文章生成主要依据马尔可夫模型,所以使用了2-gram,这样可以统计出一个单词出现在另一个单词后的概率。生成新文章是基于分析大量随机事件的马尔可夫模型。随机事件的特点是在一个离散事件发生之后,另一个离散事件将在前一个事件的条件下以一定的概率发生。

Python实现文章自动生成

  from urllib.request import urlopen

  from random import randint

  from bs4 import BeautifulSoup

  import re

  def wordListSum(wordList):

  sum = 0

  for word, value in wordList.items():

  sum = sum + value

  return sum

  def retrieveRandomWord(wordList):

  randomIndex = randint(1, wordListSum(wordList))

  for word, value in wordList.items():

  randomIndex -= value

  if randomIndex <= 0:

  return word

  def buildWordDict(text):

  text = re.sub('(\n|\r|\t)+', " ", text)

  text = re.sub('\"', "", text)

  punctuation = [',', '.', ';', ':']

  for symbol in punctuation:

  text = text.replace(symbol, " " + symbol + " ")

  words = text.split(' ')

  words = [word for word in words if word != ""]

  wordDict = {}

  for i in range(1, len(words)):

  if words[i-1] not in wordDict:

  wordDict[words[i-1]] = {}

  if words[i] not in wordDict[words[i-1]]:

  wordDict[words[i-1]][words[i]] = 0

  wordDict[words[i-1]][words[i]] = wordDict[words[i-1]][words[i]] + 1

  return wordDict

  def randomFirstWord(wordDict):

  randomIndex = randint(0, len(wordDict))

  return list(wordDict.keys())[randomIndex]

  html = urlopen("http://www.guancha.cn/america/2017_01_21_390488_s.shtml")

  bsObj = BeautifulSoup(html, "lxml")

  ps = bsObj.find("div", {"id": "cmtdiv3523349"}).find_next_siblings("p");

  content = ""

  for p in ps:

  content = content + p.get_text()

  text = bytes(content, "UTF-8")

  text = text.decode("ascii", "ignore")

  wordDict = buildWordDict(text)

  length = 100

  chain = ""

  currentWord = randomFirstWord(wordDict)

  for i in range(0, length):

  chain += currentWord + " "

  currentWord = retrieveRandomWord(wordDict[currentWord])

  print(chain)

  buildWordDict(text)函数接收文本内容,生成的内容如下

  {‘itself’: {‘,’: 1}, ‘night’: {‘sky’: 1}, ‘You’: {‘came’: 1, ‘will’: 1}, ‘railways’: {‘all’: 1}, ‘government’: {‘while’: 1, ‘,’: 1, ‘is’: 1}, ‘you’: {‘now’: 1, ‘open’: 1, ‘down’: 1, ‘with’: 1, ‘.’: 6, ‘,’: 1, ‘that’: 1},

  主要就是生成一个字典,键是文章中所有出现的词语,值其实也是一个字典,这个字典是所有直接出现在键后边的词语及其出现的频率。这个函数就是ngram模型思想的运用。

  retrieveRandomWord(wordList)函数的wordList代表的是出现在上一个词语后的词语列表及其频率组成的字典,然后根据统计的概率随机生成一个词。这个函数是马尔可夫模型的思想运用。

  然后运行这个程序会生成一个长度为100的文章,如下面所示

  fail . We will stir ourselves , but we will never before . Do not share one heart and pleasant it back our jobs . We are infused with the orderly and railways all of the gangs and robbed our jobs for their success will determine the civilized world . We will their success will be a great men and highways and millions to all bleed the world . It belongs to great national effort to defend our products , constantly complaining , D . We will be ignored again . It belongs to harness the expense of America .

  生成的文章看起来语法混乱,这也难怪,因为只是抓取分析统计了一篇的文章。我想如果可以抓取足够多的英文文章,数据集足够大那么语法准确度会大大提高。

  以上内容为大家介绍了Python实现文章自动生成,希望对大家有所帮助,如果想要了解更多Python相关知识,请关注多测师。https://www.e70w.com/xwzx/

返回列表
在线客服
联系方式

热线电话

17727591462

上班时间

周一到周五

二维码
线