3.Python爬虫抓取百度网页.md · zo-gode/python爬虫

# Python爬虫抓取百度网页

**完整代码:**
```py
from urllib import request,parse 
from fake_useragent import FakeUserAgent # 提供随机一个UA信息

# 拼接URL地址
def get_url(word):
  url = 'https://baidu.com/s?{}'
  #此处使用urlencode()进行编码
  params = parse.urlencode({'wd':word})
  url = url.format(params)
  return url

# 发请求,保存本地文件
def request_url(url,filename):
  ua = FakeUserAgent()

这为一个完整的请求头，以防请求出错
#   Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
#   Accept_Language = "zh-CN,zh;a=0.9" 
#   Accept_Encoding = "gzip, deflate, br"
#   headers = {
#     'User-Agent':ua.google,
#     'Accept': Accept,
#     'Accept-Encoding':Accept_Encoding,
#     'Accept_Language':Accept_Language
#     }
  # 请求对象 + 响应对象 + 提取内容
#   req = request.Request(url=url,headers=headers)
  req=request.Request(url=url,headers={
    'User-Agent':ua.google
    })
  res = request.urlopen(req)
  html = res.read().decode("utf-8")
#   html = requests.get(url,headers=headers)
#   html.encoding = 'utf-8'
  # 保存文件至本地
  with open(filename,'w',encoding='utf-8') as f:
    f.write(html)

# 主程序入口
if __name__ == '__main__':
  word = input('请输入搜索内容:')
  url = get_url(word)
  filename = word + '.html'
  request_url(url,filename)
```

## 拼接URL地址
```py
def get_url(word):
    url = 'http://www.baidu.com/s?wd={}' # 百度搜索
    #想要搜索的内容
    word = input('请输入搜索内容:')
    params = parse.quote(word)
    full_url = url.format(params)
```

## 向URL发送请求
发送请求主要分为以下几个步骤：
* 创建请求对象-Request
* 获取响应对象-urlopen
* 获取响应内容-read
```py
#重构请求头
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0'}
#创建请求对应
req = request.Request(url=full_url,headers=headers)
#获取响应对象
res = request.urlopen(req)
#获取响应内容
html = res.read().decode("utf-8")
```

## 保存为本地文件
```py
filename = word + '.html'
with open(filename,'w', encoding='utf-8') as f:
    f.write(html)
```

zo-gode/python爬虫

简介

发行版

贡献者

近期动态

zo-gode/python爬虫 .gitee-modal { width: 500px !important; }

简介

发行版

贡献者

近期动态

搜索帮助

zo-gode/python爬虫