1 Star 0 Fork 0

bit212/emailscraper

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
文件
克隆/下载
chrome.go 1.09 KB
一键复制 编辑 原始数据 按行查看 历史
Law Zava 提交于 2021-03-07 00:34 +08:00 . :tada:first commit
package emailscraper
import (
"context"
"fmt"
"time"
"github.com/chromedp/chromedp"
"github.com/gocolly/colly"
)
func initiateScrapingFromChrome(response *colly.Response, timeout int) error {
opts := []chromedp.ExecAllocatorOption{
// nolint:lll // allow longer line here
chromedp.UserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3830.0 Safari/537.36"),
// nolint:gomnd // allow magic number here
chromedp.WindowSize(1920, 1080),
chromedp.NoFirstRun,
chromedp.Headless,
chromedp.DisableGPU,
}
ctx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
defer cancel()
ctx, cancel = chromedp.NewContext(ctx)
defer cancel()
if timeout > 0 {
ctx, cancel = context.WithTimeout(ctx, time.Duration(timeout)*time.Second)
defer cancel()
}
var res string
if err := chromedp.Run(ctx, chromedp.Navigate(response.Request.URL.String()),
chromedp.InnerHTML("html", &res), // Scrape whole rendered page
); err != nil {
return fmt.Errorf("executing chromedp: %w", err)
}
response.Body = []byte(res)
return nil
}
Loading...
马建仓 AI 助手
尝试更多
代码解读
代码找茬
代码优化
1
https://gitee.com/bit212/emailscraper.git
git@gitee.com:bit212/emailscraper.git
bit212
emailscraper
emailscraper
main

搜索帮助