监听网页请求与结果
2023-11-07 21:42:19 94
监听浏览器打开网页时所发出的请求跟响应结果。能够抓取请求url、请求headers、请求body、响应headers、响应body。
监听结果以json对象的形式返回。监听通过url关键字进行匹配。符合条件的url将会被设置到监听结果中。
匹配url方式
文本查找---通过文本关键字的方式进行匹配url链接
正则匹配---通过正则表达式的方式进行匹配url链接
规则或url
如果是文本查找的方式，填写完整的url链接或者关键字，例如 ：jquery.js
如果是正则匹配的方式，请填写正则表达式，例如：\S+jquery\.js
监听结果保存至
保存监听到的json对象。json对象完整数据如下：
{
"req_method": "GET",
"request": {
"headers": [{
"Accept": "application/json, text/javascript, */*; q=0.01"
}, {
"Accept-Language": "zh-CN,zh;q=0.9"
}, {
"Cookie": "BIDUPSID=1DEF119D4F925601E46E396E69E01C3C; PSTM=1698854821; BAIDUID=1DEF119D4F9256012F2255CD833AE876:FG=1; BAIDUID_BFESS=1DEF119D4F9256012F2255CD833AE876:FG=1; ZFY=KOweg:AtVt7iXh0zi5UAVsLY6:AQc:AwD5GT:BLWfMShzDQ:C; BA_HECTOR=00212h05a42g0l2h01a10h8h1ikfdlr1q; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598"
}, {
"User-Agent": "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
}, {
"X-Requested-With": "XMLHttpRequest"
}, {
"sec-ch-ua": "\"Chromium\";v=\"107\""
}, {
"sec-ch-ua-mobile": "?0"
}, {
"sec-ch-ua-platform": "\"Windows\""
}]
},
"respone": {
"content": "{\"errno\":0,\"request_id\":\"3590354227\",\"timestamp\":1699277664,\"data\":{\"LocalNews\":{\"errno\":0,\"data\":{\"cityid\":3373,\"name\":\"\\u798f\\u5dde\",\"rows\":{\"pic\":{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774642373494363782\",\"title\":\"\\u201c\\u6d77\\u4e1d\\u597d\\u5e74\\u534e\\u2014\\u2014\\u7b2c\\u4e00\\u5c4a\\u2018\\u6d77\\u4e1d\\u9752\\u5e74\\u6c47\\u2019\\u4ea4\\u6d41\\u8425\\u201d\\u5f00...\",\"time\":\"15:16\",\"imgUrl\":\"https:\\/\\/t11.baidu.com\\/it\\/u=3454788165,212425516&fm=30&app=106&f=JPEG?w=312&h=208&s=FB91F3AA3E5930C60A870B50030070FA\"},\"first\":[{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774634857351164703\",\"title\":\"\\u9648\\u7389\\u6811\\u5f53\\u9009\\u798f\\u5efa\\u7701\\u4fa8\\u8054\\u517c\\u804c\\u526f\\u4e3b\\u5e2d\\uff1a\\u4ee5\\u201c\\u4fa8\\u201d\\u4e3a\\u201c\\u6865...\",\"time\":\"14:02\",\"imgUrl\":\"http:\\/\\/t11.baidu.com\\/it\\/u=2977994430,212335637&fm=30&app=106&f=JPEG?w=312&h=208&s=1B8B8645040157570024FD3F0300C04B\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774748353161668453\",\"title\":\"\\u798f\\u5dde\\u5e02\\u4ed9\\u6e38\\u5546\\u4f1a\\u7b2c\\u4e09\\u5c4a\\u7b2c\\u4e00\\u6b21\\u4f1a\\u5458\\u5927\\u4f1a\\u5728\\u6995\\u4e3e\\u884c\",\"time\":\"19:45\",\"imgUrl\":\"http:\\/\\/t11.baidu.com\\/it\\/u=1777932924,212441336&fm=30&app=106&f=JPEG?w=312&h=208&s=351C5F9604F81F8220936AE10300F028\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774833601562793311\",\"title\":\"33\\u5bb6\\u56fd\\u4f01\\u9080\\u9000\\u5f79\\u519b\\u4eba\\u5165\\u804c \\u6211\\u7701\\u9996\\u573a\\u9000\\u5f79\\u519b\\u4eba\\u53ca\\u519b...\",\"time\":\"18:21\",\"imgUrl\":\"http:\\/\\/t11.baidu.com\\/it\\/u=1869647517,212520731&fm=30&app=106&f=JPEG?w=312&h=208&s=2EE7099A5E0F58C80C41ECCB03000035\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774795475686729517\",\"title\":\"\\u798f\\u6e053.0\\u7248\\u57ce\\u5e02\\u516c\\u56ed\\u4eae\\u76f8\\uff01\\u905b\\u5f2f\\u53c8\\u6709\\u597d\\u53bb\\u5904\\u4e86\",\"time\":\"08:15\",\"imgUrl\":\"http:\\/\\/t12.baidu.com\\/it\\/u=934646211,212485227&fm=30&app=106&f=JPEG?w=312&h=208&s=580626D00841DB471299DB14030060DD\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774730410298209613\",\"title\":\"\\u51dd\\u805a\\u5171\\u8bc6\\u52a9\\u529b\\u53d1\\u5c55\\uff0c\\u5e7f\\u5dde\\u5e02\\u5404\\u6c11\\u4e3b\\u515a\\u6d3e\\u3001\\u5de5\\u5546\\u8054\\u8d1f\\u8d23...\",\"time\":\"15:01\",\"imgUrl\":\"http:\\/\\/t10.baidu.com\\/it\\/u=3473745129,212424632&fm=30&app=106&f=JPEG?w=312&h=208&s=31D7887E04B12D880DB299D90300D09B\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774836778035266086\",\"title\":\"\\u798f\\u5efa\\u4e00\\u6279\\u4ea4\\u901a\\u8fd0\\u8f93\\u5de5\\u7a0b\\u8fdb\\u5ea6\\u6761\\u5237\\u65b0\\u4e86\",\"time\":\"19:11\",\"imgUrl\":\"http:\\/\\/t11.baidu.com\\/it\\/u=2491620381,212523692&fm=30&app=106&f=JPEG?w=312&h=208&s=2F325F805A1928CE283C1CA103003082\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774852760972304743\",\"title\":\"\\u798f\\u53a6\\u7ebf3\\u6807\\u987a\\u5229\\u8d2f\\u901a\\uff01\\u6e56\\u5357\\u9001\\u53d8\\u7535\\u516c\\u53f8\\u4fdd\\u969c\\u5de5\\u7a0b\\u987a\\u5229...\",\"time\":\"23:25\",\"imgUrl\":\"http:\\/\\/t12.baidu.com\\/it\\/u=521400763,212538587&fm=30&app=106&f=JPEG?w=312&h=208&s=B326D300DCAD7B0D23351CD10300C0BA\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774818487353301261\",\"title\":\"\\u7834\\u9898\\u201c\\u51b7\\u70ed\\u4e0d\\u5747\\u201d\\uff0c\\u5404\\u5730\\u535a\\u7269\\u9986\\u591a\\u5411\\u53d1\\u529b\",\"time\":\"14:21\",\"imgUrl\":\"http:\\/\\/t11.baidu.com\\/it\\/u=2625878613,212506659&fm=30&app=106&f=JPEG?w=312&h=208&s=04705A8142C8335FC2B8C41F030080C1\"}],\"second\":[{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774846844927137147\",\"title\":\"\\u6b4c\\u5267\\u300a\\u5f04\\u81e3\\u300b\\u4e2d\\u6587\\u7248\\u5728\\u6995\\u5c55\\u6f14 \\u201c\\u6d0b\\u620f\\u4e2d\\u5531\\u201d\\u89c2\\u4f17\\u76f4...\",\"time\":\"21:51\",\"imgUrl\":\"http:\\/\\/t11.baidu.com\\/it\\/u=1577197647,212533066&fm=30&app=106&f=JPEG?w=312&h=208&s=457019C64FAAA2D00CDD990903007083\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774733220946765996\",\"title\":\"\\u90d1\\u5de7\\u7684\\u89d2\\u68b3\\u201c\\u65b0\\u201d\\u68a6\",\"time\":\"15:45\",\"imgUrl\":\"http:\\/\\/t11.baidu.com\\/it\\/u=1225374385,212427244&fm=30&app=106&f=JPEG?w=312&h=208&s=CCCA722B4E60369214A19DC60100A091\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774640455504879591\",\"title\":\"\\u798f\\u5efa\\u4e3e\\u529e\\u201c\\u4e2d\\u56fd\\u533b\\u5e08\\u8282\\u201d\\u6d3b\\u52a8\\u5927\\u4f1a\",\"time\":\"15:10\",\"imgUrl\":\"http:\\/\\/t12.baidu.com\\/it\\/u=4157607778,212340860&fm=30&app=106&f=JPEG?w=312&h=208&s=49619346CF210C8C33E9150B03007090\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774839359139396485\",\"title\":\"\\u798f\\u5dde\\u5730\\u94c1\\u5ba3\\u5e03\\uff1a\\u514d\\u8d39\\u9886\\uff0c\\u53d1\\u5b8c\\u4e3a\\u6b62\\uff01\",\"time\":\"19:52\",\"imgUrl\":\"http:\\/\\/t11.baidu.com\\/it\\/u=87024306,212526095&fm=30&app=106&f=JPEG?w=312&h=208&s=8372C536094B404D5245B0C00100E0B1\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774802258322267237\",\"title\":\"\\u300c\\u70df\\u706b\\u4eba\\u95f4\\u300d\\u7a7f\\u884c\\u4e8e\\u4e09\\u574a\\u4e03\\u5df7\",\"time\":\"10:06\",\"imgUrl\":\"http:\\/\\/t10.baidu.com\\/it\\/u=2646765359,212491543&fm=30&app=106&f=JPEG?w=312&h=208&s=91022BF946E3D6E61E846239030000D7\"},{\"url\":\"http:\\/\\/baijiahao.baidu.com\\/s?id=1774712492948944773\",\"title\":\"\\u798f\\u5efa5\\u4f4d\\u7701\\u7ea7\\u975e\\u9057\\u4ee3\\u8868\\u6027\\u4f20\\u627f\\u4eba\\uff0c\\u88ab\\u53d6\\u6d88\\u8d44\\u683c\\uff01\",\"time\":\"10:16\",\"imgUrl\":\"http:\\/\\/t10.baidu.com\\/it\\/u=653678190,212407940&fm=30&app=106&f=JPEG?w=312&h=208&s=4AE788561AEE4E8C22B82FEE0300B03E\"}]}},\"ad\":null}}}",
"headers": [{
"Connection": "keep-alive"
}, {
"Content-Encoding": "br"
}, {
"Content-Length": "1707"
}, {
"Content-Type": "application/json;charset=utf-8"
}, {
"Date": "Mon, 06 Nov 2023 13:34:24 GMT"
}, {
"Server": "nginx/1.9.1"
}, {
"Set-Cookie": "LOCALGX=%u798F%u5DDE%7C%33%33%37%33%7C%u798F%u5DDE%7C%33%33%37%33; expires=Mon, 13-Nov-2023 13:34:24 GMT; Max-Age=604800; path=/; domain=.news.baidu.com"
}, {
"Tracecode": "359035422707229935370110621"
}, {
"X-Bd-Api": "news_widget_localnews"
}, {
"X-Bd-Status": "200"
}, {
"X-Logic-No": "null"
}],
"status": 200
},
"url": "https://news.baidu.com/widget?id=LocalNews&ajax=json&t=1699277664212"
}
其中列表项包含:
'req_method'(请求方式POST、GET)
'url'(发起请求的url)
'request' （请求对象,里面还包含列表项:'headers'(请求头部)、如果是post方式还包含'post_param'(post参数)）
'respone'(响应对象,里面还包含列表项:'status'(响应状态码)、'headers'(响应头部)、'content'(响应内容))
使用方式
设置好监听请求后，再操作网页界面，rpa会在后台持续监听网页的请求链接，当符合设置的条件后会生成上面的json格式对象保存到变量中。
如果后续流程不需要再继续监听请求。建议停止监听网页，提高网页的加载速度跟效率。
数据的使用方式
