对于io_data_processor:

1.将请求相关封装成了Request类
2.新增了获取cookies后使用aiohttp请求的机制,理论上可以提速+减少不必要的性能开销,并且将cookies存储至cache
3.整理代码
新增config.py,用于配置cache路径
添加依赖项Brotli
This commit is contained in:
2022-08-08 13:32:31 +08:00
parent 68028cf3d9
commit e473dca4df
5 changed files with 267 additions and 88 deletions

View File

@@ -18,6 +18,7 @@ lxml = "^4.9.1"
pandas = "^1.4.3"
playwright = "^1.24.1"
ujson = "^5.4.0"
Brotli = "^1.0.9"
[tool.poetry.dev-dependencies]
mypy = "^0.971"