对于io_data_processor:

1.将请求相关封装成了Request类
2.新增了获取cookies后使用aiohttp请求的机制,理论上可以提速+减少不必要的性能开销,并且将cookies存储至cache
3.整理代码
新增config.py,用于配置cache路径
添加依赖项Brotli
This commit is contained in:
2022-08-08 13:32:31 +08:00
parent 68028cf3d9
commit e473dca4df
5 changed files with 267 additions and 88 deletions

View File

@@ -2,13 +2,13 @@
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code.
extension-pkg-allow-list=lxml
extension-pkg-allow-list=lxml, pydantic
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code. (This is an alternative name to extension-pkg-allow-list
# for backward compatibility.)
extension-pkg-whitelist=lxml
extension-pkg-whitelist=lxml, pydantic
# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime