逆向工程|抖音app协议逆向分析 android|安卓|爬虫

1.准备工具

抖音app（app下载连接) 提取码：6666

博客所写的所有算法还原均已开源在GitHub，地址

https://github.com/YotaGit/AlgorithmRestore

2.本次主要演示抓取某个话题下的数据没有图片的话是将图片过滤掉了，这样加载会更快
3.先抓包话题首页

GET http://aweme.snssdk.com/aweme/v1/challenge/aweme/?ch_id=1603132116358228&query_type=0&cursor=0&count=20&type=5&retry_type=retry_http&iid=361063819794455&device_id=1126379551206542&ac=wifi&channel=yingyonghui&aid=2329&app_name=douyin_lite&version_code=180&version_name=1.8.0&device_platform=android&ssmix=a&device_type=Nexus+6P&device_brand=google&language=zh&os_api=25&os_version=7.1.1&openudid=df2eb2a465f9b18f&manifest_version_code=180&resolution=1440*2392&dpi=560&update_version_code=1800&_rticket=771495793&ts=1630245339&as=aac81339d8000bc5a7c813&cp=3a28c81339d8c8133a2032&mas=0133332319a3f3f3b323f9b9b93fef9cdaf3b323f9d3b993f97933 HTTP/1.1 Accept-Encoding: gzip X-SS-QUERIES: dGMCA76ot3awALq29Cjed6Wxk9vvcxlI%2BcrVbsSshXPE367hsCodObDobByuYNDsyCWTS76ot3awAOm2rKzdv5e1E7B4tcJ4rILc5cqsABMctdUrsE%2BsYPhZcFX06I%2F6XKH1ew%3D%3D X-SS-REQ-TICKET: 771495805 Cookie: 你自己的cookie X-Gorgon: 03742f2b00051aeec7dc6b6c24d8a111044edde107abede12ba7 X-Khronos: 771495 Host: aweme.snssdk.com Connection: Keep-Alive User-Agent: okhttp/3.10.0.1

翻页后的请求与响应

GET http://aweme.snssdk.com/aweme/v1/challenge/aweme/?ch_id=1603132116358228&query_type=0&cursor=20&count=20&type=5&retry_type=retry_http&iid=361063819794455&device_id=1126379551206542&ac=wifi&channel=yingyonghui&aid=2329&app_name=douyin_lite&version_code=180&version_name=1.8.0&device_platform=android&ssmix=a&device_type=Nexus+6P&device_brand=google&language=zh&os_api=25&os_version=7.1.1&openudid=df2eb2a465f9b18f&manifest_version_code=180&resolution=1440*2392&dpi=560&update_version_code=1800&_rticket=771550466&ts=1630245393&as=aac81339d8000bc5dec813&cp=3a28c81339d8c8133a2032&mas=0133332319a3f3f3b323f9b9b93fef9c44f3b323f9991993f97933 HTTP/1.1 Accept-Encoding: gzip X-SS-QUERIES: dGMCA76ot3awALq29Cjed6Wxk9vvcxlI%2BcrVbsSshXPE367hsCodObDobByuYNDsyCWTS76ot3awAOm2rKzdv5e1E7B4tcJ4rILc5cqsABMctdUrsE%2BsYPhZcFX06I%2F6XKH1ew%3D%3D X-SS-REQ-TICKET: 771550474 Cookie:你自己的cookie X-Gorgon: 03742f2b000547608bad6b6c24d8a111044edde107abede1b5f4 X-Khronos: 771550 Host: aweme.snssdk.com Connection: Keep-Alive User-Agent: okhttp/3.10.0.1

跑一段时间你会发现出不了数据
逐步替换url请求参数会发现是as cp mas这三个参数失效了，得逆向分析一波
4.逆向分析步骤查壳
查壳的方式很多种，可以先用apktool解开查看特侦，如果应用被加固了，那么需要脱壳
脱壳的教程很多，可以优先考虑FDex2

文章图片

5.jadx使用搜索大法搜索as cp mas三个参数搜索需要讲究技巧，先搜索as和cp会直接死亡

文章图片

在搜索时按照java的尿性会有以下几种形式(仅供参考)先找长字符的搜索

"mas "mas= "mas" ="mas"

果然给蒙对了

文章图片

双击进入发现就是我们想要的

文章图片

上图的代码如下

package com.ss.android.ugc.aweme.app.a; import android.text.TextUtils; import com.meituan.robust.ChangeQuickRedirect; import com.meituan.robust.PatchProxy; import com.meituan.robust.PatchProxyResult; import com.meizu.cloud.pushsdk.notification.model.AdvanceSetting; import com.ss.android.common.applog.EagleEye; import com.ss.android.common.applog.GlobalContext; import com.ss.android.common.applog.UserInfo; import com.ss.android.deviceregister.a; import com.ss.android.ugc.aweme.app.AwemeApplication; import com.ss.android.ugc.aweme.app.b.e; import com.ss.sys.ces.out.ISdk; import com.ss.sys.ces.out.StcSDKFactory; import e.t; import java.io.UnsupportedEncodingException; import java.net.URLDecoder; import java.util.ArrayList; import java.util.List; /* compiled from: ApiAntiSpamUtils */ public final class b {/* renamed from: areason: collision with root package name */ public static ChangeQuickRedirect f13081a; public static t a(t tVar, List list, int i) { String str; PatchProxyResult proxy = PatchProxy.proxy(new Object[]{tVar, list, new Integer(i)}, null, f13081a, true, 866, new Class[]{t.class, List.class, Integer.TYPE}, t.class); if (proxy.isSupported) { return (t) proxy.result; } String decode = URLDecoder.decode(tVar.toString()); String b2 = a.b(); if (b2 == null) { b2 = ""; } if (decode.contains("&device_id=") || decode.contains("?device_id=")) { str = UserInfo.getUserInfo(i, decode, (String[]) list.toArray(new String[list.size()]), b2); } else { str = UserInfo.getUserInfo(i, decode, (String[]) list.toArray(new String[list.size()]), ""); } t.a k = tVar.k(); int length = str.length(); if (TextUtils.isEmpty(str)) { k.a(AdvanceSetting.ADVANCE_SETTING, "a1iosdfgh").a("cp", "androide1"); } else if (length % 2 == 0) { int i2 = length >> 1; String substring = str.substring(0, i2); ISdk sdk = StcSDKFactory.getSDK(GlobalContext.getContext(), (long) AwemeApplication.p().m()); sdk.setSession(e.a()); k.a(AdvanceSetting.ADVANCE_SETTING, substring).a("cp", str.substring(i2, length)).a("mas", EagleEye.byteArrayToHexStr(sdk.encode(substring.getBytes()))); } else { k.a(AdvanceSetting.ADVANCE_SETTING, "a1qwert123").a("cp", "cbfhckdckkde1"); } return k.b(); }public static List a(String str) { int i = 0; PatchProxyResult proxy = PatchProxy.proxy(new Object[]{str}, null, f13081a, true, 867, new Class[]{String.class}, List.class); if (proxy.isSupported) { return (List) proxy.result; } ArrayList arrayList = new ArrayList(); while (i <= str.length()) { int indexOf = str.indexOf(38, i); if (indexOf == -1) { indexOf = str.length(); } int indexOf2 = str.indexOf(61, i); if (indexOf2 == -1 || indexOf2 > indexOf) { arrayList.add(URLDecoder.decode(str.substring(i, indexOf))); arrayList.add(null); } else { arrayList.add(URLDecoder.decode(str.substring(i, indexOf2))); arrayList.add(URLDecoder.decode(str.substring(indexOf2 + 1, indexOf))); } i = indexOf + 1; } return arrayList; }public static String b(String str) { PatchProxyResult proxy = PatchProxy.proxy(new Object[]{str}, null, f13081a, true, 868, new Class[]{String.class}, String.class); if (proxy.isSupported) { return (String) proxy.result; } if (TextUtils.isEmpty(str)) { return str; } try { return URLDecoder.decode(str, "UTF-8"); } catch (UnsupportedEncodingException e2) { com.google.b.a.a.a.a.a.a(e2); return URLDecoder.decode(str); } } }

这里有一个AdvanceSetting.ADVANCE_SETTING
我们查找一下声明

文章图片

这一节我们只做fridaRpc调用下一节我们直接用python还原这三个参数

文章图片

现在我们已经很清楚需要hook UserInfo.getUserInfo()这个方法

import os import sysimport fridadef on_message(message, data): if message['type'] == 'send': print("[*] {0}".format(message['payload'])) else: print(message)js_code = """ Java.perform(function(){ var UserInfo = Java.use('com.ss.android.common.applog.UserInfo'); UserInfo.getUserInfo.overload('int', 'java.lang.String', '[Ljava.lang.String; ', 'java.lang.String').implementation=function(a,b,c,d){ send('a:'+a); send('b:'+b); send('c:'+c); send('d:'+d); user=this.UserInfo(a,b,c,d); return user; } }); """process = frida.get_remote_device().attach('com.ss.android.ugc.aweme.lite') script = process.create_script(js_code) script.on('message', on_message) script.load() sys.stdin.read()

getUserInfo的各个参数如下

文章图片

第一个是时间戳
第二个是你传入的请求url
第三个是请求的参数构成的字符串
第四个是一堆数字其实就是deviceId,从下面这个if else可以得知

文章图片

frida.rpc hook模板如下

hook_code = ''' rpc.exports = { getSign: function(str){ send('getSign'); Java.perform(function(){} ) } }; '''

于是结合djax代码可的代码如下

def on_message(message, data): if message['type'] == 'send': print("[*] {0}".format(message['payload'])) else: print(message)js_code = """ rpc.exports = { getsign: function (timestamps, base_url, params, device_id) { var sig = "aa"; Java.perform( function () { send('getsign'); var StcSDKFactory = Java.use('com.ss.sys.ces.out.StcSDKFactory'); var GlobalContext = Java.use('com.ss.android.common.applog.GlobalContext'); var AwemeApplication = Java.use('com.ss.android.ugc.aweme.app.AwemeApplication'); var EagleEye = Java.use('com.ss.android.common.applog.EagleEye'); var sdk = StcSDKFactory.getSDK(GlobalContext.getContext(), AwemeApplication.p().m()); var UserInfo = Java.use('com.ss.android.common.applog.UserInfo'); var aac=params.toArray(params.length); var user_info = UserInfo.getUserInfo(timestamps,base_url,params,device_id); var as = user_info.substring(0, 22); var cp = user_info.substring(22, user_info.length); var mas = EagleEye.byteArrayToHexStr(sdk.encode(getBytes(as))); sig = '&as='+as+'&cp='+cp+'&mas='+mas; } ) return sig; } }function getBytes(s) { var bytes = []; for (var i = 0; i < s.length; i++) { bytes.push(s.charCodeAt(i)); } return bytes; }"""process = frida.get_remote_device().attach('com.ss.android.ugc.aweme.lite') script = process.create_script(js_code) script.on('message', on_message) script.load()a= int(time.time())b="https://aweme-eagle.snssdk.com/aweme/v1/feed/?type=0&max_cursor=0&min_cursor=0&count=6&volume=0.0&pull_type=1&need_relieve_aweme=0&filter_warn=0&ts=1630343714&app_type=lite&os_api=25&device_platform=android&device_type=Xiaomi&iid=361063819794455&ssmix=a&manifest_version_code=180&dpi=560&version_code=180&app_name=douyin_lite&version_name=1.8.0&openudid=df2eb2a465f9b18f&device_id=1126379551206542&resolution=1440*2392&os_version=7.1.1&language=zh&device_brand=google&ac=wifi&update_version_code=1800&aid=2329&channel=yingyonghui&_rticket=1630343715767"c="type,0,max_cursor,0,min_cursor,0,count,6,volume,0.0,pull_type,1,need_relieve_aweme,0,filter_warn,0,ts,1630343714,app_type,lite,os_api,25,device_platform,android,device_type,Xiaomi,iid,361063819794455,ssmix,a,manifest_version_code,180,dpi,560,version_code,180,app_name,douyin_lite,version_name,1.8.0,openudid,df2eb2a465f9b18f,device_id,1126379551206542,resolution,1440*2392,os_version,7.1.1,language,zh,device_brand,google,ac,wifi,update_version_code,1800,aid,2329,channel,yingyonghui,_rticket,1630343715767"d="1126379551206542" sig = script.exports.getsign(a, b, c, d) print(sig)

【逆向工程|抖音app协议逆向分析】最终的结果如下图所示，每一次的调用都是生成不同的参数