BabyDuncan
- 浏览: 574021 次
- 性别:
- 来自: 北京
-
文章分类
最新评论
-
liuzeyuss：
好文章，顶顶
java 线程死锁的检测 -
ysite：
hugh.wang 写道请问你这个能还原吗？短网址用的是HAS ...
Java版短网址(ShortUrl)的算法 -
lv双：
没有注释，看的费劲
java 线程死锁的检测 -
dgj：
好帖子竟然没人顶...
java 线程死锁的检测 -
天空趋虚：
不好,你这只是记录式的博客,不是分享式的博客,对浏览着不友好. ...
一些技术牛人的博客
使用hive分析nginx 日志文件
数据格式
分析共有几种数据格式：
more access.log | awk -F "[\"]" '{print $2}' | awk -F "[?]" '{print $1}'| sort | uniq | sort
共有以下种请求：
****************************************************
stat
api/databus/stat/click
GET /click.html
api/databus/stat/display
GET /display.html
api/databus/stat/game
GET /game.html
ane
api/databus/ane/end
GET /end_ane.html √
api/databus/ane/run
GET /run_ane.html √
api/databus/ane/login
GET /login_ane.html √
api/databus/ane/loader
GET /loader_run.html
***************************************************
GET /click.html
举例：
124.88.232.193 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*925008604*1*nlink*https://itunes.apple.com/cn/app/xue-di-zi-sheng-dan-ban/id588116262?ls=1&mt=8
HTTP/1.1" 301 0 "-"
1.205.1.200 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*838001018*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0 "-"
125.85.93.141 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*838000737*1*nlink*https://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0 "-"
——————————————————————————————————————————————————————————————————————————————
GET /click.html HTTP/1.0
62.24.222.131 - - [13/Jan/2013:23:26:18 +0800] "GET /click.html HTTP/1.0" 200 3 "http://tj.pet.mop.com/click.html"
——————————————————————————————————————————————————————————————————————————————
GET /display.html
115.45.201.131 - - [13/Jan/2013:23:00:01 +0800] "GET /display.html?80134|952000101|0|receiver|0|mop HTTP/1.1" 304 0
36.40.191.155 - - [13/Jan/2013:23:00:02 +0800] "GET /display.html?500293|1|0|receiver|0|mop HTTP/1.1" 200 23
116.22.217.54 - - [13/Jan/2013:23:00:05 +0800] "GET /display.html?9|9040010018|0|receiver|0|mop HTTP/1.1" 200 23
——————————————————————————————————————————————————————————————————————————————
GET /end_ane.html
221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&i
mei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885
001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=13580
88689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-"
114.221.147.156 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
roleid=295553123592712996&uuid2=null&starttime=1358089089&apptype=null&domain=&endtime=1358089095&uuid1=null&onlinetime=6&devicetype=null&launch
time=1358080763&deviceversion=null&startcount=12&f=838000707&activetime=436&detaildevicetype=null&gameid=ms&jailbroken=null&versionid=1.0.8&connec
ttype=null&userid=497886004&appid=com.msqy.weiduan HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1"
175.142.180.206 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?
connecttype=wifi&uuid1=0d40e54d523a48428ad10ecc5253e153&mac=A467069329AE&detaildevicetype=iPad2,2&apptype=ios&versionid=1.1.2&deviceversion=4.3.1&ime
i=null&uuid2=ee5bdbc81fb02ee3bc0b362a5281a935&devicetype=iPad&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x21.cs
j.renren.com&userid=null&roleid=null&starttime=1358089192&endtime=1358089200&onlinetime=8&launchtime=1358089192&startcount=1&activetime=8 HTTP/1.1"
200 46 "-"
——————————————————————————————————————————————————————————————————————————————
GET /favicon.ico HTTP/1.1
120.33.159.158 - - [13/Jan/2013:23:00:45 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
14.118.120.122 - - [13/Jan/2013:23:01:25 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
116.230.225.94 - - [13/Jan/2013:23:05:25 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
——————————————————————————————————————————————————————————————————————————————
GET /game.html
222.209.150.169 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?0|932000101|3400|renren|0|http://x20.war.renren.com/ HTTP/1.1" 304 0
222.209.150.169 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?0|932000101|2400|renren|1|http://x20.war.renren.com/ HTTP/1.1" 304 0
182.242.227.214 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?-1|855004201|3400|0|0|0 HTTP/1.1" 200 20
——————————————————————————————————————————————————————————————————————————————
GET / HTTP/1.1
113.108.11.50 - - [13/Jan/2013:23:09:21 +0800] "GET / HTTP/1.1" 200 20
112.90.84.224 - - [13/Jan/2013:23:09:21 +0800] "GET / HTTP/1.1" 200 20
123.190.215.78 - - [13/Jan/2013:23:13:21 +0800] "GET / HTTP/1.1" 200 0 "-"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.imop.com/click.html
112.97.192.31 - - [13/Jan/2013:23:11:07 +0800] "GET /click.html?9*855000865*1*nlink*http://itunes.apple.com/cn/app/lu-bu-yu-diao-chan/id555912419?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*855000865*1*nlink*http://itunes.apple.com/cn/app/lu-bu-yu-diao-chan/id555912419?mt=8"
118.116.81.28 - - [13/Jan/2013:23:11:39 +0800] "GET /click.html?9*838001010*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*838001010*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8"
125.126.32.95 - - [13/Jan/2013:23:12:00 +0800] "GET /click.html?9*838001017*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*838001017*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/click.html
119.97.151.160 - - [13/Jan/2013:23:54:42 +0800] "GET http://tj.pet.mop.com/click.html?60101*150101*1*nlink*http://itunes.apple.com/cn/app/id588116262?mt=8 HTTP/1.1" 301 0
"-"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/end_ane.html
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=4&endtime=1357999139&deviceversion=6.0&version
id=8.1.1&starttime=1357999135&launchtime=1357998799&f=838000707&detaildevicetype=iPhone3,1&userid=467415015&activetime=220&apptype=ios&jailbroken=f
alse&startcount=5&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=295834598569411941 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
216.36.142.199 - - [13/Jan/2013:23:07:07 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=157&endtime=1358089586&deviceversion=6.0&versi
onid=8.1.1&starttime=1358089429&launchtime=1358089429&f=838000707&detaildevicetype=iPhone3,1&userid=null&activetime=157&apptype=ios&jailbroken=fals
e&startcount=1&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=null HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:11 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=51&endtime=1358089676&deviceversion=6.0&versio
nid=8.1.1&starttime=1358089625&launchtime=1358089429&f=838000707&detaildevicetype=iPhone3,1&userid=454219948&activetime=208&apptype=ios&jailbroken=
false&startcount=2&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886914 HTTP/1.1" 200 43 "app:/com.renren.gameskit
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/login_ane.html
216.36.142.199 - - [13/Jan/2013:23:07:22 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:25 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
119.97.151.160 - - [13/Jan/2013:23:08:55 +0800] "GET http://tj.pet.mop.com/login_ane.html?
roleid=42a040000008200&f=926004501&userid=71461626@qq.com&jailbroken=false&appid=522657364&devicetype=iPhone&uuid2=3c648eb51633153a9a3cba3c0476004
e&deviceversion=6.0.1&detaildevicetype=iPhone4,1&versionid=4.2.0&uuid1=4f94d34e68724d8488782b7632a3935f&domain=x42%20-%20%E7%AA%81%E8%A2%AD
%E8%8B%8F%E8%81%94-60.29.251.165&gameid=com.imop.game.IronMarshal&connecttype=wwan&apptype=ios HTTP/1.1" 200 43 "app:/com.renren.gameskit"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/run_ane.html
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
216.36.142.199 - - [13/Jan/2013:23:07:08 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:11 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
——————————————————————————————————————————————————————————————————————————————
GET /loader_run.html
180.172.254.182 - - [13/Jan/2013:23:00:01 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021510&sdkver=0.0.16&gameid=null&f=8560021510&apptype=android&versionid=0.0.16&uuid2=354356050087566&detaildevicetype=HTC
+T329t&imei=354356050087566&uuid1=810816ca06a04f2a3b468adbb3616aff&mac=a0f450ff66e7&jailbroken=false&devicetype=HTC
+T329t&domain=&deviceversion=&connecttype=4.0.4 HTTP/1.1" 200 3 "-"
1.192.245.84 - - [13/Jan/2013:23:00:02 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021519&sdkver=0.0.14&gameid=null&f=8560021519&apptype=android&versionid=0.0.14&uuid2=866508010354368&detaildevicetype=ONN
+K7&imei=866508010354368&uuid1=44d6b021cc7d04d1fb78d2bfeb1adfee&mac=5ce2f4f4ab17&jailbroken=false&devicetype=ONN
+K7&domain=&deviceversion=&connecttype=4.0.4 HTTP/1.1" 200 3 "-"
112.93.35.68 - - [13/Jan/2013:23:00:03 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021518&sdkver=0.0.15&gameid=null&f=8560021518&apptype=android&versionid=0.0.15&uuid2=359548049282598&detaildevicetype=GT-
N7000&imei=359548049282598&uuid1=946161396c64523669bc52f7cf39a6cb&mac=50CCF8AABE67&jailbroken=false&devicetype=GT-
N7000&domain=&deviceversion=&connecttype=2.3.6 HTTP/1.1" 200 3 "-"
——————————————————————————————————————————————————————————————————————————————
GET /login_ane.html
101.85.147.146 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=70f784738fe0d2f645e15df6de3ffade&devicetype=iPad&mac=40300426A829&detaildevicetype=iPad2,2&connecttype=wifi&sdkver=1.0.7&uuid1=0e7431baeb7846c
0b1caa73a5647684b&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=447393827&roleid=294567961174220907 HTTP/1.1" 200 46 "-"
218.83.227.131 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
f=925004401&gameid=lstx&domain=http://x66.lstx.renren.com&appid=588116262&versionid=1.5.0.1&devicetype=iPad&deviceversion=6.0&jailbroken=false&connectt
ype=wifi&uuid1=93f3dc7b9e2547149a980e94cbb02cbe&udid2=6137ccd66654bc518207c33e460c8986&mac=4c:b1:99:08:bc:d8 HTTP/1.1" 200 43 "-"
210.242.214.26 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=09b67afe070498e4c2601f1be7d33f30&devicetype=iPad&mac=A4D1D25BF26B&detaildevicetype=iPad2,1&connecttype=wifi&sdkver=1.0.7&uuid1=65fe6e70e77c457
89fe988c1ace8bb38&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=8089268&roleid=290768048988618933 HTTP/1.1" 200 46 "-"
——————————————————————————————————————————————————————————————————————————————
GET /login.html
115.197.239.203 - - [13/Jan/2013:23:00:01 +0800] "GET /login.html?
lhsg*ios*870001315*null*efc073686f92362f2663fcc7cc6e64c97eae0c8d*iPad*5.1.1*com.renren.game.nqj2hd*1.0.1 HTTP/1.1" 200 3 "app:/lhsg.swf"
115.197.239.203 - - [13/Jan/2013:23:00:01 +0800] "GET /login.html?
lhsg*ios*870001315*x10.lhsg.renren.com*efc073686f92362f2663fcc7cc6e64c97eae0c8d*iPad*5.1.1*com.renren.game.nqj2hd*1.0.1 HTTP/1.1" 200 3 "app:/lhsg.swf"
111.110.226.84 - - [13/Jan/2013:23:00:03 +0800] "GET /login.html?
lstx*android*8820017*http://x14.tx.rrgame.jp*SSHGE005483_a1000016c6b272_78:1C:5A:3C:34:74*ANDROID_IS17SH*4.0.4*and10301*1.3.317 HTTP/1.1" 200 3 "-"
——————————————————————————————————————————————————————————————————————————————
GET /robots.txt HTTP/1.0
62.24.222.131 - - [13/Jan/2013:23:26:17 +0800] "GET /robots.txt HTTP/1.0" 404 564 "http://tj.pet.mop.com/robots.txt"
GET /robots.txt HTTP/1.1
180.76.5.91 - - [13/Jan/2013:23:51:58 +0800] "GET /robots.txt HTTP/1.1" 404 162 "-"
180.76.5.49 - - [13/Jan/2013:23:52:01 +0800] "GET /robots.txt HTTP/1.1" 404 345
——————————————————————————————————————————————————————————————————————————————
GET /run_ane.html
112.96.131.124 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
appid=com.imop.game.IronMarshalX&gameid=ms&uuid1=abea9df0f87e49b7984f4ca373421a33&devicetype=iPhone&deviceversion=5.1.1&versionid=3.1.1&f=838000729&de
taildevicetype=iPhone4,1&apptype=ios&jailbroken=true&uuid2=a98488fc51257fc2ac574ab1983d054d&domain=&connecttype=wwan HTTP/1.1" 200 43
"app:/com.renren.gameskit"
114.221.147.156 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
uuid2=null&apptype=null&domain=&uuid1=null&devicetype=null&deviceversion=null&f=838000707&detaildevicetype=null&gameid=ms&jailbroken=null&versio
nid=1.0.8&connecttype=null&appid=com.msqy.weiduan HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1"
221.194.31.155 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?f=925004443&gameid=jzsg&domain={domain}
&appid=564762410&versionid=1.0.8&devicetype=iPhone&deviceversion=4.3.3&jailbroken=false&connecttype=wwan&uuid1=c95a9e14a1e34702999b57fb1e9ddb53&udid
2=041357b8146f516b3ca9a96b9e88cc90&mac=14:5a:05:e7:6c:25 HTTP/1.1" 200 43 "app:/musouClient.swf"
——————————————————————————————————————————————————————————————————————————————
GET /run.html
175.141.66.138 - - [13/Jan/2013:23:00:01 +0800] "GET /run.html?
lstx*android*925010017*http://x82.lstx.renren.com*cff19b4b01dc53f_358401049472325_98:0C:82:F0:BA:B2*ANDROID_GT-I9100G*4.0.3*and10101*1.3.318 HTTP/1.1" 200 3 "-"
210.242.214.5 - - [13/Jan/2013:23:00:03 +0800] "GET /run.html?lstx*android*9250082140**HT243W104686_359188048172357_18:87:96:8b:dc:6d*ANDROID_HTC%20One
%20X*4.1.1*and10101*1.3.299 HTTP/1.1" 200 3 "-"
89.24.14.32 - - [13/Jan/2013:23:00:05 +0800] "GET /run.html?lstx*android*862004601**4df1eea16a789f3f_354833058446879_20:02:AF:B9:F7:F8*ANDROID_GT-
N7100*4.1.1*and02501*1.3.263 HTTP/1.1" 200 3 "-"
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
统计的url仅限：
次数 url
50924 GET /login_ane.html
72678 GET /end_ane.html
78684 GET /run_ane.html
13 GET http://tj.pet.mop.com/login_ane.html
16 GET http://tj.pet.mop.com/end_ane.html
16 GET http://tj.pet.mop.com/run_ane.html
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/run_ane.html √
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/end_ane.html √
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=4&endtime=1357999139&deviceversion=6.0&version
id=8.1.1&starttime=1357999135&launchtime=1357998799&f=838000707&detaildevicetype=iPhone3,1&userid=467415015&activetime=220&apptype=ios&jailbroken=f
alse&startcount=5&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=295834598569411941 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/login_ane.html √
216.36.142.199 - - [13/Jan/2013:23:07:22 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /run_ane.html √
112.96.131.124 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
appid=com.imop.game.IronMarshalX&gameid=ms&uuid1=abea9df0f87e49b7984f4ca373421a33&devicetype=iPhone&deviceversion=5.1.1&versionid=3.1.1&f=838000729&de
taildevicetype=iPhone4,1&apptype=ios&jailbroken=true&uuid2=a98488fc51257fc2ac574ab1983d054d&domain=&connecttype=wwan HTTP/1.1" 200 43
"app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /end_ane.html √
221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&i
mei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885
001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=13580
88689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /login_ane.html √
101.85.147.146 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=70f784738fe0d2f645e15df6de3ffade&devicetype=iPad&mac=40300426A829&detaildevicetype=iPad2,2&connecttype=wifi&sdkver=1.0.7&uuid1=0e7431baeb7846c
0b1caa73a5647684b&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=447393827&roleid=294567961174220907 HTTP/1.1" 200 46 "-"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
hadoop fs -ls /data/raw/adwall/test
hadoop fs -rm /data/raw/adwall/test/adwall-1.0.jar
hadoop fs -put ./adwall-1.0.jar /data/raw/adwall/test
hadoop fs -put /data/var/zghtest/adwallSerde/target/adwall-1.0.jar /data/raw/adwall/test
hive> add jar /data/var/zghtest/adwall-1.0.jar;
Added /data/var/zghtest/adwall-1.0.jar to class path
Added resource: /data/var/zghtest/adwall-1.0.jar
cp /data/var/zghtest/adwallSerde/target/adwall-1.0.jar /data/var/zghtest
删除的话就是delete jar
delete jar /data/var/zghtest/adwall-1.0.jar;
建表语句相关
CREATE EXTERNAL TABLE raw.t_adwall_test
(
ip STRING ,
datee STRING ,
interfaze STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall_test2
(
ip STRING ,
datee STRING ,
interfaze STRING,
ipanddate STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall_test3
(
ip STRING ,
datee STRING ,
interfaze STRING,
ipanddate STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall
(
f_ip STRING ,
f_datee STRING ,
f_interfaze STRING,
f_appid STRING,
f_gameid STRING,
f_uuid1 STRING,
f_devicetype STRING,
f_deviceversion STRING,
f_versionid STRING,
f_f STRING,
f_detaildevicetype STRING,
f_apptype STRING,
f_jailbroken STRING,
f_uuid2 STRING,
f_domain STRING,
f_connecttype STRING,
f_roleid STRING,
f_mac STRING,
f_sdkver STRING,
f_userid STRING,
f_starttime BIGINT,
f_endtime BIGINT,
f_onlinetime BIGINT,
f_launchtime BIGINT,
f_startcount INT,
f_activetime BIGINT,
f_formatright SMALLINT,
f_formaterrorreason STRING,
f_raw STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
LOAD DATA LOCAL INPATH '/data/var/zghtest/access.log' OVERWRITE INTO TABLE raw.t_adwall;
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
hive> select * from t_adwall where f_formatright=0 limit 10;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201206261006_286591, Tracking URL = http://h1-1.io8.org:50030/jobdetails.jsp?jobid=job_201206261006_286591
Kill Command = /data/opt/hadoop/bin/hadoop job -Dmapred.job.tracker=h1-1.io8.org:9001 -kill job_201206261006_286591
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 0
2013-01-16 22:30:26,367 Stage-1 map = 0%, reduce = 0%
2013-01-16 22:30:31,394 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 11.23 sec
2013-01-16 22:30:32,401 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 11.23 sec
2013-01-16 22:30:33,408 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 11.23 sec
MapReduce Total cumulative CPU time: 11 seconds 230 msec
Ended Job = job_201206261006_286591
MapReduce Jobs Launched:
Job 0: Map: 2 Accumulative CPU: 11.23 sec HDFS Read: 33726867 HDFS Write: 2843 SUCESS
Total MapReduce CPU Time Spent: 11 seconds 230 msec
OK
1024null null null com.msqy.weiduan ms null null null 1.0.8 838000707 null null null null null null null null NULL 1358089234
NULL 0 1358089234 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null 520025006 jzsg 39bfe5c7178c4882a06a19202f0aee6b iPad 5.1.1 1.0.6 iPad3,1 ios false ec147b2821c8030b4c6812538ef5d9ed
http://x41.s.jzsg.renren.com wifi null null null NULL 1358089273 1358089273 0 1358089273 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null elite.gen.ms.shzr ms 27329507945c40afbf588a5330a2246b iPhone 6.0.1 8.1.1 838000707 iPhone4,1 ios false
d38a920af438c405b08ee7d150d35bd4 wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1357915585 1357915585 0 1357915585
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wosiphone ms 407c7060d95946fdae79054f307bb1e3 iPad 6.0.1 1.0.7 838000707 iPad2,2 ios false
338955105bad19c4f57641b05eb0a77e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358077428 1358077428 0 1358077428
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wosiphone ms 8c81ef15e8ea4c5fbcf8856153bef980 iPhone 4.3.1 1.0.6 838000707 iPhone3,1 ios true
f4db9de6d3d08e956e3378f03a3ceb70 wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1357871821 1357871821 0 1357871821
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.msqy.weiduan ms null null null 1.0.8 HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1" 838000707 null null null null
null null null null NULL 1358089493 1358089493 0 1358089493 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null elite.gen.ms.shzr ms 01bd91e55d564dde83d665851328f4b7 iPhone 6.0.1 8.1.1 838000707 iPhone4,1 ios false
f427006b61c116e6ea3c53d18e5e2f9e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358089478 1358089478 0 1358089478
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wos1iphone ms 970b9f38cd464251a53bf39c2f3ebddc iPhone 6.0.1 1.1.6 838000707 iPhone4,1 ios false
5133ffcde7cf5c69d7fd141e436c265e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358088195 1358088195 0 1358088195
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null wostest ms HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1" null null null 00.00.04 838000707 null null null null
nullnull null null NULL 1358089515 1358089515 0 1358089515 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null and10101 lstx 2842f80102034bc1b59f674ae2649f27 MB525 2.3.7 HTTP/1.1" 200 43 "app:/com.renren.gameskit" 1.6.075 925010018
MB525 android false 2842f80102034bc1b59f674ae2649f27 lstx.renren.com 2G null 984b4a1561ab 1.0.7 NULL 1358088814 1358088814 0 1358088814
1 0 1 {}
Time taken: 14.202 seconds
hive>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
hive -S -e "use raw;add jar /data/var/zghtest/adwall-1.0.jar; select * from t_adwall where f_formatright=1 limit 10;" >> /data/var/zghtest.cvs
set hive.cli.print.header=true;
set hive.cli.print.row.to.vertical=true;
set hive.cli.print.row.to.vertical.num=1;
开启显示列名的功能。
CREATE EXTERNAL TABLE dw0.t_adwall
(
f_ip STRING ,
f_datee STRING ,
f_interfaze STRING,
f_appid STRING,
f_gameid STRING,
f_uuid1 STRING,
f_devicetype STRING,
f_deviceversion STRING,
f_versionid STRING,
f_f STRING,
f_detaildevicetype STRING,
f_apptype STRING,
f_jailbroken STRING,
f_uuid2 STRING,
f_domain STRING,
f_connecttype STRING,
f_roleid STRING,
f_mac STRING,
f_sdkver STRING,
f_userid STRING,
f_starttime BIGINT,
f_endtime BIGINT,
f_onlinetime BIGINT,
f_launchtime BIGINT,
f_startcount INT,
f_activetime BIGINT,
f_formatright SMALLINT,
f_formaterrorreason STRING,
f_raw STRING
)
建立洗过一次的数据表：
insert overwrite table dw0.t_adwall
select f_ip, f_datee,f_interfaze, f_appid, f_gameid , f_uuid1, f_devicetype , f_deviceversion
, f_versionid,f_f , f_detaildevicetype , f_apptype , f_jailbroken, f_uuid2, f_domain, f_connecttype,
f_roleid , f_mac , f_sdkver , f_userid , f_starttime , f_endtime , f_onlinetime , f_launchtime , f_startcount ,
f_activetime , f_formatright , f_formaterrorreason , f_raw
from raw. t_adwall
新的查询结果：
hive> select * from dw0.t_adwall where f_formatright=1 limit 10;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201206261006_287114, Tracking URL = http://h1-1.io8.org:50030/jobdetails.jsp?jobid=job_201206261006_287114
Kill Command = /data/opt/hadoop/bin/hadoop job -Dmapred.job.tracker=h1-1.io8.org:9001 -kill job_201206261006_287114
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 0
2013-01-17 17:01:10,566 Stage-1 map = 0%, reduce = 0%
2013-01-17 17:01:13,580 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.55 sec
2013-01-17 17:01:14,585 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.55 sec
MapReduce Total cumulative CPU time: 2 seconds 550 msec
Ended Job = job_201206261006_287114
MapReduce Jobs Launched:
Job 0: Map: 2 Accumulative CPU: 2.55 sec HDFS Read: 140255 HDFS Write: 6541 SUCESS
Total MapReduce CPU Time Spent: 2 seconds 550 msec
OK
f_ip f_datee f_interfaze f_appid f_gameid f_uuid1 f_devicetype f_deviceversion f_versionid f_f f_detaildevicetype f_apptype f_jailbroken f_uuid2 f_domain f_connecttype f_roleid f_mac f_sdkver f_userid f_starttime f_endtime f_onlinetime f_launchtime f_startcount f_activetime f_formatright f_formaterrorreason f_raw
221.194.30.138 - null "GET /end_ane com.imop.game.pirate2iphone rrnhzf d51c772dd73e48c29dd76269e021d418 iPhone 6.0.1 1.2.1 885001201 iPhone3,1 ios falsbdd0d548eb1edbcd459dbf7f32cc4751 x39.nhzf.renren.com null 295271648616025261 28E02C12BE8D 1.0.7 221106049 1358088689 1358089200 511 1358088689 1 NULL{} 221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&imei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=1358088689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-"
175.142.180.206 null "GET /end_ane com.renren.game.csj csj 0d40e54d523a48428ad10ecc5253e153 iPad 4.3.1 1.1.2 null iPad2,2 ios false ee5bdbc81fb02ee3bc0b362a5281a935 x21.csj.renren.com null null A467069329AE 1.0.8 null 1358089192 1358089200 8 1358089192 1 NULL 1 {} 175.142.180.206 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wifi&uuid1=0d40e54d523a48428ad10ecc5253e153&mac=A467069329AE&detaildevicetype=iPad2,2&apptype=ios&versionid=1.1.2&deviceversion=4.3.1&imei=null&uuid2=ee5bdbc81fb02ee3bc0b362a5281a935&devicetype=iPad&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x21.csj.renren.com&userid=null&roleid=null&starttime=1358089192&endtime=1358089200&onlinetime=8&launchtime=1358089192&startcount=1&activetime=8 HTTP/1.1" 200 46 "-"
221.5.86.37 - - null "GET /end_ane com.renren.games.csj4 com.renren.games.csj4 e368ba2a6ec14a02bb32cb5cef284027 iPhone 5.0.1 1.1.4 8560011158 iPhone4,1 nulltrue eaacae952f7b421abf062978cb8898d1 x32.csj.renren.com wifi 297241973452978776 60FACD22FC83 1.0.9 505529877 1358088339 1358089202 863 1358088339 NULL 1 {} 221.5.86.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=e368ba2a6ec14a02bb32cb5cef284027&mac=60FACD22FC83&detaildevicetype=iPhone4,1&versionid=1.1.4&platformid=null&roleid=297241973452978776&deviceversion=5.0.1&imei=null&uuid2=eaacae952f7b421abf062978cb8898d1&devicetype=iPhone&sdkver=1.0.9&gameid=com.renren.games.csj4&jailbroken=true&appid=com.renren.games.csj4&f=8560011158&domain=x32.csj.renren.com&userid=505529877&connecttype=wifi&starttime=1358088339&launchtime=1358088339&endtime=1358089202&onlinetime=863&startcount=1&activetime=863 HTTP/1.1" 200 53 "-"
112.96.30.111 - null "GET /end_ane air.nceo.as.android8560021732 air.nceo.as.android8560021732 f440d14885466879b11dc79e1b7b2346 GT-I9100 2.3.3 1.1.4 8560021732 GT-I9100 android false f440d14885466879b11dc79e1b7b2346 x30.csj.renren.com 3G 296679023499553278 null 1.0.9 503021509 1358089185 1358089253 68 1358088880 3 NULL 1 {} 112.96.30.111 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?activetime=230&appid=air.nceo.as.android8560021732&detaildevicetype=GT-I9100&endtime=1358089253&startcount=3&sdkver=1.0.9&gameid=air.nceo.as.android8560021732&jailbroken=false&connecttype=3G&apptype=android&f=8560021732&domain=x30.csj.renren.com&starttime=1358089185&uuid2=f440d14885466879b11dc79e1b7b2346&versionid=1.1.4&launchtime=1358088880&imei=359090046272710&platformid=null&uuid1=f440d14885466879b11dc79e1b7b2346&mac=null&devicetype=GT-I9100&userid=503021509&onlinetime=68&deviceversion=2.3.3&roleid=296679023499553278 HTTP/1.0" 200 43 "app:/com.renren.gameskit"
58.243.250.37 - null "GET /end_ane com.renren.game.csj csj 60124ca558884db38eb5e50c024f72bc iPhone 6.0.1 1.1.2 null iPhone4,1 ios false 5ef2936b2d7fe62766648bd07394e59b x10.csj.renren.com null 291049523965365363 5C95AE8A5860 1.0.8 487981608 1358086504 1358089197 2693 1358086504 1 NULL 1 {} 58.243.250.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=60124ca558884db38eb5e50c024f72bc&mac=5C95AE8A5860&detaildevicetype=iPhone4,1&apptype=ios&versionid=1.1.2&deviceversion=6.0.1&imei=null&uuid2=5ef2936b2d7fe62766648bd07394e59b&devicetype=iPhone&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x10.csj.renren.com&userid=487981608&roleid=291049523965365363&starttime=1358086504&endtime=1358089197&onlinetime=2693&launchtime=1358086504&startcount=1&activetime=2693 HTTP/1.1" 200 46 "-"
58.243.250.37 - null "GET /end_ane com.renren.game.csj csj 60124ca558884db38eb5e50c024f72bc iPhone 6.0.1 1.1.2 null iPhone4,1 ios false 5ef2936b2d7fe62766648bd07394e59b x10.csj.renren.com null 291049523965365363 5C95AE8A5860 1.0.8 487981608 1358086504 1358089197 2693 1358086504 1 NULL 1 {} 58.243.250.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=60124ca558884db38eb5e50c024f72bc&mac=5C95AE8A5860&detaildevicetype=iPhone4,1&apptype=ios&versionid=1.1.2&deviceversion=6.0.1&imei=null&uuid2=5ef2936b2d7fe62766648bd07394e59b&devicetype=iPhone&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x10.csj.renren.com&userid=487981608&roleid=291049523965365363&starttime=1358086504&endtime=1358089197&onlinetime=2693&launchtime=1358086504&startcount=1&activetime=2693 HTTP/1.1" 200 46 "-"
112.224.67.29 - null "GET /end_ane com.renren.game.csjhd com.renren.game.csjhd d23c7659b1d944e190a89c22ea1b4997 iPhone 4.3.5 1.1.3 856001130 iPhone3,1 nullfalse 4305f44c04c402abcf58b36e005fcf68 x23.csj.renren.com wwan 294708698662585180 148FC67E6E0A 1.0.9 496599565 1358089157 1358089199 42 1358066165 NULL 1 {} 112.224.67.29 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=d23c7659b1d944e190a89c22ea1b4997&mac=148FC67E6E0A&detaildevicetype=iPhone3,1&versionid=1.1.3&platformid=null&roleid=294708698662585180&deviceversion=4.3.5&imei=null&uuid2=4305f44c04c402abcf58b36e005fcf68&devicetype=iPhone&sdkver=1.0.9&gameid=com.renren.game.csjhd&jailbroken=false&appid=com.renren.game.csjhd&f=856001130&domain=x23.csj.renren.com&userid=496599565&connecttype=wwan&starttime=1358089157&launchtime=1358066165&endtime=1358089199&onlinetime=42&startcount=2&activetime=721 HTTP/1.1" 200 53 "-"
58.22.114.175 - null "GET /end_ane 494778270 cq dc06015fd37842eea06b2143eacd1c2a iPhone 5.1.1 1.5.3 855000877 iPhone4,1 null false 1cd5a157ec972135bba67b722b50ba99 http://s51.cq.renren.com/ wwan 043304000000453c 0C771A63EFF8 1.0.9 499040717 1358083127 1358084580 1453 1358083127 1 NULL{} 58.22.114.175 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=dc06015fd37842eea06b2143eacd1c2a&mac=0C771A63EFF8&detaildevicetype=iPhone4,1&versionid=1.5.3&platformid=null&roleid=043304000000453c&deviceversion=5.1.1&imei=null&uuid2=1cd5a157ec972135bba67b722b50ba99&devicetype=iPhone&sdkver=1.0.9&gameid=cq&jailbroken=false&appid=494778270&f=855000877&domain=http://s51.cq.renren.com/&userid=499040717&connecttype=wwan&starttime=1358083127&launchtime=1358083127&endtime=1358084580&onlinetime=1453&startcount=1&activetime=1453 HTTP/1.1" 200 53 "-"
211.140.5.110 - null "GET /run_ane 511848500 cq null GT-I9018 2.2.2 1.5.3 855002850 GT-I9018 android false 537dfb91f38b2964d6fe60ad4c0c8558 http://s59.cq.renren.com/ 2G null 1234569D4E60 1.1.0 null 1358089277 NULL NULL 1358089277 NULL NULL 1 {} 211.140.5.110 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?uuid1=537dfb91f38b2964d6fe60ad4c0c8558&uuid2=537dfb91f38b2964d6fe60ad4c0c8558&versionid=1.5.3&imei=867715001384978&gameid=cq&mac=1234569D4E60&detaildevicetype=GT-I9018&f=855002850&userid=null&connecttype=2G&platformid=null&roleid=null&starttime=1358089277&jailbroken=false&devicetype=GT-I9018&sdkver=1.1.0&deviceversion=2.2.2&appid=511848500&domain=http://s59.cq.renren.com/&launchtime=1358089277&apptype=android HTTP/1.1" 200 53 "app:/com.renren.gameskit"
182.249.240.38 - null "GET /run_ane 563138738 cq aa78cb3cfb8347bfddebe4f4b686defc URBANO%20PROGRESSO 4.0.3 1.2.1 821000401 URBANO%20PROGRESSO android false null http://x2.jcq.rrgame.jp/ 2G null 245fdf41e931 1.0.7 null NULL NULL NULL NULL NULL NULL 1 {} 182.249.240.38 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?uuid2=aa78cb3cfb8347bfddebe4f4b686defc&devicetype=URBANO%20PROGRESSO&deviceversion=4.0.3&versionid=1.2.1&f=821000401&appid=563138738&detaildevicetype=URBANO%20PROGRESSO&apptype=android&gameid=cq&uuid1=aa78cb3cfb8347bfddebe4f4b686defc&sdkver=1.0.7&domain=http://x2.jcq.rrgame.jp/&jailbroken=false&imei=35228005085670&connecttype=2G&mac=245fdf41e931 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
Time taken: 8.832 seconds
hive>
分析共有几种数据格式：
more access.log | awk -F "[\"]" '{print $2}' | awk -F "[?]" '{print $1}'| sort | uniq | sort
共有以下种请求：
****************************************************
stat
api/databus/stat/click
GET /click.html
api/databus/stat/display
GET /display.html
api/databus/stat/game
GET /game.html
ane
api/databus/ane/end
GET /end_ane.html √
api/databus/ane/run
GET /run_ane.html √
api/databus/ane/login
GET /login_ane.html √
api/databus/ane/loader
GET /loader_run.html
***************************************************
GET /click.html
举例：
124.88.232.193 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*925008604*1*nlink*https://itunes.apple.com/cn/app/xue-di-zi-sheng-dan-ban/id588116262?ls=1&mt=8
HTTP/1.1" 301 0 "-"
1.205.1.200 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*838001018*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0 "-"
125.85.93.141 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*838000737*1*nlink*https://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0 "-"
——————————————————————————————————————————————————————————————————————————————
GET /click.html HTTP/1.0
62.24.222.131 - - [13/Jan/2013:23:26:18 +0800] "GET /click.html HTTP/1.0" 200 3 "http://tj.pet.mop.com/click.html"
——————————————————————————————————————————————————————————————————————————————
GET /display.html
115.45.201.131 - - [13/Jan/2013:23:00:01 +0800] "GET /display.html?80134|952000101|0|receiver|0|mop HTTP/1.1" 304 0
36.40.191.155 - - [13/Jan/2013:23:00:02 +0800] "GET /display.html?500293|1|0|receiver|0|mop HTTP/1.1" 200 23
116.22.217.54 - - [13/Jan/2013:23:00:05 +0800] "GET /display.html?9|9040010018|0|receiver|0|mop HTTP/1.1" 200 23
——————————————————————————————————————————————————————————————————————————————
GET /end_ane.html
221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&i
mei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885
001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=13580
88689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-"
114.221.147.156 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
roleid=295553123592712996&uuid2=null&starttime=1358089089&apptype=null&domain=&endtime=1358089095&uuid1=null&onlinetime=6&devicetype=null&launch
time=1358080763&deviceversion=null&startcount=12&f=838000707&activetime=436&detaildevicetype=null&gameid=ms&jailbroken=null&versionid=1.0.8&connec
ttype=null&userid=497886004&appid=com.msqy.weiduan HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1"
175.142.180.206 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?
connecttype=wifi&uuid1=0d40e54d523a48428ad10ecc5253e153&mac=A467069329AE&detaildevicetype=iPad2,2&apptype=ios&versionid=1.1.2&deviceversion=4.3.1&ime
i=null&uuid2=ee5bdbc81fb02ee3bc0b362a5281a935&devicetype=iPad&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x21.cs
j.renren.com&userid=null&roleid=null&starttime=1358089192&endtime=1358089200&onlinetime=8&launchtime=1358089192&startcount=1&activetime=8 HTTP/1.1"
200 46 "-"
——————————————————————————————————————————————————————————————————————————————
GET /favicon.ico HTTP/1.1
120.33.159.158 - - [13/Jan/2013:23:00:45 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
14.118.120.122 - - [13/Jan/2013:23:01:25 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
116.230.225.94 - - [13/Jan/2013:23:05:25 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
——————————————————————————————————————————————————————————————————————————————
GET /game.html
222.209.150.169 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?0|932000101|3400|renren|0|http://x20.war.renren.com/ HTTP/1.1" 304 0
222.209.150.169 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?0|932000101|2400|renren|1|http://x20.war.renren.com/ HTTP/1.1" 304 0
182.242.227.214 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?-1|855004201|3400|0|0|0 HTTP/1.1" 200 20
——————————————————————————————————————————————————————————————————————————————
GET / HTTP/1.1
113.108.11.50 - - [13/Jan/2013:23:09:21 +0800] "GET / HTTP/1.1" 200 20
112.90.84.224 - - [13/Jan/2013:23:09:21 +0800] "GET / HTTP/1.1" 200 20
123.190.215.78 - - [13/Jan/2013:23:13:21 +0800] "GET / HTTP/1.1" 200 0 "-"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.imop.com/click.html
112.97.192.31 - - [13/Jan/2013:23:11:07 +0800] "GET /click.html?9*855000865*1*nlink*http://itunes.apple.com/cn/app/lu-bu-yu-diao-chan/id555912419?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*855000865*1*nlink*http://itunes.apple.com/cn/app/lu-bu-yu-diao-chan/id555912419?mt=8"
118.116.81.28 - - [13/Jan/2013:23:11:39 +0800] "GET /click.html?9*838001010*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*838001010*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8"
125.126.32.95 - - [13/Jan/2013:23:12:00 +0800] "GET /click.html?9*838001017*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*838001017*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/click.html
119.97.151.160 - - [13/Jan/2013:23:54:42 +0800] "GET http://tj.pet.mop.com/click.html?60101*150101*1*nlink*http://itunes.apple.com/cn/app/id588116262?mt=8 HTTP/1.1" 301 0
"-"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/end_ane.html
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=4&endtime=1357999139&deviceversion=6.0&version
id=8.1.1&starttime=1357999135&launchtime=1357998799&f=838000707&detaildevicetype=iPhone3,1&userid=467415015&activetime=220&apptype=ios&jailbroken=f
alse&startcount=5&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=295834598569411941 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
216.36.142.199 - - [13/Jan/2013:23:07:07 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=157&endtime=1358089586&deviceversion=6.0&versi
onid=8.1.1&starttime=1358089429&launchtime=1358089429&f=838000707&detaildevicetype=iPhone3,1&userid=null&activetime=157&apptype=ios&jailbroken=fals
e&startcount=1&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=null HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:11 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=51&endtime=1358089676&deviceversion=6.0&versio
nid=8.1.1&starttime=1358089625&launchtime=1358089429&f=838000707&detaildevicetype=iPhone3,1&userid=454219948&activetime=208&apptype=ios&jailbroken=
false&startcount=2&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886914 HTTP/1.1" 200 43 "app:/com.renren.gameskit
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/login_ane.html
216.36.142.199 - - [13/Jan/2013:23:07:22 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:25 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
119.97.151.160 - - [13/Jan/2013:23:08:55 +0800] "GET http://tj.pet.mop.com/login_ane.html?
roleid=42a040000008200&f=926004501&userid=71461626@qq.com&jailbroken=false&appid=522657364&devicetype=iPhone&uuid2=3c648eb51633153a9a3cba3c0476004
e&deviceversion=6.0.1&detaildevicetype=iPhone4,1&versionid=4.2.0&uuid1=4f94d34e68724d8488782b7632a3935f&domain=x42%20-%20%E7%AA%81%E8%A2%AD
%E8%8B%8F%E8%81%94-60.29.251.165&gameid=com.imop.game.IronMarshal&connecttype=wwan&apptype=ios HTTP/1.1" 200 43 "app:/com.renren.gameskit"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/run_ane.html
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
216.36.142.199 - - [13/Jan/2013:23:07:08 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:11 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
——————————————————————————————————————————————————————————————————————————————
GET /loader_run.html
180.172.254.182 - - [13/Jan/2013:23:00:01 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021510&sdkver=0.0.16&gameid=null&f=8560021510&apptype=android&versionid=0.0.16&uuid2=354356050087566&detaildevicetype=HTC
+T329t&imei=354356050087566&uuid1=810816ca06a04f2a3b468adbb3616aff&mac=a0f450ff66e7&jailbroken=false&devicetype=HTC
+T329t&domain=&deviceversion=&connecttype=4.0.4 HTTP/1.1" 200 3 "-"
1.192.245.84 - - [13/Jan/2013:23:00:02 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021519&sdkver=0.0.14&gameid=null&f=8560021519&apptype=android&versionid=0.0.14&uuid2=866508010354368&detaildevicetype=ONN
+K7&imei=866508010354368&uuid1=44d6b021cc7d04d1fb78d2bfeb1adfee&mac=5ce2f4f4ab17&jailbroken=false&devicetype=ONN
+K7&domain=&deviceversion=&connecttype=4.0.4 HTTP/1.1" 200 3 "-"
112.93.35.68 - - [13/Jan/2013:23:00:03 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021518&sdkver=0.0.15&gameid=null&f=8560021518&apptype=android&versionid=0.0.15&uuid2=359548049282598&detaildevicetype=GT-
N7000&imei=359548049282598&uuid1=946161396c64523669bc52f7cf39a6cb&mac=50CCF8AABE67&jailbroken=false&devicetype=GT-
N7000&domain=&deviceversion=&connecttype=2.3.6 HTTP/1.1" 200 3 "-"
——————————————————————————————————————————————————————————————————————————————
GET /login_ane.html
101.85.147.146 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=70f784738fe0d2f645e15df6de3ffade&devicetype=iPad&mac=40300426A829&detaildevicetype=iPad2,2&connecttype=wifi&sdkver=1.0.7&uuid1=0e7431baeb7846c
0b1caa73a5647684b&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=447393827&roleid=294567961174220907 HTTP/1.1" 200 46 "-"
218.83.227.131 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
f=925004401&gameid=lstx&domain=http://x66.lstx.renren.com&appid=588116262&versionid=1.5.0.1&devicetype=iPad&deviceversion=6.0&jailbroken=false&connectt
ype=wifi&uuid1=93f3dc7b9e2547149a980e94cbb02cbe&udid2=6137ccd66654bc518207c33e460c8986&mac=4c:b1:99:08:bc:d8 HTTP/1.1" 200 43 "-"
210.242.214.26 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=09b67afe070498e4c2601f1be7d33f30&devicetype=iPad&mac=A4D1D25BF26B&detaildevicetype=iPad2,1&connecttype=wifi&sdkver=1.0.7&uuid1=65fe6e70e77c457
89fe988c1ace8bb38&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=8089268&roleid=290768048988618933 HTTP/1.1" 200 46 "-"
——————————————————————————————————————————————————————————————————————————————
GET /login.html
115.197.239.203 - - [13/Jan/2013:23:00:01 +0800] "GET /login.html?
lhsg*ios*870001315*null*efc073686f92362f2663fcc7cc6e64c97eae0c8d*iPad*5.1.1*com.renren.game.nqj2hd*1.0.1 HTTP/1.1" 200 3 "app:/lhsg.swf"
115.197.239.203 - - [13/Jan/2013:23:00:01 +0800] "GET /login.html?
lhsg*ios*870001315*x10.lhsg.renren.com*efc073686f92362f2663fcc7cc6e64c97eae0c8d*iPad*5.1.1*com.renren.game.nqj2hd*1.0.1 HTTP/1.1" 200 3 "app:/lhsg.swf"
111.110.226.84 - - [13/Jan/2013:23:00:03 +0800] "GET /login.html?
lstx*android*8820017*http://x14.tx.rrgame.jp*SSHGE005483_a1000016c6b272_78:1C:5A:3C:34:74*ANDROID_IS17SH*4.0.4*and10301*1.3.317 HTTP/1.1" 200 3 "-"
——————————————————————————————————————————————————————————————————————————————
GET /robots.txt HTTP/1.0
62.24.222.131 - - [13/Jan/2013:23:26:17 +0800] "GET /robots.txt HTTP/1.0" 404 564 "http://tj.pet.mop.com/robots.txt"
GET /robots.txt HTTP/1.1
180.76.5.91 - - [13/Jan/2013:23:51:58 +0800] "GET /robots.txt HTTP/1.1" 404 162 "-"
180.76.5.49 - - [13/Jan/2013:23:52:01 +0800] "GET /robots.txt HTTP/1.1" 404 345
——————————————————————————————————————————————————————————————————————————————
GET /run_ane.html
112.96.131.124 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
appid=com.imop.game.IronMarshalX&gameid=ms&uuid1=abea9df0f87e49b7984f4ca373421a33&devicetype=iPhone&deviceversion=5.1.1&versionid=3.1.1&f=838000729&de
taildevicetype=iPhone4,1&apptype=ios&jailbroken=true&uuid2=a98488fc51257fc2ac574ab1983d054d&domain=&connecttype=wwan HTTP/1.1" 200 43
"app:/com.renren.gameskit"
114.221.147.156 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
uuid2=null&apptype=null&domain=&uuid1=null&devicetype=null&deviceversion=null&f=838000707&detaildevicetype=null&gameid=ms&jailbroken=null&versio
nid=1.0.8&connecttype=null&appid=com.msqy.weiduan HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1"
221.194.31.155 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?f=925004443&gameid=jzsg&domain={domain}
&appid=564762410&versionid=1.0.8&devicetype=iPhone&deviceversion=4.3.3&jailbroken=false&connecttype=wwan&uuid1=c95a9e14a1e34702999b57fb1e9ddb53&udid
2=041357b8146f516b3ca9a96b9e88cc90&mac=14:5a:05:e7:6c:25 HTTP/1.1" 200 43 "app:/musouClient.swf"
——————————————————————————————————————————————————————————————————————————————
GET /run.html
175.141.66.138 - - [13/Jan/2013:23:00:01 +0800] "GET /run.html?
lstx*android*925010017*http://x82.lstx.renren.com*cff19b4b01dc53f_358401049472325_98:0C:82:F0:BA:B2*ANDROID_GT-I9100G*4.0.3*and10101*1.3.318 HTTP/1.1" 200 3 "-"
210.242.214.5 - - [13/Jan/2013:23:00:03 +0800] "GET /run.html?lstx*android*9250082140**HT243W104686_359188048172357_18:87:96:8b:dc:6d*ANDROID_HTC%20One
%20X*4.1.1*and10101*1.3.299 HTTP/1.1" 200 3 "-"
89.24.14.32 - - [13/Jan/2013:23:00:05 +0800] "GET /run.html?lstx*android*862004601**4df1eea16a789f3f_354833058446879_20:02:AF:B9:F7:F8*ANDROID_GT-
N7100*4.1.1*and02501*1.3.263 HTTP/1.1" 200 3 "-"
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
统计的url仅限：
次数 url
50924 GET /login_ane.html
72678 GET /end_ane.html
78684 GET /run_ane.html
13 GET http://tj.pet.mop.com/login_ane.html
16 GET http://tj.pet.mop.com/end_ane.html
16 GET http://tj.pet.mop.com/run_ane.html
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/run_ane.html √
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/end_ane.html √
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=4&endtime=1357999139&deviceversion=6.0&version
id=8.1.1&starttime=1357999135&launchtime=1357998799&f=838000707&detaildevicetype=iPhone3,1&userid=467415015&activetime=220&apptype=ios&jailbroken=f
alse&startcount=5&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=295834598569411941 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/login_ane.html √
216.36.142.199 - - [13/Jan/2013:23:07:22 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /run_ane.html √
112.96.131.124 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
appid=com.imop.game.IronMarshalX&gameid=ms&uuid1=abea9df0f87e49b7984f4ca373421a33&devicetype=iPhone&deviceversion=5.1.1&versionid=3.1.1&f=838000729&de
taildevicetype=iPhone4,1&apptype=ios&jailbroken=true&uuid2=a98488fc51257fc2ac574ab1983d054d&domain=&connecttype=wwan HTTP/1.1" 200 43
"app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /end_ane.html √
221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&i
mei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885
001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=13580
88689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /login_ane.html √
101.85.147.146 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=70f784738fe0d2f645e15df6de3ffade&devicetype=iPad&mac=40300426A829&detaildevicetype=iPad2,2&connecttype=wifi&sdkver=1.0.7&uuid1=0e7431baeb7846c
0b1caa73a5647684b&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=447393827&roleid=294567961174220907 HTTP/1.1" 200 46 "-"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
hadoop fs -ls /data/raw/adwall/test
hadoop fs -rm /data/raw/adwall/test/adwall-1.0.jar
hadoop fs -put ./adwall-1.0.jar /data/raw/adwall/test
hadoop fs -put /data/var/zghtest/adwallSerde/target/adwall-1.0.jar /data/raw/adwall/test
hive> add jar /data/var/zghtest/adwall-1.0.jar;
Added /data/var/zghtest/adwall-1.0.jar to class path
Added resource: /data/var/zghtest/adwall-1.0.jar
cp /data/var/zghtest/adwallSerde/target/adwall-1.0.jar /data/var/zghtest
删除的话就是delete jar
delete jar /data/var/zghtest/adwall-1.0.jar;
建表语句相关
CREATE EXTERNAL TABLE raw.t_adwall_test
(
ip STRING ,
datee STRING ,
interfaze STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall_test2
(
ip STRING ,
datee STRING ,
interfaze STRING,
ipanddate STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall_test3
(
ip STRING ,
datee STRING ,
interfaze STRING,
ipanddate STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall
(
f_ip STRING ,
f_datee STRING ,
f_interfaze STRING,
f_appid STRING,
f_gameid STRING,
f_uuid1 STRING,
f_devicetype STRING,
f_deviceversion STRING,
f_versionid STRING,
f_f STRING,
f_detaildevicetype STRING,
f_apptype STRING,
f_jailbroken STRING,
f_uuid2 STRING,
f_domain STRING,
f_connecttype STRING,
f_roleid STRING,
f_mac STRING,
f_sdkver STRING,
f_userid STRING,
f_starttime BIGINT,
f_endtime BIGINT,
f_onlinetime BIGINT,
f_launchtime BIGINT,
f_startcount INT,
f_activetime BIGINT,
f_formatright SMALLINT,
f_formaterrorreason STRING,
f_raw STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
LOAD DATA LOCAL INPATH '/data/var/zghtest/access.log' OVERWRITE INTO TABLE raw.t_adwall;
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
hive> select * from t_adwall where f_formatright=0 limit 10;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201206261006_286591, Tracking URL = http://h1-1.io8.org:50030/jobdetails.jsp?jobid=job_201206261006_286591
Kill Command = /data/opt/hadoop/bin/hadoop job -Dmapred.job.tracker=h1-1.io8.org:9001 -kill job_201206261006_286591
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 0
2013-01-16 22:30:26,367 Stage-1 map = 0%, reduce = 0%
2013-01-16 22:30:31,394 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 11.23 sec
2013-01-16 22:30:32,401 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 11.23 sec
2013-01-16 22:30:33,408 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 11.23 sec
MapReduce Total cumulative CPU time: 11 seconds 230 msec
Ended Job = job_201206261006_286591
MapReduce Jobs Launched:
Job 0: Map: 2 Accumulative CPU: 11.23 sec HDFS Read: 33726867 HDFS Write: 2843 SUCESS
Total MapReduce CPU Time Spent: 11 seconds 230 msec
OK
1024null null null com.msqy.weiduan ms null null null 1.0.8 838000707 null null null null null null null null NULL 1358089234
NULL 0 1358089234 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null 520025006 jzsg 39bfe5c7178c4882a06a19202f0aee6b iPad 5.1.1 1.0.6 iPad3,1 ios false ec147b2821c8030b4c6812538ef5d9ed
http://x41.s.jzsg.renren.com wifi null null null NULL 1358089273 1358089273 0 1358089273 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null elite.gen.ms.shzr ms 27329507945c40afbf588a5330a2246b iPhone 6.0.1 8.1.1 838000707 iPhone4,1 ios false
d38a920af438c405b08ee7d150d35bd4 wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1357915585 1357915585 0 1357915585
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wosiphone ms 407c7060d95946fdae79054f307bb1e3 iPad 6.0.1 1.0.7 838000707 iPad2,2 ios false
338955105bad19c4f57641b05eb0a77e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358077428 1358077428 0 1358077428
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wosiphone ms 8c81ef15e8ea4c5fbcf8856153bef980 iPhone 4.3.1 1.0.6 838000707 iPhone3,1 ios true
f4db9de6d3d08e956e3378f03a3ceb70 wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1357871821 1357871821 0 1357871821
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.msqy.weiduan ms null null null 1.0.8 HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1" 838000707 null null null null
null null null null NULL 1358089493 1358089493 0 1358089493 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null elite.gen.ms.shzr ms 01bd91e55d564dde83d665851328f4b7 iPhone 6.0.1 8.1.1 838000707 iPhone4,1 ios false
f427006b61c116e6ea3c53d18e5e2f9e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358089478 1358089478 0 1358089478
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wos1iphone ms 970b9f38cd464251a53bf39c2f3ebddc iPhone 6.0.1 1.1.6 838000707 iPhone4,1 ios false
5133ffcde7cf5c69d7fd141e436c265e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358088195 1358088195 0 1358088195
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null wostest ms HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1" null null null 00.00.04 838000707 null null null null
nullnull null null NULL 1358089515 1358089515 0 1358089515 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null and10101 lstx 2842f80102034bc1b59f674ae2649f27 MB525 2.3.7 HTTP/1.1" 200 43 "app:/com.renren.gameskit" 1.6.075 925010018
MB525 android false 2842f80102034bc1b59f674ae2649f27 lstx.renren.com 2G null 984b4a1561ab 1.0.7 NULL 1358088814 1358088814 0 1358088814
1 0 1 {}
Time taken: 14.202 seconds
hive>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
hive -S -e "use raw;add jar /data/var/zghtest/adwall-1.0.jar; select * from t_adwall where f_formatright=1 limit 10;" >> /data/var/zghtest.cvs
set hive.cli.print.header=true;
set hive.cli.print.row.to.vertical=true;
set hive.cli.print.row.to.vertical.num=1;
开启显示列名的功能。
CREATE EXTERNAL TABLE dw0.t_adwall
(
f_ip STRING ,
f_datee STRING ,
f_interfaze STRING,
f_appid STRING,
f_gameid STRING,
f_uuid1 STRING,
f_devicetype STRING,
f_deviceversion STRING,
f_versionid STRING,
f_f STRING,
f_detaildevicetype STRING,
f_apptype STRING,
f_jailbroken STRING,
f_uuid2 STRING,
f_domain STRING,
f_connecttype STRING,
f_roleid STRING,
f_mac STRING,
f_sdkver STRING,
f_userid STRING,
f_starttime BIGINT,
f_endtime BIGINT,
f_onlinetime BIGINT,
f_launchtime BIGINT,
f_startcount INT,
f_activetime BIGINT,
f_formatright SMALLINT,
f_formaterrorreason STRING,
f_raw STRING
)
建立洗过一次的数据表：
insert overwrite table dw0.t_adwall
select f_ip, f_datee,f_interfaze, f_appid, f_gameid , f_uuid1, f_devicetype , f_deviceversion
, f_versionid,f_f , f_detaildevicetype , f_apptype , f_jailbroken, f_uuid2, f_domain, f_connecttype,
f_roleid , f_mac , f_sdkver , f_userid , f_starttime , f_endtime , f_onlinetime , f_launchtime , f_startcount ,
f_activetime , f_formatright , f_formaterrorreason , f_raw
from raw. t_adwall
新的查询结果：
hive> select * from dw0.t_adwall where f_formatright=1 limit 10;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201206261006_287114, Tracking URL = http://h1-1.io8.org:50030/jobdetails.jsp?jobid=job_201206261006_287114
Kill Command = /data/opt/hadoop/bin/hadoop job -Dmapred.job.tracker=h1-1.io8.org:9001 -kill job_201206261006_287114
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 0
2013-01-17 17:01:10,566 Stage-1 map = 0%, reduce = 0%
2013-01-17 17:01:13,580 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.55 sec
2013-01-17 17:01:14,585 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.55 sec
MapReduce Total cumulative CPU time: 2 seconds 550 msec
Ended Job = job_201206261006_287114
MapReduce Jobs Launched:
Job 0: Map: 2 Accumulative CPU: 2.55 sec HDFS Read: 140255 HDFS Write: 6541 SUCESS
Total MapReduce CPU Time Spent: 2 seconds 550 msec
OK
f_ip f_datee f_interfaze f_appid f_gameid f_uuid1 f_devicetype f_deviceversion f_versionid f_f f_detaildevicetype f_apptype f_jailbroken f_uuid2 f_domain f_connecttype f_roleid f_mac f_sdkver f_userid f_starttime f_endtime f_onlinetime f_launchtime f_startcount f_activetime f_formatright f_formaterrorreason f_raw
221.194.30.138 - null "GET /end_ane com.imop.game.pirate2iphone rrnhzf d51c772dd73e48c29dd76269e021d418 iPhone 6.0.1 1.2.1 885001201 iPhone3,1 ios falsbdd0d548eb1edbcd459dbf7f32cc4751 x39.nhzf.renren.com null 295271648616025261 28E02C12BE8D 1.0.7 221106049 1358088689 1358089200 511 1358088689 1 NULL{} 221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&imei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=1358088689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-"
175.142.180.206 null "GET /end_ane com.renren.game.csj csj 0d40e54d523a48428ad10ecc5253e153 iPad 4.3.1 1.1.2 null iPad2,2 ios false ee5bdbc81fb02ee3bc0b362a5281a935 x21.csj.renren.com null null A467069329AE 1.0.8 null 1358089192 1358089200 8 1358089192 1 NULL 1 {} 175.142.180.206 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wifi&uuid1=0d40e54d523a48428ad10ecc5253e153&mac=A467069329AE&detaildevicetype=iPad2,2&apptype=ios&versionid=1.1.2&deviceversion=4.3.1&imei=null&uuid2=ee5bdbc81fb02ee3bc0b362a5281a935&devicetype=iPad&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x21.csj.renren.com&userid=null&roleid=null&starttime=1358089192&endtime=1358089200&onlinetime=8&launchtime=1358089192&startcount=1&activetime=8 HTTP/1.1" 200 46 "-"
221.5.86.37 - - null "GET /end_ane com.renren.games.csj4 com.renren.games.csj4 e368ba2a6ec14a02bb32cb5cef284027 iPhone 5.0.1 1.1.4 8560011158 iPhone4,1 nulltrue eaacae952f7b421abf062978cb8898d1 x32.csj.renren.com wifi 297241973452978776 60FACD22FC83 1.0.9 505529877 1358088339 1358089202 863 1358088339 NULL 1 {} 221.5.86.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=e368ba2a6ec14a02bb32cb5cef284027&mac=60FACD22FC83&detaildevicetype=iPhone4,1&versionid=1.1.4&platformid=null&roleid=297241973452978776&deviceversion=5.0.1&imei=null&uuid2=eaacae952f7b421abf062978cb8898d1&devicetype=iPhone&sdkver=1.0.9&gameid=com.renren.games.csj4&jailbroken=true&appid=com.renren.games.csj4&f=8560011158&domain=x32.csj.renren.com&userid=505529877&connecttype=wifi&starttime=1358088339&launchtime=1358088339&endtime=1358089202&onlinetime=863&startcount=1&activetime=863 HTTP/1.1" 200 53 "-"
112.96.30.111 - null "GET /end_ane air.nceo.as.android8560021732 air.nceo.as.android8560021732 f440d14885466879b11dc79e1b7b2346 GT-I9100 2.3.3 1.1.4 8560021732 GT-I9100 android false f440d14885466879b11dc79e1b7b2346 x30.csj.renren.com 3G 296679023499553278 null 1.0.9 503021509 1358089185 1358089253 68 1358088880 3 NULL 1 {} 112.96.30.111 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?activetime=230&appid=air.nceo.as.android8560021732&detaildevicetype=GT-I9100&endtime=1358089253&startcount=3&sdkver=1.0.9&gameid=air.nceo.as.android8560021732&jailbroken=false&connecttype=3G&apptype=android&f=8560021732&domain=x30.csj.renren.com&starttime=1358089185&uuid2=f440d14885466879b11dc79e1b7b2346&versionid=1.1.4&launchtime=1358088880&imei=359090046272710&platformid=null&uuid1=f440d14885466879b11dc79e1b7b2346&mac=null&devicetype=GT-I9100&userid=503021509&onlinetime=68&deviceversion=2.3.3&roleid=296679023499553278 HTTP/1.0" 200 43 "app:/com.renren.gameskit"
58.243.250.37 - null "GET /end_ane com.renren.game.csj csj 60124ca558884db38eb5e50c024f72bc iPhone 6.0.1 1.1.2 null iPhone4,1 ios false 5ef2936b2d7fe62766648bd07394e59b x10.csj.renren.com null 291049523965365363 5C95AE8A5860 1.0.8 487981608 1358086504 1358089197 2693 1358086504 1 NULL 1 {} 58.243.250.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=60124ca558884db38eb5e50c024f72bc&mac=5C95AE8A5860&detaildevicetype=iPhone4,1&apptype=ios&versionid=1.1.2&deviceversion=6.0.1&imei=null&uuid2=5ef2936b2d7fe62766648bd07394e59b&devicetype=iPhone&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x10.csj.renren.com&userid=487981608&roleid=291049523965365363&starttime=1358086504&endtime=1358089197&onlinetime=2693&launchtime=1358086504&startcount=1&activetime=2693 HTTP/1.1" 200 46 "-"
58.243.250.37 - null "GET /end_ane com.renren.game.csj csj 60124ca558884db38eb5e50c024f72bc iPhone 6.0.1 1.1.2 null iPhone4,1 ios false 5ef2936b2d7fe62766648bd07394e59b x10.csj.renren.com null 291049523965365363 5C95AE8A5860 1.0.8 487981608 1358086504 1358089197 2693 1358086504 1 NULL 1 {} 58.243.250.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=60124ca558884db38eb5e50c024f72bc&mac=5C95AE8A5860&detaildevicetype=iPhone4,1&apptype=ios&versionid=1.1.2&deviceversion=6.0.1&imei=null&uuid2=5ef2936b2d7fe62766648bd07394e59b&devicetype=iPhone&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x10.csj.renren.com&userid=487981608&roleid=291049523965365363&starttime=1358086504&endtime=1358089197&onlinetime=2693&launchtime=1358086504&startcount=1&activetime=2693 HTTP/1.1" 200 46 "-"
112.224.67.29 - null "GET /end_ane com.renren.game.csjhd com.renren.game.csjhd d23c7659b1d944e190a89c22ea1b4997 iPhone 4.3.5 1.1.3 856001130 iPhone3,1 nullfalse 4305f44c04c402abcf58b36e005fcf68 x23.csj.renren.com wwan 294708698662585180 148FC67E6E0A 1.0.9 496599565 1358089157 1358089199 42 1358066165 NULL 1 {} 112.224.67.29 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=d23c7659b1d944e190a89c22ea1b4997&mac=148FC67E6E0A&detaildevicetype=iPhone3,1&versionid=1.1.3&platformid=null&roleid=294708698662585180&deviceversion=4.3.5&imei=null&uuid2=4305f44c04c402abcf58b36e005fcf68&devicetype=iPhone&sdkver=1.0.9&gameid=com.renren.game.csjhd&jailbroken=false&appid=com.renren.game.csjhd&f=856001130&domain=x23.csj.renren.com&userid=496599565&connecttype=wwan&starttime=1358089157&launchtime=1358066165&endtime=1358089199&onlinetime=42&startcount=2&activetime=721 HTTP/1.1" 200 53 "-"
58.22.114.175 - null "GET /end_ane 494778270 cq dc06015fd37842eea06b2143eacd1c2a iPhone 5.1.1 1.5.3 855000877 iPhone4,1 null false 1cd5a157ec972135bba67b722b50ba99 http://s51.cq.renren.com/ wwan 043304000000453c 0C771A63EFF8 1.0.9 499040717 1358083127 1358084580 1453 1358083127 1 NULL{} 58.22.114.175 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=dc06015fd37842eea06b2143eacd1c2a&mac=0C771A63EFF8&detaildevicetype=iPhone4,1&versionid=1.5.3&platformid=null&roleid=043304000000453c&deviceversion=5.1.1&imei=null&uuid2=1cd5a157ec972135bba67b722b50ba99&devicetype=iPhone&sdkver=1.0.9&gameid=cq&jailbroken=false&appid=494778270&f=855000877&domain=http://s51.cq.renren.com/&userid=499040717&connecttype=wwan&starttime=1358083127&launchtime=1358083127&endtime=1358084580&onlinetime=1453&startcount=1&activetime=1453 HTTP/1.1" 200 53 "-"
211.140.5.110 - null "GET /run_ane 511848500 cq null GT-I9018 2.2.2 1.5.3 855002850 GT-I9018 android false 537dfb91f38b2964d6fe60ad4c0c8558 http://s59.cq.renren.com/ 2G null 1234569D4E60 1.1.0 null 1358089277 NULL NULL 1358089277 NULL NULL 1 {} 211.140.5.110 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?uuid1=537dfb91f38b2964d6fe60ad4c0c8558&uuid2=537dfb91f38b2964d6fe60ad4c0c8558&versionid=1.5.3&imei=867715001384978&gameid=cq&mac=1234569D4E60&detaildevicetype=GT-I9018&f=855002850&userid=null&connecttype=2G&platformid=null&roleid=null&starttime=1358089277&jailbroken=false&devicetype=GT-I9018&sdkver=1.1.0&deviceversion=2.2.2&appid=511848500&domain=http://s59.cq.renren.com/&launchtime=1358089277&apptype=android HTTP/1.1" 200 53 "app:/com.renren.gameskit"
182.249.240.38 - null "GET /run_ane 563138738 cq aa78cb3cfb8347bfddebe4f4b686defc URBANO%20PROGRESSO 4.0.3 1.2.1 821000401 URBANO%20PROGRESSO android false null http://x2.jcq.rrgame.jp/ 2G null 245fdf41e931 1.0.7 null NULL NULL NULL NULL NULL NULL 1 {} 182.249.240.38 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?uuid2=aa78cb3cfb8347bfddebe4f4b686defc&devicetype=URBANO%20PROGRESSO&deviceversion=4.0.3&versionid=1.2.1&f=821000401&appid=563138738&detaildevicetype=URBANO%20PROGRESSO&apptype=android&gameid=cq&uuid1=aa78cb3cfb8347bfddebe4f4b686defc&sdkver=1.0.7&domain=http://x2.jcq.rrgame.jp/&jailbroken=false&imei=35228005085670&connecttype=2G&mac=245fdf41e931 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
Time taken: 8.832 seconds
hive>
分享到：
发表评论
相关推荐
Flume采集Nginx日志到Hive的事务表时需要导入到Flume下的Jar文件，具体使用方式可参见博文：https://blog.csdn.net/l1028386804/article/details/97975539
这是我的一份课程作业，需要事先下载搜狗日志文件。有问题，可以问百度。里面也参考了其他博客主的文章，在最后有相关链接。
Flume采集Nginx日志到新版Hive，Flume中需要添加的Jar包，各软件版本为：Hadoop 3.2.0、Flume 1.9.0、Hive 3.1.2、Nginx 1.17.2。
Hive 分析搜狗搜索日志与结果可视化，利用 hive 命令行完成搜狗日志各项数据分析，使用 Python 进行数据可视化。主要步骤 包括：安装部署 Hive、启动 Hadoop 集群、进入 Hive 命令行、创建数据库和数据表、加载 或...
1、典型的离线流数据分析系统 2、技术分析 - Hadoop - nginx - flume - hive - mysql - springboot + mybatisplus+vcharts nginx + lua 日志文件埋点的 基于Hadoop网站流量日志数据分析系统 1、典型的离线流数据...
针对传统分布式模型在海量日志并行处理时的可扩展性和并行程序编写困难的问题, 提出了基于Hive的Web海量搜索日志分析机制。利用HQL语言以及Hadoop分布式文件系统(HDFS)和MapReduce编程模式对海量搜索日志进行分析...
本资源适合hive使用log4j2作为日志去开启metastore审计日志，那么本文章就值得(不管是hive2还是hive3)
基于Hadoop／Hive的web日志分析系统的设计，为广大大数据爱好者提供更好的帮助，喜欢可以下载
09_Hive中数据文件的存储格式介绍及对比 10_常见的压缩格式及MapReduce的压缩介绍 11_Hadoop中编译配置Snappy压缩 12_Hadoop及Hive配置支持snappy压缩 13_Hive中的常见调优 14_Hive中的数据倾斜及解决方案-三种join...
Hive内部表合并小文件Java程序
至此，我们通过Python网络爬虫手段进行数据抓取，将我们网站数据（2013-05-30,2013-05-31）保存为两个日志文件，由于文件大小超出我们一般的分析工具处理的范围，故借助Hadoop来完成本次的实践。 使用python对原始...
该实验数据主要用于Hive进行Apache Web日志的统计分析学习使用，数据量不是大。
hive 分析 hive 分析 血缘关系 血缘关系 血缘关系 ，太难了，太难了
windows平台下，hive可执行文件，完全没有问题，可以使用。 将这个bin目录覆盖hive原有的bin目录即可
hive日志系统实战
hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+...
基于Hadoop网站流量日志数据分析系统项目源码+教程.zip网站流量日志数据分析系统 典型的离线流数据分析系统 技术分析 hadoop nginx flume hive sqoop mysql springboot+mybatisplus+vcharts 基于Hadoop网站流量日志...
一种基于Hive日志分析的大数据存储优化方法，王正也，李书芳，从2002年起，某金融机构积累了大量的银行卡支付日志数据。随着业务的不断增长，数据集高速增长，原生的Hive的查询性能已经不能令人�
基于Hadoop Hive健身馆可视化分析平台项目源码+数据库文件.zip启动方式 环境启动 hadoop hive2元数据库 sql导入 导入hivesql脚本,修改application.yml 启动主程序 HadoopApplication 基于Hadoop Hive健身馆可视化...
Hive的配置，选定一个Hive的master，其他的...这些节点在访问元数据库时，先会内部跳转到Hive服务器的节点，该包命名成了hive-site_master.xml和hive-site_client.xml以区分，使用的时候请注意改回名字hive-site.xml。