繞過 Web 上「防機器人」機制的資料

這兩天的 Hacker News 冒出一些討論在講 Web 上「防機器人」機制要怎麼繞過:

第一篇主要是從各種面向都一起討論,從大方向的分類討論 (「Where to begin building undetectable bot?」),另外介紹目前有哪些產品 (在「List of anti-bot software providers」這邊)。

在文章裡有提到一個有意思的工具「puppeteer-extra-plugin-stealth」,主要是在 Node.js 類的環境,查了一下在 Python 上也有 pyppeteer-stealth,不過 Python 版本直接講了不完美 XDDD

Transplanted from puppeteer-extra-plugin-stealth, Not perfect.

第二篇文章在開頭就提到他不是很愛 Proxy,因為 Proxy 很容易偵測。在文章最後面則是提到了兩個方案,第一個是用大量便宜的 Android 手機加上 Data SIM 來跑,另外一個是直接用 Android 模擬器加上 4G 網卡跑。

依照這些想法,好像可以來改善一下手上的 RSS 工具...

AWS 推出了 Anomaly Detection

AWS 這次在 CloudWatch 上推出了新功能,可以直接透過機器學習的演算法,對 CloudWatch 所記錄的值提供異常偵測 (anomaly detection) 的能力:「New – Amazon CloudWatch Anomaly Detection」,對應的文件則可以在「Using CloudWatch Anomaly Detection」這邊讀到。



CloudFormation 可以偵測手動的變更了...

CloudFormation 宣佈支援 Drift Detection,也就是偵測非 CloudFormation 所產生的改變:「New – CloudFormation Drift Detection」。

以往要修正這個問題得花不少功夫 workaround,現在可以透過 Drift Detection 解了。不過目前並不是所有資源都有提供 Drift Detection:

We are launching with support for a core set of services, resources, and properties, with plans to add more over time. The initial list of resources spans API Gateway, Auto Scaling, CloudTrail, CloudWatch Events, CloudWatch Logs, DynamoDB, Amazon EC2, Elastic Load Balancing, IAM, AWS IoT, Lambda, Amazon RDS, Route 53, Amazon S3, Amazon SNS, Amazon SQS, and more.

這是在 console 上偵測到改變的操作:

話說隔壁棚的 Terraform 什麼時候要支援啊... 喔不對,要賣 Enterprise 服務所以應該不太想支援...

Netflix 用 CloudTrail 記錄找出 AWS key 外洩的小工具

在「aws-credential-compromise-detection – Detecting Credential Compromise in AWS」這邊看到可以抓漏的專案 Netflix-Skunkworks/aws-credential-compromise-detection

透過分析 CloudTrail 記錄找出有哪些可疑的 AWS key 被外部使用,看起來預設值會過濾掉 Private IP range 以及 (設給給 CGNAT 使用的網段)。

不過 Netflix-Skunkworks 的定位是什麼啊,裡面好像有不少有趣的東西...

拿來分析 Android APK 檔的 Droidefence

忘記在哪邊看到的,分析 Android APK 檔的軟體:「Droidefense: Advance Android Malware Analysis Framework」。

Droidefense (originally named atom: analysis through observation machine)* is the codename for android apps/malware analysis/reversing tool. It was built focused on security issues and tricks that malware researcher have on they every day work. For those situations on where the malware has anti-analysis routines, Droidefense attemps to bypass them in order to get to the code and 'bad boy' routine. Sometimes those techniques can be virtual machine detection, emulator detection, self certificate checking, pipes detection. tracer pid check, and so on.

Droidefense uses an innovative idea in where the code is not decompiled rather than viewed. This allow us to get the global view of the execution workflow of the code with a 100% accuracy on gathered information. With this situation, Droidefense generates a fancy html report with the results for an easy understanding.

看起來是輔助用的工具... 先記錄下來 XD

AWS 推出 Amazon GuardDuty 進行內部網路監控

AWS 推出 Amazon GuardDuty 監控內部網路:「Amazon GuardDuty – Continuous Security Monitoring & Threat Detection」。

從示意圖可以看到結合了許多 log 資料,然後綜合判斷:

In combination with information gleaned from your VPC Flow Logs, AWS CloudTrail Event Logs, and DNS logs, this allows GuardDuty to detect many different types of dangerous and mischievous behavior including probes for known vulnerabilities, port scans and probes, and access from unusual locations.

所以連 Bitcoin 相關網站也當作條件之一 XD

開了相當多區 (相較於之前 AWS Elemental MediaOOXX 系列...):

Amazon GuardDuty is available in production form in the US East (Northern Virginia), US East (Ohio), US West (Oregon), US West (Northern California), EU (Ireland), EU (Frankfurt), EU (London), South America (São Paulo), Canada (Central), Asia Pacific (Tokyo), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), and Asia Pacific (Mumbai) Regions and you can start using it today!

Windows 將引入 TruePlay,推出作弊偵測的 API

在「Windows now includes gaming cheat detection at the system level」這邊看到微軟將會引入 TruePlay (然後跟 Sonostrueplay 衝名 XDDD) 作弊偵測機制。


As Microsoft notes, "to protect customer privacy, no data is shared or transmitted until permission is granted," and no information is sent until "processing has determined cheating is likely to have occurred."

這不是把人當傻子嗎,遊戲一開始就會要求你同意才能玩啊,所以資料一定會送出的啊... 而且 TruePlay 變成作業系統的標準配備後,作弊程式就會找 workaround 才會推出 :o

偵測 Chrome Headless

作者因為種種原因,想要偵測 Headless 模式的 Google Chrome:「Detecting Chrome Headless」。

之前因為主要是 PhantomJS,有很多地方跟一般的瀏覽器不同,可以利用這些不同的地方來判斷出是不是 PhantomJS:

Until now, one of the most popular headless browser was PhantomJS. Since it is built on the Qt framework, it exhibits many differences compared to most popular browsers. As presented in this post, it is possible to detect it using some browser fingerprinting techniques.

但從 Google Chrome 59 以後因為支援 Headless,使得大多數的判斷的失效:

Since version 59, Google released a headless version of its Chrome browser. Unlike PhantomJS, it is based on a vanilla Chrome, and not on an external framework, making its presence more difficult to detect.

所以作者找了不少方式想要判斷兩者的相異之處... 不過這些方式看起來不太穩定,加上 Firefox 也在準備了,之後只會愈來愈困難吧 :o


作者用 OpenCV 學習老闆的臉,然後當老闆走過來的時候把畫面切到努力工作中的 screenshot XDDD:「Deep Learning Enables You to Hide Screen when Your Boss is Approaching」。

“My boss left his seat and he was approaching to my seat.”

“OpenCV has detected the face and input the image into the learned model.”

“The screen has switched by recognizing him! ヽ(‘ ∇‘ )ノ ワーイ”

作者是個日本人 (要說不意外嗎 XDDD),這套軟體的程式碼在「Hironsan/BossSensor」這邊 XDDD

超級浪費才能 XDDD

Amazon Rekognition:圖片辨識 API

GoogleVision API,到 MicrosoftComputer Vision API (參考「微軟也推出圖片辨識的 API 了」),AWS 也推出類似的服務了:「Amazon Rekognition – Image Detection and Recognition Powered by Deep Learning」。

與其他兩家都是類似的方式,丟圖進去然後用系統已經 train 好的資料給你分析結果... 然後依照次數算錢。