AWS 對 Elastic Stack 實作免費的開源版本 Open Distro for Elasticsearch

Elasticsearch 的主體是 Apache License 2.0,但 Elastic Stack (以前叫做 X-Pack) 則是需要付費使用的功能,其中包括了不少跟安全有關的項目在裡面,所以其實有不少人抱怨過產品凌駕安全性的問題,像是「ES 6.3: X-Pack Licence is "Expired" on New Install」這篇官方回應的:

A basic license is not entitled to security features. To try out security you need to use a trial license or obtain a subscription.

AWS 這次則是出手實作了他們自己的版本,叫做 Open Distro for Elasticsearch:「New – Open Distro for Elasticsearch」。

如果你看文章說明,他列出來的 feature 全部都是在 Elastic Stack 這頁上列出來的項目,針對性的意思其實很清楚了:

In addition to Elasticsearch and Kibana, the first release includes a set of advanced security, event monitoring & alerting, performance analysis, and SQL query features (more on those in a bit).


Security – This plugin that supports node-to-node encryption, five types of authentication (basic, Active Directory, LDAP, Kerberos, and SAML), role-based access controls at multiple levels (clusters, indices, documents, and fields), audit logging, and cross-cluster search so that any node in a cluster can run search requests across other nodes in the cluster.

目前支援 Docker Image 與 RPM,之後看看有沒有機會出 deb 版本:

In addition to the source code repo, Open Distro for Elasticsearch and Kibana are available as RPM and Docker containers, with separate downloads for the SQL JDBC and the PerfTop CLI.

這樣應該會讓 Elasticsearch 的服務模式受到很大的影響,來看 Elastic N.V. Ordinary Shares Real Time Stock Quotes 這邊會掉多少...

Amazon Elasticsearch 支援 I3 instance (i.e. 1.5 PB Disk) 了

Amazon Elasticsearch 支援 I3 instance 了:「Run Petabyte-Scale Clusters on Amazon Elasticsearch Service Using I3 instances」。

Amazon Elasticsearch Service now supports I3 instances, allowing you to store up to 1.5 petabytes of data in a single Elasticsearch cluster for large log analytics workloads.

i3.16xlarge 單台是 15.2 TB 的硬碟空間,100 台就會是 1.5 PB,不知道跑起來會多慢 XDDD

Amazon Elasticsearch Service – Amazon Web Services (AWS) | FAQs 這邊還沒修正 XD:

You can request a service limit increase up to 100 instances per domain by creating a case with the AWS Support Center. With 100 instances, you can allocate about 150 TB of EBS storage to a single domain.

用 Amazon Elasticsearch 看 VPC Flow Logs

在「How to Visualize and Refine Your Network’s Security by Adding Security Group IDs to Your VPC Flow Logs」這篇雖然是講特定功能,但還是把怎麼架設從頭到尾都講了一次...


然後再回來看怎麼串: 的 Elasticsearch

在「State of Elasticsearch Systems 2016」這邊描述他們 Elasticsearch 的架構。

有五個 cluster 打散,有跑 1.3.x 也有 1.7.x。把一般使用者與 VIP 分開,而全站的資料又是一組。另外在 2.3.x 的測試機上跑 的資料 (看起來是短時間炸掉沒關係?XD)。

由於是自己生機器出來,所以機器的選擇上用大量的記憶體與 SSD 硬碟來換各種效能:

Typical data server config:
* 96GB RAM with 31GB for ES heap. Remaining gets used for file system caching
* 1-3 TB of SSD per server. In our testing SSDs are very worthwhile.

另外上面還是有疊 cache:

memcache timeouts vary from 30 seconds to 36 hours depending on use case

Amazon API Gateway 在東京也可以用了...

算是 AWS re:Invent 2015 上比較小的消息。Amazon API Gateway 在東京啟用:「Amazon API Gateway now available in the Asia Pacific (Tokyo) AWS Region」。

由於 API Gateway 可以接 Lambda,而這次 re:Invent 又發表了許多對 Lambda 的新功能 (參考先前的「AWS Lambda 大躍進」),這使得 API Gateway 的用途多出不少...

由於可以存取 VPC 內部資源,這表示 Lambda 可以去 RDS 或是 memcached 上抓資料,或是存取內部的 Elastic (Elasticsearch)...

多開了東京的點代表很多現有的服務也可以改接 Lambda + API Gateway...

AWS 推出 Amazon Elasticsearch Service

AWS 推出了 Amazon Elasticsearch Service,也就是把 Elasticsearch (現在叫做 Elastic) 包裝起來的服務:「New – Amazon Elasticsearch Service」。

並不是所有 EC2 的 instance 種類都支援 (像是 m4.* 系列就不支援),不過也算夠多了,然後安裝時也包括了 Kibana

另外一個比較重要的整合是可以把 CloudWatch 的資料倒進去,於是舊可以在 Kibana 裡面看這些數據了:

旁邊的 Amazon CloudSearch 哭哭了...

把 AWS 的紀錄串起來倒進 Elasticsearch,用 Kibana 呈現

在「CloudWatch Logs Subscription Consumer + Elasticsearch + Kibana Dashboards」這篇文章裡,AWS 官方給了很棒的 screenshot,用一堆服務包起來後可以得到這樣的效果,提供給有興趣的人:

MySQL 5.7 的 InnoDB 的全文搜尋

在「InnoDB Full-Text : N-gram Parser」這邊看到對 MySQL 5.7 InnoDB 的全文搜尋功能介紹。開頭就有很重要的說明:

I’m now very happy to say that in MySQL 5.7.6 we’ve made use of the new pluggable full-text parser support in order to provide you with an n-gram parser that can be used with CJK!

這對資料量在中等或是更少的公司相當方便,你可以架 replication server 專門跑 search,而不需要利用 reliable queue 確保更新後推進 SolrElastic (改名了,之前叫 ElasticSearch)。

不過,如果資料量很大的話應該還是得用 Solr 或 Elastic 的方案...

維基基金會的 2014 年八月月報

維基基金會釋出八月月報 (好像晚了三個月?):「Wikimedia Foundation Report, August 2014」,在「Wikimedia Highlights, August 2014」有比較精簡的版本。

維基基金會在報告裡有提供一些 PV 相關的數據,包括 comScore 的數字與自己 server log 所統計出來的數據。另外也包含了財務狀況。

其中技術相關的是取自「Wikimedia Engineering/Report/2014/August」這頁。另外因為這是八月的資料,我順便偷看了九月與十月的「Wikimedia Engineering/Report/2014/September」與「Wikimedia Engineering/Report/2014/October」。

可以看到在測試 HHVM 的計畫,而且目前看起來還不錯:「[Wikitech-l] [Engineering] Migrating to HHVM」,拿了 測試,其中 speed test 的部份有大幅改善:

1) Speed test: measure the time taken to request the page 1000 times over just 10 concurrent connections:

                        HHVM    Zend    diff
Mean time (ms):         233     441     -47%
99th percentile (ms):   370     869     -57%
Request/s:              43      22.6    +90%


2) Load test: measure how much thoughput we obtain when hogging the appserver with 50 concurrent requests for a grand total of 10000 requests. What I wanted to test in this case was the performance degradation and the systems resource consumption

                        HHVM    Zend    diff
Mean time (ms):         355     906     -61%
99th percentile (ms):   791     1453    -45%
Request/s:              141     55.1    +156%
Network (Mbytes/s)      17      7       +142%
RAM used (GBs):         5(1)    11(4)
CPU usage (%):          90(75)  100(90)

維基百科之所以沒有遇到太多問題,主要是因為所使用的軟體是 open source 而且夠大的關係,直接成為 HHVM 測試的一環:「Compatibility Update」。

不過目前看起來應該還是跑 PHP,沒有看到整個都轉換過去的計畫。

另外一方面,搜尋引擎的更換就沒有這麼順利,雖然換到 Elasticsearch 後改善不少,不過可以看到八月的報告這樣寫:

tarted deploying Cirrus as the primary search back-end to more of the remaining wikis and we found what looks like our biggest open performance bottleneck. Next month's goal is to fix it and deploy to more wikis (probably not all). We're also working on getting more hardware.


In September we worked to mitigate the performance bottleneck that we found in August. We found there to be no silver bullet but used the information we learned to pick and order appropriate hardware to handle the remaining wikis. We also implemented out significantly improved wikitext Regular Expression search. In October we've begun rolling out the wikitext Regular Expression search and received some of the hardware we need to finish cutting over the remaining wikis. We believe we'll get it all installed in October and cut the remaining wikis over in November.


In October we prepared for November in which we deployed Cirrus to all the remaining wikis by installing new servers installing new versions of Elasticsearch and our plugins. We also fixed up regex search which had caused a search outage.

這些報告的連結裡面其實有些不會在對外新聞稿上面的評語... XD

Elasticsearch 1.2.0

由於 Elasticsearch 的想法與實做比起 Solr 吸引人,可以看到愈來愈多團體換過去...

而前幾天 Elasticsearch 的官方放出 1.2.0 與 1.1.2 的消息:「elasticsearch 1.2.0 and 1.1.2 released」。

1.2.0 最大的改變是強制使用 Java 7 了,也就是不能在 Ubuntu 12.04 下安裝 default-jre 了,變成要裝 openjdk-7-jre。(要注意,官方建議的是 Oracle 官方的 JDK,而非 OpenJDK)

如果是 Ubuntu 14.04 就沒這個問題。(因為 default-jre 會裝 Java 7)

另外一個大改變是,之前產生安全問題的 dynamic scripting 預設關掉了,也就是 CVE-2014-3120

目前我的進度只到看完 mapping,但還沒實際開始塞資料進去玩...