Home » Posts tagged "access"

Amazon EFS 也要推出 IA 版本了 (Infrequent Access)

Amazon EFS 也要推出 IA (Infrequent Access) 版本了,Infrequent Access 指的是不常存取的資料:「Coming Soon – Amazon EFS Infrequent Access Storage Class」。

這剛好配合上很多人拿 Amazon EFS 來堆 log 的行為... AWS 是說有機會到省到 85%,不過應該是非常大的量才有機會有這個價錢?

EFS IA reduces storage costs for files not accessed every day, with savings up to 85% compared to the EFS Standard storage class.

其實用過 Amazon EFS 的人都對效能抱怨頗嚴重 (透過 NFS 有太多操作沒辦法 cache,於是 network latency issue 就出現了),堆 log 或是當作跨機器的空間大概是目前的主流用法...

Netflix 用 CloudTrail 記錄找出 AWS key 外洩的小工具

在「aws-credential-compromise-detection – Detecting Credential Compromise in AWS」這邊看到可以抓漏的專案 Netflix-Skunkworks/aws-credential-compromise-detection

透過分析 CloudTrail 記錄找出有哪些可疑的 AWS key 被外部使用,看起來預設值會過濾掉 Private IP range 以及 100.64.0.0/10 (設給給 CGNAT 使用的網段)。

不過 Netflix-Skunkworks 的定位是什麼啊,裡面好像有不少有趣的東西...

AWS 推出可以直接操作機器的 Shell Access

AWS 推出了 Shell Access,讓你可以直接操作 EC2 上的機器,而且又符合稽核上的要求:「New – AWS Systems Manager Session Manager for Shell Access to EC2 Instances」。

看起來機器上只要有 ssm-agent 就可以做這件事情了。在官方的圖可以看到用法:

另外 AWS 也在開發新的功能,一個是繞過 security group 的 SSH client:

SSH Client – You will be able to create SSH sessions atop Session Manager without opening up any inbound ports.

另外一個是給外部機器用的版本:

On-Premises Access – We plan to give you the ability to access your on-premises instances (which must be running the SSM Agent) via Session Manager.

這功能最主要還是稽核面的需求... 以前要作到這件事情會需要透過跳板之類的東西記錄使用者行為,現在 AWS 做出來後就省下這件事情。

歐洲研究機構的資助者推動研究論文的開放存取

在「Radical open-access plan could spell end to journal subscriptions」這邊看到歐洲 11 個研究機構資助者成立了「cOAlition S」,推動研究論文的開放存取。

目標是在 2020 年開始,由這些機構所資助的研究都必須投在符合完全開放條件的平台上:

cOAlition S signals the commitment to implement, by 1 January 2020, the necessary measures to fulfil its main principle: “By 2020 scientific publications that result from research funded by public grants provided by participating national and European research councils and funding bodies, must be published in compliant Open Access Journals or on compliant Open Access Platforms.

而現在大約只有 15%:

According to a December 2017 analysis, only around 15% of journals publish work immediately as open access (see ‘Publishing models’) — financed by charging per-article fees to authors or their funders, negotiating general open-publishing contracts with funders, or through other means.

用這種方式降低那些收錢才能下載的平台的影響力...

Amazon Aurora (MySQL) 推出的 Asynchronous Key Prefetch

Amazon Aurora (MySQL) 推出新的效能改善,可以改善 JOIN 時的效能:「Amazon Aurora (MySQL) Speeds Join Queries by More than 10x with Asynchronous Key Prefetch」。

看起來像是某個情況的 optimization,將可能的 random access 換成 sequential access 而得到大量的效能:

This feature applies to queries that require use of the Batched Key Access (BKA) join algorithm and Multi-Range Read (MRR) optimization, and improves performance when the underlying data set is not in the main memory buffer pool or query cache.

其實記憶體還是最好用的加速器,能加大硬拼就先硬拼... XD

Amazon Device Farm 支援讓使用者直接連上去 debug 了...

Amazon Device Farm 推出這樣的功能又朝著設備租賃服務更進一步了:「Amazon Device Farm Launches Direct Device Access for Private Devices」。

Now, with direct device access, mobile applications developers can use individual devices in their private test set as if they were directly connected to their local machine via USB. Developers can now test against a wide array of devices just like they would as if the devices were sitting on their desk.

這樣就可以使用更底層的東西了...

nginx 記錄 TLS 連線資訊

想要在 nginx 的 access log 裡面記錄使用者在 HTTPS 連線使用的 TLS protocol 與 cipher。

在「How can I let nginx log the used SSL/TLS protocol and ciphersuite?」這邊有提到方向是 $ssl_protocol$ssl_cipher (出自「Module ngx_http_ssl_module」內的 Embedded Variables 章節)。

他的方式是在前面就插入 protocol,但我希望前面的欄位保持不變,把 protocol & cipher 放到後面,所以我就加了一個 /etc/nginx/conf.d/combined_ssl.conf (這邊我用 ondrej 的 PPA,在設定檔裡會撈 /etc/nginx/conf.d/ 下的設定,不確定其他的情況如何):

#
log_format combined_ssl '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $ssl_protocol/$ssl_cipher';

然後本來用 combined 的 HTTPS 設定就改成 combined_ssl

來放一陣子再來分析,然後想看看要怎麼調整 cipher...

StackOverflow 上離開 Vim 方法的文章...

被拿出來當 PR 宣傳了:「Stack Overflow: Helping One Million Developers Exit Vim」。

由於 Vim 是 Unix-like 系統一定會內建的 editor,所以常常被拿來放在 tutorial 裡面 (考慮到普及性,但完全不熟的初學者就...),或是不小心在輸入 vipw 或是 visudo 之類的指令就中獎了:

可以看到 pageview 破一百萬次了 XDDD 而且流量也都很穩定:

依照地區來拆開的話:(不過沒有照人口數正規化...)

然後做交叉分析,看這些卡在 Vim 的人平常是看什麼其他的文章:

回到資料分析的角度來看,這些東西可以透過有 cookie 的 access log 做到。有 access log 後可以用 Google CloudBigQuery,也可以用 AWS 家的 Amazon Athena 做。

Swap 對 InnoDB 的影響

Percona 的老大拿 5.7 版做實驗,確認 swap 對 InnoDB 的影響:「The Impact of Swapping on MySQL Performance」。

測試的機器是 32GB RAM,作業系統 (以及 swap) 裝在已經有點年紀的 Intel 520 SSD 上,而 MySQL 則是裝在 Intel 750 NVMe 上。透過對 innodb_buffer_pool 的調整來看情況。

可以看到設為 24GB (記憶體 75% 的量) 時很穩定的在 44K QPS 與 3.5ms (95%):

This gives us about 44K QPS. The 95% query response time (reported by sysbench) is about 3.5ms.

而當設成 32GB 的時候開始可以觀察到 swap i/o,掉到 20K QPS 與 9ms (95%):

We can see that performance stabilizes after a bit at around 20K QPS, with some 380MB/sec disk IO and 125MB/sec swap IO. The 95% query response time has grown to around 9ms.

當拉到 48GB 的時候就更掉更多,6K QPS 與 35ms (95%):

Now we have around 6K QPS. Disk IO has dropped to 250MB/sec, and swap IO is up to 190MB/sec. The 95% query response time is around 35ms.

作者發現掉的比率沒有想像中大:

When I started, I expected severe performance drop even with very minor swapping. I surprised myself by getting swap activity to more than 100MB/sec, with performance “only” halved.

這邊測試用的是 SSD,如果是傳統用磁頭的硬碟,對 random access 應該會很敏感而掉更多:

This assumes your swap space is on an SSD, of course! SSDs handle random IO (which is what paging activity usually is) much better than HDDs.

基本上還是要避免碰到 swap 啦,另外 comment 的地方剛好有提到前陣子在猜測的 best practice,測試時的 vm.swappiness 是設成 1,這應該是作者的 best practice:

Swappiness was set to 1 in this case. I was not expecting this to cause significant impact as swapping is caused by genuine (intended) missconfiguration with more memory required than available.

Archives