Melon blog

Attention is all you need

Sun, 04 Jan 2026 00:00:00 +0000

Introduction

Transformer, an encoder and decoder model, solely based on attention mechanisms.
Not using recurrence and convolutions, more parallelizable, capturing long-distance context in $O(1)$ . Therefore, it requires less time to train.

One layer contains two sub-layers: attention and Feed Forward (FFN).
Employ a residual connection around each of the two sub-layers, then layer normalization.
- The output of each sub-layer is $\text{LayerNorm}(x + \text{Sublayer}(x))$
Multiple $N$ layers. e.g., Llama-3-8B has 32 layers.

Input preparation

Input is a natural language sequence with $n$ tokens.
Input is first converted to a matrix $X$ of dimension $[n, d_{model}]$ . $d_{model}$ is the dimension of token embedding vector. e.g., 768, 4096.
Add positional embeddings to the input matrix to provide position info for each token. The dimension doesn’t change.
1. Using fixed sinusoid function. Can also use learned ones.

Self-attention

While processing a sequence, automatically learn the contexts, relations between different parts.

Calculate $Q$ (Query), $K$ (Key), $V$ (Value) using the input matrix and projection matrices $W^Q, W^K, W^V$ .

, the query, for each position, what context it wants to search for
1. $W^Q$ is of dimension $[d_{model}, d_{model}]$ . $Q$ is of dimension $[n, d_{model}]$ .
, the key, for each position, what info it can provide for the search, like a search index
1. $W^K$ is of dimension $[d_{model}, d_{model}]$ . $V$ is of dimension $[n, d_{model}]$ .
$V = X W^V$$, the value, for each position, if selected, what detailed info it can provide, like the payload
1. $W^V$ is of dimension $[d_{model}, d_{model}]$ . $V$ is of dimension $[n, d_{model}]$ .

With training, $W^Q$ and $W^K$ learn about searching and matching contexts. $W^V$ learns about information providing.

\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\]

$Q K^T$ is calculating similarity between query and all keys.
$\sqrt{d_k}$ is for scaling to prevent gradient vanishing.
Softmax is for normalizing similarity scores to probabilities, the total is 1.
$\times V$ gets the final value. Values with higher similarities occupy larger portion in the final value. The dimension is $[n, d_{model}]$ .

Multi-head attention

Learn the same sequence from multiple aspects. Multi-head attention allows the model to learn these aspects independently, in parallel. There could be shallow heads focusing on local info, deep heads for global info.

Linear projection for heads: project the initial for each head with .
1. The dimension $d_{model}$ is usually split equally across heads. $d_h = \frac{d_{model}}{h}$ . So the dimension of $QW_i^Q$ is $[n, d_h]$ . So multi-head won’t increase computation cost.
Parallel attention:
1. The dimension is $[n, d_h]$ .
Concatenation: Concatenate the outputs of all heads. The dimension is $[n, d_{model}]$ .
Final linear: use a $W^O$ to project the output back to the original dimension, also synchronize info from all heads.

Diversity: learns in various aspects.
Robustness: If one head fails due to noise, other heads are still working.
Sub-space learning: Easier to capture complex info by learning in multiple lower-dimensional spaces.

Feed Forward network (FFN)

FFN is after the attention layer. It’s position/point-wise, each position only cares about itself, learning deeper information. This layer adds nonlinear to the overall model.

\[\text{FFN}\left(x\right) = \text{activation}\left(xW_1+b_1\right)W_2+b_2\]

Dimension expansion
- Using $W_{up}$ of dimension $[d_{model}, d_{ff}]$ to expand the input dimension to $[n, d_{ff}]$
- $d_{ff}$ is usually larger (~4X) than $d_{model}$ . It’s easier to identify and extract complex details in higher-dimensional spaces.
Dimension reduction
- Use $W_{down}$ of dimension $[d_{ff}, d_{model}]$ to reduce to the original dimension.
- Filter noise and compress useful info.

The activation function is nonlinear, e.g., ReLU, $MAX()$ .
Token has gained context info from attention layer; FFN is for further digesting these info.
Dimension expansion:
Store knowledge and memories in FFN weight matrices.
~2/3 weights in a LLM are in FFN.

Training

Adam optimizer
Regularization
- Residual dropout on (1) the output of each sub-layer and (2) the addition of input embedding and positional embedding.
- Label smoothing

References

paper

Design patterns

Fri, 25 Jul 2025 00:00:00 +0000

Books

Advantages

More flexible
- Easier to add/remove/modify behavior
- Easier to reuse behavior
Easier to understand
Less code to write/debug/maintain
More efficient method of creating software
More value per line-of-code

Strategy, proxy, decorator

Strategy

Strategy
- Strategy pattern is just polymorphism.
- “Code to interfaces, not to their implementations”
Proxy
- is the delegation strategy: A strategy which uses a strategy.
- Lets you swap implementation without the knowledge of clients
Decorator
- A proxy which adds behavior.
- Add new behaviors to objects dynamically at runtime, flexible alternative to subclassing/inheritance.
- Decoration reuses an Object (dynamic); inheritance reuses a Class (static, compile time).
- Separation of concerns: breaking a program into distinct features that overlap as little as possible. A concern can be considered as a feature of behavior.
- Implement features separately, small, focused classes.
- Avoids # class explosion.
  - 10 classes, adding 5 behaviors
    - Inheritance: 10 * 2^5
    - Decorator: 10 + 5
- Advantages:
  - Efficient to implement behavior
  - Less code to write/maintain.
  - Separation of Concerns
  - Easier to extend
  - Testable

// Decoration
public class SynchronizedRunnable extends RunnableProxy {
    public SynchronizedRunnable(Runnable delegate) { // Takes an object
        super(delegate);
    }
    synchronized public void run() {
        this.delegate.run();
    }
}

// Inheritance
public class SynchronizedMyRunnable extends Runnable {
    public SynchronizedMyRunnable() {}
    synchronized public void run() {
        this.super();
    }
}

Path to MVC

Observer Pattern
- Define a one-to-many dependency between objects so that when one object changes state, all its dependents are notified and updated automatically.
- Observerble{attach(Observer); detach(Observer); notify()}; Observer{update()}
Mediator Pattern
- A mediator serves as an intermediary that keeps objects in a group from referring to each other explicitly.
- (Mediated object 1) -> (Mediator) <- (Mediated object 2)
Adaptor and Bridge
- Special types of strategy.
- Rely on another object to do the job(interface), which has a different interface.
Composite Pattern
- Compose objects into tree structures to represent part-whole hierarchies. Clients can then treat individual objects and compositions of objects uniformly.
- Any idea is better when made recursive – Brian Randell.
Flyweight
- Use sharing to support large numbers of fine-grained objects efficiently.
- Often implemented as a stateless Singleton.
- Object to be worked with is passed as an argument.
- Useful for creating extensions to existing classes.
- Java comparator example: public interface Comparator { int compare(Object o1, Object o2)}

MVC

Model: Data model
- Adaptor/Bridge strategy representing data, state and logic
View: UI
- Adaptor strategy for interacting with users
Controller: control logic, mediator
- Mediator between model and view

Good to always use Factory to create objects, which is more flexible and can have different creation options.

读人类简史

Sun, 13 Jul 2025 00:00:00 +0000

第一部分：认知革命 (The Cognitive Revolution)

核心观点： 大约在7万年前，智人（Homo sapiens）的思维和沟通方式发生了革命性的变化，这使得我们能够超越其他人类物种，并最终主宰地球。

第一章：无足轻重的人类 (An Animal of No Significance)
- 在历史长河的大部分时间里，智人只是一种生活在非洲角落、毫不起眼的动物。
- 地球上曾同时存在多种人类物种（如尼安德特人、丹尼索瓦人等）。
- 人类位于食物链的中端，直到学会用火后地位才开始显著提升。
第二章：知善恶树 (The Tree of Knowledge)
- 认知革命的发生是智人崛起的关键，可能源于一次偶然的基因突变。
- 这场革命赋予了智人全新的虚构语言能力 (Fictive Language)，能够讨论不存在的事物，如部落的神、国家、法律和金钱：想象的现实。
- 正是这种对“虚构故事”的共同信仰，使得大规模、灵活的合作成为可能，突破了约150人的生理社群规模限制。
第三章：亚当和夏娃的一天 (A Day in the Life of Adam and Eve)
- 描述了采集狩猎者的生活方式。他们知识广博，工作时间可能比现代人短，营养也更均衡。
- 他们是“最初的富裕社会”，但生活也充满危险，如野兽、意外和暴力。
第四章：毁天灭地的人类洪水 (The Flood)
- 随着智人走出非洲，他们所到之处，大型动物（巨型动物群）纷纷灭绝。
- 这表明智人早在农业革命和工业革命之前，就已经是地球生态系统中最具破坏力的物种。

第二部分：农业革命 (The Agricultural Revolution)

核心观点： 农业革命是“史上最大的骗局”。虽然它为人类提供了更多的粮食，但也带来了更辛苦的劳动、更差的营养、更严重的疾病和更深重的社会不平等。

第五章：史上最大骗局 (History’s Biggest Fraud)
- 农业革命并非人类主动选择，而是一个缓慢的、诱人深入的陷阱。
- 从某种意义上说，并非人类驯化了小麦，而是小麦“驯化”了人类。人类被迫定居下来，生活变得更辛苦，饮食更单一。
- 种群的成功并不等同于个体的幸福。
第六章：盖起金字塔 (Building Pyramids)
- 农业带来的剩余粮食催生了更大规模的社会组织，如村庄、城镇和王国。
- 人们开始为“未来”而焦虑，担心收成和天气。
- 出现了不事生产的统治精英阶层，并利用想象的秩序 (Imagined Order)，如法律和神话，来维持社会阶级和不平等。
第七章：记忆过载 (Memory Overload)
- 随着社会规模扩大，人类大脑无法处理海量的信息（如税收、财产记录）。
- 为了管理复杂的社会，文字被发明出来。最早的文字主要用于记账和行政管理。
第八章：历史从无正义 (There is No Justice in History)
- 想象的秩序（如种族、阶级、性别制度）往往会造成不公正的歧视，并通过恶性循环自我强化。
- 一个偶然的历史事件导致某个群体处于劣势，然后通过法律和习俗将这种劣势固化，并宣称其是“自然的”或“神定的”。

第三部分：人类的融合统一 (The Unification of Humankind)

核心观点： 历史的大方向是走向融合与统一。三种普世秩序——金钱、帝国和宗教——将原本分散的文化连接成一个全球性的文明。

第九章：历史的方向 (The Arrow of History)
- 历史的大趋势是融合，原本数千个独立的小世界逐渐合并成一个“全球村”。
- 这种融合是通过金钱、帝国和宗教三种普世秩序实现的。
第十章：金钱的味道 (The Scent of Money)
- 金钱是历史上最成功的“虚构故事”，是一种普世的信任体系。
- 它能跨越文化和地理障碍，让陌生人高效合作，因为所有人都相信它的价值。
- 为什么黄金的价值在欧亚大陆都被认可？
  - 因为远距离贸易的存在，使各区域对价值的认知趋同。
  - 相比之下，美洲印第安文明只把黄金作为装饰。
第十一章：帝国的愿景 (Imperial Visions)
- 帝国虽然常伴随着暴力和压迫，但也极大地促进了思想、技术、法律和文化的传播与融合。
- 几乎所有现代文化都是帝国遗产的混合体，帝国创造的标准化体系为今天的全球化奠定了基础。
第十二章：宗教的法则 (The Law of Religion)
- 宗教是另一种强大的统一力量，它赋予人类社会超人类的合法性，使其规范和价值观具有普世性。
  - 成为全球性宗教的关键是普世性和传播欲望。
- 近代以来，新形式的“人文主义宗教”（如自由主义、社会主义）开始取代神本位宗教，它们崇拜的是“人性”。

第四部分：科学革命 (The Scientific Revolution)

核心观点： 科学革命并非知识的革命，而是一场“承认无知”的革命。它催生了欧洲的崛起、资本主义的扩张，并彻底改变了人类社会和生态。

第十三章：发现自己的无知 (The Discovery of Ignorance)
- 现代科学的核心特征是：愿意承认“我们不知道”，并以观察和数学为中心来寻求新知。
- 这与传统知识体系（如宗教）形成鲜明对比，后者认为所有重要知识都已存在于古代经典中。
第十四章：科学与帝国的联姻 (The Marriage of Science and Empire)
- 科学革命和欧洲帝国主义相互促进，形成了强大的共生关系。
- 科学为帝国提供了技术和合法性，而帝国为科学提供了资金、数据和保护。
第十五章：资本主义的教条 (The Capitalist Creed)
- 资本主义为科学发展提供了经济动力，其核心信念是对“未来”和“增长”的信任。
- 科学、帝国和资本主义形成了一个强大的“铁三角”，共同驱动了现代历史的进程。
第十六章：工业的巨轮 (The Wheels of Industry)
- 工业革命的核心是能源转换的革命，人类学会了利用各种新能源，摆脱了对自然能量循环的依赖，导致生产力空前爆发。
  - 不再主要需要生物肌肉进行转换。
- 消费主义兴起，家庭和社区的功能被国家和市场所取代。
第十七章：国家的胜利 (A Permanent Revolution)
- 工业革命彻底改变了社会结构，传统的家庭和社群解体，取而代之的是国家和市场这两个想象的共同体。
  - 国家、市场势力强大 -> 个人能力强大 -> 传统社群实力弱 -> 国家、市场势力强大。
- 我们依赖国家提供教育、医疗和安全，依赖市场提供商品和服务。不再继续依赖传统社群提供保障。
第十八章：和平的时代 (And They Lived Happily Ever After)
- 尽管20世纪有两次世界大战，但二战后，人类进入了有史以来最和平的时期。
- 战争的成本（核武器）急剧上升，而利润却在下降（财富形式转向知识和技术），使得大规模战争变得不划算。
第十九章：智人的终结 (The End of Homo Sapiens)
- 尽管人类在物质上取得了巨大进步，但我们是否比过去更快乐并不确定。快乐更多取决于期望与现实的差距，以及身体的生物化学机制。
- 人类正开始挑战自然选择的法则，通过生物工程（基因改造）、仿生工程（人机结合）和无机生命工程（人工智能）来升级自己。
- 智人可能即将终结自己的历史，被我们自己创造出的“神人”（Homo Deus）所取代。
后记：变成神的动物 (The Animal that Became a God)
- 我们从一种无足轻重的动物，通过一系列革命，成为了地球的主宰，正走在将自己升级为“神”的道路上。
- 赫拉利最后提出了一个开放性问题：我们希望自己成为什么样的“神”？拥有神一般的能力，却不知道自己想要什么，这比以往任何时候都更加危险。

Mixture of Experts model papers

Sat, 12 Jul 2025 00:00:00 +0000

Switch Transformers

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Precursor of Switch Transformers

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Mixtral SOTA

Mixtral of Experts blog post

Adaptive mixtures of local experts

Hierarchical Mixtures Of Experts And The Em Algorithm

Blackboard design pattern

Two complementary patterns to build multi-expert systems

Querying various data sources

English phrases

Thu, 26 Jun 2025 00:00:00 +0000

“Severance” TV show

Season 1

Should I have consulted you?
Good initiative
Keep it between us
That’s one less horrible thing
Weird energy
Careful out there
Thanks for noticing
He was coming at you
I know what you’re up to
Save it
I haven’t worked it out yet
Are you sweet on this guy
You disapprove?
Get the hell out of here.
You aware what MDR is up to right now?
I thought that’s what you were paid for
I will have a word with
Looks like we both made it through
Congratulations there
I am a leader who cares about his employees
must be held accountable for the actions
Change can be disorienting
That’s poetic as shit, man
We’re not children, we didn’t do anything wrong
I trusted you, you abused the trust
I’m good with water, thank you
It’s a privilege
If it’s lame, we will leave
How are you holding up
I think we gonna take off
I’ve had a day
You are being weird
It’s kind of a big deal for me
Just getting a head start on the day
Just see how the day goes
You look troubled
That we shall
You deserve something special
Stay tuned
Not knowing is probably the best
Lead the way
How about a round of applause for the man of the hour
Come and circle up, gather around
I come home feeling tired but fulfilled, feel satisfied
Bon voyage
We made it, despite a staffing shake-up
Just now, just a moment ago
You do all sorts of wonderful things
I just have so much to do today
I’m over her, just wanna be friends
On your go, then
In case we don’t come back. Or, I don’t know, in case we do?
Sorry to just pop over
I’m not feeling quite myself
I don’t want to be nosy(curious)
I got a little woozy
I’m gonna stop by the bathroom
So nice to finally meet you. Likewise.
This better be working, assholes
You don’t have to weigh in on that
Discussing whether it might have been a wiser move for .. to ..
This is gonna sound weird
I know
I’m certainly gonna try
Family is both my bedrock and my inspiration
I can crane
It’s going great
The drink just hit me a bit
Spared from the pain
Lumon has their hands in so many pies
I will tee you up
Just stick to the talking points
Use the line that …
Bumps in the roads
With any transformative technology, there have been setbacks
We are on a verge of a revolution

Season 2

I think it’s best to save any queries till
Been a minute
We have much to discuss, I don’t want them distracting you.
But we must be cut to heal
I’d like to hear it from them
Please see that it’s taken care of.
I will put some thoughts on it.
If anything of note occurred
I feel confident that he’d approve me saying so
Let’s Occam’s razor this shit
I imagine you have a few questions
You could say that
You have my word
We do need to know what we’re dealing with
We’ve just said all there is to say
I will circle back
If we can just get a half-step more confirmation
My third question concerns …
It just takes time, I hope you give us that time
Hopefully there will be some chemistry
Of course, I bear full responsibility
This is unheard of
Everything chill?
Looks like you can use some coffee
Not like we have a lot of options right now
You don’t have an objective perspective due to
I’m teasing
xx has forced our hand
I don’t give three dry fuck to xx
Wow, some actual honesty
If you are taking feedback, I hate it
Words has consequences
A little sugar with your usual salt
Well, I hope that won’t be necessary
But the bulk of our day will revolve around the following
It’s quite a legacy you will leave
I’m tightening the leash
Have a restful evening
I’m just trying to speak their language
I will reflect on your words
This is a fiscal and creative opportunity unlike any I’ve ever seen, I’m not inclined to walk away from it
Call it a quirk
Using your time well, focusing on your own duty
She tricked both of us
You have a lot of stages, diverse
He is looking for the exit
I heard nothing but good things about you
Please don’t mess with me
It’s a settled fxxking law, the end
Locked and loaded
That’s kind of you
I’m not sick, I just needed the day, ok?
I have your word …?
I took their word for it

Common English phrases for daily life

“English in Comfort” podcast

“How’s it going?” - A casual way to ask “how are you?”
“I’m just chilling” - Relaxing and taking it easy.
“Grab a bite” - Getting a quick meal or snack.
“Call it a day” - Stopping work or an activity.
“Take a break” - Pausing to rest.
“Run errands” - Completing small tasks.
“Keep in touch” - Staying connected with someone.
“In a hurry” - Rushing or not having much time.
“Pick up” - Collecting something or someone.
“Just in case” - Being prepared for a possibility.
“Get things done” - Being productive.
“Catch up” - Reconnecting or getting back on track.
“Out of order” - Something is broken.
“On the go” - Being busy and moving.
“Make the most of it” - Taking full advantage of a situation.
“Settle down” - Calming down or establishing a stable lifestyle.
“Hang out” - Spending time casually with someone.
“Hit the road” - Leaving or starting a journey.
“Break the ice” - Starting a conversation in an awkward situation.
“Get the ball rolling” - Starting a project or discussion.
“What’s up” - A casual greeting.
“Long time no see” - Used when you haven’t seen someone in a while.
“I’m beat” - Being very tired.
“It’s a piece of cake” - Something is very easy.
“Call it a night” - Stopping an activity for the evening.
“Running late” - Being behind schedule.
“Hold on a second” - Asking someone to wait.
“It slipped my mind” - Forgetting something.
“I’m all ears” - Ready to listen attentively.
“It’s on me” - Offering to pay.
“Play it by ear” - Deciding plans as you go.
“Feeling under the weather” - Not feeling well.
“Hit the hay” - Going to bed.
“Not a big deal” - Something isn’t important.
“Pulling your leg” - Joking or teasing.
“Get down to business” - Focusing on a task.
“In the same boat” - Being in the same situation.
“It’s up to you” - Letting someone else decide.
“I’m swamped” - Being very busy.
“Wrap it up” - Finishing something.
“Throw someone under the bus” - Blaming someone else.
“Spill the beans” - Revealing a secret.
“Hit the nail on the head” - Being exactly right.
“Think outside the box” - Being creative.
“Take it with a grain of salt” - Not believing something completely.
“Once in a blue moon” - Something happens rarely.
“On the same page” - Agreeing or understanding a plan.
“Cutting edge” - Super modern or advanced.
“Back to the drawing board” - Starting over after a failure.
“Burn the midnight oil” - Working late into the night.
“The ball is in your court” - It’s your turn to decide or act.
“Cost an arm and a leg” - Something is very expensive.
“Sit on the fence” - Unable to decide.
“Bite the bullet” - Doing something difficult you have to do.
“In a nutshell” - Summarizing something briefly.
“Take it easy” - Relaxing or not worrying.
“That’s a win-win” - A situation where everyone benefits.
“You’re spot on” - You’re exactly right.
“Not my cup of tea” - A polite way to say you don’t like something.
“That’s the last straw” - When a small thing finally makes you lose patience.
“Hang in there” - Encouraging someone who is struggling.
“I’m on it” - You’ll handle something right away.
“Break a leg” - Wishing someone good luck.
“What do you have in mind” - Asking for someone’s idea.
“It’s a no-brainer” - An easy decision.
“I’ll sleep on it” - Needing time to decide.
“That rings a bell” - Something sounds familiar.
“Go the extra mile” - Putting in more effort.
“Cut corners” - Doing something cheaply and badly.
“Cross that bridge when we come to it” - Dealing with a problem later.
“Let’s face it” - Acknowledging an uncomfortable truth.
“I’m on the fence” - Being unsure.
“Let’s shake on it” - Informally agreeing.
“It’s up in the air” - Plans are uncertain.
“Running on fumes” - Being exhausted.

C++ data structures for leetcode

Sun, 14 Jan 2024 00:00:00 +0000

std::vector

// constructors
std::vector<int> v1;
std::vector<int> v2(5); // size of 5
std::vector<int> v3(5,1); // 5 ints with value 1
std::vector<int> v4(v3.begin(), v3.end()); // copying v3
std::vector<int> v5(v3); // copying v3
// construct from arrays
int int_arr[] = {2, 5, 7};
std::vector<int> v6(int_arr, int_arr + sizeof(int_arr) / sizeof(int));

v6[2] = 3;
std::cout << v6[2];
v6.size();
v6.push_back(3);
v6.emplace_back(...);
v6.back();
v6.pop_back(); 
v6.clear();
v6.erase(iter); // return the iterator of the next element.
v6.insert(iter, 5); // insert 5 before the iter element.
sort(v6.begin(), v6.end());
for (auto iter = v6.begin(); iter != v6.end(); iter++) {}

std::unordered_map

std::unordered_map<int, int> map;
map[1] = 100;
auto found = map.find(2);
if (found != map.end()) {}
std::cout << found->first << found->second; // key, value

std::heap

std::string

std::queue

std::stack

arrays

string persons[5];
string cars[3] = {"Volvo", "BMW", "Ford"};

// Pass string array as argument
bool myFunction(string *cars);

string functions

// string to integer
std::string str = "-12.3";
const int i = std::stoi(str); // -12
// string to long
const int i2 = std::stol(str); // -12

// Append characters
int n = 5;
char my_char = '!';
str.append(n, my_char);

new data

/**
 * Definition for a binary tree node.
 * struct TreeNode {
 *     int val;
 *     TreeNode *left;
 *     TreeNode *right;
 *     TreeNode() : val(0), left(nullptr), right(nullptr) {}
 *     TreeNode(int x) : val(x), left(nullptr), right(nullptr) {}
 *     TreeNode(int x, TreeNode *left, TreeNode *right) : val(x), left(left), right(right) {}
 * };
 */
class Solution {
public:
    TreeNode* createNewRoot(TreeNode* root, int val) {
        TreeNode *new_node = new TreeNode(val, root, nullptr);
        return new_node;
    }
};

Reading "Clean code"

Sat, 28 May 2022 00:00:00 +0000

Clean codes are elegant and efficient.
- Straightforward, minimal dependencies, articulated error handling, optimal performance, does one thing well.
Boy Scout Rule: Leave the campground cleaner than you found it.

Meaningful names

Use intention-revealing names
- Answers all big questions; no comment required
Avoid disinformation
- Like single-letter names
Make meaningful distinctions
- Not sufficient to add numbers, noise words or prefix conventions (“a”, “the”, etc) to distinguish two similar variables
  - e.g., list_1, list_2; product_info, product_data
Use pronounceable names
- Avoid names like genymdhms
Use searchable names
- MAX_CLASSES_PER_STUDENT instead of 7.
- Single-letter names may be fine as a local variable inside short methods.
Avoid encodings
- Avoid encode type or scope information into names
No member prefixes like m_address
Interfaces and implementations
- Prefer to leave interfaces unadorned, ShapeFactory instead of IShapeFactory. Users don’t need to know we are handling them an interface. Implementations can be encoded like ShapeFactoryImpl.
- Avoid mental mapping
  - Readers shouldn’t have to mentally translated a name into other name they already know, like t for time.
Class names
- Should be noun or noun phrases like Customer, Account, AddressParser.
- Choose a more descriptive names, avoid Manager, Processor Data or Info.
Method names
- Should be verb or verb phrases.
- Try use get, set prefixes.
- When the constructors are overloaded, prefer static factory methods with names that describe the arguments.
  - e.g., Complex.FromRealNumber(23.0) is better than new Complex(23.0).
  - Consider making corrsponding constructors private.
Don’t be cute
- Say what you mean. Avoid using slangs. Just DeleteItems instead of HolyHandGrenade.
Pick one word per concept
- and stick with it. get, fetch, retrieve; controller, manager, driver etc.
Use solution domain names
- CS terms, algorithm names etc. Visitor, Queue
Using problem domain names
Add meaningful context
- Use addrState to note the state is for address. Using an Address class is even better.
Don’t add gratuitous context
- Don’t use the same prefix for many methods and classes. Shorter names are generally better. Add no more context to a name than is necessary.

Functions

Small
- Functions should hardly ever be 20 lines long.
- The indent level of a function should not be greater than one or two.
Do one thing
- Do it well and do it only.
- Sections within functions, a clear indication that a function is doing more than one thing.
One level of abstraction per function
The stepdown rule, reading code from top to bottom
- We want the code to read like a top-down narrative.
- Every function to be followed by those at the next level of abstractions.

读浪潮之巅

Mon, 01 Nov 2021 00:00:00 +0000

帝国余晖，AT&T

1877年创始，贝尔电话公司
1885年为长途电话业务设立AT&T子公司
1916年，成为道琼斯指数成分股
1925年成立Bell Labs，同时分离加拿大电信业务
- 后来加拿大部分发展成北方电信
- 收缩可能帮助躲过大萧条
1956年，司法部协议限制活动范围
- 垄断法逼着公司追求技术进步，而不是单纯利用垄断资源
1984年反垄断拆分，市话业务剥离成8家Baby Bell公司（包括SBC），主体更名为AT&T
- 因垄断美国电信业
- 之后10年间， AT&T在电话，网络与移动通信业务上一直世界领先
1995年，内部提出拆分计划

三部分：

AT&T: 电信服务
- 2004年拆分为：
  - AT&T：个人和企业业务
    - 传统长途电话业务
    - 没有发展潜力
  - AT&T Wireless
    - 没有资金支持扩张
  - AT&T宽带
- 2004年，道琼斯指数除名
- 2005年，被SBC吞并，基本是没有技术含量的基础设施公司
朗讯：设备制造
- 这样就可以做其他电信商的生意，销售额增长，股价暴涨10+倍
- 职业经理人，董事会，投资公司看重短期利益
- 朗讯利润不足以支撑两万人的贝尔实验室
- 为了促进收入增长，借钱给其他公司购买其设备
  - 互联网泡沫破裂后转为亏损
- Carly Fiorina杀鸡取卵，再次拆分无线设备部门Avaya
- 互联网冲击下，数据交换设备（思科）需求超过语音交换设备
- 2006年，阿尔卡特和朗讯合并
- 2016年，被诺基亚收购
NCR：计算机业务

一些想法：

幸运的是发现并顺应潮流
追求短期利益对公司的伤害很大
- 当无人拥有控制权时，长期发展会有问题

蓝色巨人，IBM

保守
to B的先天基因，与政府、军方保持良好关系，客户主要是政府，银行，科研机构，经营终端消费品经验不足。
二战后在机械和电子产业中选择电子
引领计算机革命，把计算机推向民间，商用
司法部发起反垄断诉讼，长达10年
推出IBM PC，但是公司不重视
和微软的合作中吃亏了
- 微软一方面为IBM开发操作系统，一面自己开发Windows
1993年，Louis Gerstner接任CEO，把公司转型为以软件和服务为主
- 裁撤项目，变卖资产，回收分拆的服务公司
用户目标定位为企业，放弃个人消费者
申请很多进攻性专利，收益不菲
不断剥离利润低业务，总利润增长
在金融危机中股价也很稳定，
- 因为是给大客户卖服务，有持续收入
- 把很多工作岗位迁到印度
- 全球化收入
发展云计算，人工智能沃森

八叛徒与硅谷

William Shockley在贝尔实验室发明晶体管
1955年Shockley辞去工作在湾区创建公司生产半导体
- 期间招来包括Robert Noyce等人
1956年，八人受不了Shockley的独裁作风，集体离职
1957年，Traitorous Eight，八人创办仙童公司，得到IBM大股东仙童家族的投资
仙童公司拿到军工和IBM的晶体管合同，不到一年盈利
1958年诺伊斯发明集成电路
硅谷的发展
- 1951年斯坦福设立斯坦福工业园，吸引科技公司
- 1952年IBM创立Almanden lab
- 仙童公司流失的人才在当地创业
  - 当时集成电路成本高，仙童公司投资人希望优先发展成熟的晶体管产业赚钱。多人离职建立集成电路和投资公司
    - Teledyne，Signetics，Molectro，Kleiner Perkins，Sequoia Capital，Intel
革命性技术出现，诞生一家巨无霸公司，既有可能压榨本地区其他新公司的产生，也有可能开枝散叶，促进地区发展

科技时尚，苹果

1976年，乔布斯和Steve Wozniak，Ron Wayne创办苹果
- 推出几百美金的Apple I
1984年推出Macintosh
- 图形交互系统，优于IBM PC，但是是施乐开发的
- 封闭硬件系统，不允许兼容机
1985年，乔布斯与CEO产生矛盾，被董事会踢出公司
乔布斯买下Pixar动画工作室，最后被迪士尼收购
苹果公司市场不断萎缩，开始亏损
1997年乔布斯回归，开始打造性能和时尚
- 同时采用Intel通用处理器和FreeBSD系统内核，方便开发者
2001年推出iPod，很成功
2006年，推出Apple TV，但是不太兼容市场上的设备
2007年开始陆续推出iPhone，iPad，大获成功
擅长发现事物的价值，背后是长时间的思考沉淀，还有对品质的追求

信息产业的生态链

Moore’s Law
- 每18个月，集成电路集成度翻一番，IT产品性能翻一番
- 技术公司基础架构升级要按目前计算能力和存储量的10倍来设计
Andy and Bill’s Law
- What Andy gives，Bill takes away。 Andy Grove是Intel CEO。Bill是Bill Gates。
- 微软等软件公司的新软件把硬件提示的好处几乎用光了，实际使用体验差别不大。
  - 迫使用户更新机器，使硬件公司收益，形成WinTel产业格局。
  - 安卓联盟也类似
  - 同时因为Windows很久没更新，导致Intel等硬件厂商升级PC产品的动力不大
Reverse Moore’s Law
- Eric Schmidt：如果一个IT公司今天和18个月前卖掉同样多的同样产品，营业额会下降一半。
- 硬件公司生计比较艰难

奔腾的芯，Intel

1968年，Gordon Moore和Robert Noyce创办，主打低端微处理器。
1981年，IBM PC采用Intel 8086处理器。大量兼容机厂商都选用Intel处理器
80年代开发80386和80486，坐上半导体行业头把交椅。
- 期间和摩托罗拉竞争，技术上相对落后却占据更多市场份额
  - 强援IBM、微软
  - 管理层和公司激励制度优势
  - 专注于处理器
指令集
- 复杂指令集CISC
  - 设计复杂，需要很高集成度
  - 指令执行时间不一样长，造成不必要等待
  - 高功耗
- 精简指令集RISC
  - John Hennessy‘s MIPS， David Patterson
- Intel x86系列是CISC，后来产品选择继续兼容
  - 研发投入大，提高处理器性能
- 6大其他坚持RISC的厂商的处理器做不下去
和AMD
- AMD主要做兼容英特尔的便宜替代品
- 反垄断给AMD带来好处
- AMD在高端服务器市场发起挑战
- Intel推出酷睿系列，和AMD打价格战，夺回主动权
错失移动时代
- 在PC处理器以外的芯片研发一直失败，把移动和通信处理器业务卖给Marvell
- 商业模式是大投入大批量，所以新兴市场一开始很难盈利
- ARM-based RISC芯片占据智能手机市场98%以上份额
- PC市场逐渐萎缩，远景不容乐观

罗马帝国，微软

1981年乔布斯曾向盖茨展示过图形操作系统
- 给别的公司做报告的内容确保都是公开发表过的
垄断操作系统就间接垄断了整个PC行业
对苹果三管齐下
- 承诺为Macintosh开发应用软件
- 和IBM合作开发OS/2
- 悄悄开发Windows
  - 利用DOS争取大概10年时间
    - 近乎免费给IBM PC提供BASIC和DOS，麻痹IBM，使用户和开发者产生依赖
- 苹果的失误
  - 封闭系统，放弃兼容机市场
    - 摩尔定律/安迪比尔定律决定了一家公司吃不下整个PC市场
      - 软件更新带动硬件发展
  - 少有第三方开发软件
  - 各代之间不兼容
  - 公司内部不稳定
1990年推出Windows 3.0成功
- 推出办公软件、浏览器等，打败Lotus，Netscape等应用软件公司
  - 全力开发IE浏览器，捆绑销售
2000年被诉垄断，败诉然后共和党上台翻案
因为抄袭行为在知识产权上侵权案很多
从普通人身上赚钱，而不是狠宰富人
内部争端，操作系统派/toB vs 浏览器派/toC
- 操作系统派获胜，因为会平稳度过金融危机或互联网泡沫
- IE部门大量核心员工离职，从此发展缓慢
苹果好比古希腊，微软好比罗马帝国
- 罗马帝国大军出征日耳曼，条顿堡之战被全歼，没有复仇
- 雅虎阻击了微软向互联网的扩张
  - 免费向用户提供服务，向内容提供者和广告主收钱
  - 华尔街看好，市盈率达到1000
- 互联网泡沫破裂时，也没有收入人才和技术
  - 傲慢
  - 反垄断限制
  - 为了财报不愿搞长线投资
  - 对互联网不看好
- 2011年Google发现Bing在抄其搜索结果。微软也发起对Google的反垄断诉求
- 搞智能家庭
  - 游戏机业务长期亏损
  - Netflix在家庭娱乐中崛起，和Youtube，Chromecast分庭抗礼
  - Google Home，Nest，Amazon Alexa
- 互联网争夺
  - 引入陆奇主管在线部门，加大研发投入，和雅虎百度在搜索上合作
  - 开始Bing每年亏损20亿美元
  - Chrome崛起，IE市场份额只占5%
- 苹果动摇微软根基
  - 苹果请回乔布斯，恳求盖茨，获得投资起死回生
  - 互联网泡沫破裂后，持有现金却没有布局新业务，而是派发股息
  - 企业部分利润做勾搭，失去开发优秀toC产品的能力
- 纳德拉扭转颓势
  - 强化企业级业务，砍掉互联网，亚洲研究院，移动业务，即时通讯
    - 利润80%以上的Office套件和数据库业务
    - 大力发展Azure
    - Office转为Online subscription模式

纯软件先驱，甲骨文

1977年埃里森等人创办公司，推出关系型数据库Oracle
- 之间占统治地位的是Hierarchical model和Network model
- 关系型讲数据库的物理层和逻辑层完全分离
  - 可以实现非常复杂的查询逻辑
  - 实现大型复杂数据库
  - 逻辑层开发难度较低
1981年Gupta加入，明确发展方向为开发通用关系型数据库系统
大量并购其他公司
2000年与微软，IBM在数据库系统三足鼎立，但此后发展速度远高于对手
- 专注于数据库市场，更容易获得用户认可
- 注重成本控制和利润
- 多次成功并购，Peoplesoft, Siebel, Hyperion, BEA, Sun
2010年并购Sun，专利诉讼谷歌，败诉
2011年开始进入云计算领域

Golden gate bridge，思科

斯坦福Leonard Bosack和Sandy Lerner发明了多协议路由器，可以支持各种网络服务器和协议。
1984年创办思科，1986年推出第一款产品，顺应了互联网刚起步的需求
CEO钱伯斯培育了健康的公司文化
- 曾在IBM和王安电脑工作，总结企业文化上失败的教训
- 对员工慷慨大度
- 无条件满足客户需求
宽容内部创业的政策
收购很多前员工创业的公司
毛利率高达60%
- 微软80%，硬件商20%，石油工业35%, 华为40%
竞争对手
- Juniper Networks
  - 规模小，定位高端
- 华为
  - 请大批IBM顾问打造国际化企业
  - 压榨整个行业的利润空间，思科没有胜算
Peter Novig：当一家公司的市场占有率超过50%以后,就不要再指望在市场占有率上翻番了
- 当公司占据大部分市场时，再进一步扩招也很难使公司再扩张一倍
- 需要开拓新市场
  - 思科的VoIP （Voice over IP)
  - 华为进军智能手机市场

英名不朽，杨致远，David Filo，Yahoo！

制定游戏规则：开放，免费和盈利。
商业模式：用户和客户可以不是同一群体
三个斯坦福同学开发了一个门户网站，流量暴增
- 无条件，开发的为全世界网站建立索引
- 制止了AOL等公司把互联网办成另一个电话网的企图
软银投资，一度占近40%股份
杨致远想到打广告赚钱的法子
- 2014年美国广告市场规模大概$180B
流量第一，专心办互联网上最好的媒体
- 流量不平等，一流品牌在一流媒体做广告
- 对流量的片面重视，导致很多网站不重视内容，雅虎也不能免俗
- 互联网泡沫破裂，市值蒸发90%
复苏
- 裁撤亏损项目
- Terry Semel和Overture合作，使用搜索结果中竞价排名的方法
- 买下搜索引擎Inktomi公司，后来买下被Google重击几次的Overture
收入主要靠
- 传统的品牌广告，是雅虎的强项
- 在线搜索广告，是技术竞争，Google占优势
  - 为了夺回搜索之王，雅虎大力扩张，最后只搞出一个令人失望的Panama广告系统；在品牌广告方面无所作为
2004年开始盈利增长不及预期，靠大量抛售Google股票来创造虚高盈利欺骗散户
Semel下台后Decker接手，大局观和对互联网市场的了解都很差
- 在Google上市前以低于开盘价（￥82.62）出手了Google股票
一直使用手工分类目录系统，手工调整搜索结果
2006年开始5年迅速衰败
- 业绩下滑，核心员工离职
- Google搅黄了微软的收购
  - Google提出雅虎可以使用自己的广告系统，这样雅虎可以精简人员；雅图提高对微软的报价；微软拒绝
  - 二流人才也基本离开了
- 大批私募基金进入董事会，不再关注公司发展
  - 选定Carol Bartz出任CEO，开始拆分卖出，减少投资人损失
  - 放弃搜索引擎改用Bing，采用微软不成型的搜索广告系统
- 唯一价值是拥有日本雅虎和阿里巴巴的大量股份
  - Bartz对阿里巴巴支付宝事件也处理的不好，占股从43%降到15%一下
- 来自Google的Marissa Mayer出任CEO，也没有扭转颓势
- 2017年被Verizon以$4.8B收购

硅谷见证，惠普

1939年Hewlett和Packard创办HP
二战后斯坦福出租土地创办Stanford Industrial Park，HP入驻
业务包括示波器，信号发生器，医疗仪器等
进入小型计算机，打印机行业
90年代后期开始转型
- 产品线太长，内部混乱，需要重组，剥离一些部门单独上市
  - 董事会和CEO Fiona剥离赖以起家的仪器部门，合并亏损的康柏公司
    - 拆分的安捷伦公司股价暴涨，同时HP股价也涨，是不理性的股市表现
- 工作站业务远远落后，个人电脑领域差距越来越大，打印机市场也不断萎缩
- 惠普资金周转不够快，一年一次，戴尔有两次以上
2002年出现首次巨额亏损
并购康柏后没有增加市场份额，反而勉强维持康柏原有的份额
亚洲制造冲击了惠普的打印机市场
新任CEO Mark Hurd
- 裁撤研究部门
- 三个主要部门：技术服务TSG，个人电脑PSG，打印设备ISG
- 强化PC代销模式，与Walmart和Costco等合作
- 恢复作为技术公司的形象
Mark离职后HP陷入颓势
- 2015年拆分为两家公司
- 个人业务的HP公司手反摩尔定律制约，利润变薄，发展不稳定
  - 大量人才流失，Alan Eustace和Sanjay Ghemawat去了Google
- 惠普企业执行力不高，被其他公司蚕食

没落贵族，摩托罗拉

1928年创立，原名Galvin Manufacturing Corp，制造Motorola品牌收音机
为军方开发无线步话机，对讲机，注重技术和品质
垄断无线双向通信市场直到90年代初
曾进军家电市场，生产彩电，不是很成功
80年代业务拓展至计算机的半导体芯片，数字信号处理芯片DSP，也发明了手机
90年代初在移动通信，数字信号处理和计算机处理器三个领域都是技术最强，不过后来三个市场都没发展好
第二代移动通信
- 在第一代移动通信没有敌手，第二代移动通信发展中，欧洲联合推出GSM技术标准(Group Special Mobile，多用户共用一个信道TDMA)，战胜了美国的标准
- 因为第一代模拟手机的成功，在数字手机研发上进展缓慢
- 同时数字手机技术差异较小。功能，可操作性，外观反而比技术更重要
  - e.g. 音箱里数字设备不同品牌差异不大，不过模拟部分的喇叭，日本厂商做不过美国的Harman Kardon，INFINITY
- 管理层能力平庸，十几年积累的模拟技术无足轻重
铱星计划
- 77颗低轨道卫星覆盖全球，每颗有三千多个信道，可以和手机直接通信，无需基站，可以在地球任何地点通信
- 1996年第一颗星上天，1998年投入商业运营
- 成本高导致定价贵，手机$5000，每分钟通话$3
- 1999年铱星公司破产
- 目前还在运营
全线溃败
- 手机领域被诺基亚，三星，LG抢走市场
  - 2001年专注手机市场，不过开发速度太慢，用Java打造的操作系统速度慢，基于Linux的平台开发进展不顺利。输给安卓
  - Sanjay Jha力主加入安卓联盟，全力支持安卓手机开发。推出的手机大获好评
- 计算机处理器上输给英特尔
- 数字信号处理器上没竞争过德州仪器
  - 一直做24位处理器，不方便使用
  - 3G手机普及时，高通一月成为芯片最大提供商
  - 半导体业务后来剥离成Freescale
2011年拆分为两个公司
- 摩托罗拉移动：手机业务和电视机顶盒
  - 苹果和微软联手挑起和安卓联盟的手机专利之战
    - 苹果和微软联合收购Nortel的移动通信专利，企图阻挡安卓联盟进入智能手机领域
    - Google收购摩托罗拉移动，反制
      - 陆续出手资产和裁撤员工
- 企业级通信和其他业务：没有任何影响力

硅谷奇迹探秘

硅谷为什么成功？

宛若似真的理由：

气候独特舒适，地中海气候
- Eric Schmit
- 其他类似气候地区没发展起来
斯坦福的存在
- 反而硅谷带动了斯坦福的发展
- 波士顿周边科研水平更高
- 特拉维夫，深圳附近没什么好大学，创业却很活跃
  - 香港好大学多，没有拿得出手的科技成就
重视知识产权
- 硅谷相较美国没有特殊性
风险投资
- 是催化剂但不是决定性因素

成王败寇

硅谷创业成功率很低，每年大概4000起投资，IPO公司一般不到30家
好区划分，一共不到4万户，不到硅谷地区家庭数量的5%
- Atherton, Hillsborough, Woodside, Los Altos Hills
- Palo Alto, Saratoga, Los Altos
创业成功必要因素
- 创始团队，领袖人物，精力过人
- 小而精的团队，技术壁垒
- 商业头脑，盈利模式
- 及时纠错
- 外部环境，时机不早不晚
- 运气好
- 饥渴感
失败的出路
- 找工作，离开硅谷，重新再来

嗜血的地方

硅谷科技公司平均工作时间较长
失业压力大
文化沙漠，娱乐文化生活贫瘠
换工作很常见，赌 SV lottery

机会均等

权威众多，不迷信权威
招聘时不太注重过往经历
其他行业比如房地产，律师，银行，装修发展也很好

半导体行业在硅谷逐渐衰退

反摩尔定律，硅谷费用高昂
- 软件业不受影响
亚洲制造

真正的奥秘

叛逆的文化，会建设的颠覆者
对叛逆和失败的宽容
- 加州基本不honor竞业禁止
  - 如必须使用某种技能才能生存时，可以使用
  - 不能明确描述过去工作细节
  - 不得使用原公司知识产权
- 司法部和加州起诉Apple，Intel，Google，Adobe公司，因为他们彼此员工跳槽很少，破幻了竞争
- 华盛顿州有竞业禁止限制
多元文化，人口组成多元
- 产品设计面向全球
创新的灵魂
- Genetech以IT公司模式经营，不断创新，业务增长快
- 辉瑞发展停滞，依赖购买小公司获得新药

短暂的春秋:与机会失之交臂的公司

Public speaking course notes

Sat, 11 Sep 2021 00:00:00 +0000

This is the notes while taking Public speaking course from University of Washington, instructed by Dr. Matt McGarrity.

A rhetorical approach

Rhetoric is the art of identifying communication needs and strategically responding to them.

While public speaking, we need to find the best or most appropriate response to current rhetoric situations, including topic, setting, audience, occasion and credibility.

Rhetorical canons are tools for preparing and performing speeches, including:

Invention: Coming up with contents.
Arrangement: Putting it in order for the audience.
Style: Finding the most effective language.
Memory: Getting your speech into your head to support good delivery.
Delivery: Performing the content effectively for this audience.

Speeches should be optimized for ears not eyes. The listeners needs cues to decode the talk. Contextual cues provide situational information, eliminating other irrelevant situations. Prosodic cues (pitch, rate, pause, stress etc) provide auditory information. e.g., We break the sentence into functional units.

Inventing and arranging main points

Key point speeches:

Determine key points we want audience to remember.
Discuss each points with concrete and interesting examples.
Deliver clearly with confidence.

A model of arguments by Stephen Toulmin can help us write clearer speeches.

Claim: an assertion you wanna audience to take as valid.
Support: evidence used to validate your claim.
Warrant: what links the support to the claim, helping understand the relationship.
Outlining: Preparing a speech in a hierarchical structure. Outlines allow you to plan and refine the speech’s logic. We can outline a few points then develop the speech hierarchically.
Flowing: Taking notes on a speech in an outline format. Flows allow you to better understand a speech.

Inventing key points

Developing key points for topics, can start with:

Short term/long term
Past/present/future
Increase/decrease
- Some types of qualitative/quantitative changes going on.
Cause/effect
Division
- Two component in the topic can be broken out, not mutually exclusive.
Definition

For how many key points to include in a speech, try 2-5 for a start; break up larger presentations into “chunks” of 2-5 key points.

Arranging key points

Subordination: All key points are related to the topic.
Coordination: Work well together.
- Having a clear rationale driving the arrangement.
Discreteness: Don’t overlap with each other.
Phrasing key points
- Use shorter phrases
- Use evocative words
- Put the key terms in important positions, starts or ends
- Use parallel phrasing, if possible
  - Having repeating parts in each key points, e.g., “I do developing work; I do testing work; I do operational work”

Handling Q&A session

Key is ethos (credibility of the speaker). The speaker should be and sound like an expert, in control of the content.
Prepare for questions
- Identify places that may get clarification or push back questions.
- Prepare basic answers, top 5 questions, evidences.
Structure the answer
- Keep it brief.
- Stay on point.
  - State it: yes or no.
  - Explain it.
  - Show it.
  - Conclude it.
- For tough questions
  - Reject loaded questions and words like “So the company is getting rid of that unfair policy?”
  - Avoid matching hostility with hostility.
    - Can break the contact with the questioner and come back to the entire audience.
      - “I get what he’s concerned with. I do. I want to be very clear about this issue …”
  - Push it to a post Q&A.
    - “I understand what you are asking here. I don’t think we have time to hash it all out here. Let’s tak more offline.”
  - Q&A is still part of the presentation. The speaker is still held accountable.
  - Just say no if don’t know.
    - “I actually don’t have that answer right now. I will find out and get back to you”

Inventing and arranging support

Support and explanation

Good support:
- shows validity and provides details.
Types of support
- Facts, statistics, testimony
- Example and illustrations
- Metaphors and analogies

When discussing your support:

Focus on relevant details. Come back if it’s too detail.
Focus on clarity over comprehensiveness.
Help the audience craft a mental image.

To perform a key point:

State it: give the claims
Explain it: unpack the claim and prime for the support
Show it: explain the support and how it relates to the claim
Conclude it: touch on the key claim again to wrap up

Making your speech easy to follow

Speech structure:

Introductions
- Core functions:
  - Open the speech, like a pause then a clap.
  - Orient the audience. Let them know the matter quickly.
  - Provide a preview. Main key points; how the time is spent.
Transitions
- demonstrate the distinctness of each point.
- help the audience know where you are.
- give the audience just a little break.
- E.g., state the previous key point then state the next key point.
Conclusion
- reinforce the key points.
- provide a sense of closure.
- should sound like a final line (usually slower and more deliberate)
- E.g., So in conclusion, …

Revising, practicing and remembering the speech

Revise for fit between claims and support.
Look for rearrangement opportunities.
Try a couple of different models.

List the structure, top down:

Topic
- Key point 1
  - Support 1
  - Support 2
- Key point 2
  - Support 3

Then do bottom up to see if the child point relates to the parent point.

Fear of public speaking and good delivery

MacOS bluetooth connectivity issues

Sat, 05 Sep 2020 00:00:00 +0000

What?

Somehow both my two Macbook Pro (2018 13-inch with High Sierra and 2019 13-inch with Catalina) have been experiencing bluetooth connectivity issues for several months. All of a sudden, all bluetooth devices disconnect for a few seconds then recover. This could happen dozens of times a day and really disturbs productivity. I have tried numerous solution mentioned online like restarting SMC, reset bluetooth module, etc and none of them works. Finally, found this article. Turning of “Handoff” feature works for me.. (System preferences -> General -> uncheck “Allow Handoff between this Mac and your iCloud devices”) You can read more in the article. Basically “Handeoff” allows you to coordinate between your iCloud devices, which I have never tried.

Read "Dynamo, Amazon’s Highly Available Key-value Store"

Sun, 21 Jun 2020 00:00:00 +0000

Reference

Link to paper

Abstract

K-V store.
Highly available, scalable.
Always on for reads and writes.
Eventual consistency.

1 Introduction

Amazon architecture
- Highly decentralized, loosely coupled SOA consisting hundreds of services.
- Failures are the norm of daily operations.
Dynamo uses techniques:
- Consistent hashing for partition and hashing.
- Consistency is facilitated with object versioning.
- Quorum-like technique and a decentralized replica synchronization protocol for consistency among replicas during updates.
- A gossip based distributed failure detection and membership protocol.

2 Background

Relational DBs are complex and inefficient for simple KV use cases.
System assumptions and requirements
- Query model
  - Simple reads and writes to a single data item.
  - Values are binary objects.
- Doesn’t need ACID transactions.
- Efficiency
  - SLAs on 99.9% percentile.
- No internal hostiles thus no security related requirements.
SLAs
- Uses 99.9% for better client experience.
Design considerations
- Use optimistic replication techniques -> conflict resolution
  - When to resolve, read/write?
    - Resolving during writes will reject some writes.
    - (Accepted) Resolving during reads to have an always-writable system.
  - Who resolves
    - Data store: Choices are limited, like last-write-wins.
    - Application: More flexible.
- Incremental scalability
- Symmetry
  - All nodes should have the same responsibilities.
- Decentralization
  - Favors decentralized peer-to-peer techniques over centralized control.
- Heterogeneity
  - e.g. work distribution must be proportional to the individual server capabilities.

Dynamo differs from some other projects as follows.

Always writable as no writes should be rejected.
All nodes are trusted since it’s used internally.
Doesn’t require hierarchical namespaces or complex schemas.
For latency-sensitive applications, targeting at 99.9% percentile.
- Multi-hop routing is not acceptable.
- Dynamo is zero-hop DHT that each node maintains enough routing info locally to route a request to an appropriate node directly.

4 System architecture

General considerations for a scalable and robust distributed system:

Data persistent component
Load balancing
Membership and failure detection
Failure recovery
Replica synchronization
Overload handling
State transfer
Concurrency and job scheduling
Request marshalling
Request routing
System monitoring and alarming

Core techniques used by Dynamo:

Problem	Technique	Advantage
Partitioning	Consistent hashing	Incremental scalability
High Availability for writes	Vector clocks with reconciliation during reads	Version size is decoupled from update rates
Handling temporary failures	Sloppy quorum and hinted handoff	Provides high availability and durability guarantee when some of the replicas are unavailable
Recovering from permanent failures	Anti-entropy using Merkle trees	Synchronizes divergent replicas in the background
Membership and failure detection	Gossip-based membership protocol and failure detection	Preserves symmetry and avoids having a centralized registry for storing membership and node liveness information

System interface

get(key)
put(key, context, object)
- context is some system metadata such as versions of objects.

Partition algorithm

Dynamo uses consistent hashing. On a ring, a node is assigned a random value representing the position on the ring. For an example ring A -> B -> C -> D -> A, B is the coordinator node for data with key in range (A, B].

This setting have two problems:

Random assignment leads to non-uniform data and load distribution.
Oblivious to the heterogeneity in the performance and capacity of nodes.

Therefore, Dynamo uses “virtual node” concept. Virtual nodes are nodes on the ring. A physical node can be responsible for one or more virtual nodes. Advantages are as follows:

If a node becomes unavailable, its load is evenly dispersed across remaining nodes.
If a node becomes available again or is added, it roughly accepts equivalent amount of load from the remaining ones.
The number of virtual nodes for a physical node can be decided considering the performance and capacity.

Replication

Each data is replicated across N nodes. Starting from the coordinator node, data are replicated to the N-1 successor nodes. For an example ring A -> B -> C -> D -> A and N=3, D is responsible for data in range (A, B], (B, C] and (C, D]. For a particular key, the list of nodes storing it is called preference list. In order to make sure data is replicated across N physical nodes, preference list is chosen by skipping virtual nodes on the same physical node.

Data versioning

To ensure the system is always writable, conflicting writes are accepted and the conflicts can be resolved during reads, e.g., merging different versions of a shopping cart. Vector clock is used to capture the causality of an object. It’s a list of (node, counter) pairs. For a put() operation, client must supply the version it’s updating in context, it could be obtained from an earlier read.

The size of a vector clock is usually limited since preference list contains only N nodes. In the cases of network partitions or multiple server failures, the request may be served by other nodes. Dynamo limits the size of vector clocks by associating timestamp to each pair. If the size reaches the threshold, discard the oldest ones.

Execution of get() and put() operations

Any storage node in Dynamo is eligible to receive client get and put operations for any key. Two strategies for a client to select the node performing operations:

Routes its request through a generic load balancer that will select a node based on load information.
- Clients don’t need links to Dynamo library.
- The random chosen node will forward the request to the top N healthy nodes in the preference list.
Uses a partition-aware client library that routes the request to the coordinator node directly.
- Lower latency.

Dynamo uses consistency protocol R + W > N.

R is the minimum number of nodes fore a successful read.
W is the minimum number of nodes fore a successful write.

Upon a read, the coordinator requests all existing versions of data for that key from the N highest-ranked nodes in the preference list. Once receiving at least R responses, reconcile the versions so the final versions are casually unrelated. Then send the response to the client and write the final versions back.

Handling failures: Hinted handoff

To improve availability, Dynamo uses sloppy quorum: all reads and writes are performed on the first N healthy nodes in the preference list.

For an example ring A -> B -> C -> D -> A and N=3, if A is unavailable, D will temporarily be hinted to receive the corresponding replica . Once A is recovered, D will transfer the replica back.

The preference list in constructed in a way that its storage nodes are spread across multiple data centers, in order to tackle data-center-level failures.

Handling permanent failures: replica synchronization

Dynamo uses an anti-entropy protocol with Merkle trees to keep replica synchronized.

Merkle tree
- A hash tree where leaves are hashes of the values of individual keys.
- Parent nodes higher in the tree are hashes of their respective children.
- It can quickly compare two trees to know which keys are out of sync.
In Dynamo
- Each node maintain a separate Merkle tree for each key range.
- A disadvantage is the tree needs to be recalculated if a node joins or leaves the system.
  - Will be addressed later.

Membership and failure detection

To add/remove a node to a ring,
- An admin connects to a node and issue membership changes.
- Change history is persisted.
- A gossip-based protocol propagates change history between nodes.
  - Also exchange and reconcile mapping of local nodes and token sets (key ranges).
A ring may be logically partitioned.
- e.g., connect node A to join A into the ring; connect node B to join B. In this case, A and B don’t know each other.
- To solve this, some nodes play the role of seeds, which is known by all nodes.
Failure detection
- To avoid attempt to communicate with unreachable nodes during operations like get(), put(), transferring partitions and hinted replicas.
- Local notion is sufficient: A pings B; if B doesn’t respond, A thinks B is unreachable. A can use B’s alternative nodes.
- Decentralized failure detection is unnecessary.
  - It uses a gossip protocol enabling each node to know arrival/departure of other nodes.
  - This obviates Dynamo’s node joining/leaving mechanism.

Adding/Removing storage nodes

When a new node joins the ring, following operations happen. Operational experience shows it distributes the load uniformly across storage nodes.

It gets assigned a number of random tokens (key ranges) from the ring.
The nodes currently in charge of these key ranges will transfer data to the new node upon its confirmation.
- Avoids duplicate transfers.

5 Implementation

Each Dynamo node contains 3 components:

Local persistence component
- Different storage engine can be plugged in: Berkeley DB, MySQL, etc. for different use cases.
Request coordination
- built on top of an event-driven messaging substrate where message processing pipeline is split into stages similar to SEDA architecture.
- Each client request triggers the creation of a state machine on the node handling:
  - identifying nodes for this key;
  - sending requests;
  - waiting for responses;
  - potentially doing retries;
  - processing and packaging responses.
- For read requests:
  - may performing read repair to fix stale data.
- For write requests:
  - Any node in the preference list can be the coordinator.
    - If using timestamp-based reconciliation, any node in the system can do the job.
  - This means writes for a key cannot be serialized at a single location.
  - Better load balancing.
  - Usually a write follows a read, thus use the node which replied fastest the the previous read, as write coordinator.
    - Better chances getting read-your-write consistency.
    - Reduce variability in the performance, improve 99.9% performance.
Membership and failure detection

6 Experience and lessons learned

Main patterns in which Dynamo uses are as follows. W, R, N values can be tuned to achieve desired level of performance, availability and durability. Typically, N = 3. Common combination is N = 3; R = 2; W = 2.

Business logic specific reconciliation
Timestamp based reconciliation
- Last writes wins.
High performance read engine
- R = 1; W = N

Balancing performance and durability

Write latencies are usually higher than reads since a disk accesses are required. However, we can trade-off durability guarantees for performance. Write operations can be buffered at memory, which gets periodically written to storage. To reduce durability risk of this approach, the coordinator can choose 1 out of N node performing “durable write”. As the coordinator only waits for W responses, the write performance won’t be impacted.

Ensuring uniform load distribution

Dynamo assumes that when significant skews of access distribution happen, there are enough keys for popular end of the distribution thus load can be distributed uniformly.
Experiments show that imbalance ratio decrease with increase load.
- With high loads, a large number of popular keys are distributed uniformly.

The evolution of partitioning schemes:

Strategy 1: T random tokens per node and partition by token value
- Two consecutive tokens define a range.
- Cons:
  - When a new node joins, it needs to steal tokens from others.
    - Partitioning would be affected.
    - This needs scanning, which is inefficient.
  - When nodes join/leave, key ranges of many nodes change and Merkel tree need to be recalculated.
  - No easy way to snapshot entire key space.
Strategy 2: T random tokens per node and equal-sized partitions
- Hash space is divided into Q equally sized partitions/ranges and each node is assigned T random tokens.
- Q >> N; Q >> S*T, S is the number of nodes.
- Partitioning and partitioning placement are decoupled.
- This is an intermediate state migrating from 1 to 3.
Strategy 3: Q/S tokens per node, equal-sized partitions
- Each node handles Q/S tokens.
- When nodes join/leave, tokens are re-arranged to satisfy the invariant.

Evaluation:

Load balancing efficiency: ratio of average number of requests of all nodes to the maximum number of requests of the hottest node.
Experiment result of efficiency: 3 > 2 > 1.
Strategy 3
- Pros:
  - Size of membership information at each node is greatly reduced. (>1000X)
  - Faster bootstrapping/recovery: Partition ranges are fixed and stored as files, which can be transferred easily.
  - Easy archival: Entire dataset can be archived by archiving those partition files separately.
- Cons:
  - Changing node membership requires coordination in order to preserve the invariant.
    - (ST): Strategy 1 requires coordination for tokens too?

Divergent versions

Divergent versions happen in case of:
- failure scenarios;
- large number of concurrent writes to a single data item.
Experiments show that divergent versions are rarely created.

Client-driven or server-driven coordination

State machine can be move to client side. Advantages:

Load balancer is not needed.
- This also avoids the extra network hop for the load balancer.
Clients poll any Dynamo node for membership updates every 10 seconds.

Balancing background tasks

Background tasks such as replica synchronization and data handoff should run only when the regular critical operations are not affected significantly.
An admission control mechanism and resource monitoring are integrated to schedule background tasks.

Discussion

Note that Dynamo works for a system of couple of hundreds of nodes. If running with tens of thousands of nodes, the routing table (O(n) to system size) is hard to maintain. This may be overcome by introducing hierarchical extensions or DHT systems.

Read "Bigtable, A Distributed Storage System for Structured Data"

Tue, 05 May 2020 00:00:00 +0000

Reference

Link to paper

Abstract

Distributed storage for managing structured data.
Scale to petabytes of data across thousands of servers.
Satisfying different client demands such as data size, latency etc.

1 Introduction

Goals:
- Wide applicability, scalability, high performance and high availability.
Simple data model, not a full relational data model.
- Dynamic control over data layout and format.
- Clients can reason about locality properties.
Indexed using row and column names.
Data are treated as uninterpreted strings.
- Clients can serialize various data types into them.
Clients can control whether to serve data out of memory or from disk.

2 Data model

Bigtable is a sparse, distributed, persistent multi-dimensional sorted map, indexed by a row key, column key and a timestamp. The value is an uninterpreted string. (row:string, column:string, time:int64) -> string

A sample piece of data for web page is as follows. Row key is url; column key is content, anchor:cnnsi.com, etc.

2.1 Row

Size: up to 64 KB.
Each read/write under a single row key is atomic.
- No matter how many columns are involved.
Data are lexicographic ordered by row key.
Tablet: dynamically partitioned row range.
- Unit of distribution and load balancing.
Data access can gain good locality.
- i.e., use reversed URL.

2.2 Column families

Column families: grouped sets of column keys.
- Basic unit of access control: different user can read/write different groups individually.
- Basic unit of disk and memory accounting.
Data in a family are usually of the same type.
Number of column families should be small (up to hundreds) and rarely changed during operations.
Column key: family:qualifier
- In the example of web page data, anchor is a family.

2.3 Timestamps

Represents multiple versions of the same data.
Can be assigned by Bigtable or clients.
Decreasing ordered.
Versions can be garbage collected automatically.

3 API

Table-related
- Creates/Deletes tables and column families.
- Changes cluster, table and column family metadata, e.g., access control.
Values
- Writes/Deletes values.
- Looks up values from individual rows.
- Iterates over a subset of data in a table.
Other
- Single-row transactions.
  - e.g., read-modify-write.
- Batching writes across row keys.
- Cells can be used as integer counters.
- Execution of client-supplied scripts.
- Can be used with MapReduce, as inputs/outputs.

4 Building blocks

GFS: stores logs and data files.
Google SSTable is used to store data.
- Persistent, ordered, immutable map from key to values.
- A SSTable contains a sequence of blocks.
  - Block indexes are used to locate blocks. (Stored at the end of SSTable)
  - Data lookup: Load index into memory -> Find the block by binary search -> Read the block.
Chubby
- Highly available, persistent distributed lock service.
- Usages for Bigtable:
  - Ensure at most one active master.
  - Store bootstrap location of data.
  - Discover tablet servers and finalize tablet server deaths.
  - Store Bigtable schema information.
  - Store access control lists.

5 Implementation

3 major components: a library linked into every clients, one master server and many tablet servers.

Master is responsible for
- assigning tablets to tablet serves;
- detecting addition and deletion of tablet servers;
- load-balancing;
- garbage collection;
- schema changes.
Tablet server
- managers a set of tablets (10 - ~1000);
  - A tablet is a row range in the table, of size 100-200 MB by default.
- handles read and writes operations;
  - Clients talk with it directly, not via the master.
- splits tablets.

Tablet location

3-level hierarchy to store tablet location information. Locations are also cached on client library.

A Chubby file contains the location of a root tablet.
- Root tablet contains the locations of all tablets in a special METADATA table.
- Root tablet is the first tablet of METADATA table, but never split.
  - So no more than 3 levels.
METADATA table
- Contains locations of user tablets.
User tablets

Tablet assignment

Master keeps tracks of:

the set of live tablet servers
- Uses Chubby exclusive locks.
- If a tablet server is unavailable, the master can detect it and reassign its tablets.
  - By trying to acquire its lock.
current assignment of tablets to tablet servers, including unassigned ones
- Master needs to discover the assignments upon startup.
  - scans and communicates with all tablet servers.
  - scans METADATA table for all tablets.

Master initiates the creation, deletion of table and merging two tablets, thus keeps tracks of these changes regarding tablet assignments. Besides them, tablet split is initialized by tablet server. This change would be known by the master upon change committing or reading the split tablet.

Tablet serving

A commit log for redo records.
A memtable for recent updates.
A sequence of SSTable.

Operations:

Recover tablet: reads the indices of SSTables into memory and reconstructs the memtable by applying all updates since redo points.
Writes:
- Write to the commit log.
- Write to the memtable.
- Group commit is good for improving throughput of small mutations.
Reads: the server reads the merged view of the sequence of SSTables and memtable.

Compactions

Minor compaction
- Write the reach-threshold memtable into SSTable.
  - Reduce memory usage.
  - Reduce amount of data to read during recovery.
Merging compaction
- Merge a few SSTables and memtable into a new SSTable.
Major compaction
- Merge all SSTables into a new SSTable.
- Remove all deletion entries(tombstone) in old SSTables.
  - Tombstone of a record in a newer SSTable prevent the use of the record in a older SSTable.

Locality groups
- Groups multiple column families to use a separate SSTable, in a tablet.
  - Accessed together frequently.
- Can be declared to be in-memory.
Compression
- Can specify whether to compress and compression format for locality groups.
- Example compression format: 1st pass using Bently & McIlroy’s scheme; 2nd pass with a fast compression algorithm.
  - Often 10-to-1 reduction in space.
Caching for read performance
- Scan cache: higher-level caching the k-v pairs returned by the SSTable interface.
- Block cache: lower-level caching the SSTable blocks that were read from GFS.
Bloom filters
- Whether a SSTable might contain any data for a specified row/column pair.
Commit-log implementation
- Appends mutations to a single commit log per tablet server, for all tablets.
  - Avoids a large number of log file being written concurrently to GFS.
  - Recovery becomes complicated since tablets will be moved to many other tablet servers.
    - Master will initialize a sort of the log entries with respect with keys to tackle this.
- Tablet server will maintain two log writing threads. If one thread performs poorly, switches to the other one.
Speeding up tablet recovery
- The source tablet server with do minor compaction on tablets and the log to speed up.
Exploiting immutability
- All generated SSTables are immutable.
  - To delete data, use garbage collection on obsolete SSTable data.
  - Can split the tablet quickly. Child tablets share the SSTable of the parent tablet.
- Memtables are mutable; we use copy-on-write to reduce contentions during reads and reads and writes can be proceeded in parallel.

Performance evaluation

Performance doesn’t increase linearly with respect to the number of servers.
- There is a significant drop in per-server throughput.
  - Imbalance in load in multiple server configurations.
  - Network links are saturated.

Lessons

Systems are vulnerable to many types of failures, expected or unexpected.
Add new features until it’s clear how they would be used.
Proper system-level monitoring.
Value of simple design.

Read "Streaming Systems" 3, Watermarks

Sun, 03 May 2020 00:00:00 +0000

Watermarks

Definition

Assumption: Any event in the streaming data has an associated logical event timestamp.
- Can use the time of the original event’s occurrence.
Definition: The watermark is a monotonically increasing timestamp of the oldest work not yet completed.
- Completeness: Watermark allows to know when it’s correct to close a window.
  - If watermark has passed a time T, we are guaranteed by its monotonic policy that no more processing would occur for an event with event time at or before T.
- Visibility
  - Watermark cannot advance if an event is being stuck in the pipeline.
  - We can find the source of problem if this happens.

Source watermark creation

Watermarks can be perfect or heuristic.

Perfect watermarks account for all data.
Heuristic watermarks admit some late-data elements.

Perfect watermark creation

This requires perfect knowledge of the input, thus is not impractical for many real-world problems. Following examples can utilize perfect watermarks.

Ingress timestamping
- Use the ingress times as the event times fot data items.
Static sets of time-ordered logs.
- Just the minimum timestamp of unprocessed records across all partitions.

Heuristic watermark creation

Using heuristic watermarks can lead to late data, but it’s still possible to build a highly accurate heuristic watermarks. For example,

Dynamic sets of time-ordered logs
- By tracking the minimum timestamp for unprocessed records in known set, growth rates, external knowledge like network topology and bandwidth.
Google Cloud Pub/Sub
- No guarantee on in-order delivery.

There is no one-fits-all solution. At least, we simplify the problem of tracking completeness in a pipeline, into the problem of creating a watermark at the source.

Watermark propagation

A pipeline may have multiple independent stages. It’s meaningful to track individual watermarks for these stages. Note that for stages come later, their watermark is older than those of prior stages, since they have seen less records.

For a stage, defining watermarks at boundaries. We can calculate the amount of event-time latency/lag introduced by a stage.

Input watermark:
- minimum of the output watermarks of its upstream input/sources/stages.
Output watermark:
- minimum of its input watermark and event times of all non-late active messages within the stage.

Within a stage, processing is not monolithic, as different components/buffer may exist and they can have their own watermarks as well. Therefore, the output watermark of the stage may be the minimum of:

Per-source watermark: for each sending stage.
Per-external input watermark.
Per-state component watermark: for each type of state that can be written.
Per-output buffer watermark: for each receiving stage.

Watermark propagation and output timestamps

Within a stage, processing can be divided into windows in event-time domain. The output timestamp of each window advances the output watermark of this stage. We have several choice for the output timestamp for a window:

End of the window
- allows the smoothest watermark progression.
Timestamp of the first non-late element
- Most conservative
- Watermark progression may be likely delayed.
  - Output watermark is held until that window processing is complete.
Timestamp of a specific element
- For some special use cases, it may make sense.

The tricky case of overlapping windows

If an element is in 3 overlapping windows, when 1st stage of 1st window complete, the input watermark of the 2nd stage of the 1st window is held by the output watermark of the 1st stage of the 2nd and 3rd windows. This delay is unnecessary. Apache Beam has a special logic for this: output timestamp of N+1 window is always greater than that of N window.

Percentile Watermarks

Instead of using the minimum of the event timestamps of active messages, we can consider the entire distribution of event timestamps and use percentile watermark.

A 90-percentile watermark means we are guaranteed to have processed this percentage of all events with earlier timestamp.
Avoids being delay by outliers.
A good way to tune the trade-off between latency of materializing results and precision of results.

Processing-time watermarks

Event-time based watermark is unable to distinguish delays caused by old data or a delayed system (processing being stuck).
Processing-time watermark is the processing-time timestamp of the oldest active message.
Can be used at system-implementation level for tasks such as garbage collections.

Case studies

Watermarks in Google Cloud Dataflow

Watermarks in Apache Flink

Source watermarks for Google Cloud Pub/Sub

Read "Streaming Systems" 1&2, Streaming 101

Sat, 07 Dec 2019 00:00:00 +0000

Streaming 101

What is streaming?

Streaming system: A type of data processing engine that is designed with infinite datasets in mind.

Two dimensions to describe a dataset:

Cardinality
- Dictates the dataset size.
- Bounded/unbounded.
Constitution
- Dictates the physical manifestation: the ways one can interact with the data in question.
  - Table
    - A holistic view of a dataset at a time point.
    - SQL
  - Stream
    - Element-by-element view of the evolution of a dataset over time.

Greatly exaggerated limitations of streaming

Streaming systems have historically been relegated to a somewhat niche market of providing low-latency, inaccurate or speculative results. (Lambda architecture)
Kappa architecture: Run a single pipeline that can replay.
A well-designed streaming system actually provide a strict superset of batch functionality. (Apache Flink)

Streaming systems need two things to beat batch ones:

Correctness
- Boils down to consistent storage.
- Streaming systems need a method for checkpointing persistent state over time, which should remain consistent in light of failures.
  - Strong consistency is required for exact-once processing.
    - (ST) Strong consistency -> unique ID -> exact-once -> correctness.
Tools for reasoning about time
- Essential for dealing with unbounded, unordered data of varying event-time skew.

Event time vs. processing time

Two domains of time:

Event time
- When the event actually occurred.
Processing time
- When the event is observed in the system.

The skews between event time and processing time are often highly variable.

Processing-time lag is always equal to event-time skew.

Windowing:

By processing time: not quite make sense if we care about event time, since some events may fall into wrong processing-time window due to skew.
By event time: Need a way to reason the completeness, how we can determine when we have observed all events in a window.

Data processing patterns

Bounded data

The overall model is quite simple, using engines like MapReduce.

Unbounded data: batch

Batch engines can be used to process unbounded data by windowing.

Fixed windows
- Process each window as a separate, bounded data source.
- Completeness problem: Some events may be delayed.
Sessions
- A period of activity terminated by a gap of inactivity.
- Not ideal since a session may be split across batches.

Unbounded data: streaming

Unbounded data sources in reality may be high unordered with respect to event times, or of varying event-time skew. 4 groups of approaches can help:

Time-agnostic
- Used when time is essentially irrelevant.
- Engines just need to support basic data delivery.
- Examples
  - Filtering, e.g., web log for a specific domain.
  - Inner joins: buffer the inner source and check the events from the other source.
    - May be time-related if we want garbage collection.
Approximation
- Algorithms: approximate Top-N, streaming k-means etc.
- Designed to be low overhead but limited and often complicated.
- Usually processing-time based and provides some provable error bounds on the approximations.
(Windowing basics)
- Fixed windows, tumbling, no overlap
- Sliding windows, hopping, with overlap
- Sessions
Windowing by processing time
- Nice things
  - Simple.
  - Judging window completeness is straightforward.
  - Exact what we want if we want to infer information as the data is observed. e.g., monitoring.
- Downside
  - Cannot reflect the reality in event time domain. Processing-time order doesn’t respect event-time order.
Windowing by event time
- Supports for it has been evolving: Flink -> Spark -> Storm -> Apex.
- Advantages
  - Gold standard of windowing.
  - Correctness
  - Dynamically sized windows like sessions.
- Problems
  - Buffering: more buffering is required
    - Good things
      - Persistent storage is cheaper now.
      - Update can be incremental, without buffering entire input set.
  - Completeness
    - No good way of knowing when we’ve seen all of data for a given window.
    - Solution: Use heuristic estimate like watermarks (MillWheel, Cloud Dataflow, Flink).
      - If we need absolute correctness, The only real option is let the pipeline builder to express when to materialize.

The What, Where, When and How of data processing

Roadmap

Five main concepts for this chapter:

Processing time vs. event time
- Use event time if we care about correctness and when things actually occur.
Windowing
- Fixed, sliding, sessions.
Triggers
- A mechanism for declaring when the output for a window should be materialized relative to some external signal.
- Can have multiple triggers to observe the output for a window multiple times as it evolves. (early update, processing late data)
Watermarks
- A notion of input completeness with respect to event times.
- Statement for a watermark with value of time X: All input data with event times less than X have been observed.
Accumulation
- Specify the relationship between multiple results that are observed for the same window.
- Different accumulation modes are available to different use cases.

Critical questions to ask for every unbounded data processing problem:

What results are calculated?
- e.g., sum, histograms, training ML models.
Where in event time are results calculated?
- Event-time windowing.
When in processing time are result materialized?
- By the use of triggers and (optionally) watermarks.
- Repeated update triggers vs watermarks.
How do refinement of results relate?
- By the types of accumulation used: discarding, accumulating, accumulating & retracting.

Batch foundations

What: transforming

Some Apache Beam primitives:

PCollections: Datasets across which parallel transformations can be performed.
PTransforms: Can be applied on PCollections to create new PCollections.
- Element-wise transformation.
- Group/aggregate.
- Composite combination of other PTransformations.

Where: windowing

Example Beam code of windowed summation:

Pcollection<KV<Team, Integer>> totals = input
    .apply(Window.into(FixedWindows.of(TWO_MINUTES)))
    .apply(Sum.integersPerKey());

Going Streaming

We would like to have lower latency and natively handle unbounded data sources.

When: triggers

Triggers declare when output for a window should happen in processing time. Two useful types of triggers:

Repeated update triggers
- Periodically generate updated panes for a window as its contents evolve.
- Can be materialized with every new record or after some processing-time delays, like once a minute.
- The most common, simple to implement and understand.
- Choose processing-time delay:
  - Aligned delays
    - Processing time domain is spliced into fixed regions by the delays.
    - Get regular updates across all modified windows at the same time.
      - Good for predictability.
      - Bad for causing bursty workloads.
  - Unaligned delays
    - The delay is relative to the data observed within a given window.
    - Better for large-scale processing: Spread the load out more evenly across time.
Completeness triggers
- Output only after the input for that window is believed to be complete to some threshold.
- Closely align with batch processing. We would need a way to reasoning about completeness, thus watermarks sound like a better idea.

When: watermarks

Watermark are temporal notions of input completeness in the event-time domain, to better support answering “When in processing time are result materialized?” Watermark can be modeled as a function: F(P) -> E, which takes a processing time P then returns a event time E. At this point, all events with event time less than E have been observed. Watermarks can be perfect (if we have enough knowledge about the input source) or heuristic (could be remarkably accurate in many cases, but may cause late data sometimes). It forms the foundation of completeness triggers.

Advantages
- A way to reason completeness. e.g., in outer joins, emit a partial join record instead of keep waiting.
Shortcomings
- Too slow
  - A watermark could be delayed due to known unprocessed data.
  - Not ideal from a latency perspective.
  - Perfect watermarks still suffer from this.
- Too fast
  - A heuristic watermark is incorrectly advanced earlier than it should be, creating late data.

Conclusion is we cannot get both low latency and correctness based on merely completeness trigger. Combination of repeated update triggers and watermarks sounds a better idea.

When: early/on-time/late triggers

The combination of repeated update/watermark triggers partitions the panes to materialize into 3 categories:

Zero or more early panes
- Repeatedly update until the watermark passes the end of window.
- Prevent being too slow.
A single on-time pane
- Fire when the watermark passes the end of window.
- Safe to reason about missing data.
Zero or more late panes
- Fire for late data.
- Compensate being too fast.

Window lifetime bounds: For heuristic watermarks, we need to keep the state of the window for some duration even after the watermark has passed. To determine how long we need to keep around for each window, allowed lateness comes in.

When: allowed lateness (garbage collection)

A real-world system needs to bound the lifetime of the windows, defining how late (relative to watermark) a piece of data could be. It would drop data arriving too late since no one cares about it. It’s a kind of garbage collection.

Using processing time is easy but vulnerable to issues within the pipeline itself (e.g., worker crashing). We should specify the horizon in event-time domain, which directly ties with the actual progress of the pipeline.

Different types of watermarks:

Low watermarks
- Pessimistically attempt to capture the event time of the oldest unprocessed record the system is aware.
- Resilient to changes in event-time skew.
High watermarks
- Optimistically track the event time of the newest record the system is aware of. (maximum event-time skew)

(ST) e.g., We have a event-time window [0, 3] and unprocessed event 1, 2, 4. Low watermark would be 1, thus 1 and 2 will be processed for the window. High watermark is 4, which passes the end of window thus causes missing 1 and 2.

If we have perfect watermarks, no need to care about late data sine there will be none. For cases that we do global aggregations over a finite number of keys, we don’t need to specify lateness horizon; no need to do garbage collection.

How: accumulation

3 modes of accumulation:

Discarding
- Every time a pane is materialized, any stored state is discarded. Panes are independent.
- e.g., each pane represents delta. Downstream is required to do their own summation.
Accumulating
- Future inputs are accumulated into the existing state.
- e.g., Pipeline does the sum itself.
Accumulating and retracting
- When produce a new pane, also produce an independent retraction for previous panes.
- Meaning “I previously told you X, but I was wrong. Get rid of the X I told you last time, and replace it with Y”.
- Two cases when it’s useful:
  - Consumers downstream are regrouping data by a different dimension, thus a new value for a key may be put in a different group.
  - Dynamic windows: the new value might be replacing more than one previous window.

Home network, media center setup

Wed, 13 Nov 2019 00:00:00 +0000

Background

Recently, I set up my home network for digital content sharing. I will take some notes here for future reference.

Devices

Router: Netgear Nighthawk AX8 AX6000
- Supports Wifi 6, MU-MIMO, link aggregation (802.3ad).
NAS: Synology DS418play
- 4-bay, 2GB RAM, 1Gbps ports, supports 4K transcoding.
- But doesn’t support Dock natively.
HDD:
- WD Red 4TB * 3
- WD Red 14TB * 1
UPS: APC Back-UPS 650VA UPS
4K player for TV: Nvidia shield pro

Upgrades in 2022

Unifi Dream Machine Pro
- Gateway, NVR
Unifi Switch Pro 24 PoE
- POE
- Level 3 switching
A bunch of Unifi G4 cameras for surveilliance.
Ziddo Z9X player

Setup

Router and NAS setup are pretty straightforward.
- One RAID 1 (4TB) to store documents I can’t lose, photos etc. Can use HyperBackup to sync it to a cloud drive as well.
- One RAID 0 (18TB) to store downloaded BT contents.
Link aggregation
- Needs to configure this on both router and NAS.
- Synology doc
Time machine backup for macOS
- Synology doc
The default file transfer protocol between macOS and NAS is SMB. Here are some tips to optimize it.
- Apple support doc
- Turn off packet signing
- Result: Read speed of a 1.48GB file increased from 58 MB/s to 95 MB/s.
UPS: just connect then enable it in Hardware settings.

Synology NAS

I tried to manually configure DNS server to 8.8.8.8 in Control Panel -> Network for external access. But this setting seemed to cause some interruptions for DSM connection. Also, transmission jobs experienced frequent 0-speed time. It may not be a good idea.
Move installed packages between volume: unofficial tricks.

BT

I found the official Download station package in DSM is not quite optimized for BT downloading. Transmission is a better choice. Here are steps to install and setup in Synology DSM.

Add “synocommunity” in Package Center -> Settings -> Package Sources.
- Location: http://packages.synocommunity.com
Set Trust Level to “Synology Inc. and trusted publishers” in Package Center -> Settings -> General.
Install “Transmission” in Package Center -> Community.
Create folders to be used.
- “download” for completed downloads.
- “watch”: Transmission will start a download task automatically once a torrent file is added here.
- “incomplete” for downloading files.
Start the installation in Package Center; fill in all the folder information and credentials.
Set up Read/Write permission for Group “sc-download” for those folders in Control Panel -> Group -> sc-download -> edit -> Permissions.
All done! View the web UI at http://your_nas_ip:9091
- The package can be started or stopped in the Package Center.
- Available GUI: transgui.

Kodi & Emby

Install emby-next-gen add-on following Emby doc.
Use Kodi native play mode instead of add-on mode:
- Set up optional shared network folder in emby.
  - e.g., map volume1/video to smb://<nas_ip>/video thus network devices can recognize the path.
- Input your credentials at kodi: File manager -> Add source -> Add a network location.
- Change the emby settings in kodi.
  - Need to turn off forced HTTP playback as well.
Cannot import external subtitles
- Make sure the subtitle file has the same name as the video file.
Garbled Chinese subtitles
- Settings -> Player -> Language -> Subtitles: Change the language, font and the Character set.
Download subtitles automatically
- Settings -> Player -> Language -> Download services: Use OpenSubtitles.org for both default services. Input your credentials for it.
If weird things happened (e.g., no playback), you may wanna reset local database.
M is the shortcut for subtitle menu, etc.
For Zidoo player, need to install ZDMC (a Kodi fork) for this setup. (apk download)

Zidoo

While playing some videos (perhaps Dolby Vision/Atmos), I experienced some stutters every a few minutes. Found a solution on zidoo forum:
Settings -> advanced settings -> Realtek developer options
- Switch “Playback with less reserved buffers” off.
- Enable “performance mode”
- Enable “playback performance mode”

Read "F1, a distributed SQL database that scales"

Mon, 05 Aug 2019 00:00:00 +0000

Reference

Link to paper

Abstract

Distributed relational database
Filial 1 hybrid, of
- System community: high availability, scalability.
- Database community: consistency and usability.
  - SQL query, automatic change tracking and publishing.
On Spanner: synchronous replication and strong consistency.
- Higher commit latency
Mitigate high latency
- Hierarchical schema model with structured types.
- Smart application design.

Introduction

Both OLTP and OLAP database for AdWords system.
- Adwords as of 2013:
  - 100s applications; 1,000s users
  - 100TB data
  - ~1M QPS
  - SQL queries scan O(10^13) rows per day.
  - Availability: 99.999%
  - Not higher latency compared with the old MySQL backend.
- Some other applications use it as well.
Built to replace old MySQL backend.

Key goals of F1’s design are as below. These goals are mutually exclusive, but F1 achieved them with trade-offs and sacrifices.

Scalability
- Scale up trivially and transparently.
Availability
- Must never go down for any reason.
Consistency
- ACID transactions.
Usability
- SQL query and other SQL features like indexes and ad hoc query.

F1 inherits features from Spanner and adds more:

Distributed SQL queries
Transactionally consistent 2nd indexes
Asynchronous schema changes
Optimistic transactions
Automatic change recording and publishing

To hide the high latency brought by the design, some techniques are developed:

F1 schema makes data clustering explicit.
- Using hierarchical relationships and columns with structured data types.
- Result: Improve data locality; reduce number of internal RPCs to read remote data.
F1 users heavily use batching, parallelism and asynchronous reads.
- A new ORM (object-relational mapping) library makes these explicit.

Basic architecture

F1 architecture:

F1 severs: receives clients’ read and write requests.
- Load balancer may choose a server far away from clients in cases of high load or failures.
- F1 servers usually co-locate with Spanner servers, but may contact other Spanner servers when necessary.
- F1 servers are usually stateless, unless a pessimistic transaction is in process and server must hold locks.
  - A client can contact different servers for different requests.
- Servers can be quickly added or removed and require no data movement.
  - Adding or removing Spanner servers requires data movement but the process is transparent to F1.
Slave pool: for distributed SQL queries.
- Distributed execution is chosen when the query planner finds that increased parallelism will reduce latency.
F1 master: manage the membership of slave pool.
Support MapReduce framework, talking to Spanner directly.

With this architecture, throughput can be scaled up by adding more Spanner servers, F1 servers or F1 slaves. Since the servers are widely distributed, the commit latencies are relatively high (50 - 150 ms).

Spanner

Check out this note.

Data model

Hierarchical schema

Originally, Spanner used a Bigtable-like model; but later Spanner actually adopts the hierarchical logical data model of F1. (Directory table, child tables, etc) F1 schema supports explicit table hierarchy and columns with Protocol Buffer data types.

Child tables are clustered with and interleaved within the rows from its parent table.
Child and grand child table share the same primary key prefix with the root table row.
A root row in root table and its hierarchy form a Spanner directory.
- In F1, make Customer (Advertiser) a root table.
Child rows are stored under their parent row ordered by primary key.

Hierarchy example:

Customer 1
  Campaign 1 1
    Ads 1 1 1 
    Ads 1 1 2
  Campaign 1 2 (interleaved; doesn't stay next to Campaign 1 1)
    Ads 1 2 1
Customer 2
  ...

Advantages:

Can fetch Campaign and Ads records (without knowing the Campaign ID) in parallel.
Use a single range read to get all Ads under a Customer.
- Don’t need indexes.
Since records are ordered, can join two tables with a simple ordered merge.
Reduce the number of Spanner groups involved in a transaction.
- Application developers should try to use single-root transaction as much as possible.
  - If must use multiple roots, we should limit the number of roots involved to reduce 2PC cost.

Protocol buffers

Columns can be Protocol buffers, a structured data type.
Can use it across storage and code.
repeated fields can be used to replace child tables.
- Number of records in a such field is limited.
- Reduce overhead and complexity to store and join child tables.
- Easier for users to use the object as an atomic unit.

A single protobuf column or multiple columns?

Many tables consist of a single protobuf column.
Others has multiple columns.
- Split by:
  - grouping fields usually accessed together.
  - static vs. frequently updated data.
  - different read/write permissions.
- Allows concurrent updates to different columns.
Using fewer columns generally reduces performance overhead.

Indexing

All indexes are transactional and fully consistent.
Store: index key -> primary key of the indexed table.
- Type:
  - Local
    - The index key contains the root row primary key as a prefix.
    - Stored in the same directory as the root row.
    - Updating cost is little.
  - Global
    - The index key doesn’t contain the root row primary key.
    - Updating is often large and rate is high.
      - Thus sharded into many directories.
    - Use it sparingly due to its cost and try to use small transaction if must do it.

Schema changes

AdWords system requires F1 to be highly available; schema changes should not lead to downtime or table locking.

Challenges:

Servers distributed in multiple geographic regions.
F1 servers have schema loaded in memory; not practical to update atomically across all servers.
Queries and transactions must continue on all tables even with ongoing schema changes.
Schema changes should not impact availability and latency.

F1 chooses to do schema changes asynchronously. Servers may update the database using different schema. Note: Why not using Spanner style? Assign a future timestamp to the change. Maybe because it will block some transactions requiring the new change.

Problem: If two F1 servers update the database with different schemas (not compatible), it will lead to anomalies like data corruption.

Example of data corruption: A schema change from schema S1 to S2, adding a new index I on table T.

Server M1 uses S1; M2 uses S2.
M2 inserts a new row r, and adding index Ir.
M1 deletes row r, leaving index Ir alone since it is not aware of it.
Index scan on I will return spurious data on r.

Algorithm to prevent such anomalies:

Enforce that at most two schemas are active across all servers.
Subdividing each schema change into multiple phases where consecutive ones are mutually compatible and cannot cause anomalies.

The example can be divided as:

Add index I and only allows delete operations on it.
Upgrade I to allow write operations on it.
Perform Offline job to backfill index entries to all rows.
- Need to carefully handle concurrent writes.
Once completed, make I visible to all read operations.

Full details are in this paper (Online, asynchronous schema change in F1).

Transactions

F1 must provide ACID transaction feature. In contrast, eventual consistent systems add a lot of burden on application developers. We should support full transactional consistency at database level.

3 types of F1 transactions:

Snapshot transaction
- Read-only
- Use Spanner snapshot timestamp.
- Default current (when timestamp is not provided)
Pessimistic transaction
- Map to Spanner transactions,
- Can use shared or exclusive locks.
Optimistic transaction
- Use last modification timestamp (LMT) on each row, stored in a hidden lock column.
  - Updated in every F1 writes.
- Check the LMT of all already read rows while trying to commit.
- Default in F1.

Advantages of optimistic transactions:

Tolerate misbehaved clients since not holding locks.
Support long-lasting transactions, e.g., with user interactions.
Easy to retry transparently on server-side.
Client can retry on different F1 servers since optimistic transactions are server-stateless.
- When servers failed or needs load balancing.
Speculative writes.
- Read values from outside; also requires their LMT.

Disadvantages of optimistic transactions:

Insertion phantoms
- LMT only exists on existing rows.
- Can use parent-table locks to avoid this.
Low throughput under high contention
- When many clients update concurrently.
- Use pessimistic ones or do batching updates.

Flexible locking granularity

Row-level locking by default.
Support column-level locking per row.
- Can update different columns on one row concurrently.
Can also selectively reduce concurrency.
- Use a lock column in a parent table to cover columns in a child table.
  - Avoid insertion phantoms.

Change history

Change history is a first-class feature and guaranteed full coverage.
Enabled by default, but can opt out some tables or columns.

How:

Every transaction creates one or more ChangeBatch protobuf.
- Primary key is the root table key + transaction commit timestamp.
  - Thus is a child table of the root row, stored in commit order.
- Stores the before and after values for each updated row.
- If multiple root rows are involved, create the protobuf for each root row.

Applications:

For pubsub.
For cache.
- If cache is stale, do incremental updating starting from the checkpoint using change history.

Client design

Simplified ORM
- Expose APIs for parallel and asynchronous read access.
- Avoid anti-patterns:
  - Serial reads, implicit traversals etc.
NoSQL interface
- A simple KV based interface for reads/writes.
- Can batch retrieval of rows from multiple tables in a single call.
SQL interface
- Supports from low-latency OLTP to large OLAP.
- Can join data from outside sources, like Bigtable, CSV files, etc.

Query processing

Some key properties of the query processing system:

Queries are executed as either low-latency centrally executed ones or distributed ones with high parallelism.
- Centrally executed queries: OLTP-style.
- Distributed queries: OLAP-style.
  - Use snapshot transactions (read-only).
Data is remote and batching is used heavily to mitigate network latency.
- Example: Lookup join
  - Load 50MB data from one table then lookup in the other table.
All input and internal data is arbitrarily partitioned and has few ordering properties.
Many hash-based repartitioning steps.
Individual query plan operators are designed to stream data to later operators as soon as possible.
- Reduce pipeline stalls.
- This decision limits operators’ ability to preserve interesting data orders.
- Maximize pipelining, read request concurrency.
- Reduce memory usage for buffering temporary data.
Optimized access on hierarchically clustered tables.
Query data can be consumed in parallel.
First-class support for structured data types, by protobuf-type columns.
Snapshot consistency provided by Spanner.

Distributed query example

SELECT agcr.CampaignId, click.Region,
      cr.Language, SUM(click.Clicks)
FROM AdClick click
  JOIN AdGroupCreative agcr
    USING (AdGroupId, CreativeId)
  JOIN Creative cr
    USING (CustomerId, CreativeId)
WHERE click.Date = '2013-03-23'
GROUP BY agcr.CampaignId, click.Region,
        cr.Language

AdGroup: a collection of ads with some shared configuration.
Creative: actual ad text.
AdGroupCreative: a table of foreign keys linking AdGroup and Creative.
AdClick: records the AdGroup and Creative.

A possible query plan:

Steps:

Scan AdClick into a lookup join operator and do the join.
- The operator looks up AdGroupCreative using secondary index key.
Repartition data stream by hashing with CustomerId and CreativeId.
Distributed hash join with Creative using the same keys.
Repartition again by hashing with GROUP BY keys (CampaignId, Region, Language).
Aggregate using aggregation operator.

Distributed execution overview

A distributed query plan may consist of tens of plan parts, forming a directed acyclic graph. The data flows up from the leaves to a single root node (query coordinator). Query coordinator is the server which received the initial request; it plans the execution and processes results to return to the client.

Co-partitioning of the stored data is helpful to push down large amount of query processing to nodes hosting the partitions. F1 cannot utilize this since all data are remote and Spanner does random partitioning. To allow efficient processing, F1 re-partitions data (hash partition). It requires heavy network traffic and hence limits the size of F1 cluster due to the network switch hardware. However, both are not causing problems.

F1 operators execute in memory without checkpointing, stream results as much as possible. Therefore, a single server failure will fail entire query.

Hierarchical table joins

Child table entries are interleaved in the parent table; thus a single Spanner request can work to join a parent table with its child table. Cluster join (like merge join) is efficient, which buffers one parent entry and one child entry.

Parent(3)
  Child1(3, 1)
  Child1(3, 2)
Parent(4)
  Child1(4, 1)

However, a single Spanner request doesn’t work to join two sibling child tables. (Note: How are sibling child tables interleaved in parent table?) Need to perform one cluster join first (Parent, Child1), then another join (join1 result, Child2? What primary key to use? parent root key?).

Partitioned customers

A single query coordinator or a single client process may be a bottleneck.
- It may receive results from many servers in parallel.
Clients can ask F1 for distributed data retrieval.
- F1 returns a set of endpoints to connect to.
- A slow reader may block entire processing.
  - F1 processes results for all readers in lock-step.

Queries with protobuf

protobuf can be used in F1 SQL path expressions like WHERE c.Info.country_code = 'US'.
F1 can query and pass entire protobuf like SELECT c.Info
Can access repeated fields with implicit join PROTO JOIN.
Disadvantages:
- Performance implications due to fetching and parsing entire protobuf, even if we only need one field.
  - Can improve by pushing the parsing and field selection to Spanner.

Deployment

5 datacenters across mainland US.
- 5-way Paxos replication.
  - 3-way is not enough: If one failed, a single machine failure may fail the second one then the system is unavailable.
Additional read-only replicas for snapshot reads.
Put clients with heavy modifications near leaders.
- User transactions require at least 2 round trips to the leader.
  - One for read, one for commit.
  - Maybe another read as part of a transaction commit.
2 at east coast; 2 at west coast; 1 centrally.
- Leader at east cost.
  - Round trip to the other datacenter at east cost and central one accounts for 50 ms minimum latency.
    - Paxos voting process is improved since majority 3/5 is required.

Latency and throughput

Read latencies: 5-10 ms.
Commit latencies: 50-150 ms.
Multi-group transaction latencies: 100-300 ms since requiring 2PC.
User-facing latencies for AdWords interactive application: 200 ms.
- Achieved much by avoiding serial reads.
- Tail latencies are much better than MySQL.
Non-interactive bulk updates:
- Optimize for throughput instead of latency.
- Do small transactions (only one directory involved) in parallel.
Query processing can be speeded up linearly with more resources.
Resource cost: CPU 1 order higher than MySQL
- Need to decompress disk data, process then recompress and send over the network.

Hybrid of relational and NoSQL.
Optimistic transactions.
Asynchrony in query processing.
MDCC (Multi-datacenter consistency) Paxos optimizations.
Protocol Buffers.

Conclusion

Highly scalable, available.
High throughput.
ACID transactional guarantee.
SQL query.
Rich column types (protobuf).

Read "Zanzibar, Google’s Consistent, Global Authorization System"

Sun, 28 Jul 2019 00:00:00 +0000

Reference

Link to paper, slides

Abstract

Zanzibar is a consistent, global authorization systems used by Google Calendar, Cloud, Drive, Maps, Photos and Youtube. It scales to billions of users, trillions of ACLs (access control list) and millions of QPS. It maintains 95th latency of less than 10 milliseconds and 5-9s availability over 3 years of production use.

Introduction

Zanzibar is a unified system that allows inter-application usages.

Goals for Zanzibar:

Correctness
- Consistency
Flexibility
- Support a rich set of access control policies.
Low latency
High availability
Large scale

Zanzibar features:

A simple data model and a powerful configuration language
- User can define arbitrary relations between users and objects, like owner.
- Support set-algebraic operators such as intersection and union.
ACL: user U has relation R to object O.
- U can also be a user group, referring to another ACL.
  - Commenter is also viewer.
- User group can contains other user groups. Nested groups may lead to long chain for ACL checks.
- An ACL check request may fan out to multiple servers.
All ACLs are globally replicated to server large QPS quickly.
Global consistency enabled by Spanner.

For low latency and high availability:

Its consistency protocol allows the vast majority of requests to be served with locally replicated data, without requiring cross-region round trips.
Handle hot spots by caching final and intermediate results, deduping simultaneous requests.
Handle deeply nested sets by hedging request and optimizing computations.

Model, language and API

Relation tuples

In Zanzibar, ACLs are represented as relation tuples.

(tuple) ::= (object)#(relation)@(user)
(object) ::= (namespace):(object_id)
(user) ::= (user_id)|(userset)
(userset) ::= (object)#(relation)

(userset) allows us to refer a group to support nested group membership.

Examples of tuples:

doc:readme#owner@10
group:eng#member@11
doc:readme#viewer@group:eng#member
- Group eng’s members are viewers of readme.

Consistency model

“new enemy problem”, considering a scenario:

Alice removes Bob from the ACL of a folder.
Alice adds a new content into the folder.

Problems:

Fail to preserve ordering of ACL changes.
- Wrong order will let Bob see the new content.
Misapply old ACL to new content.
- If ACL check is evaluated on the old ACL, Bob will see the content.

Features Zanzibar provided to avoid such problems are as below. Both are enabled by Spanner (my note).

External consistency
- Ensure casual ordering.
Snapshot reads with bounded staleness
- Won’t see stale results.

Zookie:

Zanzibar zookies are basically Spanner timestamp assigned to the ACL update.
Write path
- When the content modification is about to be saved, the client requests a zookie for each content version.
  - Via a content-change ACL check.
    - Does not need to be in the same transaction as the content change. (Note: Why?)
- The client stores the content change and the zookie in an atomic write.
Check path
- The client sends this zookie in subsequent ACL check requests to get fresh results compared with the update.

Advantages of using zookies:

Most of checks use default staleness with already replicated data.
- Low latency.
Free to choose even fresher snapshot.
- To avoid hot spots.
- High availability.

Namespace configuration

Namespace configuration specifies:

Relations
- e.g., viewer, editor
Storage parameters
- Sharding setting
- Encoding for object ID like string, integer.

Relation configs and user rewrites

Clients can define object-agnostic relationships via userset rewrite rules in relation configs. The userset rewrite rules are defined per relation. It can be used to express this relation with other relations. The expressions can be union, intersection and exclusion.

The userset rewrite rule specifies a function with an object ID as input and a userset expression tree as output. The leaf nodes of the tree can be:

_this
- All users from tuples for <object#relation> pair.
- Including indirect ACLs by usersets from tuples.
- Default when no rules specified.
computed_userset
- Compute a new userset for the input object.
- e.g., viewer relation can refer to editor relation.
tuple_to_userset
- Compute a tupleset from the input object; fetch relation tuples matching the tupleset; then compute a userset from every fetched relation tuple.

An example of relation config, expressing that viewer contains editor and viewer from parent folder; editor contains owner.

name: "doc"

relation { name: "owner" }

relation {
 name: "editor"
 userset_rewrite {
  union {
   child { _this {} }
   child { computed_userset { relation: "owner" } }
}}}

relation {
 name: "viewer"
 userset_rewrite {
  union {
   child { _this {} }
   child { computed_userset { relation: "editor" } }
   child { tuple_to_userset {
     tupleset { relation: "parent" }
     computed_userset {
       object: $TUPLE_USERSET_OBJECT  # parent folder
       relation: "viewer" 
    }}}
}}}

API

Read
- A request specifies one or more tuplesets and zookie.
- tupleset:
  - A single tuple key or all tuples with a a given object ID or userset in a namespace.
  - Can be optionally constrained by a relation name.
- Response
  - look up a specific membership entry;
  - read all entries in an ACL or group;
  - read all groups with a given user.
- Note Read doesn’t reflect userset rewrite rules, thus only retrieve direct users or objects.
  - Use Expand to get effective ACLs.
Write
- Can modify single relation tuple.
- Can also modify all tuples related to an object via a read-modify-write cycle.
  - Steps:
    1. Read all relation tuples of an object, including a per-object lock tuple.
    2. Generate tuples to write or delete. Send the writes along with the lock tuple.
    3. If the lock tuple is not modified, commit. Otherwise, back to step 1.
- A write issues a content-change check against the latest snapshot.
  - If authorized, Check will response with a zookie, to be sent back in the write response to the client.
Watch
- Request: one or more namespaces, zookie representing the time starting to watch.
- Response: all tuple modification events in time order and a heartbeat zookie (representing the end time).
- Clients can use the heartbeat zookie to send a new request to resume watching.
- Can be used to maintain secondary indexes on client side.
Check
- Request: a userset <object#relation>, a putative user and optional zookie.
Expand
- Get the effective userset given an <object#relation> pair and an optional zookie.
- Follow indirect references expressed through userset rewrite rules.
- Result: a userset tree
  - Leaf nodes: user IDs or usersets pointing to other <object#relation> pairs.
  - Intermediate nodes: expressions like union, intersection or exclusion.

Architecture and implementation

aclservers
- Serve Check, Read, Expand, Write requests.
- Organized in clusters.
- The initial server may fan out work to other servers; other server may contact another server as well.
- The initial server gathers final result and sends it back to the client.
Storage
- ACLs and metadata are stored in Spanner.
- One database per namespace.
- One database stores all namespace configurations.
- One changelog database shared across all namespaces.
watchservers
- Serve Watch requests.
- Tail the changelog and serve a stream of changes.
Offline pipelines
- e.g., dumps, garbage collection on old versions.
Leopard indexing system
- Used to optimize operations on large and deeply nested sets.

Storage

Relation tuple storage:

One database per namespace.
Primary key to identify a row:
- shard ID, object ID, relation, user, commit timestamp.
Shard ID
- Usually determined by object ID solely.
  - If a namespace stores groups with very large numbers of members, shard ID can be computed from object ID and user.

Changelog:

Primary key:
- changelog shard ID, timestamp, unique update ID.
- Shard is randomly selected for each write.
Relation tuple and changelog are both updated in a single transaction.
Use the Spanner server hosting changelog shard as the transaction manager.
- To minimize blocking of changelog reads on pending transactions.
  - Note: Save the time for sending commit timestamp to non-coordinator participant leaders in 2PC.

Namespace config storage:

Two tables:
- One stores the configs keyed by namespace IDs.
- The other is a changelog of config updates, keyed by commit timestamps.
Zanzibar servers can load all configs upon startup and monitor the changelog to refresh configs.

Replication:

Data is replicated to be close to clients.
- Dozens of locations around the world.
5 voting replicas in eastern and central US.
- in 3 metropolitan areas to isolate failures;
- within 25 ms of each other to allow Paxos transactions commit quickly.

Serving

Evaluation timestamp

Zanzibar always respects zookie if provided. Zanzibar also does statistics on out-of-zone reads probability. Default staleness bound will be updated to a safe value according to the result. Use default staleness for requests without zookies. The signal of whether it’s out-of-zone is sent back by Spanner.

Config consistency

Zanzibar chooses a single timestamp for the config to evaluate the same request on all servers. This timestamp is chosen from a range in which all timestamps are available to all servers.

This range is maintained by a monitoring job.
Each server load config changes continuously.

Check evaluation

Zanzibar converts check requests to boolean expressions. A simple example without userset rewrite rules:
$\\$
$CHECK(U, ) = \\ \hspace{2cm} \exists tuple \\ \hspace{1.9cm} \vee \exists tuple , where \\ \hspace{4cm} U' = s.t. CHECK(U, U')$

A check request fans out on all indirect ACLs or groups, recursively.
- Deep or write ACLs cases are optimized by Leopard indexing system.
All leaf nodes of the boolean expression tree are evaluated concurrently.
- Eager cancellation: If the outcome of one node determines the result of a subtree, evaluations on other nodes are cancelled.
Use pooling mechanism to group reads for the same ACL check to reduce the number of RPCs to Spanner.

Leopard indexing system

For namespaces with deeply nested groups or large number of child groups, they can be selected to use Leopard indexing system to reduce latency of checks. The idea is to flatten the group to group paths. A query to Leopard system is an expression of UNION, INTERSECTION or EXCLUSION of named sets (indexing tuples). Response is a set ordered by element ID up to a specified number of results.

An indexing tuple (T, s, e):

T: the set type.
- GROUP2GROUP(s) -> {e}: For a group s, return all directly or indirectly descendant groups {e}.
- MEMBER2GROUP(s) -> {e}: For a user s, return all direct parent groups {e}.
s: set ID.
e 64-bit element ID.

Below is the check of user $U$ on group $G$ .
$\\$
$MEMBER2GROUP(U) \cap GROUP2GROUP(G) \neq \emptyset$

3 parts of Leopard system:

A serving system for consistent and low-latency operations across sets
- Tuples are stored like a skip list, to perform operations (UNION, INTERSECTION) in $O(\min(\vert A \vert, \vert B \vert))$ .
- Index is sharded by element IDs.
  - Note: Why? This way elements in the same set don’t stay together like a list.
An offline periodic index building system
- Generate index shards from a snapshot of relation tuples and configs; and replicate.
- Respect the userset rewrite rules.
An online layer for continuously updating the serving system as tuple changes occur
- Incrementally update indexes since the offline snapshot.
- An update is (T, s, e, t, d).
  - t is the timestamp of update.
  - d is deletion maker.
- Implementation
  - Rely on Zanzibar Watch API.
  - A relation tuple update may trigger tens of thousands Leopard tuple events.
  - Every Leopard server receives the Watch update stream and do the update with minimal impact on serving.

Handling hot spots

ACL reads and checks often lead to hot spots. 4 improvements:

Distributed cache
- Distributed in a cluster of servers with consistent hashing.
  - Compute forwarding key with object ID.
    - Because a check on <object#relation> often involves checks on other relation of the same object. These can be processed on the same server to save number of internal RPCs.
- Both caller and callee of the internal RPCs cache the results.
- Encode snapshot timestamp in cache key.
  - Avoid using caches of old snapshots.
  - Round up timestamps from the zookie.
    - Ensure freshness.
    - Most checks can share cached result at the same timestamp.
Lock table
- To handle “cache stampede” problem.
  - Concurrent requests create flash hot spots before the cache is populated with results.
- When receive concurrent requests of the same cache key, process one and block others until cache is ready.
Cache all relation tuples for hot check targets.
- Many users may issue current requests on the same <object#relation> pair.
- Cache all relation tuples for the pair.
- Trade read bandwidth for cacheability.
- Hot objects are determined dynamically by tracking the number of reads on each objects.
Delay eager cancellation
- Eager cancellation: indirect ACL checks are often cancelled when parent check result is determined.
- Problem: Checks are cancelled before cache keys are populated; thus concurrent requests are blocked on the lock table entry.
  - Note: The indirect ACL check is not cached. This request is the only one allowed to perform that check. Other concurrent requests are blocked on the check. Apparently, they need to wait longer if the check is cancelled.
- Improvement: delay eager cancellation when there are waiters on the corresponding lock table entry.

Performance isolation

Requirement: If one client fails to provision enough resources for an unexpected usage pattern, other clients won’t be affected.
Control
- CPU capacity
  - Per client
  - Throttling if exceeding the limit.
- Total number of outstanding RPCs for memory usage
  - On Zanzibar server
  - Per server
  - Per client
- Number of concurrent reads
  - On Spanner server
  - Per (object, client)
  - Per client
Use different lock table keys for different clients.
- Prevent throttling on one client affecting others.

Tail latency mitigation

Zanzibar’s distributed processing needs to accommodate slow tasks.

For calls to Spanner and Leopard servers
- Request hedging: Send the same request to multiple servers; use the first response and cancel others.
- Place at least 2 replicas of these backends in every geographical region.
- Practice: Send the first request; defer sending hedged requests until the initial one is known to be slow.
  - Hedging delay threshold: dynamically calculate Nth percentile latency.
  - thus only hedge a small fraction of total traffic;
For requests to Zanzibar servers
- No hedging
  - Hedging is effective only when requests have similar costs.
  - Some Zanzibar checks are meant to be more time-consuming.
  - Therefore, hedging will worsen latency by duplicate expensive requests.
- Rely on sharding and monitoring mechanisms.
  - Detect and avoid slow servers.

Experience

Some statistics as of 2019:

Namespace: 1,500.
Relation tuples
- 2 trillion relation tuples, close to 100 terabytes.
- Number of tuples per namespace: 10 to 1 trillion.
  - Median: 15,000.
Replication location: more than 30.
QPS: more than 10 million.
- Example of a peak event:
  - Check: 4.2M.
  - Read: 8.2M.
  - Expand: 760K.
  - Write: 25K.
Server: more than 10,000.
- Organized in several dozen clusters.
- Number of servers per cluster: 100 to 1,000.

Requests

Two categories of requests are as below based on zookie oldness. The number of safe requests is about two orders of magnitude higher than that of recent requests. (QPS: 1.2M vs 13K as of 12/2018)

Safe
- Zookies are more than 10 seconds old.
- Requests can be served locally.
Recent
- Zookies are less than 10 seconds old.
- Requests need to be routed to leader replica, requiring inter-region round trips.

Latency

Measurement: server side, live traffic.
Check: 3ms at both safe and recent 50th.
Write: 127ms at the 50th.
Safe requests are much faster than recent ones.

Availability

Define an available request: succeed in 5s for safe, 15s for recent requests.
Measurement: replay some real requests with modified zookies and extended deadlines.
99.999% available for past 3 years.

Internals

Delegated RPCs: Zanzibar servers contact other servers, internally.
- 22 million QPS at peak, half for reads, half for checks.
- Caching handles 200 million lookups per second at peak.
  - 150M for checks, 50M for reads.
- Cache hit and RPCs saved by lock table
  - Checks
    - Delegate side: 10% cache hit, 12% by lock table.
    - Delegator side: 2% cache hit, 3% by lock table.
    - Save 500K RPCs per second.
  - Reads
    - Delegate side: 24% cache hit, 9% by lock table.
    - Delegator side: less than 1% cache hit.
    - For super-hot groups(0.1%), cache full set of members in advance.
Spanner side
- 20 million read RPCs to Spanner, thanks to caching and read request pooling.
- Number of rows per RPC: 1.5 at the median, 1,000 at the 99th.
- Latency: 0.5ms at the median, 2ms at the 95th.
- 1% of Spanner reads benefit from hedging.
Leopard system
- 1.56M QPS at the median, 2.22M at the 99th.
- Latency: 150 microsec at the median, under 1ms at the 99th.
- Index incremental updates: 500 updates per second at the median, 1,500 at the 99th.

Lessons learned

Flexibility to accommodate differences between clients.
- Access control patterns vary widely.
  - Continuously add new features to support clients’ use cases.
- Freshness requirements are often but not always loose.
  - Zookie ensures bounded staleness.
Performance optimization to support client behaviors observed in production.
- Request hedging is the key to reduce tail latency.
- Hot-spot mitigation is critical for high availability.
  - Distributed cache, lock table, cache prefetching for hot items, delay eager cancellation.
- Performance isolation protects against misbehaving clients.

Access control in multi-user OS
- Multics, UNIX, POSIX ACLs, VMS.
Taos supports identity in distributed systems.
Role-based access control
- Role is similar to relation in Zanzibar.
Identity and Access Management (IAM)
- Amazon, Google, Microsoft’s cloud services.
- Unified ACL storage and RPC-based API.
- Google Cloud IAM is built on Zanzibar.
TAO is a distributed storage for social graphs in Facebook.
- No external consistency.
Chubby and Zookeeper don’t have features required to be Zanzibar’s storage.

Conclusion

Unified access control data and logic
Flexible data model and configuration
External consistency
Scalability, low latency and high availability

Migrate blog from Github Pages to Gitlab Pages

Fri, 26 Jul 2019 00:00:00 +0000

Background

My Github Pro subscription expired a few days ago, which means although I can have private repo for my blog, it cannot publish new updates any more. Two choices for you from Github: either pay for Pro or change the repo to a public one.

I am happy to contribute to open source world. However, the repo contains some sensitive information, like Google analytics ID, or perhaps some passwords pushed by mistake in commit history.. Therefore I decided to find a free alternative instead of going along with Github.

Happy to find Gitlab Pages, which I think is a perfect replacement of Github Pages, or even better. However, there are still some caveats during migration. Here I will make a summary of this effort. Note that entire migration took me around 2 hours.

Where to push my code?

Gitlab provides an amazing CI/CD feature, which does CI/CD from external repo. So all you have to do is to create a new project in Gitlab by setting up the CI/CD from your Github repo. In this way, you don’t even have to change your local git settings like adding a new origin for Gitlab. But you can do it anyway.

Depending on what you choose, you can either push to Github or Gitlab. Both repos can be private.

How to generate static pages?

Github does this step automatically once receiving a new push. Gitlab needs some configurations for CI/CD (changes -> pages, not Github repo -> Gitlab), but this doesn’t mean it’s not as good as Github. Actually, I was surprised by this feature. It’s much more configurable and thus powerful.

For the example of Pages, add a new .gitlab-ci.yml file under the root directory with following configs. You can learn more about the config settings here.

image: ruby:2.3

before_script:
    - export LC_ALL="C.UTF-8"
    - export LANG="en_US.UTF-8"
    - export LANGUAGE="en_US.UTF-8"

pages:
  stage: deploy
  script:
  - bundle install
  - bundle exec jekyll build -d public
  artifacts:
    paths:
    - public
  only:
  - master

Push the changes. In order to trigger the page generation pipeline, you need to manually run one in the project’s Pipelines page. The following changes will be generated automatically. You can even view the progress or download the generated html files.

Wait, what’s the url?

The default url will be YOUR_USERNAME.gitlab.io/YOUR_PROJECT_NAME, which may cause trouble if you didn’t have correct prefix for your assets’ url (css, pictures, etc). I recommend to do following in Project setting -> Advanced. It enables us to use custom domain easily. Without this, the homepage url with custom domain is yourcustom.domain/YOUT_PROJECT_NAME.

Rename your project to YOUR_USERNAME.gitlab.io
Update its path to https://gitlab.com/YOUR_USERNAME/YOUR_USERNAME.gitlab.io

After this, the url will be YOUR_USERNAME.gitlab.io. Custom domain will work as well.

With this setting, url prefix for all your assets in your project should be "" in your _config.yml. Otherwise, it has to be /YOUR_PROJECT_NAME. An example of asset path suffix could be /assets/web/web_design_workflow.png.

Custom domain is fancy

Easy to set up custom domain with HTTPS. Just follow instruction in Setting -> Pages. Server IP address is provided here. HTTPS support is built-in or you can use your own certificate.

Other caveats

You shouldn’t name one of your top level directories to public. This is a reserved name by Gitlab Page.

Read "Spanner, Google's Globally-Distributed Database"

Sat, 06 Jul 2019 00:00:00 +0000

Reference

Introduction

Spanner is Google’s scalable, globally-distributed database. It shards data across many sets of Paxos state machines spread all over the world. Data is also replicated to support availability and geographic locality. To balance load or respond to failures, data can be re-sharded or migrated automatically according to data or server changes.

Most applications favor lower latency over higher availability, thus they use less datacenters for replication (3 to 5 across US), as long as they can survive 1 to 2 datacenter failures.

Comparison with other Google databases:

Bigtable
- Difficult to use for some applications with complex, evolving schemas.
- Doesn’t provide strong consistency in the presence of wide-area replication.
Megastore
- Good to have semi-relational data model.
- Support synchronous replication.
- Write throughput is relatively poor.
Spanner
- Schematized semi-relational tables.
- Data is versioned and each version is timestamped with its commit time.
- Applications can read data at old timestamps.
- Garbage collections on old versions.
- Support general transactions.
- Support SQL-based queries.

Other features:

Replication configurations for data can be dynamically controlled at a fine grain by applications.
- Which datacenter contains which data.
- How far data is from users.
- How far replicas are from each other.
- How many replicas.
Data can be dynamically and transparently moved across datacenters to balance.
Provide externally consistent reads and writes (linearizability).
Provide globally-consistent reads across database at a timestamp.

These features are enabled by the fact that Spanner assigns globally-meaningful commit timestamps to distributed transactions, which reflect serialization order. If a transaction T1 commits before another transaction T2, then T1’s timestamp is smaller than T2’s.

TrueTime API:

The key enabler is TrueTime API. It directly expose clock uncertainty (generally less than 10ms). If uncertainty is large, Spanner slows down to wait out that uncertainty. The API implementation is using both GPS and atomic clocks.

Note: Example of how the TrueTime helps. Say we have 2 nodes A and B and 2 transaction T1 and T2.

Without TrueTime:

At true time 10, A time 15, A updates A’s value from “A1” to “A2”, T1 committed at timestamp 15.
At true time 15, B time 10, B updates B’s value from “B1” to “B2”, T2 committed at timestamp 10.
Client performs a read with timestamp 10, It sees “A1” and “B2”, which is inconsistent with actual true time order.

With TrueTime, uncertainty 15 (larger than the difference of server timestamps):

At true time 10, A time 15, A updates A’s value from “A1” to “A2”, T1 committed at timestamp 15.
Wait uncertainty 15, true time is 25.
- Now true time 15 (which is T1’s timestamp) has passed.
At true time 30, B time 25, B updates B’s value from “B1” to “B2”, T2 committed at timestamp 25.
Client performs a read with timestamp 25, It sees “A2” and “B2”.

Note: Vector clock is another method solving this. Now a clock is represented as a vector <ts1, ts2, ..., tsn>. n is the number of nodes. In case of two nodes, the clock vector is <ts1, ts2>. The following example shows that the clock vectors reflect actual serialization order as well. However, we may have too many nodes in Spanner group, thus cost could be huge to store, write and transport the vectors. We can understand why it’s not used in Spanner. This vector clock may sound similar to Lamport timestamp, but better. Lamport timestamp can enforce total order, but cannot describe concurrent events well. Vector clock records timestamps on all nodes to represent such causality.

At true time 10, A time 15, A updates A’s value from “A1” to “A2”, T1 committed at timestamp <15, 0>.
This timestamp is broadcasted to other nodes, B updated its timestamp to <15, x>.
At true time 15, B time 10, B updates B’s value from “B1” to “B2”, committed at timestamp <15, 10>.

Implementation

A Spanner deployment is called universe, which can be global. Currently, there are 3 universes: test/playground, dev/prod and prod-only universes.

Spanner is organized as a set of zones. Zone is the unit of administrative deployment; location across which data can be replicated; unit of physical isolation. For physical isolation, one datacenter may have multiple zones as different applications’ data must be partitioned across different sets of servers in the same datacenter. Zone can be added or removed, as datacenters added or turned off.

The Spanner servers are organized as below.

Universe level
- universemaster
  - A console displaying status information about all zones for debugging.
- placement driver
  - Handle automated data movement across zones in minutes.
  - Communicate with spanservers periodically.
Zone level
- zonemaster
  - Assign data to spanservers.
  - One per zone.
- location proxies
  - Used by clients to locate spanservers to serve their data.
- spanservers
  - Serve data to clients.
  - 100 to several thousand per zone.

Spanserver software stack

Each spanserver is responsible for 100 to 1000 tablets.
A tablet is a bag of mappings.
- Stored in set of B-tree-like files and write-ahead log.
  - Both stored on Colossus.
Mapping: (key:string, timestamp:int64) -> string.
- Thus Spanner is a multi-version k-v store.

The spanserver is organized as:

A single Paxos state machine on top of each tablet.
- Stores its metadata and log in the tablet.
- Supports long-lived leaders with time-based lease (default 10s).
A set of replicas of a tablet is a Paxos group.
- Writes must initiate Paxos protocol at leader.
- Reads can access any replica.
At leader replica, spanserver implements a lock table to have concurrency control. (Critical to have long-live leaders)
- Contains state for 2PL: map key range to lock state.
  - In contrast, optimistic control will cause bad performance of long-lived transactions in case of conflicts.
- Practice
  - Synchronized operations acquire locks.
    - e.g., transactional reads.
  - Other operations bypass the lock table.

Distributed transactions:

If a transaction involves only one Paxos group (most of them), lock table and Paxos can provide enough transactionality.
Otherwise, coordination across groups is needed.
- Leader replica spanserver implements a transaction manager. The replica is a participant leader; others are participant slaves.
- Those groups’ leaders coordinate to perform 2PC.
  - One group is chosen to be coordinator (with coordinator leader and slaves).
- State of transaction manager is stored in the Paxos group and thus replicated.

Directories and placement

Directory is a bucketing abstraction of a set of contiguous keys that share a common prefix. Applications can use directories to control the locality of the data by choosing the keys carefully.

A directory is the unit of data placement.
- All data in a directory has the same replication configuration.
A Paxos group (tablet replicas) can have multiple directories.
- Frequently accessed directories can be local to each other.
Movedir is used to move directories between Paxos groups; and add or remove replicas to Paxos groups.
A directory is the unit of specified geographic-replication properties.
- e.g., Specify user group A has 3 replicas in US; group B has 2 replicas in EU.
Spanner shards a directory into multiple fragments if it grows too large.
- Different fragments may be served from different Paxos groups.
- Movedir moves fragments.

Note: How to ensure locality of a directory if its fragments are in different Paxos groups? My thinking: If those fragments are in different Paxos groups, locality cannot be guaranteed and 2PC is required if a transaction involves multiple fragments. But data in a fragment always stays in the same Paxos group.

Data model

Spanner data features:

Schematized semi-relational tables, synchronous replication across datacenters.
- Easy to manage, like Megastore.
Query language
- To support Dremel.
General-purpose transactions.
- Although 2PC is expensive, Spanner still support cross-row transactions. Application developers is responsible to deal with low-performance problems.

Application data model is layered on top of Spanner’s directory-bucketed k-v mappings.

An application creates one or more databases in a universe.
A database can contain unlimited number of schematized tables.
- A database must be partitioned into one or more hierarchies of tables.
A table looks like relational-database table, with rows, columns and versions.
A row must have one or more primary-key columns.
- Application can use keys to control data locality.

Example of a database schema:

CREATE TABLE Users {
  uid INT64 NOT NULL, email STRING
} PRIMARY KEY (uid), DIRECTORY;

CREATE TABLE Albums {
  uid INT64 NOT NULL, aid INT64 NOT NULL,
  name STRING
} PRIMARY KEY (uid, aid),
  INTERLEAVE IN PARENT Users ON DELETE CASCADE;

DIRECTORY declares a top-level directory table in the hierarchy.
- A directory is formed with each row in directory table with key K, together with all rows in descendant tables that start with K in lexicographic order.
INTERLEAVE IN declare descendant tables.
ON DELETE CASCADE says deleting a directory table row will delete all associated rows in descendant tables.
With this directory concept, we can have a user row and its album rows stay together.

TrueTime

API:

Method	Returns
`TT.now()`	TTinterval: `[earliest, latest]`
`TT.after(t)`	true if `t` has definitely passed
`TT.before(t)`	true if `t` has definited not arrived

TrueTime uses both GPS and atomic clocks as time references. They have different failure modes; thus can compensate each other.

Organization:
- TrueTime is implemented by a set of time master machines per datacenter and a timeslave daemon per machine.
- The majority of masters have GPS receivers.
  - Advertising uncertainty is typically close to zero.
- The rest of masters use atom clocks. (Armageddon masters)
  - Advertising uncertainty slowly increases, derived from conservatively worst-case clock drift.
Mechanism:
- Masters compare time references against each other.
- Each master cross-checks the divergence rate.
- Every daemons poll a variety of masters (some may be from farther datacenters).
  - Apply Marzullo’s algorithm to synchronize local clocks.

Define instantaneous error bound as $\epsilon$ , which is half of the interval’s width. Average error bound is $\bar{\epsilon}$ .

Before synchronizations, a daemon advertises a slowly increasing time uncertainty .
- is derived from
  - conservatively applied worst-case local clock drift
  - time-master uncertainty
  - communication delay to the time masters.
is a sawtooth function of time, varying from 1 to 7 ms over each poll interval (30s).
- Applied drift 0 to 6 ms.
  - Rate 20 microsec/sec * 30 sec.
- 1 ms is communication delay to the time masters.

Concurrency control

TrueTime -> (enables) correctness properties around concurrency control -> features:

Externally consistent transactions
Lock-free read-only transactions
Non-blocking reads in the past.

These features enable a guarantee that a whole-database audit read at a timestamp t will see exactly the effects of every transaction that has committed as of t.

Timestamp management

Operation	Concurrency control	Replica required
Read-Write Transaction	pessimistic	leader
Read-only transaction	lock-free	leader for timestamp; any for read
Snapshot read with client-provided timestamp	lock-free	any
Snapshot read with client-provided bound	lock-free	any

Read-only transaction
- Must be pre-declared first (not having any write).
- System (leader) chooses a timestamp for it.
- Can then be processed on any replica.
- Incoming writes are not blocked.
Snapshot read
- Client can specify a timestamp or provide an upper bound and let Spanner to choose one.
- Processed on any replica.

Paxos leader leases

Spanner’s Paxos use time leases to make leadership long-lived (10 sec by default).
Lease can be extended when approaching expiration.
Leader can abdicate by releasing its slaves.
- A leader must wait some time until it can abdicate to preserve the disjointness invariant.
Within a Paxos group, leaders’ lease intervals should be disjoint. (current leader’s and the subsequent leader’s)

Assigning timestamps to RW transactions

RW transactions use 2PL.
Spanner assigns a timestamp at any time when all locks have been acquired, but before any lock has been released.
- The timestamp of the Paxos write representing transaction commit.
Invariant 1: Within each Paxos group, Spanner assigns timestamps to Paxos writes in monotonically increasing order, even across leaders (current and the subsequent ones within a group).
- Enabled by making use of the disjointness invariant.
Invariant 2: if the start of a transaction occurs after the commit of a transaction , then the commit timestamp of must be greater than the commit timestamp of .
- Enabled by commit wait.

Serving reads at a timestamp

Every replica tracks a timestamp value called safe time $t_{safe}$ , to which it’s up to date. A replica can serve a read at a timestamp $t$ if $t \le t_{safe}$ .
is the minimum of:
- $t^{Paxos}_{safe}$ : the timestamp of the highest-applied Paxos write.
- For transaction manager, : .
  - $s_{i, g}^{prepare}$ is the prepared timestamps from a group $g$ , inferred from the prepared but not committed transactions (between two phases of 2PC).

Assigning timestamps to RO transactions

Two steps:
- Assign a timestamp.
- Execute the reads as snapshot reads at this timestamp.
Spanner usually assigns the oldest timestamp that preserves external consistency.
- A too fresh timestamp like TT.now().latest make block if $t_{safe}$ has not advanced sufficiently.

Details

RW transactions

When multiple Paxos groups are involved:

Writes in a transaction are buffered at the client until commit.
- Therefore, reads in the same transaction don’t see the effects of those writes.
Reads use wound-wait to avoid deadlocks.
Client sends keepalive messages to participant leaders when a transaction remains open.
Client begins 2PC when all reads are completed and all writes are buffered.
- Choose a coordinator group.
- Send a commit message to all participant leaders with the coordinator information and buffered writes.
  - Avoid sending data twice (if let the coordinator drive the 2PC).

2PC:

Non-coordinator-participant leader
1. Acquire write locks.
2. Choose a prepare timestamp (greater than all assigned).
3. Log a prepare record through Paxos.
4. Send the prepare timestamp to the coordinator.
5. Receive commit timestamp from coordinator.
6. Log the transaction’s outcome through Paxos.
Coordinator
1. Acquire write locks.
2. Skip prepare phase.
3. Hear prepare timestamps from all participant leaders.
4. Choose a transaction timestamp s.
  - Greater than all prepare timestamps.
  - Greater than all assigned at itself.
5. Wait until s is definitely passed.
6. Concurrent with step 5, log a commit record through Paxos.
7. Send the commit timestamp to the client and all participant leaders.

2PC adds an extra network round trip so it usually doubles observed commit latency. It scales well up to 10s of participants, but abort frequency and latency increase significantly with 100s of participants. (This is from F1 paper.)

Read-only transactions

Spanner requires a scope expression for every read-only transaction.
- Scope: summarize the keys that will be read by the entry transaction.
  - To determine the Paxos groups to serve the request.
Cases:
- If only one Paxos group is involved.
  - The leader maintains LastTS() for the timestamp of the last committed write at this group.
    - Better than TT.now().lastest().
    - Use it for read timestamp if no prepared transactions.
      - Note:
        
        If prepared transactions exist and deal with the same data as the RO transaction, the RO transaction should be assigned $t$ equal to TT.now().lastest(); then snapshot read will wait until $t \le t_{safe}$ .
        
        When $t$ is less than the commit timestamps of those previously prepared transactions. ( $t_{safe}$ advanced.)
        
        If data to read is unrelated to the prepared transactions, is it safe to use LastTS()?
        
        Could be, see refinements.
- Multiple Paxos groups
  - Simple option (used by Spanner): use TT.now().lastest().
  - Complicated option: do a round of communication with all groups to negotiate based on their LastTS().

Schema-change transactions

Spanner supports atomic schema changes.
- Not a standard transaction, since millions of participants could be involved.
- TrueTime is essential for this.
Bigtable supports atomic schema changes in one datacenter.
- But changes block all operations.

Details:

The transaction is explicitly assigned a timestamp t in the future.
- Registered in the prepared phase.
Operations relying on the new schema must wait until t is passed.

Problem of : A single prepared transaction prevents from advancing.
- Even if the read is unrelated to the prepared transaction.
- Refinement: Use a mapping from key ranges to prepared transaction timestamps.
  - Can be stored in the lock table.
- Schema change prepared transaction should block all.
Problem of LastTS(): A read must be assigned a timestamp after the recent transaction, even if they are irrelevant.
- Refinement: Use a mapping from key ranges to commit timestamps.
  - Not implemented in 2012.
Problem of : It cannot advance in the absence of Paxos writes.
- A snapshot read at t cannot be executed at Paxos group whose last write happened before t.
- Refinement: Paxos leader advances .
  - Set it based on MinNextTS() within the lease interval.
  - Enabled by the disjointness of leader lease intervals.
- MinNextTS() is advanced every 8 seconds.
  - Thus an idle Paxos group may serve reads at timestamp greater than 8 seconds old in the worst case.
    - Note: This means a snapshot read may need to wait for 8 seconds to perform the read.
      - When $t \le t_{safe}$ .

Evaluation

Microbenchmarks

Latency
- For writes, latency stays roughly constant with less std, as number of replicas increases.
  - Because Paxos executes in parallel at a group’s replicas.
Throughput
- Snapshot reads: increase almost linearly with the number of replicas.
  - Because they can be served from any up-to-date replicas.
- RO transactions: increases as number of replicas increases (corresponding number of spanservers also increases).
  - Timestamp must be assigned by leader.
  - Can then be served by other replicas.
- Write: 1 replica > 5 replicas > 3 replicas.
  - Because 1 replica doesn’t need to do replication.
  - Number of spanservers increases when 3 replicas -> 5 replicas.
    - Also leaders are randomly distributed.
2PC
- 2PC can be scaled up to 100 participants with reasonable latency in the experiment.

Availability

Failure mode:

Non-leader fail
- No effect on read throughput.
Leader soft fail (but notify all servers to handoff leadership first)
- Minor effect on read throughput (3% - 4%).
Leader hard fail
- Severe effect within failed leader lease window.
- Re-elect leader; then catch up in next lease window.

TrueTime

The 200 us/sec clock drift is a reasonable assumption.
- Bad CPU problem are 6 times more likely than bad clocks.
Uncertainty comes from:
- Time master uncertainty (generally 0);
- Communication delay to the time masters.
Experiment shows no significant problem.
- To improve:
  - Improve network;
  - Reduce causes of TrueTime spikes.

Case study: F1

Painful past:

Based on MySQL.
Manual re-sharding was costly and took long time (over two years).
Data growth was limited by the team.

Reasons choosing Spanner:

No need to manually re-shard.
Synchronous replication and automatic failover.
Transactions.
- e.g., generating secondary indexes.

Practices:

Choose replica locations to cope with potential outrages and their frontend locations.
Spanner timestamp makes F1 easy to maintain its in-memory data structures.
- Read a snapshot at a timestamp to construct;
- Then incrementally update.
A directory typically represents a client.
- Most directories contains only 1 fragment.
  - Reads and writes are guarantees to happen on a single server. (Note: a Paxos group?)
- As of 2012, 7 largest directories has 100 to 500 fragments.
Large std of write latencies: Caused by a fat tail due to lock conflicts.
Larger std of read latencies: Because Paxos leaders are spread across two data centers, one of them has SSDs.

Megastore
- Consistent replication across datacenters globally.
- Semi-relational data model.
- Similar schema language as Spanner.
- Not high performance.
- On top of Bigtable with high communication costs.
- Not support long-lived leaders.
  - Multiple replicas can initiate writes, which can cause unnecessary conflicts in Paxos protocol.
DynamoDB
- Consistent replication across datacenters within region.
- Provide a key-value interface.
Database functionality layered on distributed k-v stores.
- But Spanner integrates multiple layers.
  - e.g., integrating concurrency control with replication to reduce the cost of commit wait.
Scatter
- A DHT-based k-v store layering transactions on top of consistent replication.
- Spanner provides a higher-level interface.
Walter
- Snapshot isolation within datacenter.
Calvin, HStroe, Granola
- No external consistency.
VoltDB
- Not much general replication configurations.
Farsite
- Bounds of clock uncertainty.
- Much looser than TrueTime’s.

Future work

Improve monitoring and support tools.
Tune performance.
Schema language.
Automatic maintenance of secondary indices.
Automatic load-based resharding.
May support optimistically doing reads in parallel.
Reduce TrueTime , below 1 ms.
- Better clock crystals.
- Better network.
Improve local data structure in single node to better support SQL queries.
Automatically move clients’ application processes between datacenters in response to load changes.

Conclusions

Combine and extend ideas from two communities:
- Database
  - Semi-relational, transactions, SQL query.
  - Bigtable misses some database features.
- System
  - Scalability, Automatic sharding, fault tolerance, consistency, wide-area distribution.
TrueTime
- Enables to build distributed systems with much stronger time semantics.

记忆力抢救日更

Fri, 19 Apr 2019 00:00:00 +0000

感觉记忆力有点下降，开一个日更背诗贴，欢迎抽查嘻嘻。

20190418

行宫
元稹
寥落古行宫，
宫花寂寞红。
白头宫女在，
闲坐说玄宗。

芙蓉楼送辛渐
王昌龄
寒雨连江夜入吴，
平明送客楚山孤。
洛阳亲友如相问，
一片冰心在玉壶。

送杜少府之任蜀州
王勃
城阙辅三秦，风烟望五津。
与君离别意，同是宦游人。
海内存知己，天涯若比邻。
无为在歧路，儿女共沾巾。

20190419

登鹳雀楼
王之涣
白日依山尽，
黄河入海流。
欲穷千里目，
更上一层楼。

闺怨
王昌龄
闺中少妇不知愁，
春日凝妆上翠楼。
忽见陌头杨柳色，
悔教夫婿觅封侯。

送梓州李使君
王维
万壑树参天，千山响杜鹃。
山中一夜雨，树杪百重泉。
汉女输橦布，巴人讼芋田。
文翁翻教授，不敢倚先贤。

新嫁娘词
王建
三日入厨下，
洗手作羹汤。
未谙姑食性，
先遣小姑尝。

陋室铭
刘禹锡
山不在高，有仙则名。水不在深，有龙则灵。斯是陋室，惟吾德馨。
苔痕上阶绿，草色入帘青。谈笑有鸿儒，往来无白丁。可以调素琴，阅金经。
无丝竹之乱耳，无案牍之劳形。南阳诸葛庐，西蜀子云亭。
孔子云，何陋之有？

20190420

相思
王维
红豆生南国，
春来发几枝。
愿君多采撷，
此物最相思。

春宫曲
王昌龄
昨夜风开露井桃，
未央前殿月轮高。
平阳歌舞新承宠，
帘外春寒赐锦袍。

汉江临眺
王维
楚塞三湘接，荆门九派通。
江流天地外，山色有无中。
郡邑浮前浦，波澜动远空。
襄阳好风日，留醉与山翁。

20190421

杂诗三首，其二
王维
君自故乡来，
应知故乡事。
来日绮窗前，
寒梅著花未？

九月九日忆山东兄弟
王维
独在异乡为异客，
每逢佳节倍思亲。
遥知兄弟登高处，
遍插茱萸少一人。

凉州词二首，其一
王翰
葡萄美酒夜光杯，
欲饮琵琶马上催。
醉卧沙场君莫笑，
古来征战几人回？

终南别业
王维
中岁颇好道，晚家南山陲。
兴来每独往，胜事空自知。
行到水穷处，坐看云起时。
偶然值林叟，谈笑无还期。

20190422

鹿柴
王维
空山不见人，
但闻人语响。
返景入深林，
复照青苔上。

后宫词
白居易
泪湿罗巾梦不成，
夜半前殿按歌声。
红颜未老恩先断，
斜倚薰笼坐到明。

终南山
王维
太乙近天都，连山接海隅。
白云回望合，青霭入看无。
分野中峰变，阴晴万壑殊。
欲投人处宿，隔水问樵夫。

20190423

竹里馆
王维
独坐幽篁里，
弹琴复长啸。
深林人不知，
明月来相照。

宫词
朱庆馀
寂寂花时闭院门，
美人相并立琼轩。
含情欲说宫中事，
鹦鹉前头不敢言。

酬张少府
王维
晚年唯好静，万事不关心。
自顾无长策，空知返旧林。
松风吹解带，山月照弹琴。
君问穷通理，渔歌入浦深。

20190424

送别
王维
山中相送罢，
日暮掩柴扉。
春草明年绿，
王孙归不归？

近试上张水部
朱庆馀
洞房昨夜停红烛，
待晓堂前拜舅姑。
妆罢低声问夫婿，
画眉深浅入时无。

过香积寺
王维
不知香积寺，数里入云峰。
古木无人径，深山何处钟。
泉声咽危石，日色冷青松。
薄暮空潭曲，安禅制毒龙。

20190425

问刘十九
白居易
绿蚁新醅酒，
红泥小火炉。
晚来天欲雪，
能饮一杯无？

逢入京使
岑参
故园东望路漫漫，
双袖龙钟泪不干。
马上相逢无纸笔，
凭君传语报平安。

辋川闲居赠裴秀才迪
王维
寒山转苍翠，秋水日潺湲。
倚杖柴门外，临风听暮蝉。
渡头馀落日，墟里上孤烟。
复值接舆醉，狂歌五柳前。

20190426

哥舒歌
西鄙人
北斗七星高，
哥舒夜带刀。
至今窥牧马，
不敢过临洮。

黄鹤楼送孟浩然之广陵
李白
故人西辞黄鹤楼，
烟花三月下扬州。
孤帆远影碧空尽，
唯见长江天际流。

山居秋暝
王维
空山新雨后，天气晚来秋。
明月松间照，清泉石上流。
竹喧归浣女，莲动下渔舟。
随意春芳歇，王孙自可留。

20190427

静夜思
李白
窗前明月光，
疑是地上霜。
举头望明月，
低头思故乡。

怨情
李白
美人卷珠帘，
深坐颦蛾眉。
但见泪痕湿，
不知心恨谁。

早发白帝城
李白
朝辞白帝彩云间，
千里江陵一日还。
两岸猿声啼不住，
轻舟已过万重山。

夜上受降城闻笛
李益
回乐峰前沙似雪，
受降城外月如霜。
不知何处吹芦管，
一夜征人尽望乡。

归嵩山作
王维
清川带长薄，车马去闲闲。
流水如有意，暮禽相与还。
荒城临古渡，落日满秋山。
迢递嵩高下，归来且闭关。

20190428

乐游原
李商隐
向晚意不适，
驱车登古原。
夕阳无限好，
只是近黄昏。

听筝
李端
鸣筝金粟柱，
素手玉房前。
欲得周郎顾，
时时误拂弦。

贾生
李商隐
宣室求贤访逐臣，
贾生才调更无论。
可怜夜半虚前席，
不问苍生问鬼神。

次北固山下
王湾
客路青山外，行舟绿水前。
潮平两岸阔，风正一帆悬。
海日生残夜，江春入旧年，
乡书何处达，归雁洛阳边。

20190429

渡汉江
宋之问
岭外音书断，
经冬又复春。
近乡情更怯，
不敢问来人。

隋宫
李商隐
乘兴南游不戒严，
九重谁省谏书函。
春风举国裁宫锦，
半作障泥半作帆。

云阳馆与韩绅宿别
司空曙
故人江海别，几度隔山川。
乍见翻疑梦，相悲各问年。
孤灯寒照雨，深竹暗浮烟。
更有明朝恨，离杯惜共传。

20190430

八阵图
杜甫
功盖三分国，
名成八阵图。
江流石不转，
遗恨失吞吴。

瑶池
李商隐
瑶池阿母绮窗开，
黄竹歌声动地哀。
八骏日行三万里，
穆王何事不重来。

喜外弟卢纶见宿
司空曙
静夜四无邻，荒居旧业贫。
雨中黄叶树，灯下白头人。
以我独沉久，愧君相见频。
平生自有分，况是蔡家亲。

20190501

宿建德江
孟浩然
移舟泊烟渚，
日暮客愁新。
野旷天低树，
江清月近人。

嫦娥
李商隐
云母屏风烛影深，
长河渐落晓星沉。
嫦娥应悔偷灵药，
碧海青天夜夜心。

贼平后送人北归
司空曙
世乱同南去，时清独北还。
他乡生白发，旧国见青山。
晓月过残垒，繁星宿故关。
寒禽与衰草，处处伴愁颜。

20190502

春晓
孟浩然
春眠不觉晓，
处处闻啼鸟。
夜来风雨声，
花落知多少。

春怨
金昌绪
打起黄莺儿，
莫教枝上啼。
啼时惊妾梦，
不得到辽西。

夜雨寄北
李商隐
君问归期未有期，
巴山夜雨涨秋池。
何当共剪西窗烛，
却话巴山夜雨时。

赋得古原草送别
白居易
离离原上草，一岁一枯荣。
野火烧不尽，春风吹又生。
远芳侵古道，晴翠接荒城。
又送王孙去，萋萋满别情。

20190503

江雪
柳宗元
千山鸟飞绝，
万径人踪灭。
孤舟蓑笠翁，
独钓寒江雪。

秋夜寄邱员外
韦应物
怀君属秋夜，
散步咏凉天。
空山松子落，
幽人应未眠。

寄令狐郎中
李商隐
嵩云秦树久离居，
双鲤迢迢一纸书。
莫问梁园旧宾客，
茂陵秋雨病相如。

题大庾岭北驿
宋之问
阳月南飞雁，传闻至此回。
我行殊未已，何日复归来。
江静潮初落，林昏瘴未开。
明朝望乡处，应见陇头梅。

20190504

终南望余雪
祖咏
终南阴岭秀，
积雪浮云端。
林表明霁色，
城中增暮寒。

为有
李商隐
为有云屏无限娇，
凤城寒尽怕春宵。
无端嫁得金龟婿，
辜负香衾事早朝。

寄左省杜拾遗
岑参
联步趋丹陛，分曹限紫微。
晓随天仗入，暮惹御香归。
白发悲花落，青云羡鸟飞。
圣朝无阙事，自觉谏书稀。

20190505

宫词
张祜
故国三千里，
深宫二十年。
一声何满子，
双泪落君前。

江南逢李龟年
杜甫
岐王宅里寻常见，
崔九堂前几度闻。
正是江南好风景，
落花时节又逢君。

赠别
杜牧
多情却似总无情，
唯觉樽前笑不成。
蜡烛有心还惜别，
替人垂泪到天明。

听蜀僧濬弹琴
李白
蜀僧抱绿绮，西下峨眉峰。
为我一挥手，如听万壑松。
客心洗流水，余响入霜钟。
不觉碧山暮，秋云暗几重。

20190506

寻隐者不遇
贾岛
松下问童子，
言师采药去。
只在此山中，
云深不知处。

送崔九
裴迪
归山深浅去，
须尽丘壑美。
莫学武陵人，
暂游桃源里。

赠别其二
杜牧
娉娉袅袅十三余，
豆蔻梢头二月初。
春风十里扬州路，
卷上珠帘总不如。

夜泊牛渚怀古
李白
牛渚西江夜，青天无片云。
登舟望秋月，空忆谢将军。
余亦能高咏，斯人不可闻。
明朝挂帆席，枫叶落纷纷。

20190507

送灵澈上人
刘长卿
苍苍竹林寺，
杳杳钟声晚。
荷笠带斜阳，
青山独归远。

金谷园
杜牧
繁华事散逐香尘，
流水无情草自春。
日暮东风怨啼鸟，
落花犹似坠楼人。

赠孟浩然
李白
吾爱孟夫子，风流天下闻。
红颜弃轩冕，白首卧松云。
醉月频中圣，迷花不事君。
高山安可仰，徒此揖清芬。

20190508

听弹琴
刘长卿
泠泠七弦上，
静听松风寒。
古调虽自爱，
今人多不弹。

寄扬州韩绰判官
杜牧
青山隐隐水迢迢，
秋尽江南草未凋。
二十四桥明月夜，
玉人何处教吹箫。

渡荆门送别
李白
渡远荆门外，来从楚国游。
山随平野尽，江入大荒流。
月下飞天镜，云生结海楼。
仍怜故乡水，万里送行舟。

20190509

送上人
刘长卿
孤云将野鹤，
岂往人间住。
莫买沃洲山，
时人已知处。

遣怀
杜牧
落魄江南载酒行，
楚腰纤细掌中轻。
十年一觉扬州梦，
赢得青楼薄幸名。

送友人
李白
青山横北郭，白水绕东城。
此地一为别，孤蓬万里征。
浮云游子意，落日故人情。
挥手自兹去，萧萧班马鸣。

蜀道难
李白
噫吁嚱，危乎高哉。蜀道之难，难于上青天。
蚕丛及鱼凫，开国何茫然。
尔来四万八千岁，不与秦塞通人烟。
西当太白有鸟道，可以横绝峨眉巅。
地崩山摧壮士死，然后天梯石栈相钩连。
上有六龙回日之高标，下有冲波逆折之回川。
黄鹤之飞尚不得过，猿猱欲度愁攀援。
青泥何盘盘，百步九折萦岩峦。
扪参历井仰胁息，以手抚膺坐长叹。
问君西游何时还，畏途巉岩不可攀。
但见悲鸟号古木，雄飞雌从绕林间。
又闻子规啼夜月，愁空山。
蜀道之难，难以上青天。使人听此凋朱颜。
连峰去天不盈尺，枯松倒挂倚绝壁。
飞湍瀑流争喧豗，砯崖转石万壑雷。
其险也若此，嗟尔远道之人，胡为乎来哉。
剑阁峥嵘而崔嵬，一夫当关，万夫莫开。
其守或匪亲，化为狼与豺。
朝避猛虎，夕避长蛇，磨牙吮血，杀人如麻。
锦城虽云乐，不如早还家。
蜀道之难，难于上青天。侧身西望长咨嗟。

20190510

玉台体
权德舆
昨夜裙带解，
今朝蟢子飞。
铅华不可弃，
莫是藁砧归。

秋夕
杜牧
银烛秋光冷画屏，
轻罗小扇扑流萤。
天阶夜色凉如水，
卧看牵牛织女星。

喜见外弟又言别
李益
十年离乱后，长大一相逢。
问姓惊初见，称名忆旧容。
别来沧海事，语罢暮天钟。
明日巴陵道，秋山又几重。

20190511

将赴吴兴登乐游原一绝
杜牧
清时有味是无能，
闲爱孤云静爱僧。
欲把一麾江海去，
乐游原上望昭陵。

凉思
李商隐
客去波平槛，蝉休露满枝。
永怀当此节，倚立自移时。
北斗兼春远，南陵寓使迟。
天涯占梦数，疑误有新知。

酬郭给事
王维
洞门高阁霭馀辉，桃李阴阴柳絮飞。
禁里疏钟官舍晚，省中啼鸟吏人稀。
晨摇玉佩趋金殿，夕奉天书拜琐闱。
强欲从君无那老，将因卧病解朝衣.

20190512

赤壁
杜牧
折戟沉沙铁未销，
自将磨洗认前朝。
东风不与周郎便，
铜雀春深锁二乔。

北青萝
李商隐
残阳西入崦，茅屋访孤僧。
落叶人何在，寒云路几层。
独敲初夜磬，闲倚一枝藤。
世界微尘里，吾宁爱与憎。

和贾舍人早朝大明宫之作
王维
绛帻鸡人送晓筹，尚衣方进翠云裘。
九天阊阖开宫殿，万国衣冠拜冕旈。
日色才临仙掌动，香烟欲傍衮龙浮。
朝罢须裁五色诏，佩声归向凤池头。

20190513

泊秦淮
杜牧
烟笼寒水月笼沙，
夜泊秦淮近酒家。
商女不知亡国恨，
隔江犹唱后庭花。

蝉
李商隐
本以高难饱，徒劳恨费声。
五更疏欲断，一树碧无情。
薄宦梗犹泛，故园芜已平。
烦君最相警，我亦举家清。

奉和圣制从蓬莱向兴庆阁官道留春雨中春望之作应制
王维
渭水自萦秦塞曲，黄山旧绕汉宫斜。
銮舆迥出千门柳，阁道回看上苑花。
云里帝城双凤阙，雨中春树万人家。
为乘阳气行时令，不是宸游玩物华。

20190514

征人怨
柳中庸
岁岁金河复玉关，
朝朝马策与刀环。
三春白雪归青冢，
万里黄河绕黑山。

风雨
李商隐
凄凉宝剑篇，羁泊欲穷年。
黄叶仍风雨，青楼自管弦。
新知遭薄俗，旧好隔良缘。
心断新丰酒，销愁斗几千。

积雨辋川庄作
王维
积雨空林烟火迟，蒸藜炊黍饷东菑。
漠漠水田飞白鹭，阴阴夏木啭黄鹂。
山中习静观朝槿，松下清斋折露葵。
野老与人争席罢，海鸥何事更相疑。

20190515

金陵图
韦庄
谁谓伤心画不成，
画人心逐世人情。
君看六幅南朝事，
老木寒云满故城。

落花
李商隐
高阁客竟去，小园花乱飞。
参差连曲陌，迢递送斜晖。
肠断未忍扫，眼穿仍欲归。
芳心向春尽，所得是沾衣。

望月有感
白居易
时难年荒世业空，弟兄羁旅各西东。
田园寥落干戈后，骨肉流离道路中。
吊影分为千里雁，辞根散作九秋蓬。
共看明月应垂泪，一夜乡心五处同。

20190516

滁州西涧
韦应物
独怜幽草涧边生，
上有黄鹂深树鸣。
春潮带雨晚来急，
野渡无人舟自横。

登岳阳楼
杜甫
昔闻洞庭水，今上岳阳楼。
吴楚东南坼，乾坤日月浮。
亲朋无一字，老病有孤舟。
戎马关山北，凭轩涕泗流。

奉和中书舍人贾至早朝大明宫
岑参
鸡鸣紫陌曙光寒，莺啭皇州春色阑。
金阙晓钟开万户，玉阶仙仗拥千官。
花迎剑珮星初落，柳拂旌旗露未干。
独有凤凰池上客，阳春一曲和皆难。

20190517

桃花溪
张旭
隐隐飞桥隔野烟，
石矶西畔问渔船。
桃花尽日随流水，
洞在清溪何处边。

奉济驿重送严公四韵
杜甫
远送从此别，青山空复情。
几时杯重把，昨夜月同行。
列郡讴歌惜，三朝出入荣。
江村独归处，寂寞养残生。

登金陵凤凰台
李白
凤凰台上凤凰游，凤去台空江自流。
吴宫花草埋幽径，晋代衣冠成古丘。
三山半落青天外，二水中分白鹭洲。
总为浮云能蔽日，长安不见使人愁。

20190518

寄人
张泌
别梦依依到谢家，
小廊回合曲阑斜。
多情只是春庭月，
犹为离人照落花。

别房太尉墓
杜甫
他乡复行役，驻马别孤坟。
近泪无干土，低空有断云。
对棋陪谢傅，把剑觅徐君。
唯见林花落，莺啼送客闻。

无题重帏深下莫愁堂
李商隐
重帏深下莫愁堂，卧后清宵细细长。
神女生涯原是梦，小姑居处本无郎。
风波不信菱枝弱，月露谁教桂叶香。
直道相思了无益，未妨惆怅是轻狂。

20190519

题金陵渡
张祜
金陵津渡小山楼，
一宿行人自可愁。
潮落夜江斜月里，
两三星火是瓜洲。

旅夜书怀
杜甫
细草微风岸，危樯独夜舟。
星垂平野阔，月涌大江流。
名岂文章著，官应老病休。
飘飘何所似，天地一沙鸥。

无题凤尾香罗薄几重
李商隐
凤尾香罗薄几重，碧文圆顶夜深缝。
扇裁月魄羞难掩，车走雷声语未通。
曾是寂寥金烬暗，断无消息石榴红。
斑骓只系垂杨岸，何处西南任好风。

20190520

赠内人
张祜
禁门宫树月痕过，
媚眼惟看宿鹭窠。
斜拔玉钗灯影畔，
剔开红焰救飞蛾。

至德二载甫自京金光门出问道归凤翔乾元初从左拾遗移华州掾与亲故别因出此门有悲往事
杜甫
此道昔归顺，西郊胡正繁。
至今残破胆，应有未招魂。
近得归京邑，移官岂至尊。
无才日衰老，驻马望千门。

无题相见时难别亦难
李商隐
相见时难别亦难，东风无力百花残。
春蚕到死丝方尽，蜡炬成灰泪始干。
晓镜但愁云鬓改，夜吟应觉月光寒。
蓬山此去无多路，青鸟殷勤为探看。

20190521

集灵台其一
张祜
日光斜照集灵台，
红树花迎晓露开。
昨夜上皇新授箓，
太真含笑入帘来。

月夜忆舍弟
杜甫
戍鼓断人行，边秋一雁声。
露从今夜白，月是故乡明。
有弟皆分散，无家问死生。
寄书长不达，况乃未休兵。

无题
李商隐
来是空言去绝踪，月斜楼上五更钟。
梦为远别啼难唤，书被催成墨未浓。
蜡照半笼金翡翠，麝熏微度绣芙蓉。
刘郎已恨蓬山远，更隔蓬山一万重。

20190522

集灵台其二
张祜
虢国夫人承主恩，
平明骑马入宫门。
却嫌脂粉污颜色，
淡扫蛾眉朝至尊。

天末怀李白
杜甫
凉风起天末，君子意如何。
鸿雁几时到，江湖秋水多。
文章憎命达，魑魅喜人过。
应共冤魂语，投诗赠汨罗。

无题
李商隐
飒飒东风细雨来，芙蓉塘外有轻雷。
金蟾噬锁烧香入，玉虎牵丝汲井回。
贾氏窥帘韩掾少，宓妃留枕魏王才。
春心莫共花争发，一寸相思一寸灰。

20190523

枫桥夜泊
张继
月落乌啼霜满天，
江枫渔火对愁眠。
姑苏城外寒山寺，
夜半钟声到客船。

月夜
杜甫
今夜鄜州月，闺中只独看。
遥怜小儿女，未解忆长安。
香雾云鬟湿，清辉玉臂寒。
何时倚虚幌，双照泪痕干。

无题
李商隐
昨夜星辰昨夜风，画楼西畔桂堂东。
身无彩凤双飞翼，心有灵犀一点通。
隔座送钩春酒暖，分曹射覆蜡灯红。
嗟余听鼓应官去，走马兰台类转蓬。

20190524

陇西行四首其二
陈陶
誓扫匈奴不顾身，
五千貂锦丧胡尘。
可怜无定河边骨，
犹是春闺梦里人。

春望
杜甫
国破山河在，城春草木深。
感时花溅泪，恨别鸟惊心。
烽火连三月，家书抵万金。
白头搔更短，浑欲不胜簪。

春雨
李商隐
怅卧新春白袷衣，白门寥落意多违。
红楼隔雨相望冷，珠箔飘灯独自归。
远路应悲春晼晚，残霄犹得梦依稀。
玉珰缄札何由达，万里云罗一雁飞。

20190525

杂诗
佚名
近寒食雨草萋萋，
著麦苗风柳映堤。
等是有家归未得，
杜鹃休向耳边啼。

春宿左省
杜甫
花隐掖垣暮，啾啾栖鸟过。
星临万户动，月傍九霄多。
不寝听金钥，因风想玉珂。
明朝有封事，数问夜如何。

筹笔驿
李商隐
猿鸟犹疑畏简书，风云常为护储胥。
徒令上将挥神笔，终见降王走传车。
管乐有才原不忝，关张无命欲何如。
他年锦里经祠庙，梁父吟成恨有余。

20190526

回乡偶书
贺知章
少小离家老大回，
乡音无改鬓毛衰。
儿童相见不相识，
笑问客从何处来。

旅宿
杜牧
旅馆无良伴，凝情自悄然。
寒灯思旧事，断雁警愁眠。
远梦归侵晓，家书到隔年。
沧江好烟月，门系钓鱼船。

锦瑟
李商隐
锦瑟无端五十弦，一弦一柱思华年。
庄生晓梦迷蝴蝶，望帝春心思杜鹃。
沧海月明珠有泪，蓝田日暖玉生烟。
此情可待成追忆，只是当时已惘然。

20190527

瑶瑟怨
温庭筠
冰簟银床梦不成，碧天如水夜云轻。
雁声远过潇湘去，十二楼中月自明。

春宫怨
杜荀鹤
早被婵娟误，欲妆临镜慵。
承恩不在貌，教妾若为容。
风暖鸟声碎，日高花影重。
年年越溪女，相忆采芙蓉。

隋宫
李商隐
紫泉宫殿锁烟霞，欲取芜城作帝家。
玉玺不缘归日角，锦帆应是到天涯。
于今腐草无萤火，终古垂杨有暮鸦。
地下若逢陈后主，岂宜重问后庭花。

20190528

月夜
刘方平
更深月色半人家，
北斗阑干南斗斜。
今夜偏知春气暖，
虫声新透绿窗纱。

和晋陵陆丞早春游望
杜审言
独有宦游人，偏惊物候新。
云霞出海曙，梅柳渡江春。
淑气催黄鸟，晴光转绿蘋。
忽闻歌古调，归思欲沾巾。

送魏万之京
李颀
朝闻游子唱离歌，昨夜微霜初渡河。
鸿雁不堪愁里听，云山况是客中过。
关城树色催寒近，御苑砧声向晚多。
莫见长安行乐处，空令岁月易蹉跎。

20190529

春怨
刘方平
纱窗日落渐黄昏，
金屋无人见泪痕。
寂寞空庭春欲晚，
梨花满地不开门。

杂诗三首其三
沈佺期
闻道黄龙戍，频年不解兵。
可怜闺里月，长在汉家营。
少妇今春意，良人昨夜情。
谁能将旗鼓，一为取龙城。

咏怀古迹其一
杜甫
支离东北风尘际，漂泊西南天地间。
三峡楼台淹日月，五溪衣服共云山。
羯胡事主终无赖，词客哀时且未还。
庾信平生最萧瑟，暮年诗赋动江关。

20190530

乌衣巷
刘禹锡
朱雀桥边野草花，
乌衣巷口夕阳斜。
旧时王谢堂前燕，
飞入寻常百姓家。

宿桐庐江寄广陵旧游
孟浩然
山暝闻猿愁，沧江急夜流。
风鸣两岸叶，月照一孤舟。
建德非吾土，维扬忆旧游。
还将两行泪，遥寄海西头。

咏怀古迹五首其二
杜甫
摇落深知宋玉悲，风流儒雅亦吾师。
怅望千秋一洒泪，萧条异代不同时。
江山故宅空文藻，云雨荒台岂梦思。
最是楚宫俱泯灭，舟人指点到今疑。

20190531

和乐天春词
刘禹锡
新妆宜面下朱楼，
深锁春光一院愁。
行到中庭数花朵，
蜻蜓飞上玉搔头。

留别王维
孟浩然
寂寂竟何待，朝朝空自归。
欲寻芳草去，惜与故人违。
当路谁相假，知音世所稀。
只应守寂寞，还掩故园扉。

咏怀古迹五首其三
杜甫
群山万壑赴荆门，生长明妃尚有村。
一去紫台连朔漠，独有青冢向黄昏。
画图省识春风面，环佩空归夜月魂。
千载琵琶作胡语，分明怨恨曲中论。