Read "Streaming Systems" 1&2, Streaming 101 Read "F1, a distributed SQL database that scales" Read "Zanzibar, Google’s Consistent, Global Authorization System" Read "Spanner, Google's Globally-Distributed Database" Read "Designing Data-intensive applications" 12, The Future of Data Systems IOS development with Swift Read "Designing Data-intensive applications" 10&11, Batch and Stream Processing Read "Designing Data-intensive applications" 9, Consistency and Consensus Read "Designing Data-intensive applications" 8, Distributed System Troubles Read "Designing Data-intensive applications" 7, Transactions Read "Designing Data-intensive applications" 6, Partitioning Read "Designing Data-intensive applications" 5, Replication Read "Designing Data-intensive applications" 3&4, Storage, Retrieval, Encoding Read "Designing Data-intensive applications" 1&2, Foundation of Data Systems Three cases of binary search TAMU Operating System 2 Memory Management TAMU Operating System 1 Introduction Overview in cloud computing 2 TAMU Operating System 7 Virtualization TAMU Operating System 6 File System TAMU Operating System 5 I/O and Disk Management TAMU Operating System 4 Synchronization TAMU Operating System 3 Concurrency and Threading TAMU Computer Networks 5 Data Link Layer TAMU Computer Networks 4 Network Layer TAMU Computer Networks 3 Transport Layer TAMU Computer Networks 2 Application Layer TAMU Computer Networks 1 Introduction Overview in distributed systems and cloud computing 1 A well-optimized Union-Find implementation, in Java A heap implementation supporting deletion TAMU Advanced Algorithms 3, Maximum Bandwidth Path (Dijkstra, MST, Linear) TAMU Advanced Algorithms 2, B+ tree and Segment Intersection TAMU Advanced Algorithms 1, BST, 2-3 Tree and Heap TAMU AI, Searching problems Factorization Machine and Field-aware Factorization Machine for CTR prediction TAMU Neural Network 10 Information-Theoretic Models TAMU Neural Network 9 Principal Component Analysis TAMU Neural Network 8 Neurodynamics TAMU Neural Network 7 Self-Organizing Maps TAMU Neural Network 6 Deep Learning Overview TAMU Neural Network 5 Radial-Basis Function Networks TAMU Neural Network 4 Multi-Layer Perceptrons TAMU Neural Network 3 Single-Layer Perceptrons Princeton Algorithms P1W6 Hash Tables & Symbol Table Applications Stanford ML 11 Application Example Photo OCR Stanford ML 10 Large Scale Machine Learning Stanford ML 9 Anomaly Detection and Recommender Systems Stanford ML 8 Clustering & Principal Component Analysis Princeton Algorithms P1W5 Balanced Search Trees TAMU Neural Network 2 Learning Processes TAMU Neural Network 1 Introduction Stanford ML 7 Support Vector Machine Stanford ML 6 Evaluate Algorithms Princeton Algorithms P1W4 Priority Queues and Symbol Tables Stanford ML 5 Neural Networks Learning Princeton Algorithms P1W3 Mergesort and Quicksort Stanford ML 4 Neural Networks Basics Princeton Algorithms P1W2 Stack and Queue, Basic Sorts Stanford ML 3 Classification Problems Stanford ML 2 Multivariate Regression and Normal Equation Princeton Algorithms P1W1 Union and Find Stanford ML 1 Introduction and Parameter Learning

Node.js,Express, API and MongoDB basics

2018-01-06

Node

Introduction

Node.js is a JavaScript runtime built on Chrome’s V8 JavaScript engine. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient.

Usage

Interact with Node console

Just type node to open the console(js). alert() is not available here.

Run a file with Node

Execute: node filename

NPM

Node.js’ package ecosystem, npm, is the largest ecosystem of open source libraries in the world.

  • Do installation by npm install package-name in console
  • Import package inside js file by var catMe = require("cat-me");
  • Use the variable

Express

A popular web framework for Node.

Initialization

  • Use npm init to create a new package.json
    • This file includes project info and dependencies
  • Do npm install express --save to install express and packages
    • It will save the packages as a dependency in package.json

Hello world

var express = require("express");
var app = express();

// route
app.get("/", function(req, res) {
   res.send("<h1>hi there!</h1>"); 
});

app.get("/bye", function(req, res) {
    res.send("bye");
});

// listen to a port and ip
app.listen(process.env.PORT, process.env.IP, function() {
    console.log("Server started");
});

Routing

  • * route matcher
  • route order
    • Route order matters. The app check if a route matches one by one from start to end
  • routes containing route parameters
app.get("/", function(req, res) {
   res.send("<h1>hi there!</h1>"); 
});

app.get("/bye", function(req, res) {
    res.send("bye");
});

// pages not found
app.get("*", function(req, res) {
    res.send("Star page");
})

Route parameters are using pattern matching multiple terms (but only single thing, not deeper dir).

//use colon
app.get("/r/:subTerm", function(req, res) {
    // access variable
    var subTerm = req.params.subTerm;
    res.send(subTerm + " page");
});

Templates and EJS

  • Use res.render() to render HTML(from an EJS file)
  • Pass variables to EJS templates

Step:

  1. npm install ejs --save
  2. Put EJS file inside a views folder. It’s a HTML template.
  3. Render in js file

Note = sign will return a value from js statement and add it to HTML.

<!-- variable come from js file, everything inside <%= %> is js -->
<h1>You fell in love with <%= thingVar.toUpperCase() %></h1> 
// pass variable to template
app.get("/love/:thing", function(req, res) {
    var thing = req.params.thing;
    var posts = [{id: 1}, {id: 2}, {id: 3}];
    res.render("love.ejs", {thingVar: thing, posts: posts});
});

EJS: Conditionals and Loops

If:

<!-- take away the = sign, because it doesn't return value -->
<% if (thingVar.toLowerCase() === "rusty") { %>
    <p>Good choice</p>
<% } else { %>
    <p>else..</p>
<% } %>

Loop:

<% posts.forEach(function(post) {  %>
    <li>The id is <%= post.id %></li>
<% }) %>

Styles and partials

For styles:

  • Use external style sheet and link it
    • Add css file in public folder
    • Import public folder in app.js
  • Can add style tag on single page, not recommended
// use relative path in public folder
app.use(express.static(__dirname + "/public"));
<linke rel="stylesheet" href="style.css">

Improvement:

Tell app we are using ejs ahead of time and remove .ejs

// set
app.set("view engine", "ejs");

// use
res.render("home");

Partials

File template that we can include in other template, e.g. header, footer.

  • Put partials files in views/partials
  • Make partial file
    • To make sure style sheet works, put a / before it because this file is in a sub folder
  • Include partial file in other file

Partials:

<!DOCTYPE html>
<html>
<head>
    <title>Page Title</title>
    <link rel="stylesheet" type="text/css" href="/app.css">
</head>
<body>
<% include partials/header %>

Get method

We can pass term through a GET method and access it with req.query.*

<form action="/results" method="GET">
    <input type="text" name=search placeholder="search term">
    <button>Submit</button>
</form>
app.get("/results", function(req, res) {
    var search = req.query.search;
    request("http://www.omdbapi.com/?s=" + search + "&apikey=thewdb", 
            function(error, response, body) {
        if (!error && response.statusCode == 200) {
            // parse JSON
            var data = JSON.parse(body);
            console.log(data);
            res.render("results", {data: data});
        }
    });
});

Post method

In order to parse the req.body, we need to install body-parser

  1. install, npm install body-parser --save
  2. import, var bodyParser = require("body-parser");
  3. set, app.use(bodyParser.urlencoded({extended: true}));

The form data will be sent as an object in req.body

var friends = ["a", "b", "c"];
app.get("/friends", function(req, res) {
    res.render("friends", {friends: friends});
});


app.post("/addfriend", function(req, res) {
    var newfriend = req.body.newfriend;
    friends.push(newfriend);
    // redirect to another page
    res.redirect("/friends");
});
<form action="/addfriend" method="POST">
    <input type="text" name="newfriend" placeholder="friend's name">
    <button>Add</button>
</form>

API

A collection of APIs: https://www.programmableweb.com/

XML and JSON are the data types that APIs usually return back.

XML

Extended Markup Language. It’s similar to HTML but it does not describe presendation like HTML does (bold, etc). Only key-value pairs.

<person>
    <age>21</age>
    <name>Jack</name>
</person>

JSON

It looks like Javascript objects; but everything is a string.

A Chrome plug-in: JSONview. Use it view JSON in browser.

{
    "person" : {
        "age": "21",
        "name": "Jack"
    }
}

Make HTTP request

Use terminal: curl www.google.com

In Node

Use request package

var request = require('request');
request('http://www.google.com', function (error, response, body) {
  console.log('error:', error); // Print the error if one occurred
  console.log('statusCode:', response && response.statusCode); // Print the response status code if a response was received
  console.log('body:', body); // Print the HTML for the Google homepage.
});

Example

var request = require('request');
request("https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in%20(select%20woeid%20from%20geo.places(1)%20where%20text%3D%22nome%2C%20ak%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys", function(error, response, body) {
    // check error
    if (!error && response.statusCode == 200) {
        // parse JSON
        var parsedData = JSON.parse(body);
        console.log(parsedData["query"]["results"]["channel"]);
    }
})

MongoDB

NoSQL DB, use JSON-like object to store data.

Installation

  1. sudo apt-get install -y mongodb-org
  2. cd ~; mkdir data
  3. echo 'mongod --bind_ip=$IP --dbpath=data --nojournal --rest "$@"' > mongod
  4. chmod a+x mongod

Run MongoDB on the project root by: ./mongod

Update 3.6.1

killall mongod
sudo apt-get purge -y mongodb-org*
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list
sudo apt-get update
sudo apt-get install -y mongodb-org

Shutdown

  • Shutdown, ctrl + c
  • If it is timeout and crashed, repair it by cd ~; ./mongod --repair
  • Still have trouble, going to the data folder cd ~/data, then rm mongod.lock; cd ~; ./mongod

Commands

  • mongod: open mongo daemon
  • mongo: open interactive command line
  • help: list of commands
  • show dbs: show db names
  • use: create or use a db, use new-db-name
  • insert: db.db-name.insert({name: "jack", age: 44})
  • show collections: show things in a db
  • find: db.db-name.find() will find all; db.db-name.find({name:"jack"}) conditional
  • update: db.db-name.update({name: "jack"}, {$set: {age: 19}}), (condition, update), use $set will preserve initial info, otherwise all info will be overwritten.
  • remove: db.db-name.remove({name: "jack"}).limit(1); no limit() will remove all.

Mongoose for Node

Install: npm install mongoose --save

var mongoose = require("mongoose");
mongoose.connect("mongodb://localhost/cat_app", {useMongoClient: true});

// define schema
var catSchema = new mongoose.Schema({
    name: String,
    age: Number,
    place: String
});

// compile shema into a model
var Cat = mongoose.model("Cat", catSchema);

// add a new 
var cat1 = new Cat({
    name: "Cat1",
    age: 11,
    place: "Cal"
});
cat1.save(function(err, cat){
    // cat is what is saved in db
    if (err) {
        console.log("error");
    } else {
        console.log("good");
        console.log(cat);
    }
});

    // or another method
Cat.create({
    name: "Cat1",
    age: 11,
    place: "Cal"
}, function(err, cat) {
    if (err) {
        console.log("error");
    } else {
        console.log("good");
        console.log(cat);
    }
});


// retrieve all cats
Cat.find({}, function(err, cats) {
    if (err) {
        console.log("error");
    } else {
        console.log("good");
    }
});

// find one
Cat.findOne();

// all objects have a built-in ID
cat._id;

// FindById
Cat.findById(id, function(err, foundCat){
     // if err
});

// findByIdAndUpdate
Cat.findByIdAndUpdate(id, newCatObj, function(err, foundCat){
     // if err
});

// findByIdAndRemove
Cat.findByIdAndRemove(id, function(err, foundCat){
     // if err
});

Data association

The relations between data models.

One to many:

var postSchema = new mongoose.Schema({
    title: String,
    content: String
});

var Post = mongoose.model("Post", postSchema);

var userSchema = new mongoose.Schema({
    name: String,
    email: String,
    // one to many
    posts: [postSchema]
});

var User = mongoose.model("User", userSchema);

var newUser = new User({
    email: "tsj@gmail.com",
    name: "Skin"
});

newUser.posts.push({
    title: "first post",
    content: "hi"
});

newUser.save(function(err, user) {
    // if err
});

Object references

Referencing the id of posts in user, rather than the post objects.

// use this schema
var userSchema = new mongoose.Schema({
    name: String,
    email: String,
    // ref, one to many
    posts: [
        {
            type: mongoose.Schema.Types.ObjectId,
            ref: "Post"
        }
    ]
});

aUser.posts.push(newPost._id);
aUser.save();

// find posts for a user, the post objects will be inside returned user variable
User.findOne({name: "jack"}).populate("posts").exec(function(err, user) {

});

Module exports

Helps clean code. We can move different function related code into separate files.

In models/post.js

var mongoose = require("mongoose");

var postSchema = new mongoose.Schema({
    title: String,
    content: String
});

module.exports = mongoose.model("Post", postSchema);

In other file where this model is used.

var Post = require("./models/post");

Authentication

User signup, login and session stuff.

Existing tool: Passport, it can use Facebook, Google, twitter etc account.

  • Passport
  • Passport Local
  • Passport Local Mongoose

Session tool: Express-Session

Setup

 npm install express mongoose passport passport-local passport-local-mongoose body-parser express-session ejs --save

Demo

https://github.com/melonskin/smallWebApps/tree/master/authentication

Misc

Post object

If we do <input type="text" name="blog[title]"> in HTML, data will be put inside an object like req.body.blog.title. So we can use object blog directly.

Date

There is a moment package which is more powerful to handle.

// get human readable 
date.toDateString()

Embeded code

  • Just run the code: <% %>
  • Display the content: <%= %>
  • Evaluate the content: <%- %>, dangerous

Sanitize

Remove all dangerous code from user input.

Install npm install express-sanitizer --save

var expressSanitizer = require("express-sanitizer");

app.use(bodyParser.urlencoded({extended: true}));
//after above line
app.use(expressSanitizer());


// inside create and update
    // content is the value we want to sanitize inside blog object
req.body.blog.content = req.sanitize(req.body.blog.content);

Creative Commons License
Melon blog is created by melonskin. This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2016-2019. All rights reserved by melonskin. Powered by Jekyll.