Read "Dynamo, Amazon’s Highly Available Key-value Store" Read "Bigtable, A Distributed Storage System for Structured Data" Read "Streaming Systems" 3, Watermarks Read "Streaming Systems" 1&2, Streaming 101 Read "F1, a distributed SQL database that scales" Read "Zanzibar, Google’s Consistent, Global Authorization System" Read "Spanner, Google's Globally-Distributed Database" Read "Designing Data-intensive applications" 12, The Future of Data Systems IOS development with Swift Read "Designing Data-intensive applications" 10&11, Batch and Stream Processing Read "Designing Data-intensive applications" 9, Consistency and Consensus Read "Designing Data-intensive applications" 8, Distributed System Troubles Read "Designing Data-intensive applications" 7, Transactions Read "Designing Data-intensive applications" 6, Partitioning Read "Designing Data-intensive applications" 5, Replication Read "Designing Data-intensive applications" 3&4, Storage, Retrieval, Encoding Read "Designing Data-intensive applications" 1&2, Foundation of Data Systems Three cases of binary search TAMU Operating System 2 Memory Management TAMU Operating System 1 Introduction Overview in cloud computing 2 TAMU Operating System 7 Virtualization TAMU Operating System 6 File System TAMU Operating System 5 I/O and Disk Management TAMU Operating System 4 Synchronization TAMU Operating System 3 Concurrency and Threading TAMU Computer Networks 5 Data Link Layer TAMU Computer Networks 4 Network Layer TAMU Computer Networks 3 Transport Layer TAMU Computer Networks 2 Application Layer TAMU Computer Networks 1 Introduction Overview in distributed systems and cloud computing 1 A well-optimized Union-Find implementation, in Java A heap implementation supporting deletion TAMU Advanced Algorithms 3, Maximum Bandwidth Path (Dijkstra, MST, Linear) TAMU Advanced Algorithms 2, B+ tree and Segment Intersection TAMU Advanced Algorithms 1, BST, 2-3 Tree and Heap TAMU AI, Searching problems Factorization Machine and Field-aware Factorization Machine for CTR prediction TAMU Neural Network 10 Information-Theoretic Models TAMU Neural Network 9 Principal Component Analysis TAMU Neural Network 8 Neurodynamics TAMU Neural Network 7 Self-Organizing Maps TAMU Neural Network 6 Deep Learning Overview TAMU Neural Network 5 Radial-Basis Function Networks TAMU Neural Network 4 Multi-Layer Perceptrons TAMU Neural Network 3 Single-Layer Perceptrons Princeton Algorithms P1W6 Hash Tables & Symbol Table Applications Stanford ML 11 Application Example Photo OCR Stanford ML 10 Large Scale Machine Learning Stanford ML 9 Anomaly Detection and Recommender Systems Stanford ML 8 Clustering & Principal Component Analysis Princeton Algorithms P1W5 Balanced Search Trees TAMU Neural Network 2 Learning Processes TAMU Neural Network 1 Introduction Stanford ML 7 Support Vector Machine Stanford ML 6 Evaluate Algorithms Princeton Algorithms P1W4 Priority Queues and Symbol Tables Stanford ML 5 Neural Networks Learning Princeton Algorithms P1W3 Mergesort and Quicksort Stanford ML 4 Neural Networks Basics Princeton Algorithms P1W2 Stack and Queue, Basic Sorts Stanford ML 3 Classification Problems Stanford ML 2 Multivariate Regression and Normal Equation Princeton Algorithms P1W1 Union and Find Stanford ML 1 Introduction and Parameter Learning

File manipulation with os module in python


import os

File manipulation

# Create some file
touch foo.txt
echo Hello > foo.txt
cat foo.txt
# rename file
cat bar.txt
# remove file

Change directory

# current dir
print os.getcwd() # current working directory
# go down
print os.getcwd()

# go back up
os.chdir(os.pardir) # or simply os.chdir('..')
print os.getcwd()

List all listdir

# listdir
# touch a.txt b.txt
for file in os.listdir('.'):
    # os.listdir() return a list
    if file.endswith('.txt'):
        print file

Walk os.walk

# Directory tree generator.
# For each dir in the dir tree rooted at top (including top
# itself, but excluding '.' and '..'), yields a 3-tuple
# dirpath, dirnames, filenames
for dirpath, dirnames, filenames in os.walk('.'):
    print dirnames
    print filenames
    break # only one level needed, or just use listdir
['folder1', 'folder2']
['.DS_Store', 'a.txt', 'b.txt']

Add or remove directory

Single level
# make a dir, one level, no duplication allowed
# remove a dir, one level, not empty will raise OSError
Mulitiple level
# make dirs, multipul level
# remove all empty directories above it, ensure empty
Non empty directory
# remove non empty dir, ust a new module shutil.rmtree
# copy function is also useful
import shutil
# copy a.txt to backup folder
# or just shutil.copy('a.txt','backup/')
# use shutil.copytree to copy a folder like cp -r
# remove non empty folder

os.path module

# is a dir or not
# is a file or not
# determine the presence of path(a file or dir); os.path.lexists?
# Join two or more pathname components, inserting '/' as needed.
# If any component is an absolute path, 
# all previous path components will be discarded.
# split a pathname. Returns "(head, tail)" 
# where "tail" is everything after the final slash.
('/Users/john', 'a.txt')
# split the extension from a pathname
('/Users/john/a', '.txt')
# determine the size of a path(file or dir)

Creative Commons License
Melon blog is created by melonskin. This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2016-2020. All rights reserved by melonskin. Powered by Jekyll.