天天看點

基于七牛Python SDK寫的一個同步腳本

需求背景

最近剛搭了個markdown靜态部落格,想把部落格的圖檔放到雲存儲中。

經過調研覺得七牛可以滿足我個人的需求,就選它了。

部落格要引用圖檔就要先将圖檔上傳到雲上。

雖然七牛網站背景可以上傳檔案,但每次上傳都需要先登入,然後選擇圖檔,設定連接配接位址,才能上傳。

這個過程有些繁瑣,是以我便想用七牛雲提供的SDK寫個一同步工具,友善增量同步檔案。

有了這個想法,就馬上行動。花了大概一個上午的時間,總算把這個工具給寫出來,

并放到GitOSC和github上。

實作

(部落格園的markdown代碼區顯示不友好,可以到我的個人部落格中浏覽)

#!/usr/bin/env python
#-*- coding:utf-8 -*-
# 
# AUTHOR = "heqingpan"
# AUTHOR_EMAIL = "[email protected]"
# URL = "http://git.oschina.net/hqp/qiniu_sync"

import qiniu
from qiniu import Auth
from qiniu import BucketManager
import os
import re

access_key = ''
secret_key = ''
bucket_name = ''
bucket_domain = ''

q = Auth(access_key, secret_key)
bucket = BucketManager(q)
basedir=os.path.realpath(os.path.dirname(__file__))
filename=__file__
ignore_paths=[filename,"{0}c".format(filename)]
ignore_names=[".DS_Store",".git",".gitignore"]
charset="utf8"
diff_time=2*60


def list_all(bucket_name, bucket=None, prefix="", limit=100):
    rlist=[]
    if bucket is None:
        bucket = BucketManager(q)
    marker = None
    eof = False
    while eof is False:
        ret, eof, info = bucket.list(bucket_name, prefix=prefix, marker=marker, limit=limit)
        marker = ret.get('marker', None)
        for item in ret['items']:
            rlist.append(item["key"])
    if eof is not True:
        # 錯誤處理
        #print "error"
        pass
    return rlist

def get_files(basedir="",fix="",rlist=None,ignore_paths=[],ignore_names=[]):
    if rlist is None:
        rlist=[]
    for subfile in os.listdir(basedir):
        temp_path=os.path.join(basedir,subfile)
        tp=os.path.join(fix,subfile)
        if tp in ignore_names:
            continue
        if tp in ignore_paths:
            continue
        if os.path.isfile(temp_path):
            rlist.append(tp)
        elif os.path.isdir(temp_path):
            get_files(temp_path,tp,rlist,ignore_paths,ignore_names)
    return rlist

def get_valid_key_files(subdir=""):
    basedir=subdir or basedir
    files = get_files(basedir=basedir,ignore_paths=ignore_paths,ignore_names=ignore_names)
    return map(lambda f:(f.replace("\\","/"),f),files)


def sync():
    qn_keys=list_all(bucket_name,bucket)
    qn_set=set(qn_keys)
    l_key_files=get_valid_key_files(basedir)
    k2f={}
    update_keys=[]
    u_count=500
    u_index=0
    for k,f in l_key_files:
        k2f[k]=f
        str_k=k
        if isinstance(k,str):
            k=k.decode(charset)
        if k in qn_set:
            update_keys.append(str_k)
            u_index+=1
            if u_index > u_count:
                u_index-=u_count
                update_file(k2f,update_keys)
                update_keys=[]
        else:
            # upload
            upload_file(k,os.path.join(basedir,f))
    if update_keys:
        update_file(k2f,update_keys)
    print "sync end"

def update_file(k2f,ulist):
    ops=qiniu.build_batch_stat(bucket_name,ulist)
    rets,infos = bucket.batch(ops)
    for i in xrange(len(ulist)):
        k=ulist[i]
        f=k2f.get(k)
        ret=rets[i]["data"]
        size=ret.get("fsize",None)
        put_time = int(ret.get("putTime")/10000000)
        local_size=os.path.getsize(f)
        local_time=int(os.path.getatime(f))
        if local_size==size:
            continue
        if put_time >= local_time - diff_time:
            # is new
            continue
        # update
        upload_file(k,os.path.join(basedir,f))

def upload_file(key,localfile):
    print "upload_file:"
    print key
    token = q.upload_token(bucket_name, key)
    mime_type = get_mime_type(localfile)
    params = {'x:a': 'a'}
    progress_handler = lambda progress, total: progress
    ret, info = qiniu.put_file(token, key, localfile, params, mime_type, progress_handler=progress_handler)

def get_mime_type(path):
    mime_type = "text/plain"
    return mime_type

def main():
    sync()

if __name__=="__main__":
    main()
                

這個同步腳本支援批量比較檔案,差異增量更新、批量更新。

使用方式

  • 安裝七牛Python SDK

    pip install qiniu

  • 填寫腳本檔案(qiniusync.py)的配置資訊
access_key = ''
secret_key = ''
bucket_name = ''
           
注冊後可以拿到對應的資訊
  • 将腳本檔案(qiniusync.py)拷貝到待同步根目錄
  • 運作腳本
python qiniusync.py
           

後記

寫完送出之後才發現,七牛已經提供相應的工具,我這個算是重複造輪子吧。

既然已經寫,就發出來,當做熟悉一下七牛的SDK也不錯,說不定以後還能用的上。

部落格園的markdown代碼區顯示不友好,可以到我的個人部落格中浏覽

轉載于:https://www.cnblogs.com/shizioo/p/4751776.html