Docker仓库之Harbor

Docker仓库

一、Docker单机仓库

Docker Registry作为Docker的核心组件之一负责单主机的镜像内容的存储与分发，客户端的docker pull以及push命令都将直接与registry进行交互,最初版本的registry 由Python实现，由于设计初期在安全性，性能以及API的设计上有着诸多的缺陷，该版本在0.9之后停止了开发，由新项目distribution（新的docker register被称为Distribution）来重新设计并开发下一代registry，新的项目由go语言开发，所有的API，底层存储方式，系统架构都进行了全面的重新设计已解决上一代registry中存在的问题，2016年4月份registry 2.0正式发布，docker 1.6版本开始支持registry 2.0，而八月份随着docker 1.8 发布，docker hub正式启用2.1版本registry全面替代之前版本 registry，新版registry对镜像存储格式进行了重新设计并和旧版不兼容，docker 1.5和之前的版本无法读取2.0的镜像，另外，Registry 2.4版本之后支持了回收站机制，也就是可以删除镜像了，在2.4版本之前是无法支持删除镜像的，所以如果你要使用最好是大于Registry 2.4版本的

官方文档地址: https://docs.docker.com/registry/

官方github 地址: https://github.com/docker/distribution

官方部署文档: https://github.com/docker/docker.github.io/blob/master/registry/deploying.md

• 下载docker registry 镜像

[root@localhost docker]# docker pull registry:2.7.1
2.7.1: Pulling from library/registry
79e9f2f55bf5: Pull complete 
0d96da54f60b: Pull complete 
5b27040df4a2: Pull complete 
e2ead8259a04: Pull complete 
3790aef225b9: Pull complete 
Digest: sha256:169211e20e2f2d5d115674681eb79d21a217b296b43374b8e39f97fcf866b375
Status: Downloaded newer image for registry:2.7.1
docker.io/library/registry:2.7.1
[root@localhost docker]# docker images
REPOSITORY   TAG       IMAGE ID       CREATED        SIZE
registry     2.7.1     b8604a3fe854   3 months ago   26.2MB
           

1、创建单机仓库

1.1 创建账号启动docker registry容器

[root@localhost docker]# yum -y install httpd
[root@localhost docker]# htpasswd -Bbn test 123456 > /etc/docker/auth/registry
[root@localhost docker]# cat /etc/docker/auth/registry 
test:$2y$05$ecpX/anNOrNRe7xgenNZVOqmA1DWlb8e1AAUWD2LGFrV5pFnBtMs2

[root@localhost docker]# docker run -d -p 5000:5000 --restart=always --name registry -v /etc/docker/auth:/auth -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/registry registry:2.7.1
a218fcf93146d0d2cf9888fc8bd65ef6a753ddb985996acf3a70487c469c037d

[root@localhost docker]# docker ps
CONTAINER ID   IMAGE            COMMAND                  CREATED              STATUS              PORTS                    NAMES
a218fcf93146   registry:2.7.1   "/entrypoint.sh /etc…"   About a minute ago   Up About a minute   0.0.0.0:5000->5000/tcp   registry
[root@localhost docker]# ss -tln
State      Recv-Q Send-Q                           Local Address:Port                                          Peer Address:Port              
LISTEN     0      128                                          *:5000                                                     *:*                  
LISTEN     0      128                                          *:22                                                       *:*                  
LISTEN     0      128                                       [::]:22                                                    [::]:* 
           

1.2 登录仓库

#docker login 默认使用https登录,而docker registry为http,所以默认登录失败
[root@localhost docker]# docker login 192.168.187.10:5000
Username: test
Password: 
Error response from daemon: Get "https://192.168.187.10:5000/v2/": http: server gave HTTP response to HTTPS client
           

• 将registry仓库地址加入到service单元文件

[root@localhost docker]# cat /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://boqr6s5g.mirror.aliyuncs.com"],
  "insecure-registry": ["192.168.187.10:5000"]
}
#或者

[root@localhost docker]# cat  /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry 192.168.187.10:5000


[root@localhost docker]#systemctl daemon-reload
[root@localhost docker]#systemctl restart docker

           

• 登录

[root@localhost docker]# docker login 192.168.187.10:5000
Username: test
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
           

1.3 上传下载镜像

#需要先登录（dcoker login）
[root@dockerserver2 ~]# docker tag centos:centos7.8.2003 192.168.187.10:5000/centos7:v1
[root@dockerserver2 ~]# docker push 192.168.187.10:5000/centos7:v1
The push refers to repository [192.168.187.10:5000/centos7]
fb82b029bea0: Pushed 
v1: digest: sha256:50b9a3bc27378889210f88d6d0695938e45a912aa99b3fdacfb9a0fef511f15a size: 529

#下载镜像
[root@dockerserver2 ~]# docker pull 192.168.187.10:5000/centos7:v1
v1: Pulling from centos7
9b4ebb48de8d: Pull complete 
Digest: sha256:50b9a3bc27378889210f88d6d0695938e45a912aa99b3fdacfb9a0fef511f15a
Status: Downloaded newer image for 192.168.187.10:5000/centos7:v1
192.168.187.10:5000/centos7:v1
[root@dockerserver2 ~]# docker images
REPOSITORY                    TAG       IMAGE ID       CREATED         SIZE
192.168.187.10:5000/centos7   v1        afb6fca791e0   21 months ago   203MB
[root@dockerserver2 ~]# docker run -it --rm 192.168.187.10:5000/centos7:v1
[root@14fd3b4ad4b8 /]# cat /etc/redhat-release 
CentOS Linux release 7.8.2003 (Core)
           

二、Docker分布式仓库Harbor

1、 Harhor

Harbor是一个用于存储和分发Docker镜像的企业级Registry服务器，由VMware开源，其通过添加一些企业必需的功能特性，例如安全、标识和管理等，扩展了开源 Docker Distribution。作为一个企业级私有Registry服务器，Harbor 提供了更好的性能和安全。提升用户使用Registry构建和运行环境传输镜像的效率。Harbor支持安装在多个Registry节点的镜像资源复制，镜像全部保存在私有 Registry 中，确保数据和知识产权在公司内部网络中管控，另外，Harbor也提供了高级的安全特性，诸如用户管理，访问控制和活动审计等

vmware 官方开源服务: https://vmware.github.io/

harbor 官方github 地址: https://github.com/vmware/harbor

harbor 官方网址: https://goharbor.io/

harbor 官方文档: https://goharbor.io/docs/

github文档: https://github.com/goharbor/harbor/tree/master/docs

官方功能介绍

• 基于角色的访问控制: 用户与Docker镜像仓库通过“项目”进行组织管理，一个用户可以对多个镜像仓库在同一命名空间（project）里有不同的权限
• 镜像复制: 镜像可在多个Registry实例中复制（同步）。尤其适合于负载均衡，高可用，混合云和多云的场景
• 图形化用户界面: 用户可以通过浏览器来浏览，检索当前Docker镜像仓库，管理项目和命名空间
• AD/LDAP 支: Harbor可以集成企业内部已有的AD/LDAP，用于鉴权认证管理
• 审计管理: 所有针对镜像仓库的操作都可以被记录追溯，用于审计管理
• 国际化: 已拥有英文、中文、德文、日文和俄文的本地化版本。更多的语言将会添加进来
• RESTful API: 提供给管理员对于Harbor更多的操控, 使得与其它管理软件集成变得更容易
• 部署简单: 提供在线和离线两种安装工具， 也可以安装到vSphere平台(OVA方式)虚拟设备

Harbor组成

• Proxy: 对应启动组件nginx。它是一个nginx反向代理，代理Notary client（镜像认证）、Dockerclient（镜像上传下载等）和浏览器的访问请求（Core Service）给后端的各服务

• UI（Core Service）: 对应启动组件harbor-ui。底层数据存储使用mysql数据库，主要提供了四个

  子功能:

  • UI: 一个web管理页面ui
  • API: Harbor暴露的API服务
  • Auth: 用户认证服务，decode后的token中的用户信息在这里进行认证；auth后端可以接db、ldap、uaa三种认证实现
  • Token服务（上图中未体现）: 负责根据用户在每个project中的role来为每一个dockerpush/pull命令发布一个token，如果从docker client发送给registry的请求没有带token，registry会重定向请求到token服务创建token
• Registry: 对应启动组件registry。负责存储镜像文件，和处理镜像的pull/push命令。Harbor对镜像进行强制的访问控制，Registry会将客户端的每个pull、push请求转发到token服务来获取有效的token

• Admin Service: 对应启动组件harbor-adminserver。是系统的配置管理中心附带检查存储用量，

  ui和jobserver启动时候需要加载adminserver的配置

• Job Sevice: 对应启动组件harbor-jobservice。负责镜像复制工作的，他和registry通信，从一个registry pull镜像然后push到另一个registry，并记录job_log

• Log Collector: 对应启动组件harbor-log。日志汇总组件，通过docker的log-driver把日志汇总到

  一起

• DB: 对应启动组件harbor-db，负责存储project、 user、 role、replication、image_scan、

  access等的metadata数据

2、安装Harbor

下载地址: https://github.com/vmware/harbor/releases

安装文档: https://github.com/goharbor/harbor/blob/master/docs/install-config/_index.md

• 安装docker ---- 滤过
• 安装docker compose

[root@dockerserver2 ~]# curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   664  100   664    0     0    901      0 --:--:-- --:--:-- --:--:--   900
100 12.1M  100 12.1M    0     0  4631k      0  0:00:02  0:00:02 --:--:--  9.7M
[root@dockerserver2 ~]# 
[root@dockerserver2 ~]# ll /usr/local/bin/docker-compose 
-rw-r--r--. 1 root root 12737304 Feb 18 08:32 /usr/local/bin/docker-compose
[root@dockerserver2 ~]# chmod +x  /usr/local/bin/docker-compose 
[root@dockerserver2 ~]# ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
[root@dockerserver2 ~]# docker-compose version
docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

           

• 安装Harbor

以下使用 harbor 稳定版本v1.10.10安装包

下载离线完整安装包,推荐使用

[root@dockerserver2 ~]#wget https://github.com/goharbor/harbor/releases/download/v1.10.10/harbor-offline-installer-v1.10.10.tgz
[root@dockerserver2 src]# mkdir /apps
[root@dockerserver2 src]# tar -zxf harbor-offline-installer-v1.10.10.tgz
           

• 编辑harbor配置文件

最新文档: https://github.com/goharbor/harbor/blob/master/docs/install-config/configure-yml-file.

md

[root@dockerserver2 ~]#vim /apps/harbor/harbor.cfg
#只需要修改下面两行
hostname = 192.168.187.11 #修改此行,指向当前主机IP 或 FQDN
harbor_admin_password = 123456 #修改此行指定harbor登录用户admin的,默认用户:admin/Harbor12345
#可选项
ui_url_protocol = http #默认即可,如果修改为https,需要指定下面证书路径
ssl_cert = /data/cert/server.crt #默认即可,https时，需指定下面证书文件路径
ss_cert_key = /data/cert/server.key #默认即可,https时，需指定下面私钥文件路径
           

• 运行安装脚本

[root@dockerserver2 harbor]# yum -y install python
[root@dockerserver2 harbor]# ./install.sh 

[Step 0]: checking if docker is installed ...

Note: docker version: 20.10.12

[Step 1]: checking docker-compose is installed ...

Note: docker-compose version: 1.29.2

[Step 2]: loading Harbor images ...
[Step 3]: preparing environment ...
[Step 4]: preparing harbor configs ...
prepare base dir is set to /apps/harbor
/usr/src/app/utils/configs.py:100: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  configs = yaml.load(f)
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
/usr/src/app/utils/configs.py:90: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  versions = yaml.load(f)
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /secret/keys/secretkey
Generated certificate, key file: /secret/core/private_key.pem, cert file: /secret/registry/root.crt
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating registryctl   ... done
Creating redis         ... done
Creating registry      ... done
Creating harbor-db     ... done
Creating harbor-portal ... done
Creating harbor-core   ... done
Creating harbor-jobservice ... done
Creating nginx             ... done
✔ ----Harbor has been installed and started successfully.----

#安装harbor后会自动开启很多相关容器
[root@dockerserver2 harbor]# docker ps
CONTAINER ID   IMAGE                                  COMMAND                  CREATED              STATUS                        PORTS                       NAMES
1834217183fb   goharbor/nginx-photon:v1.10.10         "nginx -g 'daemon of…"   About a minute ago   Up About a minute (healthy)   0.0.0.0:80->8080/tcp        nginx
a726b631c455   goharbor/harbor-jobservice:v1.10.10    "/harbor/harbor_jobs…"   About a minute ago   Up About a minute (healthy)                               harbor-jobservice
d93cd0e5c4c0   goharbor/harbor-core:v1.10.10          "/harbor/harbor_core"    About a minute ago   Up About a minute (healthy)                               harbor-core
62f4feeea29f   goharbor/harbor-portal:v1.10.10        "nginx -g 'daemon of…"   About a minute ago   Up About a minute (healthy)   8080/tcp                    harbor-portal
7b4100dfc0be   goharbor/harbor-db:v1.10.10            "/docker-entrypoint.…"   About a minute ago   Up About a minute (healthy)   5432/tcp                    harbor-db
9ede3db336cb   goharbor/registry-photon:v1.10.10      "/home/harbor/entryp…"   About a minute ago   Up About a minute (healthy)   5000/tcp                    registry
0a86c46beb82   goharbor/redis-photon:v1.10.10         "redis-server /etc/r…"   About a minute ago   Up About a minute (healthy)   6379/tcp                    redis
d8cc10dda240   goharbor/harbor-registryctl:v1.10.10   "/home/harbor/start.…"   About a minute ago   Up About a minute (healthy)                               registryctl
3a1e7e0eb2bc   goharbor/harbor-log:v1.10.10           "/bin/sh -c /usr/loc…"   About a minute ago   Up About a minute (healthy)   127.0.0.1:1514->10514/tcp   harbor-log
           

• 开机启动harbor

[root@dockerserver2 harbor]# vim /lib/systemd/system/harbor.service
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor

[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/bin/docker-compose -f /apps/harbor/docker-compose.yml up
ExecStop=/usr/bin/docker-compose -f /apps/harbor/docker-compose.yml down
[Install]
WantedBy=multi-user.target

[root@dockerserver2 harbor]# systemctl daemon-reload
[root@dockerserver2 harbor]# systemctl enable harbor
Created symlink from /etc/systemd/system/multi-user.target.wants/harbor.service to /usr/lib/systemd/system/harbor.service.

           

3、使用Harbor

访问：http://192.168.187.11/

• 账号：admin/123456

• 使用单机harbor
  • 建立项目

harbor上必须先建立项目，才能上传镜像

• 登录harbor上传镜像

[root@ubuntu1804 ~]#vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
--insecure-registry 10.0.0.101 --insecure-registry 192.168.187.11

[root@dockerserver2 src]# systemctl daemon-reload
[root@dockerserver2 src]# systemctl restart docker
[root@dockerserver2 src]# docker login 192.168.187.11
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[root@dockerserver2 ~]# cat .docker/config.json
{
	"auths": {
		"192.168.187.10:5000": {
			"auth": "dGVzdDoxMjM0NTY="
		},
		"192.168.187.11": {
			"auth": "YWRtaW46MTIzNDU2"
		}
	}
}

#上传镜像
#修改 images 的名称，不修改成指定格式无法将镜像上传到 harbor 仓库
#格式： Harbor主机IP/项目名/image名字:版本

[root@dockerserver2 ~]# docker images
REPOSITORY                    TAG       IMAGE ID       CREATED         SIZE
ubuntu                        20.04     ba6acccedd29   4 months ago    72.8MB
192.168.187.10:5000/centos7   v1        afb6fca791e0   21 months ago   203MB
[root@dockerserver2 ~]# docker tag ubuntu:20.04 192.168.187.11/test/ubuntu-base:v1
[root@dockerserver2 ~]# docker push 192.168.187.11/test/ubuntu-base:v1
The push refers to repository [192.168.187.11/test/ubuntu-base]
9f54eef41275: Pushed 
v1: digest: sha256:7cc0576c7c0ec2384de5cbf245f41567e922aab1b075f3e8ad565f508032df17 size: 529

           

• 验证是否成功上传

**注意：**如果不事先建立项目，上传镜像会失败

• 可查看日志记录

• 下载镜像

下载前必须修改docker的service 文件，加入harbor服务器的地址才可以下载

[root@ubuntu1804 ~]#vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
--insecure-registry 10.0.0.101 --insecure-registry 192.168.187.11

[root@dockerserver2 src]# systemctl daemon-reload
[root@dockerserver2 src]# systemctl restart docker
[root@dockerserver1 src]# docker images
REPOSITORY   TAG       IMAGE ID       CREATED        SIZE
registry     2.7.1     b8604a3fe854   3 months ago   26.2MB
[root@dockerserver1 src]# docker pull 192.168.187.11/test/ubuntu-base:v1
v1: Pulling from test/ubuntu-base
7b1a6ab2e44d: Pull complete 
Digest: sha256:7cc0576c7c0ec2384de5cbf245f41567e922aab1b075f3e8ad565f508032df17
Status: Downloaded newer image for 192.168.187.11/test/ubuntu-base:v1
192.168.187.11/test/ubuntu-base:v1
[root@dockerserver1 src]# docker images
REPOSITORY                        TAG       IMAGE ID       CREATED        SIZE
registry                          2.7.1     b8604a3fe854   3 months ago   26.2MB
192.168.187.11/test/ubuntu-base   v1        ba6acccedd29   4 months ago   72.8MB
           

• 创建自动上传镜像脚本

[root@ubuntu1804 ~]#cd /data/dockerfile/web/nginx/1.16.1-alpine/
[root@ubuntu1804 1.16.1-alpine]#vim build.sh
[root@ubuntu1804 1.16.1-alpine]#cat build.sh
#!/bin/bash
TAG=$1
docker build -t 10.0.0.101/example/nginx-alpine:1.16.1-${TAG} .
docker push 10.0.0.101/example/nginx-alpine:1.16.1-${TAG}
docker rmi -f 10.0.0.101/example/nginx-alpine:1.16.1-${TAG}
[root@ubuntu1804 1.16.1-alpine]#bash build.sh v1
           

• 修改harbor配置

#后期如果修改harbor配置，比如: 修改IP地址等，可执行以下步骤生效
[root@ubuntu1804 ~]#cd /apps/harbor/
[root@ubuntu1804 harbor]#docker-compose stop
Stopping nginx ... done
Stopping harbor-portal ... done
Stopping harbor-jobservice ... done
Stopping harbor-core ... done
Stopping harbor-adminserver ... done
Stopping harbor-db ... done
Stopping registryctl ... done
Stopping registry ... done
Stopping redis ... done
Stopping harbor-log ...


#修改harbor配置
[root@dockerserver2 harbor]# vim /apps/harbor/harbor.yml

#更新配置
[root@dockerserver2 harbor]#/apps/harbor/prepare

#重新启动docker compose
[root@dockerserver2 harbor]#docker-compose start
           

4、Harbor 高可用

Harbor支持基于策略的Docker镜像复制功能，这类似于MySQL的主从同步，其可以实现不同的数据中心、不同的运行环境之间同步镜像，并提供友好的管理界面，大大简化了实际运维中的镜像管理工作，已经有用很多互联网公司使用harbor搭建内网docker仓库的案例，并且还有实现了双向复制功能

• 安装第二台harbor主机
• 注意：harbor.cfg中配置 hostname = 192.168.187.12
• 创建相同的项目

• 参考第一台主机信息,新建复制（同步）目标信息,将第一台主机设为复制的目标

• 输入第一台主机信息

• 第二台harbor上新建复制规则实现到第一台harbor的单向复制

• 在第一台harbor主机上重复上面操作，在第一台harbor上再执行下面操作，才实现双向同步
• 确认同步成功

5、Harbor https 配置

#安装docker步骤省略

#生成私钥和证书
[root@dockerserver2 harbor]#touch /root/.rnd
[root@dockerserver2 harbor]#mkdir /apps/harbor/certs/
[root@dockerserver2 harbor]#cd /apps/harbor/certs/

#生成CA证书
[root@dockerserver2 harbor]#openssl req -newkey rsa:4096 -nodes -sha256 -keyout
ca.key -x509 -subj "/CN=ca.org" -days 365 -out ca.crt

#生成harbor主机的证书申请
[root@dockerserver2 harbor]#openssl req -newkey rsa:4096 -nodes -sha256 -subj
"/CN=harbor.magedu.org" -keyout harbor.org.key -out harbo.org.csr

#给harbor主机颁发证书
[root@dockerserver2 harbor]#openssl x509 -req -in harbor.org.csr -CA ca.crt -
CAkey ca.key -CAcreateserial -out harbor.org.crt

[root@dockerserver2 harbor]#tree /apps/harbor/certs
/apps/harbor/certs
├── ca.crt
├── ca.key
├── ca.srl
├── harbor.org.crt
├── harbor.org.csr
└── harbor.org.key
0 directories, 6 files
[root@dockerserver2 harbor]#vim /apps/harbor/harbor.cfg
hostname = harbor.magedu.org
ui_url_protocol = https
ssl_cert = /apps/harbor/certs/harbor.org.crt
ssl_cert_key = /apps/harbor/certs/harbor.org.key
harbor_admin_password = 123456
[root@dockerserver2 harbor]#apt -y install python
[root@dockerserver2 harbor]#/apps/harbor/install.sh
           

• 上传下载镜像需要在客户端下载证书，否则会报错

[root@ubuntu1804 ~]#mkdir -pv /etc/docker/certs.d/