在树莓派的Docker Swarm上运行Eureka

之前已经折腾过docker-compose在单机上部署应用了，但是这种方式在集群环境下就不行了，一台台机器累死人。于是使用了Docker官方提供的Docker Swarm，之所以没上k8s，一是因为树莓派不够用，带不动。二是也没有那种需求，能用上k8s怎么也得几十台机器吧。再就是k8s的学习也是需要时间，如果现在学了不用不久就忘了，还不如用到时再看。
vet3yuzuyeusshwiut3j.png

References:

先来说下我的目标，是在3台树莓派上跑的Docker Swarm中跑3个Eureka服务，每台一个，并且使用stack部署，方便管理。

关于Eureka的代码这里就不贴了，网上多的是，也可以参考我的https://github.com/zhengjunlu/bilibili/tree/master/microservice-eureka-server
由于最终是要打包成Docker镜像，所以这里用了Maven的一个插件，可以很方便的打包镜像。执行mvn clean install就会自动打包镜像并安装到本地的Docker中。
插件代码如下：

<plugin>
    <groupId>com.spotify</groupId>
    <artifactId>dockerfile-maven-plugin</artifactId>
    <version>${dockerfile-maven-version}</version>
    <executions>
        <execution>
            <id>default</id>
            <goals>
                <goal>build</goal>
                <goal>push</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <repository>racecoder/${project.name}</repository>
        <tag>${project.version}</tag>
        <buildArgs>
            <JAR_FILE>${project.build.finalName}.jar</JAR_FILE>
        </buildArgs>
    </configuration>
</plugin>

具体使用可以参考上面链接或网上资源。
eureka的配置如下：

#app config
spring:
  application:
    name: microservice-eureka-server
  profiles:
    active: dev0
    
eureka:
  instance:
    preferIpAddress: false
    IpAddress: "${HOST}"
  client:
    service-url:
      defaultZone: http://raspberrypi30:8760/eureka/, http://raspberrypi31:8761/eureka/, http://raspberrypi32:8762/eureka/
      
#  开发环境配置
---
spring:
  profiles: dev0

server:
  port: 8761

eureka:
  instance:
    hostname: eureka30

#  生产环境配置
---
spring:
  profiles: prod0
  
server:
  port: 8760
  
eureka:
  instance:
    hostname: raspberrypi30
      
---
spring:
  profiles: prod1

server:
  port: 8761
  
eureka:
  instance:
    hostname: raspberrypi31
    
---
spring:
  profiles: prod2

server:
  port: 8762
  
eureka:
  instance:
    hostname: raspberrypi32

默认的profile是dev0，实际使用的是prod0，prod1，prod2。并且配置了IpAddress为${HOST}，表示从环境变量中获取，因为Docker的容器中不能正确获取ip地址，所以这里我直接手动设置。另外在配置端口时，一开始是想着全部配置8761端口，因为我确实有3台服务器，但是后来在swarm中使用时，发现swarm的ingress网络和实际的端口比较混乱，而且docke swarm自带了负载均衡，更使端口非常混乱。一旦某个地方的端口配错误(特别是defaultZone部分的配置)就可能导致这台机器Unavailable，所以最后决定每台机器的eureka端口都不一样。
本地的代码完了就提交到gayhub上，然后在我的树莓派上pull就行了。
然后执行mvn clean install等待编译代码和安装镜像。当然提前要装好JDK和Maven。

[root@raspberrypi30 microservice-eureka-server]# mvn clean install
[INFO] Scanning for projects...
[INFO] 
[INFO] --------------< com.racecoder:microservice-eureka-server >--------------
[INFO] Building microservice-eureka-server 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ microservice-eureka-server ---
[INFO] Deleting /opt/bilibili/microservice-eureka-server/target
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ microservice-eureka-server ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ microservice-eureka-server ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /opt/bilibili/microservice-eureka-server/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ microservice-eureka-server ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /opt/bilibili/microservice-eureka-server/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ microservice-eureka-server ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ microservice-eureka-server ---
[INFO] No tests to run.
[INFO] 
[INFO] --- maven-jar-plugin:3.1.1:jar (default-jar) @ microservice-eureka-server ---
[INFO] Building jar: /opt/bilibili/microservice-eureka-server/target/microservice-eureka-server.jar
[INFO] 
[INFO] --- spring-boot-maven-plugin:2.1.4.RELEASE:repackage (repackage) @ microservice-eureka-server ---
[INFO] Replacing main artifact with repackaged archive
[INFO] 
[INFO] --- dockerfile-maven-plugin:1.4.12:build (default) @ microservice-eureka-server ---
[INFO] dockerfile: null
[INFO] contextDirectory: /opt/bilibili/microservice-eureka-server
[INFO] Building Docker context /opt/bilibili/microservice-eureka-server
[INFO] Path(dockerfile): null
[INFO] Path(contextDirectory): /opt/bilibili/microservice-eureka-server
[INFO] 
[INFO] Image will be built as racecoder/microservice-eureka-server:0.0.1-SNAPSHOT
[INFO] 
[INFO] Step 1/7 : FROM dpsmyth/raspberrypi3-alpine-java
[INFO] 
[INFO] Pulling from dpsmyth/raspberrypi3-alpine-java
[INFO] Digest: sha256:d951c4541306b4c3c8310922f3497172e5d13cb0bbf598119a4c5a6e56ae5695
[INFO] Status: Image is up to date for dpsmyth/raspberrypi3-alpine-java:latest
[INFO]  ---> 5b6b20fca4b6
[INFO] Step 2/7 : VOLUME ["/tmp"]
[INFO] 
[INFO]  ---> Using cache
[INFO]  ---> bc1eac55c157
[INFO] Step 3/7 : RUN echo "#aliyun" > /etc/apk/repositories &&     echo "https://mirrors.aliyun.com/alpine/v3.8/main/" >> /etc/apk/repositories &&     echo "https://mirrors.aliyun.com/alpine/v3.8/community/" >> /etc/apk/repositories &&     apk update &&     apk add tzdata &&         ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&         echo "Asia/Shanghai" > /etc/timezone
[INFO] 
[INFO]  ---> Using cache
[INFO]  ---> 3fa981bf6b53
[INFO] Step 4/7 : COPY target/*.jar app.jar
[INFO] 
[INFO]  ---> c8b5a417007a
[INFO] Step 5/7 : RUN bash -c 'touch /app.jar'
[INFO] 
[INFO]  ---> Running in 3b4f605e33cc
[INFO] Removing intermediate container 3b4f605e33cc
[INFO]  ---> 3c53ab0e2df4
[INFO] Step 6/7 : EXPOSE 8761
[INFO] 
[INFO]  ---> Running in db42dfe8a621
[INFO] Removing intermediate container db42dfe8a621
[INFO]  ---> 4e528e3082bf
[INFO] Step 7/7 : ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/app.jar"]
[INFO] 
[INFO]  ---> Running in fe260c51bc46
[INFO] Removing intermediate container fe260c51bc46
[INFO]  ---> 80dc94baa17d
[INFO] [Warning] One or more build-args [JAR_FILE] were not consumed
[INFO] Successfully built 80dc94baa17d
[INFO] Successfully tagged racecoder/microservice-eureka-server:0.0.1-SNAPSHOT
[INFO] 
[INFO] Detected build of image with id 80dc94baa17d
[INFO] Building jar: /opt/bilibili/microservice-eureka-server/target/microservice-eureka-server-docker-info.jar
[INFO] Successfully built racecoder/microservice-eureka-server:0.0.1-SNAPSHOT
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ microservice-eureka-server ---
[INFO] Installing /opt/bilibili/microservice-eureka-server/target/microservice-eureka-server.jar to /root/.m2/repository/com/racecoder/microservice-eureka-server/0.0.1-SNAPSHOT/microservice-eureka-server-0.0.1-SNAPSHOT.jar
[INFO] Installing /opt/bilibili/microservice-eureka-server/pom.xml to /root/.m2/repository/com/racecoder/microservice-eureka-server/0.0.1-SNAPSHOT/microservice-eureka-server-0.0.1-SNAPSHOT.pom
[INFO] Installing /opt/bilibili/microservice-eureka-server/target/microservice-eureka-server-docker-info.jar to /root/.m2/repository/com/racecoder/microservice-eureka-server/0.0.1-SNAPSHOT/microservice-eureka-server-0.0.1-SNAPSHOT-docker-info.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:46 min
[INFO] Finished at: 2019-09-10T19:03:56+08:00
[INFO] ------------------------------------------------------------------------
[root@raspberrypi30 microservice-eureka-server]#

这样就是Maven已经帮我们打包好并且安装好Docker镜像了，然后因为我安装了Docker私有仓库，方便我更快速的发布镜像，也省得来回拷。
先给要push的镜像打个tag，再推到仓库去：

[root@raspberrypi30 ~]# docker tag racecoder/microservice-eureka-server:0.0.1-SNAPSHOT 127.0.0.1:5000/racecoder/microservice-eureka-server:0.0.1-SNAPSHOT
[root@raspberrypi30 ~]# docker push 127.0.0.1:5000/racecoder/microservice-eureka-server:0.0.1-SNAPSHOT
The push refers to repository [127.0.0.1:5000/racecoder/microservice-eureka-server]
02b09df5c963: Pushed 
f25c1ecd8d03: Pushed 
8d9a9964ec75: Layer already exists 
b7fae5a10eb1: Layer already exists 
62cccd28d309: Layer already exists 
7dec9693384b: Layer already exists 
ed073222c9bc: Layer already exists 
6a3d82fa764f: Layer already exists 
4c05c8420146: Layer already exists 
4d6f2b7ff2f4: Layer already exists 
e093aa48fce2: Layer already exists 
0.0.1-SNAPSHOT: digest: sha256:b777dd8289388fe0eaf3caf2189b43bf415ad747e4ad7951f4f58d194580316c size: 2632
[root@raspberrypi30 ~]#

这样swarm中的每台机器都可以pull这个镜像，就可以很方便的发布，同时我们的配置都是在同一个配置文件中的，所以只要在环境中指定profile就行，别的什么都不用改。
然后是使用docker stack启动集群了，先来看下stack的配置，和compose的配置相似，但是支持更多的操作和指令。如下：

version: '3'
services:
  # service name
  eureka30:
    # image
    image: 127.0.0.1:5000/racecoder/microservice-eureka-server:0.0.1-SNAPSHOT
    # mapping ports
    ports:
      - "8760:8760"
    deploy:
      placement:
        constraints:
          - node.hostname == raspberrypi30
    extra_hosts:
      - "raspberrypi30:192.168.1.30"
      - "raspberrypi31:192.168.1.31"
      - "raspberrypi32:192.168.1.32"
    dns:
      - 192.168.1.1
      - 114.114.114.114
      - 8.8.8.8
    volumes:
      - /var/log/webapp:/var/log/webapp
    environment:
      - HOST=192.168.1.30
      - SPRING_PROFILES_ACTIVE=prod0
  eureka31:
    # image
    image: 127.0.0.1:5000/racecoder/microservice-eureka-server:0.0.1-SNAPSHOT
    # mapping ports
    ports:
      - "8761:8761"
    deploy:
      placement:
        constraints:
          - node.hostname == raspberrypi31
    extra_hosts:
      - "raspberrypi30:192.168.1.30"
      - "raspberrypi31:192.168.1.31"
      - "raspberrypi32:192.168.1.32"
    dns:
      - 192.168.1.1
      - 114.114.114.114
      - 8.8.8.8
    volumes:
      - /var/log/webapp:/var/log/webapp
    environment:
      - HOST=192.168.1.31
      - SPRING_PROFILES_ACTIVE=prod1
  eureka32:
    # image
    image: 127.0.0.1:5000/racecoder/microservice-eureka-server:0.0.1-SNAPSHOT
    # mapping ports
    ports:
      - "8762:8762"
    deploy:
      placement:
        constraints:
          - node.hostname == raspberrypi32
    extra_hosts:
      - "raspberrypi30:192.168.1.30"
      - "raspberrypi31:192.168.1.31"
      - "raspberrypi32:192.168.1.32"
    dns:
      - 192.168.1.1
      - 114.114.114.114
      - 8.8.8.8
    volumes:
      - /var/log/webapp:/var/log/webapp
    environment:
      - HOST=192.168.1.32
      - SPRING_PROFILES_ACTIVE=prod2

可以看到这里指定了3个服务eureka0，eureka1，eureka2，镜像为刚刚我们push上去那个镜像，然后限制了每个应用的发布节点，因为每台机器的IP和profile都不一样，所以这里指定了每个hostname对应机器上的IP和环境配置。另外为了方便使用了hostname而不是直接用IP，额外加了hosts信息，并且将日志直接输出到宿主机的目录中，方便查看和搜集，最后指定了环境变量。
然后使用命令启动：

[root@raspberrypi30 ~]# docker stack deploy --resolve-image never --compose-file eureka-compose.yml eureka
Creating network eureka_default
Creating service eureka_eureka31
Creating service eureka_eureka32
Creating service eureka_eureka30
[root@raspberrypi30 ~]#

使用--compose-file指定stack命令使用的文件，指定--resolve-image never是不校验镜像的架构信息，因为这个Maven插件生成的镜像默认是"amd64"架构，然后我的树莓派架构是"armv7l"，就导致执行stack deploy命令后swarm一直无响应，查看提示“no suitable node (unsupported platform on 3 nodes)”，意思是swarm由于平台不支持镜像，所以没有找到合适的节点启动，就一直pending在那。
正常启动后查看服务：

[root@raspberrypi30 ~]# docker stack ls
NAME                SERVICES            ORCHESTRATOR
eureka              3                   Swarm
[root@raspberrypi30 ~]# docker stack ps eureka 
ID                  NAME                IMAGE                                                                NODE                DESIRED STATE       CURRENT STATE         ERROR               PORTS
in33mng187yq        eureka_eureka30.1   127.0.0.1:5000/racecoder/microservice-eureka-server:0.0.1-SNAPSHOT   raspberrypi30       Running             Running 2 hours ago                       
y93p03fryy3s        eureka_eureka32.1   127.0.0.1:5000/racecoder/microservice-eureka-server:0.0.1-SNAPSHOT   raspberrypi32       Running             Running 2 hours ago                       
l2f3r8fjo6vo        eureka_eureka31.1   127.0.0.1:5000/racecoder/microservice-eureka-server:0.0.1-SNAPSHOT   raspberrypi31       Running             Running 2 hours ago                       
[root@raspberrypi30 ~]#

看到服务正常启动了，再查看eureka的面板，其中一个如下：
eureka.png

面板显示正常，并且确实成功运行了。

标签: none

添加新评论

ali-01.gifali-58.gifali-09.gifali-23.gifali-04.gifali-46.gifali-57.gifali-22.gifali-38.gifali-13.gifali-10.gifali-34.gifali-06.gifali-37.gifali-42.gifali-35.gifali-12.gifali-30.gifali-16.gifali-54.gifali-55.gifali-59.gif

加载中……