go|istio部署grpc服务时遇到的问题 linux|kubernetes

个人博客原文地址：http://www.lampnick.com/php/916 问题一：mycaller服务请求myresponser服务时，http接口可以通，grpc服务不通

使用sleep容器对mycaller服务进行请求

# kubectl exec -it -c sleep $(kubectl get pod-l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605" {"code":10000000,"error":"内部错误","detail":"grpc: error while marshaling: proto: Marshal called with nil"}

返回错误，于是查看mycaller服务容器的日志

[root@master /root]# kubectl logs -f mycaller-6659dc46b-h4jzp mycaller 2020/02/28 22:07:49 [info] Start http server listen :59130 mycaller请求==>1 请求myresponser服务连接信息: {"ServerName":"myresponser","ServerPort":"53606","CertFile":"","CertServerName":"","IsSsl":false} mycaller get请求url==>http://myresponser:53605/testResponser/GetHello?type=1 mycaller get请求fileBody==>{"res":"[2020-02-28 22:08:47] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:"} 请求myresponser服务客户端-获取Hello失败 err: rpc error: code = Unimplemented desc = Not Found: HTTP status code 404; transport: received the unexpected content-type "text/plain; charset=utf-8" ERROR: 2020/02/28 22:08:47 grpc: server failed to encode response:rpc error: code = Internal desc = grpc: error while marshaling: proto: Marshal called with nil {"level":"error","caller":"/usr/local/go/src/git.myscrm.cn/trade/mycaller/vendor/git.myscrm.cn/golang/common/yklog/logger_context.go:202","time":"2020-02-28 22:08:47.084","msg":"grpc-gateway err: grpc: error while marshaling: proto: Marshal called with nil"}

看日志通过http get请求返回了正常的数据，但是grpc调用Hello方法时报错：rpc error: code = Unimplemented desc = Not Found: HTTP status code 404; transport: received the unexpected content-type “text/plain; charset=utf-8”，于是再查看mycaller pod中的envoy日志

2020-02-28T14:07:32.934346Z info Envoy proxy is ready [2020-02-28T14:07:49.471Z] "- - -" 0 - "-" "-" 136 566 5 - "-" "-" "-" "-" "10.10.3.61:18011" PassthroughCluster 10.36.0.10:49416 10.10.3.61:18011 10.36.0.10:49414 - - [2020-02-28T14:07:49.483Z] "GET /v2/keys/?quorum=false&recursive=false&sorted=false HTTP/1.1" 200 - "-" "-" 0 26283 2 1 "-" "Go-http-client/1.1" "45b99d06-d24a-4a9e-af17-430b582976c6" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - - [2020-02-28T14:07:49.486Z] "PUT /v2/keys/service.mycaller.default HTTP/1.1" 200 - "-" "-" 137 413 0 0 "-" "Go-http-client/1.1" "511e9a64-8294-4fa9-97db-2710474c6816" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - - [2020-02-28T14:08:47.063Z] "GET /v2/keys/service.myresponser.default?quorum=false&recursive=false&sorted=false HTTP/1.1" 200 - "-" "-" 0 216 1 1 "-" "Go-http-client/1.1" "aeb99ac6-17bf-4aea-8f6b-b58b6f3d49c8" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - - [2020-02-28T14:08:47.066Z] "GET /testResponser/GetHello?type=1 HTTP/1.1" 200 - "-" "-" 0 113 9 9 "-" "Go-http-client/1.1" "82b0a59f-223c-4944-b630-59d2c728d059" "myresponser:53605" "10.36.0.11:53606" outbound|53605||myresponser.default.svc.cluster.local - 10.97.218.153:53605 10.36.0.10:35778 - default [2020-02-28T14:08:47.077Z] "POST /testResponser.TestService/GetHello HTTP/2" 404 - "-" "-" 8 10 6 5 "-" "grpc-go/1.26.0" "be930121-1b34-4697-a909-3ec80dccbf18" "myresponser:53606" "10.36.0.11:53606" outbound|53606||myresponser.default.svc.cluster.local - 10.97.218.153:53606 10.36.0.10:37922 - default [2020-02-28T14:08:47.060Z] "GET /testCaller/GetHello?type=1&orgcode=private&port=53605 HTTP/2" 500 - "-" "-" 0 112 24 23 "-" "curl/7.64.0" "744ba535-8d37-4fa4-8e87-2a7e755c432a" "mycaller.default:59130" "127.0.0.1:59130" inbound|59130|http2-59130|mycaller.default.svc.cluster.local - 10.36.0.10:59130 10.36.0.12:52096 outbound_.59130_.v1_.mycaller.default.svc.cluster.local default

发现是HTTP/2协议报的404，于是查看svc，是配置的grpc协议，不是http2

[root@master /root]# kubectl get svc myresponser -o yaml apiVersion: v1 kind: Service metadata: ... labels: app: myresponser name: myresponser namespace: default resourceVersion: "1982215" selfLink: /api/v1/namespaces/default/services/myresponser uid: 6fbd7ac2-0895-4d5c-b99f-2cdb6fc07e0d spec: clusterIP: 10.97.218.153 ports: - name: http-53605 port: 53605 protocol: TCP targetPort: 53606 - name: grpc-53606 port: 53606 protocol: TCP targetPort: 53606 selector: app: myresponser sessionAffinity: None type: ClusterIP status: loadBalancer: {}

于是修改svc，将name从grpc-53606修改为http2-53606

apiVersion: v1 kind: Service metadata: name: myresponser namespace: default labels: app: myresponser spec: selector: app: myresponser ports: - name: http-53605 port: 53605 protocol: TCP targetPort: 53606 - name: http2-53606 port: 53606 protocol: TCP targetPort: 53606

再次请求mycaller服务，发现还是报一样的错误。搞了半天，索性把http部分去掉，只保留http2的port试试

apiVersion: v1 kind: Service metadata: name: myresponser namespace: default labels: app: myresponser spec: selector: app: myresponser ports: #- name: http-53605 #port: 53605 #protocol: TCP #targetPort: 53606 - name: http2-53606 port: 53606 protocol: TCP targetPort: 53606

再请求mycaller服务，发现grpc请求成功了

# kubectl exec -it -c sleep $(kubectl get pod-l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605" {"res":"[2020-02-28 22:30:31] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}

mycaller pod日志, 由于取消了http的端口，所以http请求不通，grpc正常

mycaller请求==>1 请求myresponser服务连接信息: {"ServerName":"myresponser","ServerPort":"53606","CertFile":"","CertServerName":"","IsSsl":false} mycaller get请求url==>http://myresponser:53605/testResponser/GetHello?type=1 mycaller get请求Do err:==> Get http://myresponser:53605/testResponser/GetHello?type=1: read tcp 10.36.0.10:46620->10.97.218.153:53605: read: connection reset by peer [2020-02-28 22:30:29] mycaller请求==> hostname:mycaller-6659dc46b-h4jzp, req:1, myresponser响应==>[2020-02-28 22:30:31] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private

mycaller中envoy的日志

[2020-02-28T14:30:29.863Z] "GET /v2/keys/service.myresponser.default?quorum=false&recursive=false&sorted=false HTTP/1.1" 200 - "-" "-" 0 216 1 1 "-" "Go-http-client/1.1" "0b71ef78-1cbc-4f40-a731-153d0833637d" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - - [2020-02-28T14:30:29.866Z] "- - -" 0 UF,URX "-" "-" 0 0 1000 - "-" "-" "-" "-" "10.97.218.153:53605" PassthroughCluster - 10.97.218.153:53605 10.36.0.10:46620 - - [2020-02-28T14:30:30.866Z] "POST /testResponser.TestService/GetHello HTTP/2" 200 - "-" "-" 8 117 301 300 "-" "grpc-go/1.26.0" "ea990043-c6f7-416b-8aeb-d1c7b59844ba" "myresponser:53606" "10.5.24.224:32483" outbound|53606||myresponser.default.svc.cluster.local - 10.97.218.153:53606 10.36.0.10:48764 - default [2020-02-28T14:30:29.861Z] "GET /testCaller/GetHello?type=1&orgcode=private&port=53605 HTTP/2" 200 - "-" "-" 0 120 1307 1307 "-" "curl/7.64.0" "2ea39954-58cc-4a32-8db9-96a0ed1c160b" "mycaller.default:59130" "127.0.0.1:59130" inbound|59130|http2-59130|mycaller.default.svc.cluster.local - 10.36.0.10:59130 10.36.0.12:52096 outbound_.59130_.v1_.mycaller.default.svc.cluster.local default

参考文档
- https://istio.io/docs/ops/deployment/requirements/

问题二：当grpc通了之后，使用istio的VirtualService进行分流的时候出现流量没有按照规则进行

当时的VirtualService规则如下

apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: myresponser-virtualservice spec: hosts: - myresponser gateways: - myresponser-gateway http: - match: - headers: orgcode: exact: private route: - destination: host: myresponser subset: v2 - route: - destination: host: myresponser subset: v1

请求及响应如下

[root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kubectl get pod-l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605" {"res":"[2020-03-02 10:57:55] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}[root@master /root/yunke-istio-manifests-problem]# [root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kubectl get pod-l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605" {"res":"[2020-03-02 10:58:02] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"} [root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kupe=1&orgcode=private&port=53605"sonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?typ {"res":"[2020-03-02 10:58:06] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"} [root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kubecpe=1&orgcode=private&port=53605"path='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?typ {"res":"[2020-03-02 10:58:10] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}[root@master /root/yunke-istio-manifests-problem]#

经过排查，发现上面的yaml文件多配置了网关，后来把网关去掉，问题解决

apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: myresponser-virtualservice spec: hosts: - myresponser #gateways: #- myresponser-gateway http: - match: - headers: orgcode: exact: private route: - destination: host: myresponser subset: v2 - route: - destination: host: myresponser subset: v1

【go|istio部署grpc服务时遇到的问题】请求如下：

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod-l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605" {"res":"[2020-03-03 09:09:59] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod-l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605" {"res":"[2020-03-03 09:10:04] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod-l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605" {"res":"[2020-03-03 09:10:07] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod-l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=&port=53605" {"res":"[2020-03-03 09:10:16] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:prod"}[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod-l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=ttt&port=53605" {"res":"[2020-03-03 09:10:23] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:ttt"}[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod-l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=prod&port=53605" {"res":"[2020-03-03 09:10:30] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:prod"}