Winse Blog

走走停停, 熙熙攘攘, 忙忙碌碌, 不知何畏.

Nginx Https With Tomcat Http

昨天配置了HTTPS了,nginx https代理访问应用的http登录页也确实没有问题的。一切都是那么的完美,然而今天一早测试的姐姐告诉我:登录失败报错400 Bad Request The plain HTTP request was sent to HTTPS port.

  • nginx 1.10.2
  • tomcat 8.0.38

初步定位问题

然后想起有看到过红薯蜀黍的 https://www.oschina.net/question/12_213459 如下:(注:最终版在最后,如果有兴趣可以看看心路历程)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
# tomcat server.xml
  <Service name="Catalina">
    <Connector port="9000" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8" 
               redirectPort="14443"
               scheme="https" 
               proxyPort="14443" />
  ...

    <Engine name="Catalina" defaultHost="localhost">

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

      <Valve className="org.apache.catalina.valves.RemoteIpValve"
                remoteIpHeader="x-forwarded-for"
                remoteIpProxiesHeader="x-forwarded-by"
                protocolHeader="x-forwarded-proto"
            />
  
# nginx 
    server {
        listen       14443 ssl;
        server_name localhost;

        ssl on;
        ssl_certificate      nginx.crt;
        ssl_certificate_key  nginx.key;

        ssl_session_cache    shared:SSL:10m;
        ssl_session_timeout  5m;

        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers  on;

        location / {
            root   html;
            index  index.html index.htm;
        }

        location /omc {
          proxy_set_header Host $http_host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto https;
          proxy_connect_timeout      240;
          proxy_send_timeout         240;
          proxy_read_timeout         240;
  
          proxy_pass http://localhost:9000; #默认连的是http的端口
          proxy_redirect off;
          #proxy_redirect https://$host/ / ;
      }
    }

登录请求确实都是https请求了,但是重定向(302)返回的https的端口丢失了(即被替换为默认的443)。查了很多资料,先弄了一个折中的处理方式,把hostname替换掉,即最后注释的那一行proxy_redirect。

查看的文章多半是http问题和多了端口的问题。我这是少端口了,但是还是有参考价值对proxy_redirect和port_in_redirect多了解一点:

还有一些没啥卵用,还带点误导性质的,但是还是得把它帖出来(蜜汁尴尬):(注:不是说人家的有错,而是说和上面的Valve结合后不行了)

关于redirectPort

翻到一篇关于Valve的文章,看不明白,感觉应该用远程调试看看为什么端口变成默认的了。

先看redirectPort,仅当http请求有安全约束才会转到端口使用SSL传输。so,redirectPort在这里没啥卵用。就当拓展知识了

1
2
redirectPort 
If this Connector is supporting non-SSL requests, and a request is received for which a matching <security-constraint> requires SSL transport, Catalina will automatically redirect the request to the port number specified here.

web.xml里面可以配置security-constraint节点

1
2
3
4
5
6
7
8
9
<security-constraint> 
<web-resource-collection> 
<web-resource-name>services</web-resource-name> 
<url-pattern>/login/*</url-pattern> 
</web-resource-collection> 
<user-data-constraint> 
<transport-guarantee>CONFIDENTIAL</transport-guarantee> 
</user-data-constraint> 
</security-constraint>

但是终究不是一种的解决问题的办法,而且怎么看怎么感觉Connector的redirectPort咋一点作用都没有呢?

Valve问题所在,解决https以及端口问题的源泉

由于是https请求,tcpdump从端口查到的数据都是看不懂的。并且不知道问题是在tomcat还是nginx,只能想着远程调试看看端口是在什么时刻被修改的。

1
2
3
4
5
6
7
8
9
10
11
[root@cu1 apache-tomcat-8.0.38]# export JPDA_ADDRESS=8000
[root@cu1 apache-tomcat-8.0.38]# bin/catalina.sh jpda run

----

# 本地pom.xml增加如下,要来查看tomcat的源码
<dependency>
    <groupId>org.apache.tomcat</groupId>
    <artifactId>tomcat-catalina</artifactId>
    <version>8.0.38</version>
</dependency>

然后本地maven项目临时加入tomcat的包,开启VPN后,在eclipse的 org.apache.catalina.connector.ResponseFacade.sendRedirect(String) 打断点调试。然后定位到 org.apache.coyote.Request.setServerPort(int), 最终确定在 org.apache.catalina.valves.RemoteIpValve.setPorts(Request, int)

RemoteIpValve类里面的Header与nginx中配置Header对应就行的。

最终成果

注:nginx/tomcat配置https的方法,请查看前一篇文章。

注2:还有tomcat里面Header是不区分大小写的: org.apache.tomcat.util.http.MimeHeaders.getValue(String)

注3:如果配置proxyPort(而不是Valve的话)取到协议好像不对(没验证),并且配置Valve可以不影响Connector。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
# nginx
    server {
        listen       14443 ssl;
        server_name localhost;

        ssl on;
        ssl_certificate      nginx.crt;
        ssl_certificate_key  nginx.key;

        ssl_session_cache    shared:SSL:10m;
        ssl_session_timeout  5m;

        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers  on;

        location / {
            root   html;
            index  index.html index.htm;
        }

        location /omc {
        port_in_redirect on;

        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Port $server_port; # !!自定义port header
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_connect_timeout      240;
        proxy_send_timeout         240;
        proxy_read_timeout         240;

        proxy_pass http://localhost:9000;
        #proxy_redirect default;
        proxy_redirect off;
        #proxy_redirect https://$host/ / ;
        }

    }

# tomcat server.xml
  <Service name="Catalina">
    <Connector port="9000" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               redirectPort="8443"
                />
  ...
    <Engine name="Catalina" defaultHost="localhost">
  ...
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
          ...
            <Valve className="org.apache.catalina.valves.RemoteIpValve"
                  portHeader="x-forwarded-port"
                  remoteIpHeader="x-forwarded-for"
                  proxiesHeader="x-forwarded-by"
                  protocolHeader="x-forwarded-proto"
            />

福利

nginx https 代理tomcat https: 其实就是和http代理一样,很简单。记得删掉上面的removeipvalve。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# tomcat server.xml
<Connector
           protocol="org.apache.coyote.http11.Http11AprProtocol"
           port="9443" maxThreads="200"
           scheme="https" secure="true" SSLEnabled="true"
           SSLCertificateFile="/home/cu/tools/apache-tomcat-8.0.38/conf/nginx.crt"
           SSLCertificateKeyFile="/home/cu/tools/apache-tomcat-8.0.38/conf/nginx.key"
           SSLVerifyClient="optional" SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"/>

# nginx
location /omc {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_connect_timeout      240;
proxy_send_timeout         240;
proxy_read_timeout         240;

proxy_redirect off;
proxy_pass https://localhost:9443;
}

nginx websockt:

1
2
3
4
5
6
7
8
    location /omc/webSocket {
            proxy_pass http://localhost:8888/omc/webSocket;
            proxy_redirect off;

            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
    }

文件大小:

1
2
3
4
proxy_read_timeout 86400;
proxy_send_timeout 86400;

client_max_body_size 1024m;

–END

Comments