Google’s GCLB doesn't comply with RFCs

When you deploy a backend system in GCP behind Google Cloud Load Balancer (GCLB) you can use TLS to secure the communication between them. The RFC that defines how HTTP over TLS works is RFC 2818. This mechanism involves sending a certificate to validate from the server, your backend in this case, to the client, the GCLB. It turns out we don't validate that certificate.

A paradox situation

The GCLB is our HTTP(s) load balancer. It is a reverse proxy, so external user connections terminate on it and then a new connection is created from the GCLB to the backend. You can choose to enable TLS encryption (HTTPs) on this communication path.

On the TLS exchange, a digital certificate and its corresponding public key is used to generate a session key to secure the connection. You can find all the details in RFC 5246 and other related RFCs. This certificate needs to be validated by the client before proceeding, but as we state in our public documentation:

When a GFE connects to backends that are within Google Cloud, the GFE accepts any certificate your backends present. GFEs do not perform certificate validation. For example, the certificate is treated as valid even in the following circumstances:
- The certificate is self-signed.
- The certificate is signed by an unknown certificate authority (CA).
- The certificate has expired or is not yet valid.
- The CN and subjectAlternativeName attributes don't match a Host header or DNS PTR record.

Note: GFE stands for Google Front End, the system that implements a GCLB

What? How come? Google's security is top-notch, and your security is a top priority for us. However this seems to contradict that. It goes against RFCs, it doesn't check the server identity, and leaves the door open to man-in-the-middle-attacks!

I'm sure this can cause some people to get nervous, especially security teams in charge of companies' security. No panic, I said this is a paradox so let me explain.

The Wild Wild Web

HTTP over TLS and digital certificates were designed to secure communications mainly in the Internet and the WWW. Of course there is widespread use also in internal networks, but the WWW represents a more general (and wild) framework.

When users access e.g. their online bank accounts, they want to be sure they are reaching the right server and that their communications can't be decrypted and their data stolen. This is not an easy task to achieve since there are many threats out there: malware, DNS spoofing, BGP hijacking…

To help on this task we all use a security framework based on certificates and the TLS protocol. That gives us a good level of protection:

We are not totally secure yet, sophisticated attacks can happen, but this raises the security bar significantly.

A new scenario

Running backend servers in GCP is a very different scenario to the WWW. You will see there are significant differences that change the threat model.

To start with, the client (GCLB) and the server (backend) run in the same network, a network managed and secured by Google. We control the systems communicating and route traffic properly. That means some of the threats mentioned before, particularly those related to intercepting traffic, don't apply in our scenario. To better understand how GCP VPC networks route traffic you can read my article on DMZs.

At the network layer our virtual network authenticates all traffic within GCP. This authentication, achieved via security tokens, protects a compromised host from spoofing packets on the network. A packet is sent directly from VM to VM, and security tokens contain authentication information about the sender and receiver. The control plane on the sending side sets the token, and the receiving host validates the token. If a host receives a packet from a rogue sender or not destined to it, the packet is dropped.

If the related topic of compromised systems in multi-tenant architectures is a concern for you, you may want to read my article on confidential computing.

When traffic goes outside of facilities that Google physically controls, we encrypt at the network layer in a similar fashion to IPsec so customer's traffic traversing our data centers is protected. You can find more details about encryption in transit in Google Cloud in this document. I recommend you to read it to understand how much effort we put in securing our infrastructure and keep customer data protected.

And remember you are in control of what backends are registered to the GCLB. Either VMs or pods running in a cluster, this is all part of the customer's GCP configuration. Neither Google nor another customer on GCP is able to modify this component.

Conclusion

As you can see, we have a secure platform and several alternative mechanisms in place to reassure ourselves that only the real backend could be talking to the GCLB, which makes it unnecessary to validate the certificate and the identity it carries as usually required. That also simplifies design, implementation and management to the customer.

In reality, this is in conformance with RFC 2818, Section 3.1:

If the client has external information as to the expected identity of the server, the hostname check MAY be omitted.

Security is not a matter of going through a security checklist. That can help you organize but Cloud has changed some rules of the game and you need to analyze and understand this new context. I hope this article will help you with that venture!