Exposing the client behind PSC

Private Service Connect (PSC) enables a producer-consumer model for services in VPCs without network connectivity between them. GCP makes it possible, but it implies hiding the client behind a request. That is a problem for services that need the source IP of the client who made the request. In this article I will explain a way to expose clients to services.

Connecting in a Cloud way

In GCP you can achieve private connectivity between a service producer’s VPC and a service consumer’s VPC using VPC Network Peering or VPNs. Both methods exist for different use cases, have their own advantages, and several limitations. Notably, the need for non-overlapping IP ranges.

PSC overcomes many of those limitations allowing private consumption of services across multiple VPC networks that can belong to different projects or organizations. Of course, you can't ask multiple organizations to coordinate when assigning IP ranges to their projects. PSC doesn't require it.

I won't analyze all the benefits and limitations of PSC vs other solutions, but I will briefly describe how it works to set the stage for the rest of discussions. Take into account that there are several flavors of PSC, here I will talk about "PSC to publish and consume services".

PSC architecture

The simplest form of PSC is pretty straightforward. There is a service offered from a producer VPC, and a consumer VPC which will consume the service. The producer creates a service attachment to expose the service, and the consumer creates an endpoint targeting this attachment.

Basic PSC architecture.

The service needs to be a load balanced service, for GCP this can be a L4 or L7 internal load balancer (ILB). The service attachment points to the forwarding rule of this load balancer. The endpoint itself is a forwarding rule whose target, instead of a backend service or proxy, is a service attachment URI.

Note that there is no layer 3 network connectivity between these VPCs, each has its own isolated network domain. You only need to deal with APIs to establish the relationship and Andromeda, GCP’s Software Defined Networking platform, takes care of the rest. Now these VPCs can use the same IP ranges, overlapping is not an issue anymore. In fact, PSC allows to deploy a multi-tenant model where your service is consumed by multiple consumers irrespective of their subnets' ranges.

Multi-tenant model with overlapping addresses.

Where things may break

I mentioned PSC exposes a service behind a L4 or L7 ILB. The L7 ILB is an HTTP proxy so backends will see the proxy IP as source IP in the requests. If they need the original source IP of the client making the request the header “X-Forwarded-For” contains that information.

The L4 ILB is not a proxy though. And a service attachment needs to define a NAT range that is used by PSC to translate the source IP of the request to one that is part of the producer VPC. Backends will see an IP from that NAT range as the source, meaning that in general non-HTTP applications will not see the original client IP.

There’s a workaround though. The PROXY protocol was designed to forward information from the client connection to the next host. This is achieved by adding a header at the L4 at the beginning of each connection. PSC is compatible with the PROXY protocol so you can enable it when creating the service attachment. Of course, backend applications receiving that info should be compatible with the protocol, otherwise the connection will fail.

Here is where things can get complicated. Many applications are not compatible with the PROXY protocol, and if you have such an app and the need to keep the original client IP, PSC and NAT in general can become a problem. This use case is the focus of this article.

Test deployment

I will deploy a test scenario to show you how all this looks like. Although I mentioned overlapping addresses are not an issue, I will use different subnet ranges to make easier to follow explanations.

Initial test scenario.

I won't type here all the instructions to set it up, only the ones more related to PSC and to test it. I think you will be able to fill in the gaps.

* Service

The service consists of two VMs as part of an instance group and an internal TCP/UDP load balancer pointing to it called "ilb-myservice". For test purposes I run a simple HTTP server on each. As I said the focus of this article is on L3/L4 applications in general, but for testing it is very convenient to run a simple web server with some logging and the tool 'curl':

$ python3 -m http.server 8080

* Service attachment

First you need a NAT subnet for the service attachment. This subnet is only for PSC, you can't use it for other resources such as VMs:

$ gcloud compute networks subnets create producer-subnet-psc --network vpc-producer --region europe-west1 --range 10.10.0.0/24 --purpose PRIVATE_SERVICE_CONNECT

The service attachment points to "ilb-myservice-forwarding-rule" and uses the previous NAT subnet "producer-subnet-psc" (10.10.0.0/24):

$ gcloud compute service-attachments create myservice-attachment --region europe-west1 --producer-forwarding-rule ilb-myservice-forwarding-rule --connection-preference=ACCEPT_AUTOMATIC --nat-subnets producer-subnet-psc

* Endpoint

Reserve an internal IP address for the endpoint in the consumer VPC:

$ gcloud compute addresses create ip-psc-endpoint --region europe-west1 --subnet consumer-subnet-1 --addresses 172.16.0.10

Create a forwarding rule to connect the endpoint to the producer’s service attachment:

$ gcloud compute forwarding-rules create fr-psc-endpoint --region europe-west1 --network vpc-consumer --address ip-psc-endpoint --target-service-attachment projects/[PRODUCER_PROJECT_ID]/regions/europe-west1/serviceAttachments/myservice-attachment

* Testing a connection

If we make a request from a consumer VM to the service via the endpoint, we can check it works although the VPCs are not connected!

Connection test from consumer to producer VPC.

Note how the request is seen as coming from IP 10.10.0.6, from the NAT range. Also the request is made to the endpoint IP, 172.16.0.10, however it reaches the backends through the ILB VIP, 10.1.0.50. You can use 'tcpdump' or VPC Flow Logs if you don't trust me :)

Exposing the client

PSC is a great way of connecting consumers and services in Cloud, so it is worth to leverage it also for those apps that need to know the client IP. We need to use the PROXY protocol to recover that info, but since we are considering non-compatible apps we need a proxy service that understands the PROXY protocol and handles it for them, and some plumbing.

I will deploy a proxy VM running HAProxy. I choose this software because it is a great TCP/HTTP proxy and load balancer, and an HAProxy developer designed the PROXY protocol so it is well supported. You can use the free version or the paid version to get more enterprise features and support.

The idea is that the proxy will receive the connection and its information through the PROXY protocol enabled in the service attachment. Then I will use that information to translate the request and connect to the service.

Exposing the client through a proxy service. Note there are two TCP connections now.

* Configuring the proxy VM

The proxy VM will forward packets with sources different from its own internal IP so it needs IP forwarding enabled at the VM level:

$ gcloud compute instances create ... --can-ip-forward

Also, it will need to accept responses to those sources. For that we will configure AnyIP feature:

proxy$ sudo ip route add local 172.16.0.0/24 dev lo

We are adding a local route for the consumer IP range, so that our VM will accept traffic on IPs not explicitly configured on it. To remove dependency with consumer VPC(s), you could use a range like 172.16.0.0/12 instead.

* Installing and configuring HAProxy

Install HAProxy following instruction from here. It comes with a default configuration, typically in '/etc/haproxy/haproxy.cfg'. HAProxy has a lot of features and options I won't cover, only the minimum you need to modify to make our setup work:

global
  # Comment out these two lines to be root and spoof client IP
  #user haproxy
  #group haproxy
defaults
  # Use TCP load balancing
  #mode http
  #option httplog
  mode tcp
  option tcplog
frontend myfrontend
  # Accept PSC connections on this port. Enable PROXY protocol, PSC will send client connection information on it
  bind :8080 accept-proxy
  # Dispatch the accepted connections to this backend
  default_backend myservice
backend myservice
  # Set source address for outgoing connections, using source IP (and port) from client connection
  source 0.0.0.0 usesrc client
  # The backend is the ILB
  server fw_rule 10.1.0.50

Save the file and restart the service:

proxy$ sudo systemctl restart haproxy

* Some more plumbing

Make the proxy VM part of an instance group and put an internal TCP/UDP load balancer pointing to it called “ilb-proxy”. It will be used by PSC. You also need to create a route in the producer VPC to this ILB to send the traffic back:

$ gcloud compute routes create route-via-proxy --network=vpc-producer --destination-range=172.16.0.0/24 --next-hop-ilb=ilb-proxy-forwarding-rule

Again, here you could use a range like 172.16.0.0/12 instead so that you don't need to specify a new route for every consumer.

* Configuring PSC

Delete the previous service attachment and create a new one pointing to “ilb-proxy-forwarding-rule”, this time enabling the PROXY protocol:

$ gcloud compute service-attachments create myservice-attachment --region europe-west1 --producer-forwarding-rule ilb-proxy-forwarding-rule --connection-preference=ACCEPT_AUTOMATIC --nat-subnets producer-subnet-psc --enable-proxy-protocol

Since the attachment changed, you need to delete the endpoint and create it again:

$ gcloud compute forwarding-rules create fr-psc-endpoint --region europe-west1 --network vpc-consumer --address ip-psc-endpoint --target-service-attachment projects/[PRODUCER_PROJECT_ID]/regions/europe-west1/serviceAttachments/myservice-attachment

* The litmus test

Let's make a request from a consumer VM. If everything works, the backends should see the client IP as the source of the request.

Connection test using PROXY protocol.

\o/

Make it available

Now that we have our HAProxy forwarding requests from clients, the next logical step is to improve the availability of the system by deploying more than one proxy. I would let the reader do that, but there are some caveats to consider.

I said PSC allows to deploy a multi-tenant model where overlapping IP ranges are not an issue. However, when we use the PROXY protocol to expose the client IP the situation is different. Think on this, the backends want to identify the client by its IP, how would that be possible if multiple clients use the same IP?

Deploying several proxies poses a related problem. If two proxies can forge requests for the same IP ranges, the response from the backends may hit the wrong proxy.

One solution to this is to deploy an active/passive configuration. We can configure a second proxy as failover backend on the ILB. When the primary backend is unhealthy GCP performs a failover and new connections are directed to the backup VM. In case the primary VM recovers and passes its health check, GCP performs a failback. This way, you have only one active proxy at any time.

I mentioned health check. Until now, I didn't talk about how to configure it for HAProxy because it was not essential. But you need it for failover to work properly. The following snippet for ‘/etc/haproxy/haproxy.cfg’ shows an example to configure a response to a health check on port 8081:

frontend health-checks
  mode http
  bind :8081
  http-request return

HAProxy will bind to port 8081, where you need to point your ILB health checks, and will immediately return a 200 OK to GCP probes.

After deploying the failover backend, you can test it works by simply stopping and starting the HAProxy service on the primary backend and observing the different route traffic takes ('tcpdump' is your friend ;):

proxy$ sudo systemctl stop haproxy
proxy$ sudo systemctl start haproxy

Final notes

You should take into account that PSC has some limitations. Some of them may be removed in the future (maybe they are not a limitation by the time you are reading this article). For example:

• PSC supports PROXY protocol for TCP services only, not for UDP.
• The PSC endpoint must be in the same region as the service attachment.
• Accessing a PSC endpoint from on-premises using Interconnect attachments (VLANs) is not supported.

Please refer to the public documentation to have an up to date information.

In summary, PSC still has some room for improvement but it is a great way to create a producer-consumer model in GCP. I hope you will start using it more after reading this article.