Direct VPC egress

Direct VPC egress provides a high-performing networking solution for your App Engine service to send traffic to a Virtual Private Cloud (VPC) network. Direct VPC egress lets workloads seamlessly access VPC network resources and removes the need to configure Serverless VPC Access connectors.

Key benefits

  • Simplified management: Eliminates the operational overhead of managing connector instances, machine types, and scaling settings. App Engine handles configurations directly within your service's app.yaml file.
  • Cost efficiency: Using Direct VPC egress has no additional charge, and you don't have to pay fixed monthly charges for connector VMs.
  • Improved performance and reliability: By eliminating the need to use a connector, Direct VPC egress offers a faster, more reliable connection to your VPC network resources. It scales as quickly as your App Engine service and avoids connection drops that might occur with connectors during maintenance.

  • Granular security: You can apply network tags directly to your App Engine service versions, enabling precise, service-specific firewall rules and network policies.

  • Supports VPC Flow Logs: You can enable VPC Flow Logs on your subnet to log egress traffic from your App Engine service.

  • Supports Cloud NAT: You can configure a Cloud NAT gateway on your subnet to provide NAT services, such as Public NAT or Private NAT for outbound traffic from your App Engine service.

Limitations

  • IP address consumption: Your service's IP address usage scales directly with the number of running instances, and your scaling ability is limited by the number of available IP addresses in your chosen subnet.

  • Maintenance events: Your service might experience brief connection breaks during networking infrastructure maintenance events. We recommend that you use client libraries for handling occasional connection resets.

  • Cold starts: Initial cold start times are dependent on the region and the specific use case. In rare cases, cold starts might last up to a minute.

  • Direct VPC ingress: App Engine doesn't support Direct VPC ingress.

  • Number of instances: By default, you can only configure up to 100 instances per App Engine version to use Direct VPC egress. You can request a quota increase for this limit if your workload requires more instances.

  • VPC Flow Logs: Doesn't display your App Engine service version in the log records. The service name is displayed in the AppEngineServiceDetails field.

  • Regional availability: Direct VPC egress is available only by invitation in certain regions. To request access in these regions, contact support.

IP address allocation

To place your App Engine service on a VPC network, specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network. App Engine allocates IP addresses from your subnet.

IP addresses are ephemeral, so don't create policies based on individual IPs. If you need to create a policy based on IPs, such as in firewall rules, you must use the IP address range of the entire subnet.

To change the network or subnet that your service uses, deploy a new version that uses the new network and subnet values.

Scale up and scale down

For faster scale up during a traffic surge, App Engine reserves IP addresses in blocks of 16 (28 subnet mask) at a time. To ensure that you have enough IPv4 addresses available for use across App Engine, your subnet's IPv4 address range must be /26 or larger.

For IP allocation efficiency and ease of management, place multiple resources on the same subnet. If your IPv4 address space is limited, see Supported IPv4 ranges for more options.

To delete the subnet, you must first delete or redeploy your App Engine service to stop using the subnet, and then wait 1-2 hours.

IP address consumption for services

At steady state, App Engine uses 2 times (2X) as many IP addresses as the number of instances. When a version scales down, App Engine retains its IP addresses for up to 20 minutes. In total, reserve at least 2X the number of IP addresses, plus a buffer to account for version updates.

For example, if you upgrade versions so that version 1 scales from 100 instances down to zero while version 2 scales from zero up to 100, App Engine retains the version 1 IP addresses for up to 20 minutes after scaling down. During the 20-minute retention window, you must reserve at least 400 IP addresses ((100 + 100) * 2).

Supported IPv4 ranges

App Engine supports the following IPv4 ranges for your subnet:

  • RFC 1918
    • 10.0.0.0/8
    • 172.16.0.0/12
    • 192.168.0.0/16
  • RFC 6598
    • 100.64.0.0/10
  • Class E
    • 240.0.0.0/4

Before you begin

  1. Make sure you have an existing VPC network and a subnet in your project. If you don't already have an existing VPC, follow the instructions to create one in Create a VPC network.

  2. Enable the Compute Engine API and the Cloud Build API:

    Enable APIs

  3. To use Direct VPC egress, make sure you are running the most recent version of the Google Cloud CLI:

    gcloud components update

Required roles

Ensure that App Engine has access to the VPC network by granting the following roles to the App Engine service agent:

  • App Engine Service Agent role: By default, the App Engine service agent has the App Engine Service Agent role (roles/appengine.serviceAgent) that contains the necessary permissions.

  • Custom permissions: For more granular control, grant the App Engine service agent the following additional permissions on the project:

    • compute.networks.get
    • compute.subnetworks.get
    • compute.subnetworks.use on the project or the specific subnet
    • compute.addresses.get
    • compute.addresses.list
    • compute.addresses.create
    • compute.addresses.delete
    • compute.addresses.createInternal
    • compute.addresses.deleteInternal
    • compute.regionOperations.get

  • Compute Network User role: If you don't use the default App Engine Service Agent role or the custom permissions, grant the Compute Network User role (roles/compute.networkUser) to the App Engine service agent. Subnets with external IPv6 also require the Compute Public IP Admin role (roles/compute.publicIpAdmin).

    For example, to grant the Compute Network User role, run the following command:

    gcloud projects add-iam-policy-binding PROJECT_ID \
--member "serviceAccount:service-PROJECT_NUMBER@gcp-gae-service.iam.gserviceaccount.com" \
--role "roles/compute.networkUser"

    Replace the following:

    • PROJECT_ID: the ID of your project.
    • PROJECT_NUMBER: the project number where you deploy your App Engine service.

Configure an App Engine service with Direct VPC egress

You can configure Direct VPC egress for an App Engine service by using YAML or Terraform. To enable or disconnect Direct VPC egress, deploy a new version and migrate traffic to it. You can also delete the deployed version first, instead of overwriting an existing version ID.

YAML

  1. Add the following vpc_access setting to your app.yaml file:

    vpc_access:
network_interface:
 network: NETWORK
 subnet: SUBNET
 tags:
     - NETWORK_TAGS
vpc_egress: EGRESS_SETTING

    Replace the following:

    • NETWORK: the name of the existing network your application instances connect to, for example, default. Specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network.

    • SUBNET: the name of the existing subnetwork your application instances connect to, for example, default. Specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network.

    • Optional: NETWORK_TAGS lists network tags to associate with your App Engine service's instances for use in firewall rules and routing policies.

    • Optional: EGRESS_SETTING controls how outbound traffic is routed. This field supports the following configuration settings:

      • all-traffic: All outbound requests are routed through the VPC network.
      • private-ranges-only (default): Only traffic to internal IP addresses is routed through the VPC network. Internet traffic uses the default App Engine path.

  2. Deploy to App Engine by running the following command:

    gcloud app deploy

Terraform

To learn how to apply or remove a Terraform configuration, see Basic Terraform commands.

  1. Add the following vpc_access block to a google_app_engine_standard_app_version resource in your Terraform configuration:

    vpc_access {
vpc_egress = "EGRESS_SETTING"
network_interfaces {
 network    = "NETWORK"
 subnetwork = "SUBNET"
 tags       = ["TAG_1", "TAG_2"]
}
}

    Replace the following:

    • NETWORK: the name of the existing VPC network your application instances connect to, for example, default. Specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network. You can also reference a google_compute_network resource name (for example, google_compute_network.custom.name).
    • SUBNET: the name of the existing subnetwork your application instances connect to, for example, default. Specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network. You can also reference a google_compute_subnetwork resource name (for example, google_compute_subnetwork.custom.name).
    • Optional: TAG_1, TAG_2 lists network tags to associate with your App Engine service's instances for use in firewall rules and routing policies.
    • Optional: EGRESS_SETTING controls how outbound traffic is routed. This field supports the following configuration settings:
    • ALL_TRAFFIC: All outbound requests are routed through the VPC network.
    • PRIVATE_IP_RANGES (default): Only traffic to internal IP addresses is routed through the VPC network. Internet traffic uses the default App Engine path.

  2. If the version is already deployed without Direct VPC egress, update the version_id field to a new value. You can't enable Direct VPC egress on an existing version.

  3. Apply the configuration:

    terraform apply

Disconnect a service

To disconnect your service from the VPC network, use YAML or Terraform:

YAML

  1. Remove the vpc_access section from your app.yaml file.

  2. Redeploy your service:

    gcloud app deploy

Terraform

  1. Remove the vpc_access block from the google_app_engine_standard_app_version resource in your Terraform configuration.

  2. Update the version_id field to a new value. You can't disable Direct VPC egress on an existing version.

  3. Apply the configuration:

    terraform apply

Isolate outbound internet traffic

By default, when vpc_egress is set to private-ranges-only, App Engine sends outbound internet traffic through a regional pool of dynamic IP addresses shared across serverless workloads. Don't use this default shared pool for workloads that depend on IP reputation or generate high-volume probing traffic.

Instead, set vpc_egress: all-traffic and route outbound traffic through Cloud NAT with dedicated static outbound IP addresses in these cases:

  • Your service connects to reputation-sensitive external endpoints, such as payment gateways, hosted databases, or partner APIs behind third-party web application firewalls (WAFs). Dedicated static IP addresses prevent noisy traffic from other tenants on the shared pool from triggering external WAF blocks or HTTP 403 errors.
  • Your service runs authorized network scanning, port enumeration, host fingerprinting, vulnerability probing, web scraping, or load testing. Running these workloads over the default shared IP pool degrades IP reputation, can disrupt service for other customers in violation of the Google Cloud Acceptable Use Policy, and might cause Google to throttle or suspend your workload.

When you run authorized scanning or high-connection workloads through Cloud NAT, allocate enough static IP addresses and configure port reservation to prevent port exhaustion.

Troubleshoot

This section describes the common errors you might encounter while deploying your App Engine service with Direct VPC egress.

Cannot delete subnet

To delete a subnet, you must first delete or redeploy all resources that use it. If App Engine is using a subnet, disconnect the App Engine service from the VPC network or move it to a different subnet before deleting the subnet.

After you delete or move your App Engine service, wait 1-2 hours for App Engine to release the IP addresses before you delete the subnet.

Deployment failures

If your deployment fails, the Google Cloud CLI displays error messages indicating the root cause. Some common issues include the following:

  • Incorrect VPC metadata, such as a misspelled network or subnet name in your app.yaml file. To fix potential errors, review the VPC configuration in your app.yaml file.

  • Insufficient IAM permissions. Make sure you grant the required permissions to the App Engine service agent. If you encounter permission errors during deployment, ensure that you grant the following additional roles to your deploying service account:

IP address exhaustion

If your subnet runs out of available IP addresses, App Engine fails to start new instances and logs an error. To resolve this issue, expand the IP range of your subnet or move your service to a larger subnet.

IP address leaks

IP address leaks might lead to IP address exhaustion. IP address leaks are unlikely during standard operations, but might occur due to the following causes:

  • The App Engine service agent only has permissions to reserve addresses (create, createInternal) and lacks permissions to release addresses (delete, deleteInternal).
  • The App Engine service agent was deleted while other Google Cloud address reservations were active.
  • The Cloud Billing account was removed and later re-enabled in the project while Serverless address reservations were active.
  • Your Google Cloud project was deleted and later restored while Serverless address reservations still existed.

To resolve this issue, follow these steps:

  1. Query for the following logs in the Logs Explorer to identify the leaked addresses:

    protoPayload.authorizationInfo.permission=~"compute.addresses.delete.*"
protoPayload.authorizationInfo.resourceAttributes.type="compute.addresses"
protoPayload.resourceName=~"projects/.*/regions/.*/addresses/serverless-.*"
severity>=WARNING

    If this query doesn't return results, there are no IP address leaks, and no further action is required.

  2. If your query returns results, make sure that the App Engine service agent has the App Engine Service Agent role (roles/appengine.serviceAgent). If you can't use this role, ensure that you grant the other required roles and permissions to the App Engine service agent.

  3. Manually delete leaked IP addresses by using the Google Cloud console or the Google Cloud CLI:

    Console

    1. Go to the IP addresses page in the Google Cloud console:

      Go to IP addresses

    2. Select the leaked IP addresses you identified in the previous step by running the query.

    3. Click Release static address to remove the leaked addresses.

    gcloud

    Run the gcloud compute addresses delete command:

    gcloud compute addresses delete ADDRESS_NAME --region=REGION

    Replace the following:

    • ADDRESS_NAME: the name of the leaked IP address.
    • REGION: the region of the leaked IP address.