Apply policy to forwarded traffic

Big picture

Enforce Calico Cloud policy on traffic transiting a host that is used as a router or NAT gateway.

Value

If your host has multiple network interfaces, and is configured as a router, or NAT gateway between two different networks, you may want to enforce policy on traffic as it moves between the networks. In this configuration, often neither the source or destination is a Calico Cloud endpoint, so policy enforcement at the endpoint is not available. You can centrally manage the firewall policy on a fleet of such hosts using the same policy language as the rest of Calico Cloud.

Concepts

Workload endpoints and host endpoints

The following figure shows a host with two network interfaces: eth0 and eth1. We call these host endpoints (HEPs). The host also runs two guest workloads (VMs or containers). We call the virtual interfaces to the guests, workload endpoints (WEPs). Each has a corresponding configuration object on the Calico Cloud API called HostEndpoint and WorkloadEndpoint, respectively.

The HostEndpoint API object is optional, and Calico Cloud does not enforce any policy on the HEP if the API object is missing. The WorkloadEndpoint API object is required, and is automatically managed by the cluster orchestrator plugin (for example, Kubernetes or OpenStack).

Several connections are shown in the figure, numbered 1 through 4. For example, connection 1 ingresses over HEP eth0, is forwarded, and then ingresses Workload A’s WEP. Calico policies select which WEPs or HEPs they apply to. So, for example an ingress policy that selects Workload A’s WEP will apply to connections as shown in number 1.

applyOnForward

By default, Calico Cloud global network policies set applyOnForward to false. When set to false on policies that select HEPs, the policies are applied only to traffic that originates or terminates on the host, for example: connection 4 (Node process). Connections 1-3 are unaffected by policies that select the HEP, but have applyOnForward set to false.

In contrast, if applyOnForward is set to true for a policy that selects a HEP, that policy can apply to all connections 1-4. For example:

• Ingress policy on HEP eth0 affects connections 1 and 2
• Egress policy on HEP eth1 affects connections 2, 3, and 4

There are also different default action semantics for applyOnForward: true policy versus applyOnForward: false policy. An applyOnForward: true policy affects all traffic through the HEP (connections 1-4). If no applyOnForward policy selects the HEP and direction (ingress versus egress), then forwarded traffic is allowed. If no policy (regardless of applyOnForward) selects the HEP and direction, then local traffic is denied.

HEP defined?	Traffic Type	applyOnForward defined?	Any policy defined?	Default Action
No	Any	n/a	n/a	Allow
Yes	Forwarded	No	Any	Allow
Yes	Forwarded	Yes	Yes	Deny
Yes	Local	n/a	No	Deny
Yes	Local	n/a	Yes	Deny

Calico Cloud namespaced network policies do not have an applyOnForward setting. HEPs are always cluster global, not namespaced, so network policies cannot select them.

preDNAT policy

Hosts are often configured to perform Destination Network Address Translation before forwarding certain packets. A common example of this in cloud computing is when the host acts as a reverse-proxy to load balance service requests for a set of backend workload instances. To apply policy to a specific example of such a reverse-proxy, see Kubernetes nodePorts.

When preDNAT is set to false on a global network policy, the policy rules are evaluated on the connection after DNAT is performed. False is the default. When preDNAT is set to true, the policy rules are evaluated on the connection before DNAT has been performed.

If you set preDNAT policy to true, you must set applyOnForward to true, and preDNAT policy must only include Ingress policies.

Host Endpoints with interfaceName: *

HostEndpoint API objects can be created with the name of the host interface (as reported by ip link or similar), or they can be created with interfaceName set to *, which means all host interfaces on the given node, including the interfaces between the host to any WEPs on that host.

With interfaceName set to a particular interface, any policies that select the HEP apply only if the traffic goes through the named interface. With it set to *, policies that select the HEP apply regardless of the interface.

This is particularly relevant when you want to enforce policy for a host that also runs guest workloads like VMs or Pods. Traffic from local workloads to reverse-proxy IPs or ports do not traverse any external interfaces, and thus a HEP with interfaceName set to * is required in order for policy to apply to them.

How to

Control forwarded traffic in or out of particular networks

1. Choose a labeling scheme for your Host Endpoints (network interfaces).
   For example, if you have an application network and management network, you might choose the labels network = application and network = management.
2. Write GlobalNetworkPolicies expressing your desired rules.
   • applyOnForward set to true.
   • Use the selector: to choose which Host Endpoints to apply policy.
3. Create the HostEndpoint objects on the Calico Cloud API.
   • Label the HostEndpoints according to the label scheme you developed in step 1.
   • We recommend that you create policies before you create the Host Endpoints. This ensures that all policies exist before Calico Cloud starts enforcing.

Tutorial

Let’s say I have a host that has two network interfaces:

• eth0 - connects to the main datacenter network for application traffic
• eth1 - connects to a special maintenance network

My goal is to allow SSH traffic to be forwarded to the maintenance network, but to drop all other traffic.

I choose the following label scheme:

• network = application for the main datacenter network for application traffic
• network = maintenance for the maintenance network

I create the GlobalNetworkPolicy that allows SSH traffic (default deny is implicit in this case).

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-ssh-maintenance
spec:
  selector: network == 'maintenance'
  applyOnForward: true
  types:
  - Ingress
  ingress:
  # Allow SSH
  - action: Allow
    protocol: TCP
    destination:
      ports:
      - 22

Save this as allow-ssh-maintenance.yaml.

Apply the policy to the cluster:

kubectl create -f allow-ssh-maintenance.yaml

Finally, create the host endpoint for the interface that connects to the maintenance network.

apiVersion: projectcalico.org/v3
kind: HostEndpoint
metadata:
  name: myhost.eth1
  labels:
    network: maintenance
spec:
  interfaceName: eth1
  node: myhost
  expectedIPs:
  - 192.168.0.45

Replace myhost with the node name Calico Cloud uses, and the expected IPs with the actual interface IP address(es). Save this file as hep.yaml.

Apply the host endpoint to the cluster:

kubectl create -f hep.yaml

For completeness, you could also create a HostEndpoint for eth0, but because we have not written any policies for the application network yet, you can omit this step.

Viewing flow logs for host forwarded traffic

When using applyOnForward and preDNAT policies with host endpoints, you can monitor and troubleshoot policy enforcement by examining flow logs. Calico Cloud captures detailed information about how policies are applied to forwarded traffic.

Configure policy information in flow logs

To see policy information in flow logs, you must enable flow logs for forwarding decisions in the FelixConfiguration resource:

apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  name: default
spec:
  flowLogsPolicyScope: AllPolicies

This configuration enables tracking the flows transiting through the node interfaces within policies field in flow logs. The policies field contains detailed information about which policies acted on the traffic.

Understanding the transit_policies field in flow logs

The policies field in flow logs contains sub-fields that show different policy evaluation results:

• transit_policies: All host endpoint policies with applyOnForward: true that applied an action to the flow while in transit through the node (for forwarded traffic between network interfaces). Note that policies with preDNAT: true will also appear here when they apply to ingress traffic before DNAT processing.

Each policy entry in these fields follows this format:

<index>|<tier_name>|<tier_name>.<global_policy_name>|<action>|<rule_index>

Example flow log for applyOnForward policy

Using the tutorial example from this document (SSH traffic to maintenance network), here's what a flow log entry might look like:

{
  "source_ip": "10.0.0.1",
  "source_name": "client-pods-gkh45",
  "source_name_aggr": "client-pods-*",
  "source_namespace": "client-pods-namespace",
  "dest_ip": "192.168.1.10",
  "dest_name": "maintenance-server-gkh45",
  "dest_name_aggr": "maintenance-server-*",
  "dest_namespace": "maintenance-server-namespace", 
  "dest_port": 22,
  "dest_type": "wep",
  "proto": "tcp",
  "action": "allow",
  "reporter": "dst, fwd",
  "policies": {
    "all_policies": [
      "0|__PROFILE__|__PROFILE__.kns.maintenance-server|allow|0"
    ],
    "enforced_policies": [
      "0|__PROFILE__|__PROFILE__.kns.maintenance-server|allow|0"
    ],
    "pending_policies": [
      "0|__PROFILE__|__PROFILE__.kns.maintenance-server|allow|0"
    ],
    "transit_policies": [
      "0|default|default.allow-ssh-maintenance|allow|0"
    ]
  },
  "bytes_in": 1234,
  "bytes_out": 5678,
  "packets_in": 10,
  "packets_out": 8,
  "host": "my-host"
}

Key points about this flow log:

• reporter is dst, fwd, indicating this flow was reported by both the destination workload endpoint and the forwarding node. The enforced_policies shows policies that applied to the destination workload's ingress traffic, while transit_policies shows the host endpoint policy that applied as the traffic was forwarded through the node.
• The policies.transit_policies field shows that the allow-ssh-maintenance GlobalNetworkPolicy allowed the traffic
• The rule index 0 indicates the first rule in the policy matched (the SSH allow rule)

Example flow log for preDNAT policy

Here's an example of a GlobalNetworkPolicy with preDNAT: true that allows HTTP traffic incoming to the cluster from external sources:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-external-http
spec:
  selector: network = 'maintenance'
  applyOnForward: true
  preDNAT: true
  types:
  - Ingress
  ingress:
  # Allow HTTP from external sources to NodePort services
  - action: Allow
    protocol: TCP
    source:
      nets:
      - 34.0.113.0/24  # External network
    destination:
      ports:
      - 30080  # NodePort

For preDNAT policies, flow logs display the original destination IP and port before DNAT is performed. When traffic originates from outside the cluster (for example, an external client accesses a NodePort service), the flow log captures the pre-DNAT source and destination details. Here's an example:

{
  "source_ip": "34.0.113.100",
  "source_name": "-",
  "source_name_aggr": "pub",
  "source_namespace": "-",
  "source_type": "net",
  "dest_ip": "66.11.155.1",
  "dest_name": "-",
  "dest_name_aggr": "pub",
  "dest_namespace": "-",
  "dest_type": "net",
  "dest_port": 30080,
  "proto": "tcp",
  "action": "allow",
  "reporter": "fwd",
  "policies": {
    "transit_policies": [
      "0|default|default.allow-external-http|allow|0"
    ]
  },
  "bytes_in": 2048,
  "bytes_out": 4096,
  "packets_in": 15,
  "packets_out": 12,
  "host": "my-host"
}

Key points about this preDNAT flow log:

• source_name_aggr is pub, indicating external traffic from outside the cluster
• dest_name_aggr is pub, indicating the NodePort's external facing IP
• dest_port shows the original NodePort (30080) before DNAT translation to the service port
• The reporter is fwd, indicating this policy acted on forwarded ingress traffic at the host endpoint
• The policies.transit_policies field shows the preDNAT policy that allowed the traffic before DNAT processing

Additional resources

• Host endpoint
• Workload endpoint
• Global network policy