Redundancy design

[u/Particular-Book-2951]

Hello all,

I have this case where I need to set up a redundant path to our partner.

Our DCs: DC in LA and in NY. We have this subnet in LA: 10.160.10.0/24 and an another subnet in NY 10.180.10.0/24 that needs to reach 10.10.10.0/24 (our partner).

Setup: https://imgur.com/zkOiApf

Between our core switches in LA and NY, we run MPLS L3 VPN.

In LA, we have an internal FW (internal_FW_LA) and core switch (Core-sw_LA) that just run static routes between each other. There will be a direct fiber connection to the partner (Parrtner_A_LA router) from Core-sw_LA. I was thinking that I will run L3 (static) between internal_FW_LA and Partner router Partner_A_LA. This will be the primary link to the partner.

In NY, we have a internal FW (internal_FW_NY) core switch (Core_sw_NY), and VPN HUB (VPN_hub_NY). We run static routes as well between these three. From the VPN hub, I will run an IPsec tunnel to the Partner_A_NY Fortigate.

Now the idea is that we will only use one link to the partner at a time and in this case, it would be the primary link which is in LA. The 10.180.10.0/24 (in NY) needs to be routed through the MPLS and then from there through the primary link (via internal_FW_LA) to the partner router Partner_A_LA. 10.160.10.0/24 will of course take the same path.

In case the primary link goes down, then traffic from LA will need to be routed through the MPLS to NY and from there over the IPsec tunnel (so both 10.160.10.0/24 & 10.180.10.0/24) will be routed over the IPsec tunnel.

Now, I was thinking that I would need to use some kind of dynamic routing and in this case BGP on the internal firewalls but the ssue is that these internal firewalls are not allowed to use BGP (due to some policy). Running BGP on the core switches may make sense but that will break standardizing our way of designing things.

I am stuck and cannot figure out how to proceed here.

Any input?

[u/alphaxion]

If you're not allowed to use BGP, could you accomplish this with static routes that have a sort of monitor profile on them that disable your usual path if ping fails and allows a second route that is usually a higher admin value to take over?

[u/Particular-Book-2951 → u/alphaxion]

Yes, I was thinking that I would like to avoid anything anything static but this is an option as well.

Basically it would be two static routes from each firewall, one that points to the other DC with higher AD and some kind of IP SLA for tracking the status of the ping?

[u/alphaxion → u/Particular-Book-2951]

Basically yeah. Palo alto firewalls allow you to apply monitoring to a route and will disable it if the ping fails, as an example.

That would remove it from the table until the route is reinstated. This is the only way I've found to have reliable failover for IPSec tunnels between sites over separate internet connections using a Palo firewall.

[u/curry9906 → u/alphaxion]

Mikrotik as well, I guess most vendors nowadays

[u/Particular-Book-2951 → u/alphaxion]

I believe Checkpoint has it as well, I’ll take a look at it. Thank you.

[u/WendoNZ]

That no BGP restriction is rough. Statics will work with sensible metrics and path monitoring, but I'd trust BGP way more than path monitoring

[u/lizardhistorian]

So use an IGP?

[u/Particular-Book-2951 → u/lizardhistorian]

I should’ve been more clear in my post, but any dynamic routing protocol is not allowed to be used.

This case is like a special case as well. If we start using any dynamic routing protocol, then we’ll break our way of designing things.

[u/nof]

Lots of orgs have brain dead blanket policies against routing protocols on firewalls. You can install a router in front of the firewalls to do the router stuff.

[u/Particular-Book-2951 → u/nof]

This is not possible unfortunately, especially not for just one partner.

But I guess we can change our way of designing connections with partners, this may be an idea to terminate partners. Thank you.

[u/Incognito_Orange]

Is OSPF an option on the firewalls? All you're doing is failover, and it's not like this will be a swarm of LSDB updates. Can still use BFD with it.

[u/Particular-Book-2951 → u/Incognito_Orange]

No dynamic routing protocol is allowed to be used unfortunately.

[u/Incognito_Orange → u/Particular-Book-2951]

That's pretty obnoxious. Maybe recursive routing w/ an IP SLA?

[u/reddit-doc]

There are two possibilities here:
1. If you cannot use BGP, what about OSPF?
2. Do you use IPSEC over the MPLS or do you trust the carrier? Same goes for the fiber, unless you or your partner own it. If you used IPSEC you would terminate it on a firewall anyway. Depending on the vendor (e.g. Fortinet has it on board) you could use SDWAN to take care of the redundancy.

[u/Particular-Book-2951 → u/reddit-doc]

1. No dynamic routing protocol is allowed.

2. No we do not run any IPsec over our MPLS, we trust our carrier. The fiber is pretty much like a direct connection (partner network is in the same DC as ours).

Both firewalls are checkpoints and as far as I’m aware, I’ve never used SD-WAN on the checkpoints but I wouldn’t run SD-WAN without BGP.

[u/defmain]

Are there latency or bandwidth concerns when the traffic fails over to the tunnel, and your LA DC traffic goes LA > NY > LA?

[u/Particular-Book-2951 → u/defmain]

No, no concerns regarding bandwidth or latency. It is within acceptable frame.

[u/NetworkMSP]

if you have no dynamic options

your core switch in LA needs a lower AD route via the fiber connection to the partner router next hop which then needs to be originated via mpls to the NY core switch for prefix 10.10.10.0/24. you could use a static route with next hop, partner router fiber ip, ip sla tracking to accomplish this. if the next hop isnt available, that route will be removed from the RIB and not originated

you can then simply have a higher AD static route to 10.10.10.0/24 via the NY path.

if the LA path is up, traffic will use the route to the partner prefix via the LA path. if the fiber is down, traffci will follow the static to the partner via the NY path

your partner would also need to setup the reverse on their end for your prefixes.