0x8000 is the new 0x4124

On new ISR routers with XE-SDWAN version 16.10.3+, you can only log in with the default password once. Over console, there is a message printed that warns you to change the password, but it can easily get lost in all the other console output. 

Breaking boot and setting the register to 0x4124 doesn’t work as you would expect. To get back in (note, your config will be gone), break boot and config the register to 0xA102 or 0x8000. 

rommon 1 > confreg 0x8000
rommon 2 > i

Then, change the configuration register back to 0x2102 and perform a sdwan software reset. This wipes out all the configuration that exists.

Router# request platform software sdwan software reset

After that, the device will reboot, login with the default admin/admin and this time—set a password. 

Source: How to Recover the Password on XE-SDWAN?

Using eapol_test to test RADIUS

The little program eapol_test can come in handy when trying to test RADIUS configurations. eapol_test is a part of wpa_supplicant.

Good information on the program can be found in the manpage, however the usage of the -N flag for attribute/value pairs deserves a few more words (and an example to remind myself in the future how to use it. Here is an example that includes sending Called-Station-Id of be-ef-be-ef-be-ef:ent-secure to 10.10.10.10 with a shared key of ‘aruba123’.

./eapol_test -c wpasupplicantconfig.conf -N 30:s:be-ef-be-e9-be-ef:ent-secure -a 10.10.10.10 -s aruba123

According to the man page, -N is for ‘attr spec’:

Send arbitrary attribute specific by attr_id:syntax:value, or attr_id alone. attr_id should be the numeric ID of the attribute, and syntax should be one of ‘s’ (string), ‘d’ (integer), or ‘x’ (octet string). The value is the attribute value to send. When attr_id is given alone, NULL is used as the attribute value. Multiple attributes can be specified by using the option several times.

So, in my example, I’m sending attribute ID of 30 (IETF Called-Station-Id), a string, with the value of be-ef-be-e9-be-ef:ent-secure.

Useful Palo Alto PAN-OS Commands

Fix terminal height/width

set cli terminal height 500
set cli terminal width 500

Update Content/Threats from CLI (update license first)

request license fetch 
request content upgrade check
request content upgrade download latest
request content upgrade install version latest

Update Anti-Virus (AV) from CLI

request anti-virus upgrade check 
request anti-virus upgrade download latest 
request anti-virus upgrade install version latest

Update PAN-OS Version from CLI

request system software info
request system software download version 9.0.3-h3
request system software install version 9.0.3-h3
request restart system

Set management port to DHCP

configure
set deviceconfig system type dhcp-client accept-dhcp-domain yes accept-dhcp-hostname yes send-client-id yes send-hostname yes
commit


Newer models – disable ZTP

request disable-ztp
Configure Panorama server and delete default config
set deviceconfig system panorama-server `IP/Hostname`

delete network virtual-wire default-vwire
delete rulebase security rules rule1
delete zone trust
delete zone untrust
delete network interface ethernet ethernet1/1
delete network interface ethernet ethernet1/2
delete network interface ethernet ethernet1/3
delete network interface ethernet ethernet1/4
delete network interface ethernet ethernet1/5
delete network interface ethernet ethernet1/6
delete network interface ethernet ethernet1/7
delete network interface ethernet ethernet1/8

FTD Authentication with Azure MFA

I recently configured Azure MFA to authenticate AnyConnect users connecting to a FTD firewall. This required some odd workarounds.

Problems to work around

  1. FTD cannot do SAML, must use RADIUS for AnyConnect AAA
  2. Microsoft NPS with Azure MFA extension must be used for RADIUS Integration to Azure MFA 
  3. Microsoft NPS has a limited number of attributes it can filter incoming RADIUS requests on
  4. Customer has a need to only allow certain AD groups access to certain tunnel groups

Authentication Flow

  1. Firewall sends Access-Request to ISE 
  2. ISE adds RADIUS:NAS Identifier attribute to Access-Request based on CVPN3000/ASA/PIX7x-Tunnel-Group-Name
  3. NPS filters incoming requests based on NAS Identifier to appropriately authenticate different tunnel groups and does primary authentication against AD.
  4. NPS requests secondary authentication from Azure MFA
  5. Azure MFA completes MFA with user, based on the user’s default MFA method. There is no way to select this during the authentication sequence.
  6. NPS sends result back to ISE
  7. ISE proxies result back to FTD

Authentication Diagram

Ftd anyconnect azuremfa

Identity Services Engine (ISE) Configuration

  • External RADIUS Servers – define your NPS server(s) and shared secret here
  • RADIUS Server Sequence
    • Create a sequence per tunnel group that needs to be differentiated 
    • In the Advanced Attribute Settings tab, add the NAS Identifier attribute

Ise modify attrib

  • Policy Sets
    • Create one policy set per tunnel group
    • Conditions: NAS-Port-Type Virtual AND CVPN3000/ASA/PIX7x-Tunnel-Group-Name = MyGroupName
    • Server Sequence: proxy sequence created above

Network Policy Server (NPS) Configuration

  • Install Azure MFA extension and link to Azure AD – https://docs.microsoft.com/en-us/azure/active-directory/authentication/howto-mfa-nps-extension]
    • Protip: use a new NPS server. Once you install the extension, all requests authenticated locally will be subject to MFA
  • RADIUS Clients
    • In the NPS console, configure your ISE servers as RADIUS clients, using the same shared secret as above. 
    • Vendor – RADIUS Standard
  • Network Policies 
    • Create one per tunnel group name
    • Type of network access server – Unspecified 
    • Conditions
      • NAS Identifier – the identifier ISE added
      • User Groups – group you want to filter on 
    • NAS-Port-Type Virtual
    • Authentication Methods – I have them all enabled, including PEAP and EAP-MSCHAPv2 You minimally need PAP, but I have lots of logs that say “Custom” for the Authentication Type.
    • Note: The Microsoft documentation says: PAP supports all the authentication methods of Azure MFA in the cloud: phone call, one-way text message, mobile app notification, OATH hardware tokens, and mobile app verification code. *CHAPV2 and EAP* support phone call and mobile app notification.”

Firepower Threat Defense (FTD) Configuration

  • Configure ISE as your RADIUS servers for AnyConnect AAA. 
  • Set the timeout to 60 seconds or longer—if users are waiting for an SMS, phone call, push notification, etc., you want to give them enough time to do it.

User Experience

  • Users will put in their username and password at the AnyConnect client 
  • If their preferred option (set at https://account.activedirectory.windowsazure.com/Proofup.aspx) uses a code (verification code form app or text message), they will see a pop-up after their first authentication. 
  • If their preferred option is a voice call or push notification, the authentication will wait until the verification is complete before granting access. Make sure all timeouts in the chain are long enough to support this. 

Troubleshooting

The NPS logs are a pain to use for troubleshooting. I wrote a quick python script to help parse the logs. You can find it here: https://github.com/gosse/interpret-nps-logs.

Dell S-Series OS10 Image Upgrade

I keep having to look this up, so I’m just going to write it here quick.

1. Download firmware (finally on dell.com)
2. Copy to usb stick or tftp server
3. Do the upgrade.

OS10# copy run start
(if tftp) OS10# image download tftp://filename.bin
OS10# image install usb://filename.bin

OS10# show image status
Image Upgrade State: install
File Transfer State: idle
State Detail: No download information available
Task Start: 0000-00-00T00:00:00Z
Task End: 0000-00-00T00:00:00Z
Transfer Progress: 0 %
Transfer Bytes: 0 bytes
File Size: 0 bytes
Transfer Rate: 0 kbps

Installation State: install
--------------------------------------------------
State Detail: In progress: Configuring root filesystem
Task Start: 2018-11-06T21:55:30Z
Task End: 0000-00-00T00:00:00Z
OS10# show image status
Image Upgrade State: install
==================================================
File Transfer State: idle
--------------------------------------------------
State Detail: No download information available
Task Start: 0000-00-00T00:00:00Z
Task End: 0000-00-00T00:00:00Z
Transfer Progress: 0 %
Transfer Bytes: 0 bytes
File Size: 0 bytes
Transfer Rate: 0 kbps

Installation State: install
--------------------------------------------------
State Detail: In progress: Installing OS10 packages
Task Start: 2018-11-06T21:55:30Z
Task End: 0000-00-00T00:00:00Z

OS10# show boot
Current system image information:
===================================
Type Boot Type Active Standby Next-Boot
-----------------------------------------------------------------------------------
Node-id 1 Flash Boot [A] 10.4.0E(R3) [B] 10.4.1.2 [A] active
OS10# boot system standby
OS10# show boot
Current system image information:
===================================
Type Boot Type Active Standby Next-Boot
-----------------------------------------------------------------------------------
Node-id 1 Flash Boot [A] 10.4.0E(R3) [B] 10.4.1.2 [B] standby
OS10# copy run start
OS10# reload

Proceed to reboot the system? [confirm yes/no]:yes

The system reboots.

show ver
OS10# show version
Dell EMC Networking OS10-Enterprise
Copyright (c) 1999-2018 by Dell Inc. All Rights Reserved.
OS Version: 10.4.1.2
Build Version: 10.4.1.2.524
Build Time: 2018-09-26T17:20:01-0700
System Type: S4112F-ON
Architecture: x86_64
Up Time: 00:02:40

Automating Certificate Install on ASA with Netmiko

Here is a little script I wrote to automate putting certificates onto ASAs. It also activates the cert on the inside interface (mine is a one-armed VPN concentrator). The cert is assumed to already in the correct format and named asabase64.cert.

#!/usr/bin/env python3

from netmiko import Netmiko
import base64
import datetime

# connection config for netmiko
asav = {
"host": "hostname",
"username": "user",
"password": "pass",
"device_type": "cisco_asa",
"secret": "enable secret"
}

# open and read the cert
with open("asabase64.cert", "r") as f:
cert = f.read().splitlines()

# name the cert with today's date
# this helps when pushing a new cert
# to not have a namespace overlap
today = datetime.datetime.now().strftime("%d-%m-%Y")
certname = today + "starcert"
certcmd = "crypto ca import " + certname + " pkcs12 asaexportpass"
# build the commands needed to install the cert
commands = []
commands.append("conf t")
commands.append(certcmd)
commands = commands + cert
commands.append("")
commands.append("quit")
trustpoint = "ssl trust-point " + certname + " inside"
commands.append(trustpoint)

# connect to the firewall
net_connect = Netmiko(**asav)
print(net_connect.find_prompt())

# always a good test
output = net_connect.send_command_timing("show int ip br")

# send the commands
for command in commands:
print(command)
net_connect.send_command_timing(command)


# clean up
net_connect.disconnect()

FDM FTD HA Upgrade

To upgrade an HA pair of Firepower firewalls managed by FDM, use the following procedure: 

1. Deploy any pending changes
2. Log into standby unit, upload the upgrade and install 
3. After the reboot and copmleted upgrade, on teh standby unit, go High Availability and Switch Mode. This will force failover. 
4. Log into new standby unit, upgrade that unit. 
5. Resume HA (if it does not automatically resume) 
6. Deploy policy if needed 

Restart FDM WebUI

Sometimes, the FDM Web UI stops responding. I’ve found this especially on 6.5 code. To restart tomcat, go into expert mode and disable/enable: 

 

> expert
admin@fp:~$ sudo su -
Password:
root@fp:~# cd /ngfw/var/cisco/ngfwWebUi/
## delete this file if it exists
root@fp:ngfwWebUi# rm .bootstrap-failed
root@fp:ngfwWebUi# pmtool disablebyid tomcat
root@fp:ngfwWebUi# pmtool enablebyid tomcat

 

The web service takes a while (10+ minutes) to come back online. 

h/t to Pieter-Jan Nefkens at https://www.nefkens.net/fdm-application-fails-after-upgrade/ 

VRF Lite

VRF-lite is just VRFs, without L3VPN. VRFs are extremely useful any time you need to have multiple routing tables (and protocols) on a single device. This also allows for overlapping IPs on the same devices. 

When talking about VRF-lite, the following terms are often used. 

* CE routers – customer edge routers, these devices provide customer access to the service provider. 
* PE routers – provider edge routers, the provider router (usually on-site) 

Each interface with an IPv4 or IPv6 address is assigned a VRF. Most are in “default”. The out-of-band management port on most Cisco devices is assigned to Mgmt-intf by default. 

Routing protocols are supported in VRF-lite as well. VRF configuration, with routing processes is below. Note how some of the VRF-aware routing protocols use address families. I’ll use the following topology to lab this out. 

UntitledImage

First, the VRFs need to be created on both devices:

vrf definition BLUE
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
vrf definition EGGPLANT
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
vrf definition ORANGE
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
vrf definition SAGE
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family

ip routing 
ipv6 unicast-routing 

Next, all the interfaces need configuring. 

R1 

interface Loopback0 
vrf forwarding SAGE
ip address 1.1.1.1 255.255.255.255
ipv6 address 2001:DB8:1:1::1/128
ipv6 enable
!
interface Loopback10
vrf forwarding ORANGE
ip address 10.10.10.1 255.255.255.255
ipv6 address 2001:DB8:10:10::10/128
ipv6 enable
!
interface Loopback20
vrf forwarding EGGPLANT
ip address 10.20.20.1 255.255.255.255
ipv6 address 2001:DB8:20:20::20/128
ipv6 enable
!
interface Loopback30
vrf forwarding BLUE
ip address 10.30.30.1 255.255.255.255
ipv6 address 2001:DB8:30:30::30/128
ipv6 enable
!
interface GigabitEthernet1
vrf forwarding SAGE
ip address 10.0.0.1 255.255.255.0
negotiation auto
ipv6 address 2001:DB8::1/64
ipv6 enable
!
interface GigabitEthernet1.10
encapsulation dot1Q 10
vrf forwarding ORANGE
ip address 10.10.0.1 255.255.255.0
ipv6 address 1001:DB8:10::10/64
ipv6 enable
!
interface GigabitEthernet1.20
encapsulation dot1Q 20
vrf forwarding EGGPLANT
ip address 10.20.0.1 255.255.255.0
ipv6 address 201:DB8:20::1/64
ipv6 enable
!
interface GigabitEthernet1.30
encapsulation dot1Q 30
ip address 10.30.0.1 255.255.255.0
ipv6 address 2001:DB8:30::1/64
ipv6 enable
!

R2

interface Loopback0
vrf forwarding SAGE
ip address 2.2.2.2 255.255.255.255
ipv6 address 2001:DB8:1:1::2/128
ipv6 enable
!
interface Loopback10
vrf forwarding ORANGE
ip address 10.10.10.2 255.255.255.255
ipv6 address 2001:DB8:10:10::11/128
ipv6 enable
!
interface Loopback20
vrf forwarding EGGPLANT
ip address 10.20.20.22 255.255.255.255
ipv6 address 2001:DB8:20:20::22/128
ipv6 enable
!
interface Loopback30
vrf forwarding BLUE
ip address 10.30.30.33 255.255.255.0
ipv6 address 2001:DB8:30:30::33/128
ipv6 enable
!
interface GigabitEthernet1
vrf forwarding SAGE
ip address 10.0.0.2 255.255.255.0
negotiation auto
ipv6 address 2001:DB8::2/64
ipv6 enable
!
interface GigabitEthernet1.10
encapsulation dot1Q 10
vrf forwarding ORANGE
ip address 10.10.0.2 255.255.255.0
ipv6 address 2001:DB8:10::2/64
ipv6 enable
!
interface GigabitEthernet1.20
encapsulation dot1Q 20
vrf forwarding EGGPLANT
ip address 10.20.0.2 255.255.255.0
ipv6 address 2001:DB8:20::2/64
ipv6 enable
!
interface GigabitEthernet1.30
encapsulation dot1Q 30
vrf forwarding BLUE
ip address 10.30.0.2 255.255.255.0
ipv6 address 2001:DB8:30::2/64
ipv6 enable

Finally, the routing protocols. 

Static (vrf SAGE) 

R1

ip route vrf SAGE 2.2.2.2 255.255.255.255 10.0.0.2
ipv6 route vrf SAGE 2001:DB8:1:1::2/128 2001:DB8::2

R1#sh ip route vrf SAGE

1.0.0.0/32 is subnetted, 1 subnets
C 1.1.1.1 is directly connected, Loopback0
2.0.0.0/32 is subnetted, 1 subnets
S 2.2.2.2 [1/0] via 10.0.0.2
10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.0.0/24 is directly connected, GigabitEthernet1
L 10.0.0.1/32 is directly connected, GigabitEthernet1

R1#sh ipv6 route vrf SAGE

IPv6 Routing Table - SAGE - 5 entries
C 2001:DB8::/64 [0/0]
via GigabitEthernet1, directly connected
L 2001:DB8::1/128 [0/0]
via GigabitEthernet1, receive
LC 2001:DB8:1:1::1/128 [0/0]
via Loopback0, receive
S 2001:DB8:1:1::2/128 [1/0]
via 2001:DB8::2
L FF00::/8 [0/0]
via Null0, receive

R2

R2(config)#ip route vrf SAGE 1.1.1.1 255.255.255.255 10.0.0.1
R2(config)#ipv6 route vrf SAGE 2001:db8:1:1::1/128 2001:db8::1

R2#sh ip route vrf SAGE
1.0.0.0/32 is subnetted, 1 subnets
S 1.1.1.1 [1/0] via 10.0.0.1
2.0.0.0/32 is subnetted, 1 subnets
C 2.2.2.2 is directly connected, Loopback0
10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.0.0/24 is directly connected, GigabitEthernet1
L 10.0.0.2/32 is directly connected, GigabitEthernet1

R2#sh ipv6 route vrf SAGE
C 2001:DB8::/64 [0/0]
via GigabitEthernet1, directly connected
L 2001:DB8::2/128 [0/0]
via GigabitEthernet1, receive
S 2001:DB8:1:1::1/128 [1/0]
via 2001:DB8::1
LC 2001:DB8:1:1::2/128 [0/0]
via Loopback0, receive
L FF00::/8 [0/0]
via Null0, receive

OSPF (vrf ORANGE)

! ospfv2 for v4
router ospf 10 vrf ORANGE
redistribute connected

interface GigabitEthernet1.10
ip ospf 10 area 0
R2#sh ip ospf nei

Neighbor ID Pri State Dead Time Address Interface
10.10.10.1 1 FULL/BDR 00:00:35 10.10.0.1 GigabitEthernet1.10

! ospfv3 for v6 
router ospfv3 10
!
address-family ipv6 unicast vrf ORANGE
redistribute connected
router-id 10.10.10.1
exit-address-family
router ospf 10 vrf ORANGE
redistribute connected

interface GigabitEthernet1.10
ospfv3 10 ipv6 area 0

R2#sh ospfv3 vrf ORANGE nei

OSPFv3 10 address-family ipv6 vrf ORANGE (router-id 10.10.10.2)

Neighbor ID Pri State Dead Time Interface ID Interface
10.10.10.1 1 FULL/BDR 00:00:31 13 GigabitEthernet1.10

EIGRP (vrf EGGPLANT)

router eigrp EGGPLANT
!
address-family ipv4 unicast vrf EGGPLANT autonomous-system 20
!
topology base
exit-af-topology
network 10.20.0.0 0.0.0.255
exit-address-family
!
address-family ipv6 unicast vrf EGGPLANT autonomous-system 20
!
af-interface Loopback20
passive-interface
exit-af-interface
!
topology base
exit-af-topology
eigrp router-id 10.20.20.1
exit-address-family

BGP (vrf BLUE) 

vrf definition BLUE
rd 65001:1
route-target export 65001:1
route-target import 65001:1
R1#sh run | sec bgp
router bgp 65001
bgp router-id 1.1.1.1
bgp log-neighbor-changes
!
address-family ipv4 vrf BLUE
bgp router-id 10.30.30.1
network 10.30.0.0 mask 255.255.255.0
redistribute connected
neighbor 10.30.0.2 remote-as 65002
neighbor 10.30.0.2 activate
exit-address-family
!
address-family ipv6 vrf BLUE
redistribute connected
bgp router-id 10.30.30.1
neighbor 2001:DB8:30::2 remote-as 65002
neighbor 2001:DB8:30::2 activate
exit-address-family
R1#sh bgp vpnv4 uni all sum    
BGP router identifier 1.1.1.1, local AS number 65001
BGP table version is 4, main routing table version 4
3 network entries using 768 bytes of memory
4 path entries using 544 bytes of memory
6/3 BGP path/bestpath attribute entries using 1824 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
1 BGP extended community entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 3184 total bytes of memory
BGP activity 6/0 prefixes, 8/0 paths, scan interval 60 secs
3 networks peaked at 02:24:50 Jul 23 2021 UTC (00:02:04.264 ago)

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.30.0.2 4 65002 8 8 4 0 0 00:02:03 2

R1#sh bgp vpnv4 uni all
BGP table version is 4, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65001:1 (default for vrf BLUE) VRF Router ID 10.30.30.1
* 10.30.0.0/24 10.30.0.2 0 0 65002 i
*> 0.0.0.0 0 32768 i
*> 10.30.30.0/24 10.30.0.2 0 0 65002 ?
*> 10.30.30.1/32 0.0.0.0 0 32768 ?

R1#show bgp vpnv6 uni all sum
BGP router identifier 1.1.1.1, local AS number 65001
BGP table version is 4, main routing table version 4
3 network entries using 840 bytes of memory
4 path entries using 672 bytes of memory
3/2 BGP path/bestpath attribute entries using 912 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
1 BGP extended community entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2472 total bytes of memory
BGP activity 6/0 prefixes, 8/0 paths, scan interval 60 secs
3 networks peaked at 02:24:45 Jul 23 2021 UTC (00:03:00.589 ago)

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
2001:DB8:30::2 4 65002 8 8 4 0 0 00:03:00 2

R1#show bgp vpnv6 uni all
BGP table version is 4, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65001:1 (default for vrf BLUE) VRF Router ID 10.30.30.1
* 2001:DB8:30::/64 2001:DB8:30::2 0 0 65002 ?
*> :: 0 32768 ?
*> 2001:DB8:30:30::30/128
:: 0 32768 ?
*> 2001:DB8:30:30::33/128
2001:DB8:30::2 0 0 65002 ?

Pretty easy, right?