Friday, July 15, 2016

AWS Security: Automating Palo Alto security rules with AWS Lambda

With the increased adoption of IaaS cloud services such as Amazon Web Services (AWS) and Microsoft Azure, there is also a greater need for security controls in the cloud. Firewall and IPS vendors such as Palo Alto, Checkpoint, Fortinet have made available virtual instances of their products ready to run in these cloud environments. These tools can provide great advantages on top of the existing security controls inherent in the cloud platform. For example, security groups in AWS control traffic flow to/from IP addresses and ports, but adding a Palo Alto or Checkpoint gateway can provide more inspection capabilities and filtering, especially on the application level. In addition to firewall capabilities, these products can provide extra features such as intrusion prevention, URL filtering, and other features that are lacking with the native security controls.
However, there is still lots of manual configuration expected from the Security or Network administrators, such as configuring interfaces or network settings, firewall and threat prevention rules, and so on. But in a cloud environment where speed and flexibility are its main assets, waiting for a security administrator to put in a rule for a newly created instance can seem to be a step back. The vendors above are definitely aware of this, and have introduced features such as Palo Alto’s capability of reading AWS attributes (for example tags or instance IDs) and then using them in dynamic rules that get updated as changes occur in the cloud environment. For example, you can create a group with the setting to include anything with a certain tag, and then use this group in a security rule to allow Web traffic to it. The gateway will then know to allow web traffic to any new or existing instance that has this tag. (A full list of monitored AWS attributes can be found here).
But there are still more features that you might want to configure automatically that are not included by the vendor. The vendors mentioned above all have API interfaces available, and so combining that with the tools from Microsoft or Amazon, we can easily write small pieces of code to automate lots of these tasks. To demonstrate this, I wrote a lambda function that monitors AWS instance starts and stops, as well as security group updates, and then pushes (and deletes) rules to a Palo Alto gateway based on these changes.
In this post, I will go over the different components of my code, its design, and then show it in action. Finally i will list a number of notes and considerations as well as a link to download the code.

Components

Palo Alto

Palo Alto instances can be accessed from the Amazon marketplace. Palo Alto provides excellent documentation on how to set up a gateway in the AWS, and I would recommend to start here for the initial configuration. Another useful case study provided by Palo Alto is on how to configure and use dynamic address groups in rules, where the groups are based on AWS attributes.
For this setup, I had a Palo Alto gateway configured as an Internet gateway in AWS, and so all internet traffic from my instances was passing through it. I also had an elastic IP assigned to its management port, and my lambda function used this IP address to configure the gateway. Using a public IP address to configure the gateway is not the best option, and I would recommend using a private IP instead (I address some of the limitations this might introduce in the notes section below).

AWS CloudTrail/S3

CloudTrail is a service from AWS to log and store the history of all API calls made in an AWS environment. CloudTrail saves all the logs to AWS S3 which is another AWS service that provides object storage. CloudTrail has to be enabled so that we can monitor when changes are made that are relevant to our code, and then act based on that information.
More details on CloudTrail and S3 can be found here and here.

AWS Lambda

Lambda is a service from AWS that lets you upload code, and AWS will run it for you based on triggers you set up (such as triggers from other AWS services or external triggers), all this without having to run a dedicated instance for your code. In our case, we will use CloudTrail (S3) as a trigger for our code, so that whenever a new change is made in the environment, the code can scan the changes and determines if a corresponding change is required in the Palo Alto gateway.
Details on AWS Lambda can be found here.

Code Features/Design

Adding rules

The lambda function will monitor the following two events for adding new rules:
  • StartInstances: Event indicating that a new instance was started
  • AuthorizeSecurityGroupIngress: Event indicating that a new rule was adding to an existing security group
Once any of these two events is detected, the function will extract the relevant information for rules required to the instances affected, and add the corresponding rules.
Rules added have the name corresponding to the type of event that triggered them. If the rule added is because of an instance started, then it’s named ‘instanceId-#’ where # is increased with every rule added, but if it is due to a change in a security group, then the naming is ‘groupid-#’. The naming convention is used by the code to track the rules it added.

Unnecessary rules

Since the Palo Alto gateway is running as an internet gateway, there are many scenarios that are not relevant, and the code will try to filter out these events so that we don’t make any unnecessary changes to the Palo Alto gateway. The following scenarios would not introduce changes to the Palo Alto gateway:
  • Instances that are started but that don’t use the Palo Alto as their internet gateway. For example, there can be multiple internet gateways configured and we're only concerns with instances that use the Palo Alto to reach the internet.
  • In instances with multiple interfaces, the code checks all the interfaces, and only includes those that use the Palo Alto instance as an internet gateway.
  • Security group rules that have a source from within the AWS VPC will be filtered out. The Palo Alto gateway in this instance is used as an internet gateway, and so traffic from within the VPC would not pass through it.
  • Security group rules that reference other security groups as a source will also not be included. These rules imply that the traffic would be local to the VPC and so would not pass the internet gateway.
Also before adding a rule, a test is made to make sure traffic is not already allowed, and only after making sure that traffic is denied, we will add a new rule.

Rule location

The code will also only add rules at the bottom, so that the security administrator can create rules at the top of the rule base that would override anything added dynamically. This can be used to control the rules automatically added. Furthermore, we specify in the code the bottom most rule that the new rules have to go above, and we can use this to control the location of the rules so that certain rules always remain at the bottom (For example, our clean up rule).

Cleaning up

When instances are stopped or rules in security groups are removed, we want any rules that we added to be removed. To avoid removing any permanent rules added by the security administrator, the code will only remove rules that it added previously to the rule base (This is can be controlled using the naming). The following events are monitored as triggers:
  • StopInstances: An instance was stopped.
  • RevokeSecurityGroupIngress: A rule was removed from a security group.

Imported Modules

I tried not to import any modules that don’t come with a default installation of Python except when needed. The only exceptions are:

Boto3

Boto3 is the AWS SDK for python. Using boto3 we can make API calls to AWS to get relevant information that will help us gather the necessary details to read events from AWS and build the rules and changes we want to push to Palo Alto. More details on boto3 at https://boto3.readthedocs.io/en/latest/.

Paloalto.py

These are functions that I wrote to interface with the Palo Alto gateway. The functions include adding/deleting rules or objects, searching rules and getting details, and saving changes.
To download the latest version of this code, refer to the github link. You can also refer to this blog post which goes over the details for writing it.

Netaddr

I used the IPAddress and IPNetwork functions from netaddr to allow quick checks on IP addresses (For example, if an IP address belongs to a certain subnet).

Event Handling Logic

The main function in the code is the lambda_handler, AWS passes to it the event that triggered it, which in this case would be adding a new entry to S3 (by CloudTrail):
  1. The event passed by AWS contains the location of the S3 object that has the new cloudtrail entries. Our first step is to extract the name and location of this file.
  2. Second we have to retrieve the file using the S3 methods from boto3, and then uncompress it using gzip.
  3. The contents are then parsed as json to allow us to read and extract properties easily.
  4. Finally we iterate through all the records in the logs provided searching for any of the following events:

  • StartInstances

  1. Call event_StartInstances which returns the list of rules relevant to the Instance in the event. In this function, a list of instance Ids are extracted, and the following is performed for each instance.
    1. First a list of relevant subnets is created. Relevant subnets are those that use the Palo Alto as their internet gateway.
    2. Second, a list of all interfaces belonging to the instance is created along with the subnets each belongs to.
    3. For each interface that belongs to a subnet in the relevant subnets, a list of security groups attached to it is compiled.
    4. Finally, the rules of all security groups compiled are parsed through, and the relevant rules are added to a list to be sent back.
  2. For each rule returned, first we have to convert the format to something that can be understood by Palo Alto. This means that we need to add zone definition, translate the destination port to a corresponding service and application, and specify the action for the rule. I used the aws_rules_to_pa function to convert the format, which in turn uses aws_to_pa_services to map port numbers to application and service combinations.
  3. Once the format is changed, we can now test the existing rule base allows this traffic. If it is already allowed, then we move to the next rule in our list from the first Step, otherwise we add the rule on the Palo Alto gateway and move it to the proper location.
  4. Finally, commit to save the changes on the Palo Alto.

  • StopInstances

  1. Call event_StopInstances to get a list of Instance Ids from the log event.
  2. For each instance id, call paloalto_rule_findbyname to get a list of all existing rules added by earlier by our code.
  3. Remove each rule returned.
  4. Commit to save changes

  • AuthorizeSecurityGroupIngress

  1. Call event_AuthorizeSecurityGroupIngress to get a list of all rules to be added. (Similar function to event_StartInstances described above).
  2. Convert each rule to Palo Alto format using aws_rules_to_pa.
  3. Find if there are already existing rules that would allow the traffic for each rule, and discard any rule that has a match.
  4. Add remaining rules and move them to the proper location.
  5. Commit to save changes.

  • RevokeSecurityGroupIngress

  1. Call event_RevokeSecurityGroupIngress to get a list of relevant security rules to be removed.
  2. Find all rules added by our code for this security group (using the rule names)
  3. Compare the matching rules on the Palo Alto with the list of relevant rules from Step 1.
  4. Remove rules that match both lists.
  5. Commit to save changes.

In action

Setup

To run the code as is, the following will be required:
  1. Palo Alto instance configured with a publicly accessible IP address for management.
  2. Lambda function created with the following settings (For help configuration the lambda function, refer to this link, and in particular Using AWS Lambda with AWS CloudTrail):
    1. Handler should be set to lambda.lambda_handler.
    2. No VPC set. (If you would like to set a VPC, refer to the Notes section below for more details).
    3. IAM role (with policy attached) to allow the lambda function access to query your S3 and EC2 resources.
    4. Timeout value of 25 seconds.
    5. Trigger set to the S3 bucket containing the CloudTrail logs.
    6. Runtime set to ‘Python 2.7’
  3. Finally you will need to upload the code as a zip file to your lambda function. Before doing so, there are some hardcoded variables that need to be set first (All of which are at the top of the lambda_handler function in lambda.py):
    1. pa_ip: IP address of your Palo Alto gateway.
    2. pa_key: Access key for the Palo Alto gateway (Refer to the Pan-OS XML API User guide for more details on this, and specifically this page).
    3. pa_bottom_rule: Name of the rule which the lambda function would be adding on top of. This would usually be the clean up rule in your security policy.
    4. pa_zone_untrust: Name of the outside zone configured on the Palo Alto gateway.
    5. pa_zone_trust: Name of the inside zone configured on the Palo Alto gateway.
    6. pa_sgp: name of security profile group in Palo Alto to be set on rules added.
    7. igwId: Instance id of the Palo Alto gateway.

Runtime

In the following example, i had a simple setup of a 3 web server instances that use a Palo Alto instance as their internet gateway. I set the variables for my lambda function to point to my Palo Alto, provided the Access Key, etc.
I had a basic security rule base configured with 4 rules initially:
  • Two rules for my web servers. One rule to allow access in (on tcp ports 80,81, and 8000), and one rule to allow access out from the web servers.
  • One rule to deny any clear text authentication protocols such as ftp, telnet, etc.
  • Finally a clean up rule so that all other dropped traffic is logged.

base rulebase.png

I set the ‘Clean up’ rule to be my bottom rule in the lambda function, so that all rules created would be added between rules 3 and 4.

Starting an Instance

I then started my 'web server 2' instance which had the IP address 172.20.200.225, and had the webserver_sg security group assigned, which allowed traffic from any internet source to destination ports 80 and 443:

Once the instance is started, you can see the Palo Alto rulebase updated with new rule #4:

Note that only one rule was added (for ssl - tcp port 443) since port 80 was already allowed by rule #2 in the rulebase.

Adding rules to a security group

Next I updated the security group ‘webserver_sg’ and added two new rules:

The lambda function adds two new rules with the security group id as the name:

Removing a rule from a security group

Finally, I removed one of the newly added rules from the security group (for port 22):

And the rulebase was updated accordingly:

Notes and considerations

  • There might be a delay from the time of an event to the time the action is seen in the Palo Alto gateway. This is because AWS can have up to 5-15 minutes delay from the time an API call is made to the time it is logged in CloudTrail. I am not aware of an easy way to overcome this, other than configure the lambda function to run on a schedule (for example every 1 or 5 minutes), or moving the code to be run continuously on a host that has access to CloudTrail and can monitor it in real time.
  • In my tests, I used a public IP address of the Palo Alto gateway to configure it. This was easier since I didn’t place my lambda function to run from within my VPC, and so it couldn’t access the private IP address. To have the lambda function access the Palo Alto gateway through a private IP address, the lambda function must be run from within the VPC, and with security groups assigned to allow it to access the private IP of the Palo Alto. Furthermore, running lambda from within the VPC might interfere with how it accesses S3 objects since those are accessed through the internet. The easiest way to get around this to have an endpoint created in the VPC for accessing S3 (See https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/).
  • All ICMP rules from AWS are treated the same when pushed to Palo Alto (configured with the application ‘icmp’ that would allow all types of ICMP regardless of the configuration in the security groups). Modify the function aws_to_pa_services to introduce more granularity.
  • Currently only inbound rules from the security groups are examined and added, but I will be adding support for outbound rule access as well.

Download
You can download the latest version of the code on github.
To use the code as is, you only need to upload the zip file to your lambda function. If you want to make modifications, you have to zip all files (lambda.py, paloalto.py, netaddr, and netaddr-0.7.18.dist-info).
I hope this has been helpful, and note that while the functionality described in this post should be fully functional, there are a number of other features that are in progress, and the github link will be updated as these features are completed.

Thursday, July 14, 2016

Writing a Palo Alto Firewall REST API Client in Python

While working on code to configure my PaloAlto instances automatically in Amazon AWS, I needed to write functions that would interact with the Palo Alto gateway (add/remove rules, create objects, commit changes, etc.). Palo Alto makes available a number of documents available to help with this, but I didn’t find any one source that would explain the process completely of how to send commands, interpret return codes and then parse outputs returned, and so I wanted to document this process.
Palo Alto gateways have a REST API available which allows you to send commands over HTTPS (or HTTP), and then returns the output in XML format making it easy to parse through and extract the information.


REST Request and Response Structure


The structure for the API requests is:
http(s)://hostname/api/?request-parameters-values
  • hostname: hostname or IP address of the Palo Alto gateway.
  • request: Can be one of 9 different request types, we will mainly use: keygen, config, op, and commit. There are others that allow you to export/import configuration or logs and other information. The request has to be specified with the 'type' paramater, for example: 'type=keygen'.
  • parameters: Each request type has different parameters available to it. For example, the 'config' request has the 'action' parameter which can be set to values such as edit,set,delete,etc. in the format (action=set,action=edit,action=delete,etc.). The op (operation) request has commands associated with it such as save, show, request, etc. 
  • values: Some parameter would require values that need to be provided, and these go here. For example, adding a new rule would require specifying parameters such as source, destination, service,etc.
Once a request is sent, the gateway will send back a response which consists of two parts: 
  1. Response status: This includes a ‘status’ such as ‘success’ or ‘error’, and a ‘code’ which is a numerical value. For example error codes 19 and 20 mean success, and other codes would specify different error reasons (For example, internal errors, invalid objects, etc.).
  2. Results. Certain commands would require a response from the gateway such as listing specific rules or objects, and these would be shown under the results section. 
Both these parts are provided in an XML tree format:

<response status="success" code="20">
<msg>command succeeded</msg>
</response>

For more details on the REST API structure, a list of all different requests and responses, refer to the PAN-OS XML API Usage Guide.

Determining Command Syntax


The XML API guide from Palo Alto is very helpful in finding out the request types, parameters available, as well as error codes and notes on the response structures. However, there are some things that might not be included in the guide, such as the structure of the values field that needs to be passed for certain commands. The easiest way to find that out is to enable debugging in the CLI, and then execute the command that would achieve the result you are looking for. Palo Alto will then show you the syntax it passed, and you can use that as a model. 
For example, to get the syntax for adding a security rule:
  1. Turn on debugging “debug cli on”
  2. Go into configure mode by running “configure”
  3. Run the command to add a rule:
set rulebase security rules "new" from untrust to trust source 1.1.1.1 destination 2.2.2.2 service tcp_1234 application any profile-setting group strict_spg

      4. From the output, the parts highlighted in red are what you would need to carry:

<request cmd="set" obj="/config/devices/entry[@name='localhost.localdomain']/vsys/entry[@name='vsys1']/rulebase/security/rules/entry[@name='new']" cookie="2476837088324585"><from><member>untrust</member></from><to><member>trust</member></to><source><member>1.1.1.1</member></source><destination><member>2.2.2.2</member></destination><service><member>tcp_1234</member></service><application><member>any</member></application><profile-setting><group><member>strict_spg</member></group></profile-setting></request>

     5. The XML API guide will also provide guidance on the values it is expecting. For the “set” action under the “config” type, it tells you that it expects two parts for values: xpath and element.

The full request would be as follows:
https:///api/?type=config&action=set&Key=&xpath="/config/devices/entry[@name='localhost.localdomain']/vsys/entry[@name='vsys1']/rulebase/security/rules/entry[@name='sg-b0f7ddcb']&element=trustuntrustany172.20.200.225anytcp_465allowstrict_spg

Authentication


To be able to send requests to the gateway, you need an access key to be included in each request. Once you have the key, the key should be included in every request and is listed right after the request, for example:

http(s)://hostname/api/?type=config&action=set&Key=KEYVALUE&cmd=


To generate a key, send the following request to the gateway:

http(s)://hostname/api/?type=keygen&user=username&password=password 

(Where username and password are the credentials of an already configured administrator account on the gateway).

The response will contain the access key. Sample response:
<response status="success"> 
 <result> 
  <key>gJlQWE56987nBxIqsdflkjsdf234ASo2BgzEA9UOnlZBhU</key> 
 </result> 
</response>

Sending a request in Python


With this information in mind, we can now turn to Python to write the code that will send requests to Palo Alto gateways and then interpret the responses. Let’s walk through the example of writing a function which will let us add security rules on a Palo Alto gateway.

Constructing a request

The request we will be sending will be as follows (which covers all parts in the request syntax mentioned above):

url = "https://"+pa_ip+cmd+"Key="+pa_key+"&"+urllib.urlencode(parameters)

Each of the items highlighted in red are variables that need to be filled in to complete the request: 
  • pa_ip: holds the IP address (or hostname) of the gateway.
  • cmd: type of request and parameter associated with the function we are trying to perform. In this case, we are writing a function to add security rules to the gateway, and so the type is config and action is set, so:
cmd = "/api/?type=config&action=set&"
  • pa_key: access key we obtained for the gateway.
  • urllib.urlencode(parameters): there are two parts here to consider:
    • parameters: holds the last part of the request which are the values required for the command or request we are sending to the gateway. In case of adding a rule, there are two values that need to be sent:
      1. xpath: path to the item we are modifying on the Palo Alto gateway.
      2. element: contains the values of the different options in the rule (source/destination IP, action, service, etc.).
    • urllib.urlencode: since we are sending these requests over HTTP, some characters need to be encoded so that the receiving end can interpret them properly. Characters such as space cannot be sent as is but have to be changed to a supported format. urlencode function from the urllib library allows us to do this, and requires that values passed to it are passed in a python dictionary format. 
parameters = {'xpath': "/config/devices/entry[@name='localhost.localdomain']/vsys/entry[@name='vsys1']/rulebase/security/rules/entry[@name='sg-b0f7ddcb']", 'element': '<to><member>trust</member></to><from><member>untrust</member></from><source><member>any</member></source><destination><member>172.20.200.225</member></destination><application><member>any</member></application><service><member>tcp_465</member></service><action>allow</action><profile-setting><group><member>strict_spg</member></group></profile-setting>'}


Sending the request


Once we have constructed the request, we can send it using the urlopen function from urllib2 library:

response = urllib2.urlopen(url)

In case you are connecting to a gateway with an untrusted SSL certificate, you will need to tell the urlopen command to ignore the SSL certificate check, otherwise the command will throw an error. To do this you can use the ssl library in python to create a context that ignores the certificate check and then pass it to the urlopen function:

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
response = urllib2.urlopen(url, context=ctx)

Parsing the Response in Python


In the previous section, we sent a request using the urlopen function, and specified the response variable to hold the return values from urllopen. ‘response’ in this case would hold the values returned from the Palo Alto gateway as well as details on the response on the HTTP level. So we can use response.code for example to print of the HTTP status of our request (200 for Okay, and almost everything else would indicate errors). 
If the HTTP status returned is 200, then we can proceed to analyze the response from the Palo Alto gateway. To do this, we need to read this as XML. Response.read() allows us to read the contents of the response from Palo Alto, and the fromstring function allows us to parse this in XML format (Need to import xml.etree.ElementTree for this function, and I have used the syntax ‘import xml.etree.ElementTree as ET’ to make it easier to reference):

contents= ET.fromstring(response.read())

contents now will hold the Palo Alto response, in an XML tree format which we can parse easily. ‘contents’ on its own access the first level, ‘contents[0]’ access the first member on the second level, contents[0][0] access the first member in the second level of the first member on the first level, and so on.

On each level, there are three values we can access: tag, attrib, and text. Consider the example of querying the Palo Alto gateway to list all configured service objects: 

Request:

https:///api/?type=config&action=get&Key=&xpath=/config/devices/entry[@name='localhost.localdomain']/vsys/entry[@name='vsys1']/service

Output:

<response status="success" code="19">
 <result total-count="1" count="1">
  <service admin="admin" dirtyId="14" time="2016/07/12 19:12:50">
   <entry name="tcp_81">
    <protocol>
     <tcp>
      <port>81</port>
     </tcp>
    </protocol>
   </entry>
   <entry name="tcp_8000">
    <protocol>
     <tcp>
      <source-port>1-65535</source-port>
      <port>8000</port>
     </tcp>
    </protocol>
   </entry>
   <entry name="tcp_465" admin="admin" dirtyId="12" time="2016/07/12 19:01:49">
    <protocol admin="admin" dirtyId="12" time="2016/07/12 19:01:49">
     <tcp admin="admin" dirtyId="12" time="2016/07/12 19:01:49">
      <port admin="admin" dirtyId="12" time="2016/07/12 19:01:49">465</port>
     </tcp>
    </protocol>
   </entry>
  </service>
 </result>
</response>

Code: 

def paloalto_service_find(pa_ip,pa_key,protocol,port):
 # Find if there are service objects that match a certain port and protocol type
 # Input: Palo Alto gateway IP, Palo Alto Access Key, IP protocol type, and port number
 # Output: Returns service object name if found or "" if there are no matches
 
 ctx = ssl.create_default_context()
 ctx.check_hostname = False
 ctx.verify_mode = ssl.CERT_NONE

 cmd = "/api/?type=config&action=get&"
 parameters = {'xpath':"/config/devices/entry[@name=\'localhost.localdomain\']/vsys/entry[@name=\'vsys1\']/service"}
 url = "https://"+pa_ip+cmd+"Key="+pa_key+"&"+urllib.urlencode(parameters)

 response = urllib2.urlopen(url, context=ctx)
 contents= ET.fromstring(response.read())

 result = ""

 for i in contents[0][0]:
  if i[0][0].tag == protocol:
   for j in i[0][0]:
    if j.tag == 'port' and j.text == port:
     result = i.attrib['name']

 return result

the response is first formatted in XML so that it can be parsed, and saved in variable 'contents'. Afterwards, we iterate through all the entries two levels down using the variable 'i', and subsequently two more levels down from i with the variable j. Throughout this process, we are comparing the protocol and port values we are reading with the port and protocol variables provided to the function. The first of these entries would be:

<entry name="tcp_81">
 <protocol>
  <tcp>
   <port>81</port>
  </tcp>
 </protocol>
</entry>

In this case, the tag is ‘entry’, attrib is a dictionary {'name':'tcp_81'}, so to access the value of 'name' we can do i. If we wanted to go one level below, we would have to reference i[0], and then for example i[0].tag would return ‘protocol’. A level below that would be i[0][0], and that would give us access to <tcp>, and so on.


Download


You can download the code from my github page.
To use it, you only need to import the paloalto.py in your code and then call the functions. The functions will continue to be updated as I work to add more functionality.