LISA’11: Security Workshop

Print Friendly, PDF & Email

Lead by Matt Disney “team lead for cybersecurity administration at the National Center for Computational Sciences”. Again as this is a workshop we introduce ourselves. My intro included detailing the organisational changes and restructuring which has had an effect on access rights, name space changes and network security implications. Others are interested in various organisational issues, OS deployment and patching, HIDS, network security. Matt wanted to see what people think about how we can make the management of security easier. Amongst the initial probe email sent to the attendees before the workshop some of the areas of work and interests mentioned were monitoring tools, patching techniques, processing netflow data. My comment of how we protect sysadmins from legal exposure, and risk. Employing security professionals and whether anyone is hiring came up. Many are obviously interested in virtualisation/cloud security. Risk management and the trade offs, commonly used security base line tests, compliance with standards such as HIPPA and ISO2000 were also mentioned.

First topic for discussion was a direction towards a comment response post by aMindWellArmed on reddit for a article about Full Disk Encryption:

Some said this is not practical , other see it just as pedantic fanboy randomness! One point made based on a discussion of the validity of the content, is to not assume that the suggestion of making a particular precaution shouldn’t be discounted just because it’s unlikely to happen. As security people we need to be open to randomness of human nature. But, “Users can’t be trusted”! Some feel they can trust their users, it depends on the user base. But Matt asked if you “can you assume that your users are not malicious”? I was expressed that if you make users life more difficult and they find a way to circumvent a policy, they will. Security guys need to not be the enemy, but be there to protect users and be on their side. Be careful about political capital, and seek the users trust. Increased risk is not that bad, as long as you can monitor and react effectively and explain to users when you’re making their life harder. However, some users will always think they know better and refuse to comply with policy. How do you override the security of VIP staff/users? There’s no stick to enforce policy at the top!

The discussion was then directed to how we define the difference between “threats” and “risks”? Threat is the external entity, threat is the factor of risk, risk is the likelihood that it will happen. When referring back to the reddit comment, the original poster isn’t applying risk analysis to the problem. We need (or are) applying that when reading it in our judgement of it’s usefulness. If we decide not to do some/all of the suggestions then it’s because our localised risk assessment states the lack of necessity. So, how do we deal with risk? “Mitigate, accept, transfer, buy insurance.”

As with other areas of IT and systems administration, implementing change in security practise or policy is difficult. You need to explain why the change is necessary, look for alternatives, make sure you get management buy in. Some have sought and been successful in getting policy accepted to say security is independent from IT and to make systems and processes not dependent or organisationally situated outside of IT departments or teams. At Los Alamos they need to have a whole team to manage security that’s separate from IT.

Final word before the morning break is that our job is to facilitate people to work securely and safely not to restrict and deny. Reserve the draconian measures as a last resort.

After break the topic moved over to patching. Most agreed the most difficult area was with application managers and developers due to the difficulty to inspect and audit what is done. It generally felt that getting involved as early as possible at the design stage will help to reduce the risk. Patching frequency in the room varies, some do yearly maintenance or between business breaks (e.g. in holidays). There are obviously some concerns over updating patches immediately from when they’re released. One person said there seems to be a sweet spot for deploying patches after release between 10-30 days. But generally a lot will have test environments for patch releases. As a reflection, back at the UoB, it would be good to have upstream package repo mirrors for prod servers which lag 1 week behind ones used for dev/test. Down side is that we’d need to manually push critical updates to prod when they come out. Chris St Piere, said they have redundant services with split repos between views of a cluster, e.g. one half of a cluster uses a pkg repo which lags behind another repo that the other half uses.

There are 2 main categories for patch deployment method:

  • release engineering (dev->qa->prod, unit testing, CI, release OS).
  • guinea pig route, staged automatic deployments (one OU gets patches immediately, the prod stuff later)

So, Matt asks for a round room description of what we have and what works well, what tools are we using, something we do that we like like. For me bcfg2 is the biggest win, it shows managed packages, unmanaged packages, coupled with fabric we can push out software updates for specific groups selectively when urgent updates are needed. e.g. the recent security update to bind the patching process was simply:

$ fab hosts:bind-server push:bind

The second thing I mentioned was ossec hids, runs on hosts and monitors logs, returns data back to central. Bcfg2 1.2 has a plugin fileprobe to help support managing the OSSEC keys transfer (OMG! love you bcfg2). This is something that we don’t do currently due to package deployment problems but have just discovered a debian source package repo on Alioth so will look at automating this at some point soon.

Some other favourites (or not) raised around the room:

  • os x server sucks it’s official! ;-)
  • users do not store local data on linux desktops, but have full disk encryption.
  • SElinux is actually quite good in RHEL 6 and getting easier to manage.

One good thing about SElinux is it enforces File System Hierarchy standards. Running in permissive mode can be useful for developers to see where apps can be taking risks. Problem, is that there’s a lack of tools and documentation to present how to deal with making localised use case deviations from the standard locations for storing files. This is my main problem with debian. The support for it is there but I doubt package developers are really thinking about this is as being enabled by default.

After lunch there were discussions about mobile devices, the cloud and externalising services to it. At one persons site moving to gmail is not considered a secure option. e.g. medical/patient data being stored on gmail is not allowed in a couple of sites. It was stated that it’s pretty much impossible to monitor what data is stored in cloud email services or indeed personal email accounts for staff.

Topic moved on to social engineering and phising emails. Some users just don’t get warnings to not reply and finding methods to get this knowledge and practise engrained into peoples minds is difficult. One person suggested that we could have a “fire drill” equivalent for social engineering threat tests. But it was argued that the data collected by such a test isn’t really valuable and just confirms that there’s a risk you already knew about. Similar issues with human behaviour and social interaction affect buildings (datacentre or otherwise) access security and can be a major issue in some setups. One could get fired or put in jail for letting unauthorised people through doors/lifts which are restricted. Motivating some people to stick to the rules doesn’t work unless there’s some kind of cost to the individual.

The conversation led then to password management, whereby most in the room groaned but Matt pushed us on. ;-) After a query to the group it seems there’s still no central way to manage passwords with multiple admins. There are some commercial solutions or alternative methods such as powerbroker.

An equally contentious topic of file integrity monitoring was raised by Matt as he has some personal desires to finally put this problem to bed. Is it actually any good or just a myth? Tripwire data can be useful as another data point for predictive/reactive support, but essentially it’s broken. The tools that monitor the machines are themselves prone to being compromised which renders the data as a untrustworthy source.

In all the workshop gave me some ideas and renewed my desire to revisit OSSEC and SElinux. It was good to hear how others are dealing with these practical problems. I had hoped to cover some of the general legal issues and protection of administrative roles in positions with elevated permission. The differences between UK and US laws however change the scope. Probably a topic to discuss back at home where it’s relevant.