Files of Michael Johnson

Lightning metric load to increasing performance!

2008-04-27T17:45:00.006-04:00

It's easy to monitor everything in Introscope, the key is finding the right balance! With most tools as you increase the metrics collected you incur some expense usually in performance. Unfortunately it becomes complicated to determine where metrics are coming from to determine how useful they are! Introscope now has a metric count type view extension to assist!

The new metric count type view breaks down your metrics into different catagories dynamically based on what its collecting. If you have a lot of metrics enabled it may tame a minute or two to render, but it's great to see where your metrics are coming from.

As you grow with Introscope you'll realize that PMI metrics become pretty useless and look to start reducing these pretty quickly! If you want this type view ask your CA/Wily Rep to provide it to you! Its a must have for agent tuning!

ChelseaPiers Hockey Schedule into your outlook or gmail cals.

2008-04-15T23:49:00.000-04:00

I get annoyed at sites that provide schedules but no way to easily import them into my outlook or google calendar. I play hockey at chelsea piers in NYC and decided to finally write up something to do this for me! Feel free to use it as well.

http://blog.michaelbjohnson.us/chelseapiers/cpschedule.php

Monitoring and Tuning Verbose GC with Introscope EPAgent

2008-02-15T13:11:00.006-05:00

Tuning garbage collection (GC) can be a difficult task without the right data, analysis tools and understanding of the runtime. With the help of the Introscope EPAgent and verbose gc logging I'll show you how to collect the data you need to be able to monitor and tune your runtimes GC!

A number of tools exist to analyze verbose gc log files, my personal favorite is IBM PMAT. PMAT gives some pretty graphs showing gc pattern analysis, allocation failures and more. Feel free to read more about PMAT here. The problem with PMAT is the time it takes to collect all the log files, and process them. Even on my loaded laptop with multiple cores and GigE connections it takes a few hours to get a report.

Our environment is pretty big. Running multiple cells, with hundreds of app servers, across multiple OS's and environments with high volumes. Our GC logs are a few hundred MB on average with the level of GC we output. We needed a way to monitor the GC logs in real time as well as having the ability to view historical data for baselines.

Introscope EPAgent came to the rescue! The EPAgent gives us the ability to monitoring log files and report the data back to the Enterprise Manager in real time. As GC data was written to the log files we would be able to collect it using Introscope. This gives us the ability to pull up historical information to use as baseline data to improve upon. As applications evolve we can easily review the data as part of our performance tuning process and report on it! As with any Introscope metrics we can also alert on them.

Here is a screenshot of the metrics this app server was collecting:

The setup is simple and I've put together a zip file of the perl script and EPAgent config file needed get done!

Download Here

Portal 5.1 NlsCannotInterpretStateException

2008-01-18T00:11:00.002-05:00

Recently I started to poke around our log files when we started having performance issues shortly after our 5.1.0.4 upgrade of portal. I started to see a number of unfamiliar errors:

2008.01.25 15:02:33.111 W com.ibm.wps.engine.Servlet doGet()
class com.ibm.wps.state.nls.inputmediators.exceptions.NlsCannotInterpretStateException: Unspecified message ()

I traced it back to before the upgrade which was good that it wasn't introduced by 5.1.0.4 but what is it?

Searching through the IBM site, it didn't give me very much on it at all. Over the years I've learned not to ignore any errors from WAS or Portal until you can safely identify what they are! On some of our lower volume portal instances we were getting a few hundred of these a day.

Using Introscope I was able to see that one user navigating a synthetic transaction (sitescope) was actually executing portlets multiple times.. some of which were not even on the same page, but on that users homepage. Using the transaction tracer for Introscope and error detector we were able to see this pretty clearly.

With some tracing we were able to see we were missing some images (arrow.gif) and a few others. Each time portal attempted to pull if they didn't exist it would throw back a home page, rather than something light such as a 404. It seems the default behavior for portal if it cannot decode a URL is to throw the home page back. ICK!

That means a few bad images could really degrade performance for your portal. This seems like a pretty tough lesson to learn about missing images especially for high volume sites! Hopefully in future releases portal will understand light weight error pages (maybe even custom ones) and throw back something like a 404!

I won't hold my breath though!

Introscope EJB's not being displayed

2007-12-14T23:49:00.000-05:00

When we added Introscope on a number of Websphere Application servers we noticed some of the EJB's were not coming up. As we poked around we saw the EJB's in the PMI data from the app server, but not flagged inside introscope itself.

Upon some research we see that introscope idenfities EJB's using the directives below:

IdentifyInheritedAs: javax.ejb.SessionBean SessionBeanTracing
IdentifyInheritedAs: javax.ejb.EntityBean EntityBeanTracing

The directives tell introscope that any object that directly inherits from javax.ejb.(SESSION|ENTITY)Bean is a (SESSION|ENTITY) EJB. The limitation is that it must *directly* inherit.

Example, if class B inherits from javax.ejb.SessionBean, then Introscope will know that class B is a Session bean. However, if class C inherits from class B, Introscope will not trace class C.

This was a big hit for us as we have a number of abstract superclasses for our EJB's. A majority of our EJB's were not being tagged. Fortunately this was an easy fix, once they were identified. We created a pbd which contains a number of directives:

IdentifyInheritedAs {abstract_superclass} SessionBeanTracing

One cool thing when we move to WebSphere 6.1 that we will be able to take advantage of is "if you are using 1.5 JVM, Introscope now supports ProbeBuilding for Multi-Level Class Hierarchies. In pre-5.0 JVMs (we are using 1.4.2 JVM), Introscope does NOT instrument classes in the deeper levels of an class hierarchy—only the classes that explicitly extend a probed class. On JVM 5.0, you can configure Introscope to instrument multiple levels of subclasses of a probed class."

Introscope java.net.MalformedURLException

2007-11-04T23:04:00.004-05:00

I'm still on my Introscope kick, trying to get us up to speed as quickly as I can. Being the lone soldier in this process its a lot of work, between doing the grunt work as well as looking forward to get the visibility we need. As we upgraded to 7.2.1 the workstation web start seems to use a newer version of java which resulted in the following error:

java.net.MalformedURLException: unknown protocol: socket

This hit both IE and FireFox users. Upgrading to the latest version of web start resolved the issue!

Wily EPAgent Stopping Unexpectedly

2007-10-26T16:05:00.000-04:00

So I started playing with the Wily EPA agent. It’s basically a way to run scripts on a remote server and report those metrics back into the Introscope EM product. It’s incredibly useful when you want to correlate date. Unfortunately installing it wasn’t as smooth as I had hoped!

For starters – I’m on windows. Easy guys – I’m trying to get off it, really I am. I have to work with what I got for now! One issue under windows is by default it doesn’t have a way to register as a service (it’s a java program).

Luckily someone on the community site already provided a service wrapper, thanks! I installed the service wrapper, started playing with a few of my WMI scripts. Never in my life have I written vbs other than when I took my VB 6 programming class in college, I just found out why I never used it beyond that.

WMI is so complex. I needed to go to MS site and do a lot of research on how to get accurate numbers from the WMI data. I provided some of my scripts but please don’t blame me if they are ideal.

I’m a geek - I wanted things like context switches per second, Processor Queue Length, Network Information, and more... Stuff I could have easily gotten with an awk command on took days of figuring out via the WMI interfaces… anyway checkout the scripts here:

System Information - http://www.hackedby.us/downloads/system.vbs
Physical Disk Information - http://www.hackedby.us/downloads/physicaldisk.vbs

So I attempted to start the service and it came up fine, but shortly after I would get a message user logged off then it shutdown. This was annoying! Come to find out windows sends a SIGNAL when a user logs off and java reads it. You can read more on this at the sun site.

The simple solution to this problem was to tell java to ignore the SIGNAL with the –Xrs option. I just added it to the wrapper conf file and was good to go.

# Java Additional Parameters
wrapper.java.additional.1=-Xrs

Hope this helps.

MS Office and Dual Monitors

2007-09-16T12:26:00.000-04:00

I find I'm much more productive with dual monitors hence I have them both at home and in the office. Unfortunately I don't think anyone at Microsoft feels the same way, at least in the Office Team. In my work environment which is Windows XP with Office 2k3, numerous products out of the box don't allow you to utilize dual displays! Here's a few tricks I do to get around the limited product issues.

PowerPoint 2003

---------------

Let me start out by saying I am not an office power user at all, nor do I plan on becoming one. Recently I've been asked to do a number of presentations and slides on various stuff which never materializes. I wanted to have 2 ppt documents open one in each window so I could steal some content easily. Powerpoint 2003 would not let me do this. I could open multiple instances but they all showed up in the same window!

To work around this:

- I created a new Windows XP user locally (pptuser).

- Whipped up a batch file to utilize the runas functionality of windows and launch powerpoint.

Contents: 'runas /user:pptuser "C:\....office11\POWERPNT.EXE"'

- I moved this batch file to an easy place in the start menu to launch it as needed.

Note: With Windows XP I had some issues with directory permissions so I needed to give permission to the pptuser for directories where I stored my documents.

Excel 2003

----------

So once I got PowerPoint working I noticed I had this same issue with Excel... SIGH... I needed to compare some workbooks, again they came up as separate instances but they were also in the same window.

- Open Windows Explorer and go to Tools -> Folder Options.

- Click on the “File Types” tab which will display the registered file types.

- Find any Microsoft Excel Worksheet (example: xls) under Extensions and File Types.

- Highlight, and Click on “Advanced”.

- Highlight the Action “Open” and Click on “Edit”.

- De select USE DDE.

- In the ‘Application used to perform action:’ section append a space “%1”

- Note: The quotes are required so it would look like:

...office11\EXCEL.EXE” /e “%1”

- Click “OK” to save the changes.

Now you should be good to go!

Dial-up BlackBerry for BroadBand Internet Access

2007-08-24T15:08:00.002-04:00

I've been using Verizon for some time now, but its extreamly expensive. So I was looking for another way to get broadband access and found that my Blackberry 8700 is equipped!

I've been using the blackberry successfully with Cingular (now AT&T) as a modem paired with my laptop computer. I had to go through a number of resources to get this to work, but now that it does I haven't had any issues and am saving a bundle compared to my Verizon plan.

Specs: As I noted I have a Cingular 8700c, in "out of the box configuration". Below are the steps to enable it to work as a broadband modem.

1. Install the Blackberry Handheld Manager and connect the blackberry via the USB to your computer. This is required for using the device as an external modem. The Blackberry Handheld Manager app needs to be running whenever you want to connect to the internet.

2. Install the modem driver. I found the drivers under c:\Program Files\Common Files\Research In Motion\Modem Drivers\. One laptop I tested it didn’t work so I just used the standard drivers which also seemed to work. If you have issues ensure you are running a recent version of the Blackberry Desktop.

3. Ensure the new modem you created now exists on your system. On my system its "Standard Modem".

4. Query the modem.

Phone and Modem Options =>

Standard Modem (click properties) =>

Click on Diagnostics => Click Query and ensure the modem responds.

5. Add advanced modem settings for Cingular.

Phone and Modem Options =>

Standard Modem (click properties) =>

Click on Advanced. In the extra settings box add:

+cgdcont=1,"IP","WAP.CINGULAR"

6. Add an internet connection for Cingular/AT&T.

Network Connections ->

Create a new connection ->

Next ->

Connect To The Internet, Next ->

Set up my connection manually, Next ->

Connect using a dial-up modem, Next ->

Standard modem (or whatever you named it above) and give this connection a name 'Blackberry Modem', Next ->

Phone number should be: *99# , Next ->

UserName: ISPDA@CINGULARGPRS.COM

password: -> (blank)

Clear all checkboxes, Make this default... ->

Finish

7. Disable Compression.
Network Connections ->

Blackberry Model (Right click properties) ->

Click on configure ->

Uncheck everything under Hardware features (flow control, error control, compression). ->

Click OK.

At this point your computer is ready to use the Blackberry devices as a modem. Now we need to configure the Blackberry.

1. Go to Settings -> Options -> Advanced Options -> TCP and add the following information.
APN:
USERNAME:WAP@CINGULARGPRS.COM
PASSWORD: CINGULAR1

2. Sit back and enjoy the internet after connecting!

Using proxies to break into the office

2007-06-22T14:44:00.000-04:00

It's amazing to me that software still comes out of the box insecure. My rant today focuses around the HTTP CONNECT method, and how to take advantage of a number of mis-configured proxy servers to get into servers behind the network from the outside.

HTTP CONNECT ( was designed to create a TCP connection that bypasses the normal application layer for proxy services. Its often used today in tunneling HTTPS (or if you read my blog SSH) connections through a standard HTTP proxy.

You can review more detailed info here: http://www.ietf.org/rfc/rfc2616.txt

So lets get started. We will be using auth-tunnel.pl again for this work, we have a fairly strait forward setup of Iplanet proxy server, setup in reverse mode per the Sun Documentation.
What we will do is connect to a proxy server using HTTP CONNECT from the outside, then establish a connection to another server on the network on port 22. We can then tunnel the traffic to ssh in from our local machine. In my setup my proxy server required Authorization, making a bit more secure :).

Tell auth tunnel the proxy server and port, the local port to tunnel, the remote ssh server then all the authentication information!

[mjohnson@shell auth-tunnel]$ ./auth-tunnel.pl -p proxytest.xyphoid.com:80 -l 7575 -r mysshserver:22 -a admin:mypassword -u mytestagent &
[1] 28208
[mjohnson@shell auth-tunnel]$ MD5 Avail. Enabling support for Digest Authentication
[mjohnson@shell auth-tunnel]$ ssh 127.0.0.1 -p 7575
Proxy authentication required...Closed proxy.
Reconnecting...using BASIC authentication.
Proxy-Authorization: Basic realm="Private Area" YWRtaW46bXlwYXNzd29yZA==

Auth tunnel establishes a connection using HTTP CONNECT to mysshserver which is only know about from the proxytest server. Its not externally accessible. Once the connection is established we can just use a standard ssh client to connect to that server.

[mjohnson@shell auth-tunnel]$ ssh 127.0.0.1 -p 7575
Proxy authentication required...Closed proxy.
Reconnecting...using BASIC authentication.
login as: mjohnson
mjohnson@mysshserver's password:
Last login: Fri Jun 22 00:49:52 2007 from 127.0.0.1
[mjohnson@mysshserver ~]$

Below is the info on the version of software I am running and what it would look like doing manually.

[mjohnson@shell auth-tunnel]$ telnet proxytest.xyphoid.com 80
Connected to proxytest.xyphoid.com.
Escape character is '^]'.
CONNECT mysshserver:22 HTTP/1.0
User-Agent: myuseragent 1.0
Proxy-Authorization: Basic realm="Private Area"
HTTP/1.1 407 Proxy authorization required
Proxy-agent: iPlanet-Web-Proxy-Server/3.6-SP7
Date: Fri, 22 Jun 2007 00:52:46
Proxy-authenticate: basic realm="Private Area"
Connection: close
Content-type: text/html
Content-length: 271

This is just one way we can get to a server using HTTP CONNECT from the outside. I hope going forward vendors start making this more secure out of the box or people becoming aware.

Windows 2003, Cygwin - SSHD Issues

2007-03-22T15:13:00.000-04:00

I was running into an issue with windows recently. I know Windows has a lot of them but specifically running SSHD (Cygwin based) on Windows 2003 and public keys. The documentation notes to run SSHD as a system account, due to privilege reasons. I’m not to fond of this, I like running things as user accounts with less privileges not to mention under the system account with Windows 2003 seems to have issues with privilege separation.

Running under the new user account I setup sshdsvc (in administrator group), when I fire SSHD up and try to connect ssh closes the connection. Same result when I try to login with a public key. I modified the log level to DEBUG and have it write to the event viewer (syslog) to get more detail.

When I attempt to login it gives me a ‘error setreuid permission denied for sshdsvc’

Trace file here:

I added the following windows privileges (Control Panel, Local Security Policy) to the sshdsvc account.

Act as part of operating system.
Create a token object.
Replace a process level token.
Log on as a service.

I then needed to change the ownership of the directory (specifically the key files), remove the log files, and start the service.

Once restarted I was able to login without issue!

Harvesting Cell Phone Numbers with Myspace.com

2007-02-02T11:22:00.000-05:00

If you use myspace.com you know how fast new features pop up on profiles. Recently all my friends (all 6) on myspace started to add this “cool” new feature to their profiles. The feature which is provided by txt2day.com, allows users to send txt messages to the cell phone of the persons page you are on. It’s pretty cool and saves you some cash if you do not have an unlimited text plan. Unfortunately it’s trivial to obtain the cell phone number of that person!

So here is how txt2day works. You go to txt2day.com and get some html code to add to your profile. Here is a snip:

<snip>

<form action="http://www.txt2day.com/send.php" method="POST" target="_blank">
<input type="hidden" name="refer" value="myspace">
<input type="hidden" name="to" value="NTU1NTU1NTU1NQ==">
<input type="hidden" name="provider" value="sprint">
<textarea style="color: #000000; background: #ffffff;" name=message rows=3 cols=20>txt msg me to my mobile</textarea>
<br><a href=http://www.txt2day.com>Txt2Day.com</a> <input style="color: #000000; background: #ffffff;" type=submit value="Text my mobile"></form>

</snip>

As you can see this is just a simple form with some hidden fields. Since we don’t really care about how txt2day works we’ll skip that discussion, what we really want is our friends’ cell phone number!

This is the simple part, the hidden form field “to” is a Base64 encoded representation of the cell phone number you want.

Example: < input type="hidden" name="to" value="NTU1NTU1NTU1NQ==" >
Base64 encoded phone number: NTU1NTU1NTU1NQ==

Now all we need is a base64 decoder. If you need a web based one go here:

http://webnet77.com/cgi-bin/helpers/base-64.pl

Enter the Base64 encoded phone number into BASE64 to decode and boom, you now have your friends phone number! In our example above its 5555555555.

If you have a private profile and restrict people that are your friends this probably isn’t a big deal since they all probably have your number. If you have people on your myspace page however that you do not want, you might want to remove this “feature”.

For those of you that are looking to collect a large list of new phone numbers for harvesting, it’s pretty easy to write a spider script across all your friends, their friends, and recurs across myspace. I’m not sure how myspace will feel about this but it’s definitely easy and doable.

Easy Spell Checking

2007-01-17T18:03:00.000-05:00

This is a short but sweet post. From time to time I have a need for a quick and dirty spell checker for forms and whatnot. We'll I finally found one that meets all my needs, SpellingCow. We all love cows, holy cow, cow-a-bunga dude (ok maybe not that one), but tucows.com (back in the day) this is just one more for your list. It uses COWS AJAX, like I know what it does?! Anyway it works great, easy to install, especially if you use the favelet.

Install:
http://www.spellingcow.com/favelet.html

See it in action:
http://buttercup.spellingcow.com/

I’m finishing up a post on snatching phone numbers from myspace accounts, so let me get back to that so you can read up on that!

Apache/IBM Http Server Lockdown

2006-12-27T23:00:00.000-05:00

Let’s take a look at locking down apache and/or the IBM http server. Installing the software and setting up a basic web server instance is beyond the scope of this, if you need that info checkout the docs first. They do a great job of explaining next, next, finish.

Install the Software
--------------------

RULE #1: Don't run software as root

So I have on Solaris 8 - Tomcat, MySQL, and the IBM HTTP server, yup same as my Iplanet install. I created a user 'www' for the web server to run as since our first part of the lockdown is not to run any software as root. You should have the user added before the software install it can be changed at any point in the httpd.conf file, be sure to secure this account such as setting the account to No password and a false shell. I suggest giving the software its own account not one of the default system ones so you can permission it accordingly and Jail/Chroot it.

Create the Instance
-------------------
RULE #2: Disable unused features

Such as mod_cgi, mod_userdir and more. Since I am running a multi-tier environment, keeping the app-server (Tomcat) content off the web server for security & scalability reasons I disable most of the modules, modify it to your own specific needs.

General Instance Lockdown
-------------------------
Since web server instances are very specific in nature to the applications they host I'm going to breakup the rest into things that could be done to lockdown all instances vs. ones that are more specific in nature.

RULE #3: Don't allow the software to modify itself.

Suppose via a php or some server side script an attacker was able to run commands on your server. We have seen this happen before, a great example is the cmd.exe exploits that came out for IIS. Ideally we don't want the attacker to get in at all but if he does we want to limit what he can do. One step in doing this is to permission the software so it can not modify itself. Apache installs new instances in my environment as Owner:www Group:www then the permissions on the files vary from 775 to 644 and so on. These default permissions would allow a vulnerable script to modify itself.

The first things I do are:

chown -R root:www /opt/IBM/live/https-instancename-443 (this is the path to my live instance)
chown www:www /opt/IBM/live/https-instancename-443/logs (so the web server can log)

Now that the software itself is somewhat locked down we can move on to some of the specific configs. If you are on windows, GWG (go with god). Hopefully you will find the correct (right click , properties.. settings to do this)

Starting with the httpd.conf to see exactly what these do refer the manual above.

Rule #4: Hide your identity
- Ensure you set ServerTokens to Prod. I must admit, I would really like the ability to set this to nothing, but IBM HTTP server does not allow this yet. IBM, at least allow me to something that does not disclose the brand of the web server. Numerous tools start by attempting to fingerprint your web server first, don't make it any easier for them hide this info! In addition set ServerSignature Off as well so you don’t disclose any more information that needed. If you really want the admin email published, use custom error pages.

Rule #5: Use strong encryption
- If you can use SSL. Not sending information in clear text makes it much more difficult to eavesdrop on.
- When using SSL disable version 2, it’s not strong.
- Use only 128 bit ciphers if you can! Only a few sites should allow less than 128bit encryption, and those should be limited too. Disable anything less than 128bit by adding the following in your IfModule

SSLCipherSpec 27
SSLCipherSpec 21
SSLCipherSpec 23
SSLCipherSpec 3A
SSLCipherSpec 34
SSLCipherSpec 35

Rule #6: Block unneeded HTTP Methods
You have your choice of options to allow browsers to use. I haven't found many others to allow than GET,HEAD, and POST. Production sites shouldn't allow webdav components or other types of features that development sites should so block them in each virtual host.

<limit>
order allow,deny
allow from all
</limit>

A critical one to block is TRACE. This provides more information that anyone honest needs to know about your systems. Add it to the main area of your httpd.conf , note you must have mod rewrite enabled.

# Disable HTTP Trace and Track and OPTIONS
RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^(TRACETRACKOPTIONS)
RewriteRule .* - [F]

The above will block TRACE, you should monitor your log files to watch the TRACE attempts.

This could arguably be a instance specific option but I add it to all of my web server configs. Microsoft doesn't play nicely with keepalive (certain versions), and rather than fixing their poor implementation they had most vendors come up with a workaround not to send a close_notify for MSIE. Some other things like realplayer had this issue as well.

BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0

Adding this to the httpd.conf will save you issues going forward in more ways than one!

Checkout http://docs.sun.com/source/817-6248-10/crobjsaf.html for all the details on this. I know it’s the sun site but you will get an idea of the problem.

Having the Web Server not send the close_notify packet may make MSIE vulnerable to a truncation attack, but that's Microsoft's Issues now isn't it.

Specific Instance Lockdown
-------------------------
So this section will contain things that you need to ensure meet your application needed. They might be tighter than you need or in the opposite direction! Modify as needed.

Rule #7: Have logs and process them.

Be sure you get all the info you need our of your web server logs. I obtain not only the request but the content length, referrer, user-agent, the processing time and cookies such as the JSESSIONID cookie for Tomcat. Keeping this info in the logs help identify problems, its important to process these logs as well with tools such as WebTrends if you have some cash to buy or Aw stats which I use here.

Here is my custom log format:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D %{JSESSIONID}C" secretagentsdefault

I also rotate my logs using cronolog, it rotates daily and does not require a restart of the web server. Previous versions used to leave cmd windows open in the background but that was fixed in the latest version. Not more fd leaks!

ErrorLog "/mycronologpath/cronolog ..path..to..logs../logs/error.%Y%m%d.log"
CustomLog "/mycronologpath/cronolog ..path..to..logs../logs/access.%Y%m%d.log" secretagentsdefault

Be aware of WebSphere's lack of security in regards to password storage

2006-10-11T17:27:00.000-04:00

As long as I can remember Websphere has used encoded not encrypted passwords when storing them. The Websphere documents clearly note passwords are not encrypted. Today with Websphere version 6.1 out passwords are still obfuscated and easily reversible encoding. In effort to raise awareness; I want to show just how easy it is to decode (and encode) these passwords in files such as soap.client.props or the misc xml files WebSphere stores its configurations in.

You can search security focus or bugtrack archives and find tickets open on versions pre WebSphere 4.x about the insecure password storage WebSphere uses.

http://www.securityfocus.com/archive/103/311216/30/180/threaded

IBM's stance seems to be if you want something more secure write a module yourself to store those passwords! Thats great thanks IBM!

They do provide some info on this here:

http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21210244

Not only do we need to worry about developers having passwords to the datasources, admin interfaces and more.. but now we need to keep them off the file system completely or ensure they do not even have read access to most of it.

Wait! That will break the solution they have for running portal as a different user and more! (Thats another blog topic in itself) Lets get back on track.

So lets open up soap.client.props. Look for the line com.ibm.ssl.keyStorePassword= . That is the XOR of the password (which you provide) and a 'secret'. If we go back to 7th grade math we would have a formula. If C = (A ^ B), then I can find A if I have C and B, so A = (B ^ C).

If you don't know the key don't worry. IBM provides a great tool to generate and recover these passwords using the default key WebSphere itself uses.

Here is a quick example on windows (for you Rad developers out there).

WAS_INSTALL_ROOT\lib>..\java\bin\java.exe -cp securityimpl.jar; iwsorb.jar ;ras.jar ;wsexception.jar;bootstrap.jar;emf.jar;ffdc.jar com.ibm.ws.security.util.PasswordEncoder unsecure

This will encode the password 'unsecure':

decoded password == "unsecure", encoded password == "{xor}KjEsOjwqLTo="

So now you know how to encode something. How do you decode a password though? Well you can write your own XOR decoder but IBM provides one for you! Thanks IBM! So lets decode that password we just encoded.

WAS_INSTALL_ROOT\lib>..\java\bin\java.exe -cp securityimpl.jar; iwsorb.jar; ras.jar; wsexception.jar;bootstrap.jar;emf.jar;ffdc.jar com.ibm.ws.security.util.PasswordDecoder {xor}KjEsOjwqLTo=

You should get an output:

encoded password == "{xor}KjEsOjwqLTo=", decoded password == "unsecure"

So now as a developer you know all the admin passwords, or as a admin who forgot what it was, you now know!

Just make note on unix to replace the ';' characters with ':'. I normally work off unix but for this example I was working on RAD under windows.

Enjoy and lets hope you won't be able to do this in future releases!

Kickstarting HP360 and Redhat 4

2006-07-20T15:46:00.000-04:00

Recently we picked up a number of HP DL360 G4p servers. We wanted to kick start all the servers with the standard Redhat kick start. This would be fairly trivial except for the fact that the HP NC340T cards as well as the onboard HP cards are Intel not Broadcom and are not supported by Redhat out of the box, so the kick starts failed with pumpsetup interface - no such device for kick start.

We did do some research with HP who noted: HP NC340T PCI-X 4-port 1000T Gigabit Server Adapter is a 4 port Broadcom card. Its really not, it’s Intel and the chipset has it stamped all over it. I'm not sure how they got this one wrong! In addition the onboard nics would be the same chipset. They were correct about the chipset on the server and the cards being the same though! Kudos to HP for getting that correct.

All the docs on Redhat site show the NC370 card is supported. Not sure why HP thought the NC340 was as well. It’s definitely not out of the box.

http://kbase.redhat.com/faq/FAQ_85_7102.shtm

Ok - So enough bashing on HP. What we needed to do was to build a custom init image to get these boxes kick starting off the network. I already had a Redhat Blade running Redhat 4 smp. This will be the box used to build the new image. If you don't have another similar Redhat 4 box, you will have to do one manual install.

We surfed HP's site for some supported Intel e1000 drivers for the quad card. At the time of this writing they were located here:

You will need to download these to the Redhat server.

1) Boot the existing Redhat 4 server off the non smp kernel. This is required since the init image does not support smp. At least the one we are working on out of the box.

2) Install the rpm source, cd to the directory and make it.
cd /usr/src/redhat/SOURCES/e1000-6.2.19b/src
make clean
make

3) Once you have the driver built the next step is to add it into the default initrd.img. Here is the process for doing this.

- Create a dir to mount into:
mkdir /mnt/loop
- Move the file so you can unzip it wherever you currently have it I will be working out of /home/mjohnson/tmp:
mv initrd.img initrd.gz
- Unzip the image:
gunzip initrd.gz
- Mount the image directory so you can work on it.
mount -o loop /home/mjohnson/tmp/initrd /mnt/loop
- cp the modules file to /tmp to work on it:
cp /mnt/loop/modules/modules.cgz /tmp
- uncompress modules cgz:
mv modules.cgz modules.gz ; gunzip modules.gz
- Modules is a cpio archive so we need to extract it:
cpio –idmv < modules

- Copy e1000.ko from the directory we compiled it earlier into this archive:
cp /usr/src/redhat/SOURCES/e1000-6.2.19b/src/e1000.ko /tmp/2.6.9-34.EL/i686

- Now we just need to put everything back together!
find 2.6.9-34.EL | cpio -o -H crc > modules.cpio
gzip modules.cpio
mv modules.cpio.gz modules.cgz
cp modules.cgz /mnt/loop/modules/
umount /mnt/loop/

Thats all there is to this. So to save you the time of doing this yourself, I uploaded my Red Hat Enterprise Linux ES release 4 (Nahant Update 3), just download it and replace your current initrd.img file with it and you should be good to go.

Sun One Proxy Server 4.x - Lockdown for reverse proxies

2006-04-04T11:35:00.000-04:00

Every now and then I come across some uses for reverse proxy servers to protect devices, app servers, IIS web servers, and more. This should cover some of the basic things to do to the default proxy instances to lock them down and clean them up. So lets dive in!

Environment
------------
I'll be doing this work on a Netra T-1 with Solaris 8 installed front ending an IIS 5 server.

Installing the Software
-----------------------
Rule #1 Don't run software as root

If you have ready some of my other posts you'll know the first rule I have is don't run as root. I created a user wsproxy for the proxy server to run as. You need to have the user added before the software install since the software will ask you, be sure to secure this account such as setting the account to no password, false shell.. the usual. So you can audit processes (such as C2 level accounting) I suggest not using one of the default system account such as nobody.

Sun's urls:
Product Info
Docs

These should guide you through the install rather than me.

Create the Instance
-------------------
RULE #2: Disable unused features

If you create a proxy instance from the gui you want to review what features it has added. You might want to disable the cache, SOCKSv5 (firewall traversal and more.. you know what you need anything else you don't!

General Instance Lockdown
-------------------------
RULE #3: Don't allow the software to modify itself.

Ensure critical config files for the proxy server can not be modified by the user you run the proxy server as! This is to prevent a bug in the proxy allowing access for an attacker to modify the proxy server itself. As a general rule the proxy server should probably only need read access to its own config files. It will need access to write logs and such so watch what you modify.

Rule #4: Hide your identity

Proxy Server 4 allows us to modify the server name being sent back to the user. Be sure you set the ServerString making it more difficult for attackers to fingerprint the software you run.

Rule #5: Use strong encryption where needed

If you're running SSL you should:
-Be running at least 128 bit
-not have any weak ciphers enabled (MD5 is a weak cipher now).
-SSL2 should be completely disabled.

Rule #6: Block unneeded HTTP Methods

Similar to web server 6 you can use client tags to block unwanted methods like trace.

<Client method="TRACE">
AuthTrans fn="set-variable" remove-headers="transfer-encoding" set-headers="content-length: -1" error="501"
</Client>

IIS also has other similar methods specific to IIS such as TRACK which provide the same information as TRACE. Its important to know what methods are available to services you are looking to protect.

Rule #7: Block bad requests

Stop wasting time processing requests that are bad before anything else! The following blocks all sorts of requests we know are just bad. Customize the string to your liking this is just an example, add it to the default object in your obj.conf before any processing. I spend a lot of time tuning this based on the IIS Logs. When I see new bad requests I expand this.

Client abort seems to be a little known but powerful feature! Play with it!

<Client uri="*(system32root.exeConsoleHelpSERVER.INISamplesWEB-INF_mem_bin_vti_biniishelpiisadmpwdiissamplesdefault.ida)*">
AuthTrans fn="set-variable" abort="true" error="412"
</Client>
Rule #8: Process and Review logs

Its important to review and monitor your logs. For capacity reasons as well as security! The only want to tighten you server even more is to see how its being attacked. If the default logging is not enough customize your log format to obtain the information you need.

Rule #9: Service Function

By default the reverse proxy servers allow many http methods to be executed upon resources. Due to this issues its critical to review the Service functions set in the configs and limit the HTTP methods that are available to only the ones required.

Service fn="proxy-retrieve" allow="POST,GET,HEAD"

Rule #10: Password files

In a production environment don't use a password.conf file for secure sites. Passwords to certificates should be kept safe and entered manually on startup.

Rule #11: Ldap authentication

In version 3.6 it was not support to use a group to query users for act protection.
Heres an example of an acl which works against a group called proxy users in the ldap.

# ACL to forces authentication and only allows GET/POST/ and HEAD.
ACL secure_GETPOST (GET, HEAD, POST) {
Default deny anyone;
Default authenticate in {
Database "default";
Method basic;
Prompt "Private Area";
};
Default allow proxy;
}

This was unsupported last I knew, but it works great. Hopefully they will honor my RFE and support it soon! I'm not going to hold my breath though since I just got ldap fail over supported!

Rule #12: Error directives

Its important to check the docs when manually modifying functions in general. I tend to see send-error incorrectly defined. The proxy server can behaved unexpectedly when paths defined do not exist or functions do not have the required arguments.

Rule #13: Mime matching

Ensure the proxy server matches the mime types as close as it can to the web server. I have seen some cases where applications behaving abnormally due to a mismatch between the servers.

That's all I got for now on this!

Discovering Portals - Jetspeed with WebSphere 5.1

2006-04-03T11:00:00.000-04:00

After I tackled getting Jetspeed up and running under tomcat which was mostly effortless, I wanted to get it running under WebSphere 5.1. This process was not an install and go as I hoped it would be! After tackling everything I must say it wasn't jetspeeds fault, it was IBM's, but I did get it running and some demo portlets deployed. Here's how!

Staring notes.
--------------

I did all this work on my Netra T1 (Solaris 8). The database I wanted to use was mysql version 5 also running on the netra. I have a default install of WebSphere 5.1.1.5.

Install the Software
--------------------
I downloaded Jetspeed2.0-MultiDb-install.jar from the Apache Portal site just like for the Tomcat install. As I noted I wanted to use mysql so I needed the MultiDb installer. I needed to install ant and Maven since this server was a fresh install. Once that was complete and installed I kicked off the jar installer.

java -jar Jetspeed2.0-MultiDB-install.jar

During the installed it did ask for the database information. I also opted in of all the demo apps this time, since with the tomcat install I didn't. For mysql I used the following info:

-username/password (root/mypassword for the install)
-The connection url:jdbc:mysql://localhost:3306t/jetspeed
-The driver: com.mysql.jdbc.Driver
-The path to the driver file I'm using: mysql-connector-java-3.0.17-ga-bin.jar.

So this created a db added a jetspeed user/password. I then modified some of the permissions to this user. Probably more than you would do if you are just playing with this.

Note: The installer as I noted with tomcat didn't like the version 5 connector.

We should now be able to bring up the install using tomcat which gets installed by default. I'm not going to though!

WebSphere Setup
--------------------
On the WebSphere side of things we need to prep the environment for the install of the portal.

Add the following jars to the WebSphere lib directory or create a shared library. Personally I used a shared library (note they are not part of the java specs yet), since these are not part of the WebSphere package and I might have different installs of jetspeed going forward which I would need to version these jars as they change.

-jetspeed-api-2.0.jar
-jetspeed-commons-2.0.jar
-portlet-api-1.0.jar
-portals-bridges-common-1.0.jar
-pluto-1.0.1.jar

Add a J2C Authentication Data Entries for the Jetspeed Datasource.

Security=>JAAS Configuration=> J2C Authenticaiton Data
Set the userid/alias and password to the jetspeed user you created. If you left the default from the script install its jetspeed/jetspeed.

After this create a MySQL JDBC Provider.

Resources=> JDBC Providers => New.
Name: MySQL
ClassPath: /www/global/lib/mysql-connector-java-3.0.17-ga-bin.jar
Implementation Classname: com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource

Once complete add a Data Source for Jetspeed.

Name: JetspeedDS
JNDI Name: jdbc/JetspeedDS
Datasource Helper Classname:com.ibm.websphere.rsadapter.ConnectJDBCDataStoreHelper
Container-managed Authentication Alias: JetspeedDS or the alias you used for J2C above.

Go to custom Properties and set the following:

databaseName: jetspeed?autoReconnect=true
username:jetspeed
password:jetspeed
serverName: localhost
port:3306
Apply, OK and Save!

The last bit of prep work we need to do is remove jdom.jar from the WebSphere lib directory. WebSphere uses a very old version of jdom.jar. If you want to replace it you may but jetspeed comes with its own packaged. Currently version 1.0 is available (http://www.jdom.org/). Without making this change portlets will not be deployed from the deploy folder and register inside the container. This was the big show stopper in my install. When you load the portal you get errors like /demo not found.

JetSpeed Deployment
----------------------
So we have everything ready now and can deploy jetspeed into an application server. I have a spare one created so I will be using that, its basically a default container with the memory bumped up to 1GB.

Applications=> Install New Application
Path: /jetspeedtmp/jetspeed.war (this should be the path to your war file on the server)
Context Root:/jetspeed (I just used this for testing)

The rest is basically a next,next,next finish routine. Just ensure you bind the datasource settings when prompted.
Click Ok and save.

JetSpeed Startup
-----------------
Check the WEB-INF/deploy directory to ensure all the demo apps reside in there. If not add them in.

Start the Application Server you deployed jetspeed into. Upon startup it should deploy all the demo applications into the ear. You should now be able to hit your portal and see some error messages as follows:
Failed to find Servlet context for Portlet Application: /demo

Stop the portal and go back to the Install New Application. For each "demo" (meaning portlet demo) war you want to install and have deployed we need to install it manually into the container as we did jetspeed itself.

You should be able to see all the portlet wars in the jetspeed_war.ear folder in installedApps. Start with demo.war and move on from there. You can start the portal after you install demo to ensure everything looks ok.

Post Issues:
I have found some issues with the default portlets such as j2-admin. Im looking at getting those running now.

Some references:
http://wiki.apache.org/portals/Jetspeed2/Fusion

Break out of the office using the proxy server!

2006-03-23T16:17:00.000-05:00

Most corporations today have proxy servers governing the access to the internet. Restricting and tracking your access to sites as well as other services the powers at be deem acceptable. I find that it’s actually pretty easy to bypass these restrictions thanks to the way most proxy servers are configured.

So lets have some fun and breaking out of the corporate network! First determine what your proxy server address is. If your internet is working currently, just go to IE’s internet options and take note of your proxy settings. Actually if you didn’t know how to do this already you might want to stop here.

For starters lets just try to access my Unix box at home without any proxy information.

eel3# ssh mjohnson@home

This just hangs since it can’t get out of the network. FYI: I added the ip address of my home system to the hosts file of the server im on.

So now lets take a look at a nice program from Philippe "BooK" Bruhat E (book at cpan.org) called connect-tunnel. Connect tunnel takes advantage of the HTTP connect command which most proxy administrators allow. It acts as a simple port forwarder. So lets give it a try to access my home system again!

Kickoff connect-tunnel in the backgroud.
eel3# ./connect-tunnel.pl --proxy webproxy.mycorp.com:8080 --tunnel 2222:home:22 &

Now lets attempt to ssh to port 2222. This should forward to port 22 on my home server.

eel3# ssh mjohnson@home -p 2222
mjohnson@home's password:
Last login: Thu Mar 23 11:38:37 2006 from mycorpproxy
[mjohnson@home mjohnson]$ hostname
home

Bingo we’re through the proxy with any port we need in this case to my home server. Its important to note if the site or server you are attempting to go to is usually blocked by the content filters then you won’t be able to connect to it. Using ssh to a server of your own we can get around that though!

If your running windows this is much easier with putty. Recent versions of putty come with an option to use proxy servers. This replaces the steps where connect tunnel is used. I want to be able to script all this stuff though, not just get outside the network to surf the web.

Lets focus on the surfing the web though since thats probably what most people are looking for. At this point its easy. We just setup a proxy server of our own such as squid on the home pc. The n open up a tunnel via ssh when we ssh to the home server for the proxy server as well. After that its just pointing whatever you want to use the proxy server port locally defined on your machine.

So we end up with a connect-tunnel going through the proxy server. Then a number of other tunnels for the services you want going over ssh!

For the few admins reading this. You can block the Connect method, but be weary that lots of applications use this and it might just break a number of things!

Happy Surfing!

Discovering Portals - Jetspeed with Tomcat

2006-01-25T00:48:00.000-05:00

Lately I have been evaluating numerous different portals on the Market. BEA, IBM, CHEF, Jetspeed and a few others. This is just an overview of some specific issues and annoyances I had with the install. Overall this was easy to get up and running!

I did all this work on my laptop WinTel. The lab is currently down since I moved and well I 'm lazy and haven't gotten a chance to get it back up. My Linux laptop is hard down; I think the memory died so if you’re reading this and have 512MB of ram for an Inspiron 8200 let me know!

Back to business, so the environment is windows XP, 2.8Mhz and 1GB of ram. The database I wanted to use was mysql so created a db where I have some mysql installs on the net and set a user/password. Basic locked down mysql instance is fine; the installer will create all the tables for you.

So I downloaded Jetspeed2.0-MultiDb-install.jar from the Apache Portal site. As I noted I wanted to use mysql so I needed the MultiDb installer. I needed to install ant and Maven since this laptop is only really used for email and a few other MS projects I have going on. I used the windows binary for those as well. Once that was complete and installed I kicked off the jar installer.

java -jar Jetspeed2.0-MultiDB-install.jar

During the installed (basically a next=>next=>Finish deal) it did ask for the database information. I also opted out of all the demo apps. I have a really nice "Hello World Portlet" I want to deploy. For mysql you need the following:
-username/password
-the connection url:jdbc:mysql://servername:port/databasename
-the driver: com.mysql.jdbc.Driver
-and the path to the driver file.

With the mysql java connector version 5.0 the install failed. I looked at it for about 3 minutes and decided I was going to use the version 3.0.1 which I use in a number of applications with mysql. Once I changed that the install went fine and completed in about 3 minutes.

The installer sets up tomcat for you in a very basic mode. You could load it up right away and tomcat will start. Doing this I had a few issues though.

One was JAVA_HOME was not set! Easy fix - I just set the environment variable for windows to fix this.

The second was when I started tomcat it could not deploy any portlets. This was due to the fact that in /conf/tomcat-users.xml the password for j2deployer was different than what was under /webapps/jetspeed/WEB-INF/conf/jetspeed.properties. I know I didn't set it in either place, so why the defaults were different out of the box is unknown to me. It’s not like its any more or less secure, it’s a default password! I set the passwords the same and closed that issue. I restarted tomcat.

The third issue is now when I start my portal I get all these errors:
"Title Error: Cannot pass a null PortletDefinition to a PortletEntity". Looking at the portlet section below it, gives me some more information. "Failed to retrieve Portlet Definition for demo::IFramePortletFailed ..." This clearly shows that even though I unchecked all of the demo applications during the install, it decided to leave some junk around which deals with the demo app. This is also an easy fix, just annoying! You can just edit the default-page.psml file inside the war or do it from the Admin Portlets, Page layout.

The fourth annoyance is again due to these demo portlets. I have all these tabs at the top of my portal. Petstore, JSF demo. Why do these guys like the demos so much? So I login to the admin, select admin portlets on the left, Click on the Portal Site Manager tab and started to delete away.

Finally I got my portal down to a stripped down version, no demos pure nuts and bolts.
I deployed my Hello World portlet just by dropping it into the deploy directory. The change was reflected in about a minute without a restart. This was great, in the past I have worked with WebSphere portal and that.. you know what I'll save that painful discussion for another time.
Since this was so easy to get up and going I am going to attempt to get it installed with WebSphere 5.1 ND edition. It says the WebSphere 5 platform is supported, but the wiki page is kinda bogus.

Restricting process information

2005-12-24T11:58:00.000-05:00

In our shared hosting environment we can see all the processes on the box. User Joe should have little or no insight as to what user Bob is doing, unless he needs to. This is a small blurb on how to restrict the process information, and make it a bit harder for a user to figure out what other users are doing on a system. We will cover BSD, Linux and our newest addition to the mix Solaris 10.

Linux - On the linux platform I have yet to find a distro that lets me do this out of the box. If you know of one let me know! The best thing I have found is OWL (openwall Linux) but its still not exactly what I want. Check it out here http://www.openwall.com/. I’m not sure if you can do this with DAC policy’s (discretionary access control) yet on Linux, but you might.

BSD - I am a big open/freebsd/BSDi fan. FreeBSD did it pretty good job at restricting process information. The introduced a kernel variable ps_showallprocs. You can set this from your sysctrl.conf file with the following:

sysctl kern.ps_showallprocs=0

This does a great job for those basic users at restricting the ps out. The slightly more advanced unix guru knows you can also get process information from /proc (procfs). To address this issue you can now disable procfs. Be warned that some programs require procfs access such as truss, w etc. I usually set this on our production systems since we try to debug issues offline as much as we can.

Solaris 10 – The guys at sun really did some good with solaris 10 and I could talk about it all day as far as security. At the same time its about time they started to come on board.

Solaris 10 provides us with a security policy we can set. Its controlled by /etc/security/policy.conf.

The ppriv option we are looking to use is proc_info. For those of you not familiar with ppriv feel free to play with it but here is a description of proc_info:

poseidon@megatron# ppriv -l -v proc_info proc_info
Allows a process to examine the status of processes other than those it can send signals to.
Processes which cannot be examined cannot be seen in /proc and appear not to exist.

Lets modify the /etc/security/policy.conf file to not allow proc_info for default users:
PRIV_DEFAULT=basic,!proc_info

We have not globally restricted ps information to all users. Sun also knows that /proc can be used to obtain process information and restricted its output to match the policy. These guys really did it right.

I have a few cron jobs that do need access to the process info for all users so how can give them that access?

Easily, for users that need access we can create a policy for them and apply it to a specific class of users with user_attr(4) file. I’ll let you do the man since you probably have all this free time being the holidays.

Something to the effect of the following just to change the defaultpriv:
moncron::::lock_after_retries=no;defaultpriv=basic

Again when you do this things like truss, pfiles, ptree are not going to work.

Something else to consider is creating jails for all of your users. You can give them each a small piece of the world to do whatever they want in. Most os’s will support this now, it’s a bit to configure but once you get it done you can automate it fairly easily. The users in the jail shouldn’t see a difference.

Wily Introscope - Portal Manager - How many users are on my site?

2005-11-11T16:04:00.000-05:00

Over the past few weeks I have been working with Wily Portal Manager for IBM WebSphere Portal, If you don't know what Introscope is by wily and you develop or host Java applications then you probably have no insight into your JVM at all. Go to http://www.wilytech.com and get on board with the products they have to get the visibility into the JVM you wish you had! They really do have a great product and one of the leaders in this space. For a quick background on the Portal Manager it is basically another plug in to the base Introscope software (10 minutes and a restart and your collecting your portal servers metrics.

I wanted to be able to get an approximate number of users logged into the site at any give time. I noticed a few people in the community were looking for this as well. My main problem was how do you know when someone is off the site. Well for this I am going to assume a few users click the logout button but the rest are going to have a session timeout.

With that in mind I started hunting around the IBM site for login and logout API's. I didn't find much but I did find this link especially useful.

http://www-1.ibm.com/support/docview.wss?uid=swg21104482&aid=3

It talks all about the login and logout functionality of portal. Since the Portal Manager already handles logins for me I decided to just use what they already have instrumented. Logout was the issue, especially for the users who do not click the logout button. From the doc above and the javadoc as well as some help from a wily utility 'PBD_Builder' I saw that com.ibm.wps.engine.commands.LogoutUser handles the timeouts of sessions as well as if the user clicks the logout button.

Note in my basic portal application I do not override the login and logout functionality. If you do you'll need to customize this a bit more.

So all I need to do now is count the number of users logged in and subtract the users that logged out and I'll get the estimated number of users on the site!

For the logged in users I'm just going to use a SimpleIncrementor:
TraceOneMethodOfClass: com.ibm.wps.engine.commands.LoginUserAuth doAuthenticate SimpleIncrementor "WebSphere PortalAuthenticationLogin:Total Count"
For the logged out users I will use a Simple SimpleDecrementor:
TraceOneMethodOfClass: com.ibm.wps.engine.commands.LogoutUser execute SimpleDecrementor "WebSphere PortalAuthenticationLogout:Negative Total count"

Note I am writing these to different places as I have other things I'm going to be doing with them but you could write them to the same place like Login:Total Users or something to that effect.

What I do now is create a metric group with both metrics Login:Total Count and Logout: Negative Total Count and use a sum calculator to get the Estimated number of users on the system.

Mileage may very!

Sun One web server 6.x Lockdown - Part I

2005-09-11T06:32:00.000-04:00

So after all your evaluations of the web servers out there you decided to use Sun One Web Server 6.1. So lets jump right into things and talk about getting this webserver ready for use. Installing the software and setting up a basic webserver instance is beyond the scope of this, if you need that info checkout the docs first. At the time of this writing they can be found here:
http://docs.sun.com/app/docs/coll/S1_websvr61sp2_en?q=schedulerd

Install the Software
--------------------

RULE #1: Don't run software as root
So I have on Solaris 8 - Tomcat, MySQL, an Sun One Web server. I created a user 'www' for the webserver to run as since our first part of the lockdown is not to run any software as root. You need to have the user added before the software install since the software will ask you, and be sure to secure this account such as setting the account to No password and a false shell. I suggest giving the software its own account not one of the default system ones so you can permission it accordingly and Jail/Chroot it later on which we will cover in Part II.

Create the Instance
-------------------
RULE #2: Disable unused features
So create a webserver instance via the GUI. Since I am running a multi-tier environment, keeping the app-server (Tomcat) content off the webserver for security & scalability reasons I disable the J2EE features of the Sun One product.

General Instance Lockdown
-------------------------
Since web server instances are very specific in nature to application I'm going to breakup the rest into things that could be done to lockdown all instances vs ones that are more specific in nature.

RULE #3: Don't allow the software to modify itself.
Suppose via a php or some server side script an attacker was able to run commands on your server. We have seen this happen before, a great example is the cmd.exe exploits that came out for IIS. Ideally we don't want the attacker to get in at all but if he does we want to limit what he can do. One step in doing this is to permission the software so it can not modify itself. I've noticed that Sun One web server installs new instances in my environment as Owner:www Group:www then the permissions on the files vary from 775 to 644 and so on. These default permissions would allow a vulnerable script to modify itself.

The first things I do is:
chown -R root:www /opt/netscape/live/$instance (this is the path to my live instance)
chown -R root:www /opt/netscape/live/httpacl/*$instance* (for acl files)
chmod 640 root:www /opt/netscape/live/httpacl/*$instance*
chmod 755 /opt/netscape/live/$instance (so www can read and write into it where it needs)
chown www:www /opt/netscape/live/$instance/logs (so the webserver can log)

Now that the software itself is somewhat locked down we can move on to some of the specific configs.

Starting with the magnus.conf to see exactly what these do refer the manual above.

Rule #4: Hide your identity
- Ensure TempDirSecurity is set to on.
- Ensure you set ServerString to something that does not disclose the version of your webserver. Numerous tools start by attempting to fingerprint your webserver first, don't make it any easier for them hide this info!

The next file we will look at is the server.xml (new to version 6) the server.xml provides information for the web server sockets.

Rule #5: Use strong encryption
- If you can use SSL. Not sending information in clear text makes it much more difficult to eavesdrop on.
- When using SSL disable version 2, its not strong ssl2="off". Be sure to change any + to - next to ssl2ciphers else you will get warning messages to the effect of SSL version 2 ciphers are enabled when ssl2 is disabled.
- Use only 128 bit ciphers if you can! Only a few sites should allow less than 128bit encryption, and those should be limited too. Disable anything less than 128bit by changing the + to a - for all the ciphers you don't need/use.

ssl3tlsciphers="-fortezza,-fortezza_rc4_128_sha, -fortezza_null, +rsa_rc4_128_md5 ,+rsa_3des_sha ,-rsa_des_sha,-rsa_rc4_40_md5, -rsa_rc2_40_md5,-rsa_null_md5, -rsa_des_56_sha, -rsa_rc4_56_sha, -fips_des_sha,-fips_3des_sha"

The last file we'll talk about in this section is the obj.conf.

Rule #6: Block unneeded HTTP Methods
You have your choice of options to allow browsers to use. I haven't found many others to allow than GET,HEAD, and POST. Production sites shouldn't allow webdav components or other types of features that development sites should so block them.

A critical one to block is TRACE. This provides more information that anyone honest needs to know about your systems. Add it to the top of your obj.conf default object.
<Client method="TRACE"\>
AuthTrans fn="set-variable" remove-headers="transfer-encoding" set-headers="content-length: -1" error="501"
</Client>
The above will block TRACE and set a nice error code you can monitor in your log files to watch the TRACE attempts.

This could arguably be a instance specific option but I add it to all of my obj.conf. Microsoft doesn't play nicely with keepalive, and rather than fixing their poor implementation they had most vendors come up with a workaround not to send a close_notify for MSIE.

AuthTrans fn="match-browser" browser="*MSIE*" SSL-unclean-shutdown="true"

Adding this to the top of the default object will save you issues going forward in more ways than one!

Checkout http://docs.sun.com/source/817-6248-10/crobjsaf.html for all the details on this.

Having the Web Server not send the close_notify packet may make MSIE vulnerable to a truncation attack, but that's Microsoft's Issues now isn't it.

Other than blocking trace restrict any Service method, the default config comes with cgi-bin references that are unprotected and more. If you need GET/POST/HEAD which most web apps that all they need append method="(GET|HEAD|POST)" to the end of all of your service tags. I hope in the future Sun One defaults to something more restrictive!

Specific Instance Lockdown
-------------------------
So this section will contain things that you need to ensure meet your application needed. They might be tighter than you need or in the opposite direction! Modify as needed.

Rule #7: Block bad requests
Stop wasting time processing requests that are bad before anything else! The following blocks all sorts of requests we know are just bad. Customize the string to your liking this is just an example, add it to the default object in your obj.conf before any processing.

<Client uri="*(system32|root.exe|ConsoleHelp|SERVER.INI
|Samples|WEB-INF|_mem_bin|_vti_bin|iishelp|iisadmpwd|iissamples
|default.ida|cmd.exe)*">
AuthTrans fn="set-variable" abort="true" error="412"
</Client>

I again specify a different error code so I can keep track of these responses and take action if I need to. Note the lines in the client tag wrap!

Rule #8: Have logs and process them.
Be sure you get all the info you need our of your webserver logs. I obtain not only the request but the content length, referrer, user-agent, the processing time and cookies such as the JSESSIONID cookie for Tomcat. Keeping this info in the logs help identify problems, its important to process these logs as well with tools such as WebTrends if you have some cash to buy or Aw stats which I use here.

More to come in Part II!

About this Blog

2005-08-25T02:17:00.000-04:00

This site was created after helping numerous people over the years with similar issues. Starting with locking down the os of systems ,webserver issues like Apache or Sun One , application server issues such as WebSphere and Coldfusion to application security related issues. Penetration testing, stress testing, finding application bugs.. I hope it all to be here and more!

Chroot Bind 9 How to - FreeBSD

2005-08-25T01:41:00.000-04:00

This document describes installing the BIND 9 nameserver to run in a
chroot jail and as a non-root user, to provide added security and minimise the potential effects of a security compromise. This is for Bind 9 only and freebsd.

Michael Johnson

V1.1 May 4, 2002

Credits:

Most people put this at the end but a lot of people don’t make it that far so I am putting them at the beginngin.
Most of this document was based off of Scott Wunsch’s chroot how-to but I found a lot of things missing. You can find his

how to with a bit more detail at: http://www.losurs.org/docs/howto/Chroot-BIND.html
Rich Mirch Looking over my setup and commenting security and configureation.

1. Introduction

This is the Chroot-BIND HOWTO for freebsd. It is assumed that you already know how to configure and use BIND (the Berkeley

Internet Name Domain). If not, please go read the DNS HOWTO and get a good understanding on DNS.

1.1 Latest version

The latest version of this can be found at http://www.setuid.us/HowTo/Chroot-Bind-Howto.html

1.2 System

The following worked on my system which is FreeBSD 4.4-stable one with intel chipset and one with cyrix chipset. It also worked on

FreeBSD 4.5-stable with smp. I hope you have as much luck as I do.

1.3 Disclaimer

The contents of this document worked for myself on the systems mentioned above. I have seen a number of different setups that

work equals as well. This is just how I decided on doing it. I have only installed bind on BSD/Solaris but with slight modifications it

should be portable across the different flavors of UNIX… sometimes those slight modifications are the ones that you can’t figure out

why its not working! If you have any comments/updates please send I will update this document in a timely manner.

2 Pre-Install
2.1 Create Non-root User

Create the user you want bind to run as. I used named. This should create a named group as well.

FreeBSD

pw user add named –s /sbin/nologin –d /usr/local/named –c “Bind User”

2.2 Create Directory Structure

I like to keep everything under /usr/local/ so mine is as follows:

/usr/local/

+-- named

      +--dev

      +-- etc

      |      +--namedb

      |      +--slave

      +--var

      +--run

Note:other directories will be created with the configure /make

FreeBSD

mkdir –p /usr/local/named
cd /usr/local/named
mkdir –p dev etc/namedb/slave var/run

2.3 Configure/Make/Install bind

Since we have not installed bind into our new directory structure lets do it now.

FreeBSD

tar -zxvf bind-9.x.x.tar.gz

cd bind-9.x.x

I configured with threads if you want to go for it if not don’t.

FreeBSD

./configure --prefix=/usr/local/named/ --enable-threads

make install

Now you should see new directories under /usr/local/named

2.4 Bind Data Files

If you already had bind installed and setup the copy the Data files from your current install to the new install path.
If you did not have bind already installed make install just put some new files into /etc.
We need to move those files from /etc and move them into our new bind home directory.
Then give the bind user we created permissions to this directory and the files in it.

FreeBSD

cp -p /etc/named.conf /usr/local/named/etc/

cp -a /var/named/* /usr/local/named/etc/namedb/

chmod 755 /usr/local/named

cd /usr/local/named

chown –R named:named etc

chown named:named var/run

2.5 System support files

For bind to work we need certain system files.

FreeBSD

mknod /usr/local/named/dev/null c 2 2

mknod /usr/local/named/dev/random c 2 3

chmod 666 /usr/local/named/dev/null

chmod 666 /usr/local/named/dev/random

You will need files from /etc as well

FreeBSD

cp /etc/localtime /usr/local/named/etc

cp /etc/passwd /usr/local/named/etc/

cp /etc/group /usr/local/named/etc/

cp /etc/spwd.db /usr/local/named/etc/

We also need some libs

FreeBSD

cd /usr/local/named/
mkdir –p usr/lib usr/libexec
cp /usr/lib/ libc_r.so.4 /usr/local/named/usr/lib
cp /usr/libexec/ ld-elf.so.1 /usr/local/named/usr/libexec

2.5.1 System support files security

I would change permissions on some of the files in our new etc directory.
chflags schg and modify them just to have named user, root and wheel.
This is up to you if you want to do this or not (it could break some things).
chflags schg /usr/local/named/etc/*(*)

2.6 Syslog Modification

Next we need to change how syslog is going to log in the chroot env.

FreeBSD

add this line or modify your current line

vi /etc/rc.conf

syslogd_flags="-s -l /usr/local/named/dev/log"

Stop and Start syslog.

3. Bind Data File Changes

3.1 named.conf

We need to edit a few lines in out named.conf file. I keep mine in /usr/local/named/etc/namedb some people like to keep it in

/usr/local/named/etc its up to you.

Add or modify the following:

FreeBSD

directory "/etc/namedb";

pid-file "/var/run/named.pid";

statistics-file "/var/run/named.stats";

4. Startup

4.1 Init startup script

I like to put this script into /usr/local/etc/rc.d/
That was it will stop and start on boot. You can do it from rc.conf or whever you like to start your daemons.

#!/bin/sh

case "$1" in

start)

# Start daemons.

echo -n "Starting named: "

chroot /usr/local/named/ sbin/named -u named –c /etc/namedb/named.conf

touch /var/run/named.pid

;;

stop)

# Stop daemons.

echo -n "Shutting down named: "

killproc named

rm -f /var/run/named.pid

echo

;;

restart)

$0 stop

$0 start

exit $?

;;

reload)

/usr/local/named/sbin/rndc reload

exit $?

;;

probe)

# named knows how to reload intelligently; we don't want

# to offer to restart every time

/usr/local/named/sbin/rndc reload >/dev/null 2>&1 || echo start

exit 0

;;

*)

echo "Usage: named {start|stop|status|restart|reload}"

exit 1

esac

exit 0

4.2 Start Bind

chroot /usr/local/named/ sbin/named -u named –c /etc/namedb/named.conf

4.3 Start Failed what should I do?

If this does not work run first look at permissions. Try to su the user you created and make sure that user can see and access everything. I had my named directory 700 root:wheel which prevented me from starting at first. I quick fix is to chmod –R named:named /usr/local/named then change back to root files you think don’t need to have that permission. If that fails run this to see if your missing files or where its breaking:

truss chroot /usr/local/named/ sbin/named -u named –c /etc/namedb/named.conf | more