Update on Fedora 18 on VMWare Fusion 5.0.3

Everything now works.

VMware FusionScreenSnapz017

 

The quick version is:

  • Create a new Fedora 18 VM
  • Do not use “Easy install”
  • Disable 3D acceleration in the VM settings (Command-E) prior to starting the install, otherwise you get a spinning idle cursor and no action upon first boot
  • Install as you see fit. I use a 64 bit F18, 4 GB of RAM (my Mac has 16 GB of RAM), 100 GB partition, and 4 virtual processors. YMMV. General rule of thumb is that 1 GB is plenty for a casual Gnome desktop, but if you’re using it for software development like me, then the more the better.
  • Once installed, update all software using
sudo yum update
  • and then reboot.
  • This gets you to kernel 3.8.4-202.fc18.x86_64 at the time of writing. This is currently incompatible with VMware tools 9.2.2-893683, but there are known patches (see below) until a VMWare Tools update comes along.
  • Let’s install the build tool chain needed to complete VMware Tool installation
sudo yum install patch kernel-devel kernel-headers gcc make
  • Reboot just to make sure we’re all good
  • Click “Install VMware Tools”
  • Unzip the tar ball to ~/Downloads/vmware-tools-distrib/
  • Visit this page http://erikbryantology.blogspot.com.au/2013/03/patching-vmware-tools-in-fedora-18.html
  • Download the vmware9.k3.8rc4.patch and vmware9.compat_mm.patch patches, and if you feel like you’re not a patch wizard, the shell scripts that apply those patches, too. 
  • Follow the directions in the above page to patch the source to the VMCI and HGFS file system sharing modules
  • Now build the patched VMware tools
 sudo ./vmware-install.pl
  • There should be no compilation ERRORS (there will be a couple of warnings, particularly for 64 bit architectures)
  • Reboot
  • Test drag and drop file sharing, cut n paste, screen resizing, shared folders, and printing to a MacOS X printer.
  • If it’s all good, shutdown the VM
  • Go to Settings for the VM (Command-E)
  • In Display, re-enable 3D acceleration
  • Start up the VM

At this point, you should have access to accelerated 3D within the VM and all VMWare Toolbox features.

How to migrate to PDO without it hurting… much

As we saw in the previous article, conversion to MySQLi is an awful lot of work. So let’s move to PDO.

Step 0. Get PDO working with your database server

Somewhere along the line, the PHP and MySQL folks decided to not be friends, so even though 99.99% of all PHP scripts require MySQL, in most cases, PDO doesn’t have MySQL support enabled.

Check that PHP doesn’t already have PDO for MySQL ready to go:

% php -i | grep -i pdo
Configure Command =>  '[...snip...]'
PDO
PDO support => enabled
PDO drivers => mysql, sqlite, sqlite2
pdo_mysql
PDO Driver for MySQL => enabled
pdo_mysql.cache_size => 2000 => 2000
pdo_mysql.default_socket => /tmp/mysql.sock => /tmp/mysql.sock
pdo_sqlite
PDO Driver for SQLite 3.x => enabled
The default socket location is fine for a development workstation, but not so good for a production host. Change php.ini and my.cnf to make it safer, such as /var/run/mysql/mysql.sock
If your PHP installation doesn’t show the above, whip out the compiler – there’s plenty of articles on the Interweb on how to get PDO and MySQL going. For now, let’s assume PDO and MySQL are good to go.
Step 1. Conversion
As in our previous article, it’s very tempting to just go through each file and make a 1:1 change from MySQL to PDO. That is actually a losing scenario.
  1. My non-trivial app has 1853 SQL queries littered through the code. At about 15 minutes to 30 minutes per query, with no test case, that’s about a year’s work with no change to the overall complexity of the app nor any improvements to the overall data architecture. Changing from one to the other is sure to introduce bugs.
  2. GaiaBB only has 13 tables. Using a DAO approach, with a list and search helper method for each (i.e forum list, search forum), that’s 6 * 13 = 78 properly written, tested DAO methods that need to be written from scratch. This is a saving of over 10 months work compared to just getting in there and getting my hands dirty.
  3. I can add security business requirements once to the DAO, such as fine grained access control, input validation, audit, and dirty output sanitization, thus fixing all my access control issues and dirty data issues in the same small piece of code. So each file that is converted to DAO gets more secure with no additional coding

You’re probably wondering about “dirty output sanitization”. Sanitization is for the weak minded! No seriously, my database is 7 years old. There’s crud in there – I can’t trust it to be safe or good. There’s some nice tools out there like Arshan’s Scrubbr to scan and clean a database, but Scrubbr is not going to be able to decode BBCode and check to see if there’s a nasty [color] based XSS or CSRF attack in there. Additionally, some versions of my software and some buggy versions of MySQL == binary blob crappiness. Blobs coming out sometimes (but not always) need to be stripslashed(). Additionally, some versions of XMB used client side checks to prevent files of the wrong type to be uploaded. I’ve found a JavaScript malicious file in my production database. So it’s worth adding checks so that I don’t serve bad things to folks. You can’t do this if you have 100+ attachment related SELECT statements alone, not without lots of bugs and lots of code. Going the DAO way, means I can do it once, for every single use of the attachment sub-system.

There’s gotta be a downside.

It’s not all sunshine and roses. You should not open both a PDO and old style connection to the database for busy servers like mine – it more than doubles the time to serve the average script, the poor little database server gets whacked, and performance will drop.

In GaiaBB, there’s a significant 7240 lines of shared code read in every single time a script does anything – header.php, functions.php, db.php, validation.php, constants.php, config.php, cache.php, english.lang.php, mail.class.php. So to only open ONE database connection requires these files to be ported to PDO first. But if I convert these files, every remaining single file totalling about 80,000 lines of PHP will also need to be converted.

So my solution for GaiaBB is to create a side-by-side approach, using kernel.php for converted files instead of header.php. kernel.php includes PDO ready files rather than the old files. This temporarily makes GaiaBB approach the 90,000 line mark, but in the long run, it will allow me to make many of the scripts smaller and more object orientated. Once converted to DAO, I can simply eliminate many security checks within the code of each page, as the DAO will simply throw an exception if you’re not privileged to do something with the data you’re trying to work with.

So my main piece of advice for you contemplating converting to PDO is to consider your data architecture. The old MySQL way is awful, buggy and insecure. Let’s exterminate it, but instead of standing still for months at a time, add in freebies like access control, audit, input validation and output sanity checking.

Converting your PHP app to MySQLi prepared statements

Okay, you’ve got like a zillion SQL queries in your PHP app, and probably 95% of them have a WHERE clause, and you need to make them safe so people will still download and use your app. Because if you don’t fix your injection issues, I will rain fire on your ass.

These are the steps you need to take to convert to prepared statements.

Step 0. PHP 4 is dead. Upgrade to PHP 5

All of the code in these samples is PHP 5 only. I will be using SPL and MySQLi, which are installed primarily on PHP 5.2.x installations. PHP 4 cannot be made safe, so it’s time to upgrade. This is non-negotiable in my view.

If you’re using register_globals, you have to stop. Do not use a function to register them for you in PHP 5, it’s time to do proper input validation. This will actually take you longer than converting all your queries to prepared statements.

To get a handle on this issue, what you need to do is:

1. Turn on error_reporting(E_ALL | E_STRICT);

2. Find all isset() and empty() and unless they are actually testing for a variable you’ve set, get rid of them. isset() and empty() are not validation mechanisms, they just hide tainted globals from view

3. Go to php.ini, and turn register_globals off. It should already be off

4. If your code has a construct like extract($GLOBALS) or some other mechanism to register globals, get rid of it

5. php -l file.php. This will give you a first pass which you will need to clean up

6. Use PHP Eclipse or PDT in Eclipse or the Zend IDE in Eclipse. This will give you warnings if you have uninitialized variables. Go to the properties, and make this into an error. Clean up all uninitialized variables

7. Start each script like this:

// Canonocalize
$dirty_variable = canonicalize($_POST['variable']);
// Validate
$variable = validate($dirty_variable);
// Use the variable
$stmt = $db->prepare("SELECT * FROM table WHERE id = ?");
$stmt->prepare("i", $variable);
$stmt->execute();
// and finally, if you need to output that sucker:
echo htmlescape($variable, $locale, $encoding); // $locale is probably 'en', and $encoding is probably UTF-8 or ISO 8859-1

Obviously, you need to canonicalize – that is make it the simplest possible form. If you have no idea about this extremely important topic, please consult the OWASP Developer Guide. Validation is essential. This replaces isset() and empty() and other mechanisms, with actual validation requirements. If you’re expecting an array of integers, make sure it’s an array of integers! If they have to be in a certain range, make it so. If the validation fails, put up an error message and do not proceed! This stuff is CS101, so please make sure you do this reliably for all variables without exception.

Step 1. Make sure your hoster has MySQLi

If your hoster is still running PHP 4, you need to see if they have the ability to run PHP 5. Most likely, your PHP 4 installation will not have ANY prepared statement compatible interface, like MySQLi or PDO. Of course, PDO is PHP 5 only… and it has a cool interesting feature – it emulates prepared statements for those databases that do not have support for it. But that’s for another post.

How do you check? Use phpinfo(). Create this small file somewhere with a random file name and upload it to your host:

<?php phpinfo(); ?>

Run the file, and note if you have the MySQLi interface. If you don’t, you can’t upgrade to prepared statements. It’s time to wage holy war on your hoster to make sure they install PHP 5.2.7 with MySQLi and PDO with MySQL 5 client libs for you … and all your other shared hosting friends.

Changing over to MySQLi

The simplest part of this process is to move to MySQLi from MySQL:

Instead of

$db = mysql_connect($db…

You have two choices: stay with functional MySQLi, or move to OO MySQLi. I think the latter is better, but that will be another post.

$db = mysqli_connect($db..

Now, this is where it’s important! You MUST check the value of $db for errors before continuing. You probably have this code today, but it’s important to realize that if $db == false, you didn’t get a connection.

if ( $db == false ) {

// Print up an error and stop

}

Simple Conversion

You may be tempted to just use the MySQLi extension, and move all your queries to place holder versions. That’s okay, but it can be a lot of work. Trust me, I’ve tried.

Although it seems like the easiest possible choice, converting MySQL queries to MySQLi’s prepared statements has a couple of issues.

Gotchya: There’s no easy way to bind lots of results when you use select *

With MySQL query, fetch_array will simply bind the current result set row to an associative array, and you can access it trivially. Most PHP apps use this data access pattern extensively, like this:

while ( $row = mysql_fetch_array($query) ) {

// do some work with $row

$blah = $row[‘column’];

}

Which brings us to the next gotchya:

Gotchya: There’s no fetch_array()

I don’t know why MySQLi does not have this most common of all the data access patterns, but it’s a right pain to fix. So let’s get a function to emulate fetch_array, including using anonymous field names as per above.

Okay, so we’ve decided that MySQLi sucks the proverbials… so in the next article, let’s talk about migrating non-trivial PHP apps to PDO.

GaiaBB and OLPC

Peter Quodling. an old friend, e-mailed out of the blue last week. I have a lot of time for Peter as he’s one of the few Australian IT architects that really knows his stuff, plus he’s a really nice guy. He is involved in OLPC in the PNG region. Last Christmas, I nearly bought an XO under the Buy One, Give One program so Mackenzie could have a cool first laptop … and somewhat more honestly, so I could play with the OLPC until she’s old enough to type let alone talk. However, circumstances prevented toy purchases of that magnitude, and I forgot all about the XO until this week.

I thought through my various abandoned projects (for I have many), trying to work out which would help the OLPC project and kids all over the world. I’ve had an itch for a while to do something for the One Laptop Per Child project (OLPC), but never really had a substantial idea that would help transform kids’ lives rather than my own. But now I think I have just the project.

So I put in a proposal for two laptops to help develop GaiaBB (which is UltimaBB++ (sic gloria transit), which is XMB++, which is awful) into a OLPC specific product.

My current plans are:

  • Get GaiaBB.com back up and make it OLPC centric. Revitalize the Sourceforge project.
  • Finish OWASP ESAPI for PHP. I need it for this project.
  • Port GaiaBB to the OLPC, porting the database to sql lite, and probably using LightHTTPd. I could use Apache + MySQL as per now, but these are huge compared to SQL Lite and LightHTTPd, and on a device with limited NAND memory, every byte counts.
  • Ensure GaiaBB works properly with Browse, the XO browser. I might need to turn the nested tables into CSS templates a bit sooner than I intended
  • Beg, borrow or steal a graphic designer to come up with a GaiaBB theme that works well with the dual color display (it’s both black and white and color and more wide than tall), and possibly work out how to detect the XO’s current screen state from the web page so I can dynamically choose a grey scale or a color theme.
  • Simplify the product so that it’s more manageable by young ‘uns without dumbing down too much or removing some of the depth and surprise features of the product
  • Write an installer that makes it easy to install on the OLPC. I want kids to be able to create their own social communities and for them to easily share their forums with their friends.
  • And this is where it gets fancy… write a web service that allows authorized folks to replicate their version of the forum with other versions of the forum … without causing major security issues. That way when you’re at home and have no Internet service, you can still read the forum and write new posts and then sync when you’re at school. This is where I will have to significantly change the way GaiaBB works as right now, it’s a single database deal and assumes that there are trusted administrators.
  • Go through the code with a fine tooth comb, replacing all the crappy security bits with ESAPI for PHP. Some parts are truly ancient (circa 2000) and need refactoring. As part of this, I will ensure the code is easily modifiable. I learnt how to code by changing other folks’ code and then starting to write my own, and I want kids to modify the heck out of this forum so as to create a new generation of coders excited about programming and IT in general.
  • Lastly, possibly write a OLPC School Server centralized GaiaBB hub for schools to run “their” forum which students can sync with in a safe fashion.

The proposal has been approved, and the laptops are on their way. They are sending me three XO’s! Awesome. Better get cracking!

Google: Don’t be evil

I work on an open source project, ESAPI for PHP. Well, “work” might be too strong a word for it, but I try to prod its lifeless carcass from time to time. That’s not the reason I write today. I write because of stupidity, and evil being conducted in the name of a “law”.

I have a fellow open sourcer, who wants to contribute to ESAPI for PHP. He’s actually completed a MVC framework for PHP (jFramework). Due to Google blocking Iran, this gentleman can’t easily contribute to our project, which hosts its repository on code.google.com. ESAPI for PHP will not help build a nuke. It does no crypto of its own. It will make PHP applications safer and more secure – but you can do that anyway if you read half a dozen pages on PHP’s website.

This is madness. ITAR is about blocking the EXPORT of sensitive MUNITIONS (i.e. weapons) TO Iran and other “hostile” countries. ITAR is NOT about blocking the GIFT of intellectual property and valuable developer cycles FROM Iran, helping everyone all over the world, including those folks in Iran (as well as Australia and the USA). This is stupidity on a scale I’ve not seen in a while.

Google: you are doing evil.

Stop this madness, now! Call in your tame congress critters and tell them how stupid and harmful this particular nonsense is and get it repealed. Grow a spine and take a chance. Unless someone open sources a command and control system for a warship, a missile guidance program, or puts Nuclear Reactors For Dummies up as a project, all of the projects should be available for download worldwide. Those one or two mythical and nonsensical projects should not block an entire library of human knowledge to the entire Iranian people just because of some imaginary evil open source project might help Iran’s nuclear program or military. The stuff we do is not rocket science.

Stupid and outdated laws / treaties like ITAR make us disrespectful of all the other laws and treaties, and make us lose all respect for those who abuse their positions of power in the name of “security”. The way to improving relations between countries is not to block them (how’s that Cuba policy going, anyway?) but to engage with them and stop the evil ignoramuses on both sides stopping everyone being happy and free, or just contributing to an open source project.

Soon, there will be one

Well, what an interesting weekend. A cold, working like a slave, and one of my co-workers is a father for the first time (Congrats, Ty!). But that’s not the most interesting news.

I will be taking sole ownership of my forum, Aussieveedubbers, sometime this week. This means that I will have to spend a bit more non-existent personal time attending to it.

This is good and bad news:

  • For UltimaBB, the underpinning forum software, it’s fabulous news. UltimaBB was never released and is now effectively a dead project. My acquisition of AVDS gives me the impetus to make the forum software as good as I can make it. Once I’m complete, as it’ll be by far the most secure PHP forum out there. Very few open source programs ever get the chance of a top to bottom code review. Once I’ve fixed all those issues, I will think about possibly adding a few flashy features and integrate it with CPanel and others, so ISPs can easily deploy it. Obviously that integration will not come cheap. Hopefully, I can start to earn a bit of income from the forum, finally.
  • For OWASP, this is not good news. I have two current projects, the Developer Guide and ESAPI for PHP. I need ESAPI for PHP to be complete to help UltimaBB, so you can guess which of the two projects will get my time.
  • For my personal life, I hope Tanya will forgive me making her do some basic accountancy work. I think it’ll ease her eventual way back into the workforce in a few years, as she is terrific at accountancy, and I would hate to see her lose all her skills for the want of a bit of work here or there. However, it’s not just delegating the book work to my poor suffering wife, it’s also a bit of an ask for the few hours I have to give right now.

So if you want to help with the Developer Guide, please join the mail list and let us know how much time you have, and where your interests are.

Validating ASVS 1.0 beta using a PHP application

A long, long time ago, I took on running Aussieveedubbers, a forum based around the love of Volkswagens. We were on EzBoard, where the adverts and performance sucked so bad, that free was no longer acceptable. Over many iterations, I now run UltimaBB, a derivative of XMB. I had various titles – including lead programmer – at both projects for a while, but my main claim to fame was to get XMB 1.9.1 out the door with Tularis, the lead dev at the time, and the primary early force behind UltimaBB along with John Briggs. John implemented the hacks, and I did the architecture changes required to make them suck a lot less and hopefully make them more secure. In a way, I have – UltimaBB does not suffer from ANY of the XMB Bugtraq failures. Therefore, I am a coding god and UltimaBB is secure, right?

I got going in forum development primarily through security fixes. By the time the UltimaBB effort died and XMB imploded in 2008, I thought I had gotten most of the security bugs out of the code. UltimaBB was a shining beacon of the bazaar approach. Or so I thought.

But like most in open source, there’s a huge difference between what I think is secure, and what is actually secure. Just because I think it is secure doesn’t make it so. I wish more folks in open source projects would get this very simple message. You suck, and as long as you know that, everything (in time) will be okay.

So, like all good scientists, I do not let my good opinion of myself get in the way of a solid slap session. I gave UltimaBB a thorough code review using OWASP’s forthcoming Application Security Verification Standard at Level 2B, which encompasses a manual code review for 83 controls with an optional use of automated static code review tools. I used Fortify’s awesome Secure Code Analyzer, which is almost certainly the best in the business today. It didn’t find everything, but it highlighted a number of deficiencies I already knew about quite reliably.

I can’t claim I did this review because I suddenly had a huge chunk of time on my hands, but because I am implementing a new service at Pure Hacking, and we’re using ASVS as our verification baseline. I’ve added a few more things in as I think they’re important, but the main thing is that I’ve now validated that the ASVS produces a reasonable outcome for an average program, one that should have been reasonably secure – because *I* wrote a great deal of it, and certainly the security bits.

The good news is that ASVS can be used as a security consultancy’s baseline for a partially automated, but primarily manual code secure code review. I think with a bit of tweaking, we can develop automated searches in most languages for about 50-60% of its requirements, which will dramatically improve the quality and reliability of secure code reviews.

The bad news is that my forum sucks. I have a bunch of extreme, high, medium and low risks to fix. Until I’ve fixed them (as there’s precisely one forum running this code out there – mine!), I can’t go into them too much, but let’s just say that I was disappointed with myself, and I’ve taken myself out the back and slapped me silly.

Luckily, I have a good client for this review (me), who will actually fix all the defects and lacks found so far.

ESAPI for PHP news

AccessReferenceMap, RandomAccessReferenceMap and IntegerReferenceMap, and enough of the other classes (FileBasedAuthenticator, StringUtilties, etc) are present and working:

ESAPI for PHP tests passing

This is very good news as although some of the other classes in Milestone 1 are complicated, these two classes were actually going to be some of the hardest to port as PHP does not have the equivalent of J2EE Set, List, HashMap, and many other basic data structures. What PHP does have is native associative arrays (somewhat like HashMaps), ArrayObject, and ArrayIterator from the SPL. The problem is that PHP doesn’t like sparse arrays with very long indexes:

$foo[“THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX. THIS IS A VERY LONG INDEX.”] = $value

So I had to make up a workable hash function and hope for zero collisions. I tried using spl_object_hash(), but that actually is too good. It uses the object’s value AND pointer position, such that:

$foo = “123”;

$bar = “123”;

spl_object_hash($foo) != spl_object_hash($bar)

I think I still need to add a few more test cases as my hash function WILL collide when there are two direct object references of the same value, and thus will not be safe for some uses.