Ramblings

‘Tis the night before TJOC and all through the apartment, not a soul was sleeping, not even me …

It’s been a long rampup to camp for me already, and camp doesn’t officially start until tomorrow (well, later today). We did the Diagnostic O written test online this year, and since I’m the one that suggested it and have the skills, I was responsible for implementing it. Not that I’m complaining … it really is a cool project, it’s just my timeline really sucked. I waited too long to really get started, and that left me almost too little time to get all the core features completed before we needed to roll it out to everyone.

But, all of that to say, I’m not the only one up late. There are still last minute test takers going right now. An email was sent out earlier today giving an official cutoff of 8AM CST on Saturday and there have been quite a few new users and test attempts since. If only more of the students had completed it … we just had barely broken 50% this morning.

I guess there are still a couple of hours ……….

As we all know (if you’ve ever used it or heard me complain), the ecommerce process (recording the payment, etc) has been running pretty slow lately. My boss and I recently had a discussion where we agreed we wanted to spend some time fixing this. I put some better logging in place to see where the bottlenecks might be and based on information gathered I have been able to make some upgrades and modifications to enhance the overall performance of the system.

It turned out the the process of sending out emails was really the killer initially. They were taking (cumulatively as there are an average of about 3 that go out per transaction) around one minute on average, and up to 2 or 3 worst case. I have redone the way those emails are being sent and have trimmed that down to an average of under 2 seconds (for all of them). That alone made a huge difference, but it isn’t all
that has changed.

There is a query that I run to determine the journal number (basically a batch number) of the transaction that was just posted. It turns out that it was taking anywhere from 3 – 10 seconds to run. Upon further investigation, the query wasn’t making use of any of the indexes on the table (and it’s a large one, so many, many records). I made a small modification there that reduced its run time to under a second.

Right now I have good data on about 32 transactions that have occurred since I made the bulk of the changes on Tuesday and Wednesday. Here’s a breakdown of the numbers and how they fall into a Service Level Agreement that we are working up:

• Trad Student Account Payments: 7 @ 18, 34, 8, 8, 5, 13, 10 seconds
• Non Trad Student Account Payments: 12 @ 6, 9, 7, 5, 4, 3, 4, 6, 9, 9, 12, 11, 5 seconds
• Non Trad Web Reg Payments: 13 @ 7, 7, 5, 8, 8, 7, 7, 8, 9, 16, 6, 7, 8 seconds

The Service Level Agreement (SLA) is currently defined as the following:

• 50% of the transactions should occur within 5 seconds
• 90% of the transactions should occur within 10 seconds
• 98% of the transactions should occur within 20 seconds

When you look at the numbers I provided previously, we are currently short of that SLA:

• 5s target: Need 16, only have 7 for a real percentage of 21.88%
• 10s target: Need 29, only have 26 for a real percentage of 81.25%
• 20s target: Need 31, have 31, but only have a 96.88% (rounding)

Only the 20 second period is currently within limits. We aren’t far off on the others, but we aren’t done with performance enhancements either. When you look at the averages, things look a little better:

• Max: 34 seconds (fluke, very high system load at the time)
• Min: 3 seconds
• Average: 8.75 seconds
• Median: 7.50 seconds

These numbers are with the posting process (what actually applies the payment to the account) still being a part of the total time. We’ve discussed having that process run sometime after a payment has been made, but within a minute or two of the actual payment. If we go that route, we should be able to subtract at least 80% of the time that we are currently seeing (as far as the student is concerned).

What say you to an update about work stuff first? No? Too bad, read another post then (more will be coming soon that aren’t work related). Sometimes it is hard for me to really step back and see just how much stuff has changed over, say, the last six months. We are in the middle of yet another web registration window. It seems that one just barely closes, and then bam, another one opens up. The real trick this time is that they decided to do summer school registration online and at the same time as fall 2006 registration. Until this time around we had only done one semester at a time. Until this time around I had no idea how badly multi-semester registration could really screw up my code. Until this time around … argh! Eh, whatever, I’m still in the process of correcting all the little problems that showed up as a result of this. The problems are almost all related to money, but aren’t they always?

More interesting things are going on, like the reason I’m in the office at 4:00 AM today. Ok, so it isn’t that much more interesting as I’m here because I couldn’t sleep and just happened to crank out some pretty nice work (if I must say so myself). Specifically, we have been having performance issues with the Web Payment stuff (pay your student account online, etc). The problems came at the very end of the process when we actually process the credit card info, send out some emails, and log all the pertinent data in the database. The user would click the submit button that began the process, then depending on how impatient they are, would wait around for up to 45 seconds while the page loaded. The really impatient ones would really screw the process up by leaving the page early. I know, I know, that shouldn’t affect anything other than what they see, but it did. Since I was in the office tonight, I decided to take some action.

We had been talking about putting some logging in place to record how long various parts of the process took to run (so we’d know where to work on things if they were in our control). So, I dropped some logging in on the testing site. Wow! The places I thought we were having issues were actually the ones performing the best. It turned out that sending the emails out was the slowest part of the whole thing, and we send quite a few emails (payment notifications, error messages, receipts, etc). So it was time to look into optimizing that section. I was using a nice cpan module (yes, this is all in Perl … oh why wouldn’t they let me use PHP!?!) that made sending the emails pretty easy, it just didn’t make it fast. Since we have Sendmail installed on this machine I figured why not pass the email off to Sendmail and let it worry about everything else. Huge speed increase! Sending a single email went from taking around seven (ya, 7) seconds down to under a single second. Makes all the difference in the world.

I have been doing a little more than just programming lately. We purchased a wiki for use here in IT called Confluence. It’s pretty nice, even though it is built on Java. Since I was the only one in the department who had any real wiki experience, the project was given to me! I was able to con talk Tech Services into giving me a machine for the wiki on which I have installed Gentoo. It’s sitting back in “DC3″ (a rack in the shop area, not a proper Data Center at all) just humming away right now. The software gave me a few problems at first. We can use our Active Directory server for authentication and group memberships, so that’s really nice, but the version of Confluence that we started with was really, really slow when I turned on AD authentication. A couple of weeks later (the wiki wasn’t really being used yet, still in testing) a new version came out that contained a ton of LDAP/AD enhancements. Those enhancements brought the speed back up to where it should be. I’m just glad it wasn’t my poor sysadmin skills that was causing the problem. So anyways, the wiki is now being used a little … and soon it will be the next big thing!

The big news around here right now are the office renovations. They are doing everything possible to fit more people in the limited space that we currently have. The problem is that we just keep growing and have no place to put any of the new hires. The construction should begin pretty soon now that the permits seem to have been properly obtained from the city. We shall see, and so will you since I’m going to try and take some pictures as the process evolves.

Look for additional posts soon that will cover the non-work related happenings around these parts.

Argh! (as I would expect to hear coming from Scum). Stupid people and their stupid pet projects with stupid drop dead-deadlines. You cannot expect a whole new ecommerce project to “just be done” in 12 days! Especially when I won’t even have time to start that project until well after that deadline. Unfortunantly, the department asking, well not so much asking, is too big a favorite with, ehem, some people in our administration to completely ignore or push off the hot plate.

OK, I have the urge to do some recreational programming again, and I think my project will be a photo gallery system. I know, I know, there are a ton of them out there already. Hey, I use one of those that already exists. I just think I can do it better, at least better for me. I want real ACL control over everything from galleries down to individual photos. I want an external API that will allow third parties,or myself, to build applications that can control the system without knowing the internals or having to hack my scripts. Right now I’m not too concerned about database abstraction, as I only have MySQL installed. Once I get things clearly defined I might consider spending the time on that item.

OK, enough ranting, here’s a quick, mostly intelligible, list of features to this point that I want:

• Batch (zip file upload) and individual photo uploads
• Add photos from filesystem (already on the machine somewhere). Don’t mess with the originals, copy them over to the webroot and deal with them there.
• Externally visible API – all functionality needed to do everything the web pages can do
• Database based config management.
• The option to choose between pre-caching resized images, on-the-fly caching, or no caching at all.
• Extensive stats tracking – galleries viewed, photos viewed, etc.
• Comments on everything from galleries to photos.
• Ratings on everything.
• Nice URLs for distribution (mod_rewrite).
• User favorites
• Keywords/descriptions on everything.
• Custom fields (unlimited number) on everything.
• Users/Groups.
• ACL control.
• User profiles (Real name, email address, physical address, website, etc).
• Galleries (equiv to multiple installations) with “Albums” that can be nested. No “Categories”.
• Group/User level control of ratings permissions, comments, uploads (no/yes/moderated), image rotation, etc.
• Basic image editing (rotation, resizing, etc).
• User defined featured photo (choose the image that represents the album).
• User defined photo order inside albums.
• Live resize on display (allow the user to select a resolution to display the current photo. Not persistent across site).
• Theme support.
• Watermark support. I don’t necessarily want it, but I know it’s a popular request.
• Backup and restore functionality.
• Offline mode for creating static web pages?
• Mirrors. Use API to add objects to remote servers or require manual sync?
• XML output for templating or just use good XHTML with CSS?
• And a whole lot more … ambitious isn’t it?

AHH!!! I’m going to go mad over this. I’ve been working on this project (and related side projects) for months, but I’m finally nearing the end. For those of you who haven’t heard, the school now offers you the ability to pay on your Student Account online. This currently only is available for Traditional Undergraduate students, but I’m working on the upgrade to allow everyone else to pay online. It’s this last piece that is driving me nuts because there are just enough changes to cause a slight ripple effect, causing me to have to revist some older code and table layouts. Errr!

Usability Guidelines for Links (Jakob Nielson)

After hearing everyone around me expound on the virtues of Python, I think it’s time to take the plunge. I’d like to see what in the way of GUI development I can do … you know, work on a front end for my P2P app. I’ve also been doing a little reading in a book called Rapid Application Development with Mozilla [ZIP link] trying to see if going with XUL and Mozilla is the way for a quick GUI …

Just did a bit of research and found a ton of commercial SFTP/SSH servers for Windows. I decided to see about getting a free one working, OSS and all. I tried using a CYGWIN & SSHD howto I ran across and what do you know, I now have a SSH/SFTP server running on my Windows XP box at work. I’m sure the network guys would love to hear that! :)

Now I suppose MacOSX has SSH/SFTP abilities native. I should check with Scum on that, as he’s my Mac resource.

Scum and I were kicking around this idea of a closed peer-to-peer filesharing network that is by invitation only. We’d like to be able to share some of our files with each other after everyone splits from school, and we aren’t really happy with anything that is out there right now (for closed p2p networks).

So Scum has embarked on a mission to create one that we could all use. It will have to be cross platform in the most extreme way (there are three operating systems represented in our small group), so he has taken a web based approach. After seeing a quick demo the other night of what he has so far, I’m convinced we will need some sort of client program to handle the actual file serving and indexing. I’d rather not have to figure out how to get a FTP server up and running and keep it secure when I could use SCP or SFTP to transfer the files.

At the same time, it isn’t efficient at all to have a remote server connect to your machine and attempt to index the files you have hosted. It would be much quicker and safer to have a client program that watches for changes in a given folder or folders and update a remote SQL server (MySQL) as needed. Then when a search is initiated on the web server, the user can feel confident that the index is the most recent possible.

The index would only contain a listing of what host has what file. It would be up to the user to launch a client program (maybe the client program initiates the query?) and specify the host and file path to actually get the file. Scum mentioned FTP (which is all he has implemented currently) because of the usual autoresume on failures on so forth. I’m not sure if SFTP can be resumed or not, but if so, that is what I would suggest. I’m not sure if there are SFTP servers for platforms other than *nix, so maybe it isn’t an option at all.

Anyways, I’m hashing this out in my head and if anyone has some input, toss it in …