Behind closed doors (a candid tech discussion)

Here's a little behind-the-scenes peek at how projects are hatched in my company. This is the transcript of an instant messaging session between me and my right-hand man. This is posted for your amusement and/or education, however you'd like to take it.

[Note: Some of the time stamps are out of sequence because I re-ordered some of the comments to make the conversation flow more logically when multiple threads were being discussed simultaneously.]

15:05 Trak3r: got a few minutes?

15:05 Kanook: yes

15:05 Trak3r: -----------------------------------------------------------

15:06 Trak3r: demarcation line for the log when i look this up later

15:05 Kanook: me over there?

15:06 Trak3r: no lets do this in chat

15:06 Kanook: ok

15:06 Trak3r: topic of discussion: images in the database

15:06 Trak3r: (let me dump my thoughts first)

15:06 Kanook: ok

15:06 Trak3r: 1. need servlet to suck them out and serve them -- it would of course cache them, keeping them in memory

15:07 Trak3r: 2. need a servlet for putting them in the database so the [managers] can upload/update them -- also this is a requirement for [client-a] (uploading new widgets/sub-cobrand images)

15:07 Kanook: first q (answer at your convenience): do we have to store them in the db?

15:07 Trak3r: 3. can we hack iis to redirect /images to the servlet so we can move all existing images into the database w/out have to re code templates?

15:08 Kanook: er

15:08 Trak3r: 4. ... shit you broke my train of thought

15:08 Kanook: then don't read my qs :-P

15:08 Trak3r: 5. do we want the database table structure to mimic a directory structure (with nested groupings)?

15:09 Trak3r: ok, discuss

15:09 Kanook: q2: is this for *all* images or something? I thought it was just for those [client-a] images in their tables

15:09 Trak3r: this discussion is about moving *all* images

15:09 Kanook: holy crap why?

15:09 Trak3r: why 1. [client-a]
why 2. so the [managers] can manage the images

15:10 Kanook: Well, 2) is asking for trouble of course

15:10 Kanook: but that's another story

15:10 Trak3r: indeed

15:10 Kanook: Please expand on 1)

15:10 Kanook: Also, please expand on why the [managers] need to manage images

15:11 Trak3r: main 1 or why 1?

15:11 Kanook: Finally, please expand on why the images need to be stored in a DB

15:11 Trak3r: one topic at a time

15:11 Kanook: why 1). Why does [client-a] mean that we have to put all images in a DB

15:11 Kanook: Ok

15:11 * Kanook is longing for good wiki software

15:11 Trak3r: [client-a] does not mean we need to put all images in the database. i'm saying [client-a] requirements are a good reason to do so.

15:12 Kanook: ok

15:12 Kanook: what are the [client-a] requirements?

15:12 Kanook: All I know is that they *currently* have some images in their DB

15:12 Trak3r: [client-a] requirements are to upload widget and cobrand images at their whim and see them live immediately

15:12 Kanook: ok, that's fine

15:12 Kanook: now, next topic

15:13 Kanook: Does it make sense to store these in a database, as opposed to a file system?

15:13 Trak3r: yes (reasons...)

15:13 Kanook: ok

15:14 Trak3r: 1. they have to be centralized
2. the only thing we have centralized is the database
3. if we centralize a web server, it becomes a single point of failure
4 .that would require a servlet that wrote files to the disk, which i consider to be a tad more problematic than writing to the db

15:14 Kanook: Ok, reason discussion begins.... now

15:14 Trak3r: one at a time

15:14 Kanook: yup

15:14 Kanook: r1) What does "centralized" mean?

15:15 Kanook: And why do they "have" to be it?

15:15 Trak3r: centralized means a single source to which to upload -- we can't upload to load balanced web servers if we use the file system since they each have their own disk

15:15 Kanook: right

15:16 Kanook: r2) We also have a centralized file share

15:16 Trak3r: 6. revision control (i sent this for logging, we'll come back to it)

15:16 Kanook: you missed 5

15:16 Trak3r: that centralized share is only for the two db servers -- there's no secure channel from the web heads to the disk array

15:17 Kanook: erm

15:17 Kanook: let me verify that

15:17 Kanook: I know for certain that I have some shared folder between all servers

15:17 Kanook: and IIRC it's on the "disk array"

15:18 Trak3r: yes your and mine accounts are authorized to access it, the iis account is not (and for good reasons)

15:18 Kanook: Ok, that may be

15:18 Kanook: however

15:18 Kanook: is there any reason why we couldn't create a folder on the shared disk array and give each IIS account read access to it?

15:19 Trak3r: which iis would write to it for uploads?

15:19 Kanook: hrm

15:19 Kanook: well, IIS wouldn't... we'd have to build something to accept the uploads & write them to the disk

15:19 Kanook: (ie: a servlet)

15:20 Kanook: (alternately, we might be able to use our [client-n] servlet, but that's a side issue)

15:20 Trak3r: sounds complicated and excessive so far

15:21 Kanook: But more complicated/excessive than writing to the db? :-P Maybe, maybe not.

15:21 Trak3r: what do we need? a centralized storage device accessible from the web servers. do we have one? yes, the database. why try to create another one via the disk?

15:21 Kanook: yes, that is a good point.

15:22 Kanook: One reason: it might take up excessive amounts of disk space for an already-large db

15:22 Trak3r: hmm

15:22 Kanook: OTOH, we could probably put this stuff in its own database

15:22 Trak3r: (thinking and clicking ... please hold)

15:22 Kanook: Ok

15:23 Kanook: I do agree with you on reason 1: centralization. The means are still up for discussion but the ends are clear in my mind

15:23 Trak3r: /images is currently 7mb -- a fraction of a fraction of the db

15:23 Kanook: 7mb? That's it?

15:23 Trak3r: yes, but...

15:23 Kanook: wow

15:23 Trak3r: i want to entertain the notion of putting other non-image binaries in there (like shockwaves and power points)

15:24 Kanook: so entertained

15:24 Kanook: However

15:24 Kanook: This is starting to get into my next point

15:24 Kanook: which is...

15:24 Kanook: Why do we want to have *all* images managed by this system?

15:25 Trak3r: 1. to avoid having to manage two image systems
2. so that the [managers] can control anything they wish

15:25 Kanook: Ok, let's talk reason 2) here...

15:25 Kanook: The [managers] managing stuff, while theoretically desirable, has certainly generated a lot of headaches

15:26 Trak3r: that relates to my point #6 -- revision control

15:26 Kanook: And, even when they have the power, they still often delegate it right back to us... as evidenced by the freaking articles that they sent me just today

15:27 Trak3r: yup, and that's something we're going to beat out of them come hell or high-water (note: i have no idea wtf that cliche means)

15:27 Kanook: re: version control. I'm not just talking about being able to undo their fuckups. I'm talking stuff like "how do I upload a file again?". Also, "do I want to roll this back or not?"

15:27 Kanook: high water is probably a Noah reference

15:27 Trak3r: ah

15:28 Trak3r: yes the admin interface will have to be fairly idiot proof, allowing them to roll forward/backward revisions, but NOT do stupid shit like rename images or move them to different directories, since their references will still be hard coded in the [html] templates

15:28 Trak3r: ...

15:28 Trak3r: ... unless we use some sort of image type class for dynamicicity [yes i just made up a word]

15:28 Kanook: I'm not confident in the [managers] being able to managing anything. I'm also not confident in us being able to teach them anything... after 5 years of [client-n] we've still had limited success.

15:29 Trak3r: ok, duly noted that the [managers] are morons and a liability. i'm not going to change your mind on that. next topic :-)

15:29 Kanook: Ok next topic...

15:30 Kanook: (note: we could not give the [managers] the power to manage images; that doesn't necessarily rule out an image database)

15:30 Trak3r: (i am SO going to blog this conversation (with names changed to protect my ass))

15:30 Kanook: heh

15:30 Kanook: Next reason: > 1. to avoid having to manage two image systems

15:30 Kanook: That's a noble goal on the surface

15:30 Trak3r: i'm superficial :-)

15:31 Kanook: However, it begins to worry me that we'll be 100% dependant on whatever we (have time to) build

15:32 Trak3r: well we obviously wont migrate over until the system is satisfactory

15:32 Kanook: but here's the thing:
1) we'll need some sort of system in place for [client-a], soon
2) We'll need plenty of time to build a system that does *everything* we need it to do, for all clients

15:33 Kanook: [client-g] is at the forefront of my mind in this respect

15:33 Kanook: They ask for some stupid stuff when it comes to images & flash files

15:34 Trak3r: 1. not necessarily. image uploading is not a requirement for this launch, just early next year. it will continue to be done manually for now.
2. i don't think it will take as long to build as it will to design ... which is what we're getting a start on here

15:34 Kanook: 1) good

15:34 Kanook: 2) what I'm worried about is the "oops, we forgot about that"s that crop up



15:35 Trak3r: ha, that's a worry of mine with every patch that hits the production server :-)

15:36 Kanook: I really don't want to be clicking a form a thousand times to upload 90 new shockwave files when [client-g] decides they want to renew their demos

15:36 Kanook: I'd rather just copy the files over

15:36 Trak3r: well, the point is -you- wouldn't, [manager] would

15:37 Kanook: But that's wishful thinking. For starters, *all* the [client-g] content changes get delegated to me

15:37 Trak3r: that's one of the points of this project: to get trivial crap like image management off your plate

15:37 Trak3r: maybe next we'll move templates into the db and the can fuxor them too ;-)

15:37 Kanook: Sometimes it's not a trivial thing though

15:38 Kanook: Here's another point: it's much harder to synchronize database records between servers than it is to fling files around

15:38 Trak3r: with the separation of verbiage and formatting templates, we could [relatively] safely let the [manager]'s munge the verbiage ones :-)

15:38 Kanook: such as Dev->Stg, stg->prod, prod->stg, dev->[client-g]

15:38 Trak3r: hmm

15:38 Trak3r: let me mull that one while i piss

15:38 Kanook: Yes, but that's a pipe dream for some other century

15:39 Kanook: Ok, I'll recap while you do:

15:39 Kanook: We've already established that we need an image management system that's accessible to someone other than you and I

15:39 Kanook: [client-a] will be the #1 user of the system.

15:39 Kanook: The [managers] may or may not be #2

15:40 Kanook: The system needs to be fault tolerant

15:40 Kanook: (which generally means load balanced and failed-over)

15:40 Kanook: I think that's it

15:41 Kanook: my points:

15:41 Kanook: 1) Not *every* image needs to be managed this way

15:41 Kanook: 1a) Most images don't change at all

15:41 Kanook: 1b) Some images change in unexpected ways

15:42 Kanook: (1b sounds goofy; let me know if you want an explanation)

15:42 Trak3r: sure, fire off some examples

15:42 Kanook: Ok, ex 1:

15:42 Kanook: last night, [client-g] sent me a crap load of Shockwave files to add to the site

15:42 Kanook: (29mb zipped; no idea how much they were unzipped. about 90 files total)

15:43 Kanook: So I add them to my server

15:43 Kanook: I change the name of the directory they're sitting in a couple of times (I wanted to pick a meaningful name)

15:43 Kanook: Then I change a JS file to point to the right directory.

15:44 Kanook: Then I check my server to make sure they're working properly

15:44 Kanook: It looks good, so I send it to the [client-g] server

15:44 Kanook: THEN I get word that what the [client-g] guy sent me was incorrect...

15:44 Kanook: some files changed, and some files were to be deleted

15:44 Kanook: so, I wipe the directory clean, and wait for him to upload the new set

15:45 Kanook: I copy those over to my server, test, and then copy them to [client-g]

15:45 Kanook: there they sit

15:45 Kanook: They're not checked in, because they're not approved; they could be changing at any time

15:45 Trak3r: these are quality use cases

15:46 Kanook: once they're approved, I'll add them to VSS and then upload them in a patch. I can do that without asking you to run any SQL on the db

15:46 Kanook: That's ex 1

15:46 Trak3r: so what i'm hearing is we might need...

15:47 Trak3r: - a command line tool for putting a file into the db
- a command line tool for copying a file-record from one db to another
- a way to batch both tools

15:47 Kanook: what we *need* is something for [client-a] to upload their images to. What we *have* already is good enough for everyone else

15:47 Trak3r: i disagree on your second point

15:48 Kanook: Ok, how so?

15:48 Trak3r: 1. image maintenance is a burden to us
2. images bloat the drops

15:48 Kanook: 1) Image management is a burden, yes. However, I think moving them to the database would *increase* the burden.

15:49 Kanook: (that is, moving *all* the images)

15:49 Kanook: obviously, there could be some cases where non-[client-a] images would be best handled by the [managers]; I'm OK with delegating that to this new image database

15:50 Kanook: However, I think it is in our best interests to *not* apply this system to every image

15:50 Trak3r: how about you write a plug-in for windows file explorer that looks and acts like a file system folder so we can drag and drop files into it but it really puts them into the database? ;-)

15:50 Kanook: Ok I'll get right on that :-P

15:50 Trak3r: hmm, ok...

15:51 Trak3r: so perhaps we're considering using [client-a] as a burn-in period for this system at first, leaving all other clients with the file based system

15:51 Kanook: File-based image management is wonderfully KISS for images that don't change often.

15:51 Kanook: Yes

15:51 Trak3r: plausible, but has a few caveats

15:52 Kanook: It does violate the "1 system" rule... but I think we'd be crazy to jump in headfirst

15:52 Trak3r: [client-a] image upload interface is going to be severely limited -- when the upload a widget image, we're already going to know where it goes and to which widget it's mapped, so they wont get the tree-structure image-naming support we'll need for later migration

15:52 Kanook: why not?

15:52 Kanook: Who says we can't build it and not use it right away?

15:52 Kanook: And who says we can't add it on later?

15:53 Trak3r: when they upload an image for widget #13, that image is going to be named 13.jpg and go into /images/widgets -- they have no say in that matter

15:53 Kanook: ok...

15:53 Trak3r: we can add it on later, but [client-a] will not use it and therefore can't be considered a "burn test" for it

15:53 Kanook: that's fine

15:54 Kanook: I'm not worried about the system not working as designed

15:54 Kanook: I'm worried about the system not being designed to do what we need it to do

15:54 Kanook: (or involving much more difficulty to get it to do what we need)

15:55 Trak3r: what [client-a] needs it to do is but a small subset of what we need it to do, so we can't consider them much of a proof of concept

15:56 Kanook: Ok

15:56 Kanook: So where does that leave us?

15:57 Trak3r: it leaves us with a few more questions than answers, but that's a good stopping point for today. i see by the time stamps we've spent exactly one hour on this conversation

15:57 Kanook: hrm

15:56 Kanook: BTW, what do we "need" it to do, beyond what [client-a] needs?

15:57 Trak3r: beyond what [client-a] needs:
- tree structure
- naming of images
- revision control

15:58 Kanook: Note that we have all of those right now :-)

15:58 Trak3r: minus the "centralized" bullet

15:58 Trak3r: ---------------------------------------------------------------------------

15:59 Kanook: yes, but right now "centralized" is not a huge concern... the web servers pull pretty much the same copy of the file from Blue on a re sync.

15:59 Kanook: don't you "------" me when I'm still typing :-P

15:59 Trak3r: LOL

16:00 Trak3r: i hear ya. we'll pick this up again soon.

"Sho ga nai" means "It can't be helped"

Peter Payne writes, "One interesting social concept I see at work a lot in Japan is the idea of 'gaman,' which means to endure or to tolerate something that's difficult to bear. The idea that is that if there's something you don't like around you, it's better to endure it stoically in an act of self-sacrifice rather than act immediately to change it. We see this every day: my wife and I will go to a restaurant that's much too cold, yet no one speaks up to ask the staff to turn the air conditioning down, preferring instead to tolerate the unpleasant situation. [...] Gaman is something that parents strive to teach to their kids at an early age here, since there are many situations when children need these skills here. The idea of an employee sacrificing himself for the good of his company or of a wife looking the other way when her husband has an affair are linked to this concept. There's a phrase the Japanese use quite often which reflects this tendency to endure something rather than change it: sho ga nai (also shikata ga nai), which means 'It can't be helped.'"

Wow. I was rather appalled when I first read that. How can you run a successful company when the employees are conditioned not to speak up about what's broken? How do you prevent embezzling when the accountants are conditioned to ignore rather than investigate or dispute discrepancies in the books? A dictatorial boss might see this as a utopian situation where drones perform their work with no questions asked, but that just wouldn't fly with me and my team. My employees' opinions are critical to my decision making process. I rely on them to fill-in the gaps of my knowledge and experience and to speak-up when they don't like something. However, Japan has certainly produced a plethora of successful technology companies, so I'm probably missing a piece of this puzzle. Would anybody care to clue me in?

"And you trust it to do that?" (another Eclipse anecdote)

An old friend stopped by the office today for an interview. During the Q&A I showed him Eclipse, and I explained some of the refactoring features. "This one," I said, "let's you move a method from one class to another, and all the dependencies are automatically updated." His response was, "And you trust it to do that!?" It just makes me wonder what sort of shoddy IDE's he's been using at his current job. Ha ha!

Ch-ch-ch-changes

This morning my colleague and I were discussing some code he had recently committed. Since we are both working on intertwined projects, I found myself stepping over (and into) his code, and when I noticed his "style" of doing things differed slightly from mine, I called him on it.

In the end, he made some changes, and I made some changes, and the topic of what to do about legacy inconsistencies came up, and he offered up this gem:

My philosophy is:
1) Change what you do in the future first.
2) If time permits, and the change is easy, change code at or near code that you're already working on next.
3) If you're dying of boredom, change old (unused?) code that's not being worked on.

I can agree with that. It's essentially what I already do, but seeing it in writing made it somehow more officious. It's department policy material if I ever saw some.

"If it ain't broke" (an Eclipse story)

Here's a little anecdote in relation to one of my old rants, heavily edited to clean up the vulgar language and abyssal spelling and grammar mistakes I sometimes make over instant messaging...

Kanook, "I get a warm fuzzy feeling every time Eclipse correctly guesses which variables to use as parameters when AutoCompleting a method call."

Trak3r, "Yeah Eclipse is the bomb; I am SO glad we switched."

Kanook, "I suggest a new credo: If it ain't broke, you probably don't know how broke it really is."

I have to admit, he makes a good point.

SpaceShipOne captures X Prize

They did it! They actually did it. They spent $30m to win a $10m prize, but an amazing precedent has been set. A small team of eccentrics with incredible talent and a tight budget proved that space exploration doesn't require a million-man agency with a trillion dollar budget.

This parable translates to the software industry as well. The similarity to my current situation is almost comical. My current company (small, cheap, and successful) compared to my last company (big, expensive, stagnant) is night and day. Companies can grow so big and so bureaucratic that they can no longer get anything done. Sometimes it takes a small, talented, ambitious, and focused team to shake things up a little.

Firefox vs. Slashdot

For the love of all things holy, somebody please fix something!

I switched to Firefox a while back. The details of my trials and tribulations are in my personal blog. The one debacle I was unable to overcome is how hideously Firefox renders the Slashdot site!

Then last week my partner in crime sends me this thread letting me know that I'm not the only person suffering this great injustice.

Woe is me. I'm not going to join the debate as to which party is in error, I just pray that one of them would step up and take some initiative! Sigh.

The two reasons I haven't switched to Gmail

Yahoo! Mail Plus still has two features that I require from a web-based e-mail interface that Gmail hasn't provided:

1. From: Spoofing -- I can send mail from my Yahoo! account and the recipient will see it as "From:" my business account. Gmail only allows you to specify a different "Reply-To:" address, which isn't the same thing.

2. Check Other Mail -- I can "harvest" e-mail from several other POP3 accounts into my Yahoo! Mail and respond to them through Yahoo! and (referring to the prior point) have them appear as if they came "From:" the respective accounts.
   

If Gmail ever adds this functionality, I'll switch in a heartbeat.

Obfuscating the package structure for clients

So you've got yourself a nice clean package structure. You're slowly building up your application organically, adding features here and there, driven by customer demand. releasing early and often, covering all the hip methodologies of the day. Then along comes a client who wants to make changes so drastic that they warrant sub-classing. We'll call this fictional client Acme. Let's say one of the classes to be altered is:

net.rudiment.calories.Calculator

So you make Calculator abstract, or an interface, and you create two sub-classes:

net.rudiment.calories.calculators.Generic
net.rudiment.calories.calculators.Acme

Seems simple enough. Perhaps years from now you'll have quite the collection of client-specific calorie calculators:

net.rudiment.calories.calculators.Cyberdyne
net.rudiment.calories.calculators.Hudsucker
net.rudiment.calories.calculators.Kumatsu
net.rudiment.calories.calculators.Oscorp
net.rudiment.calories.calculators.Tyrell
net.rudiment.calories.calculators.UAC
net.rudiment.calories.calculators.Vandelay

But what if calorie calculators aren't the only part of the system that gets commonly personalized for clients? What if you've got a few dozen classes spread out amongst the enormous source tree that have been abstracted and sub-classed like this? That makes it a little tedious to do site-wide client-specific changes. For example, if you needed to work on all the Acme-specific classes, you'd have to search them all out. It might be a little more convenient if you'd started with a structure like this:

net.rudiment.clients.acme.calories.Calculator

This also makes it easier to clean-up the code tree if you ever lose Acme as a client. You simply delete the root of the Acme-specific classes and presto, all gone. However, it makes it tricky to alter the abstract super-classes when you need to review and test all the sub-classes to ensure nothing broke; you have to search them all out rather than have them all conveniently right there in a single sub-package.

My current project is nearly five years old and contains over three-thousand classes in over six-hundred packages. Over the years we've used both of the techniques described above. And, as I explained, both have their drawbacks.

I'm curious to hear your opinions on the issue.

Checks and Balances: Don't trust yourself

I consider myself to be a pretty smart guy. Perhaps that's a little conceited. But, one of the reasons I consider myself to be so smart is that I don't trust myself to always make the best decisions. Whenever I have to make an important decision, and I have the luxury of time, I bounce it off my right hand man. More often than not, he's eager and willing to disagree with me. Debating the issue with him sometimes opens my eyes to a point of view I might not have considered. I didn’t hire him to be my lackey; I hired him to compliment my skill set.

My second in command reminds me of myself a decade ago; excited about new techniques and technologies and anxious to apply them. I, however, have become the old codger I used to despise; always choosing the safe and established route and thinking very long term about decisions, trying to see the big picture. How boring I have become. But I have much greater responsibilities now. The success of my company and its products depend on me making the right choices.

I love being the man in charge. I thrive on the responsibility, and I’m confident in myself to make the right choices. However, I’m modest enough to realize that I don’t know everything. Listening to somebody that disagrees with me helps me to make a better informed decision. I don’t become defensive; I become a sponge. I soak it up. It’s not my goal to change the other person’s mind, so arguing with them is futile. My goal is to succeed, so I listen and consider their comments.

A prime example of my mentality is this very blog. I don't post these rants because I think I know everything and I think I’m doing you a favor by sharing my wealth of wisdom. I post my ideas and opinions here with the hope that somebody will disagree with me. Not only disagree, but make an intelligent rebuttal. Open my eyes. Show me the error of my ways. Prove me wrong.

Are you up for the challenge? If so, read my last few posts and speak your mind. I'm dying to hear your opinion. Enlighten me!

How long will it take you to build this?

My colleague and I get this question a dozen times a day.

One day the big boss man pulls me aside and asks, "When I ask you how long it's going to take to build something, you tell me a long time but finish it quicker than expected. When I ask your colleague the same question, he tells me it will be quick, but takes a lot longer than expected. What's up with that?"

Well, there's several reasons, of course. First of all allow me to explain why we give differing time estimations. I can't read my colleague's mind, so I'm speculating on his thought process, but I believe that he answers the question literally. He hears the word "build" and interprets it as "code". In his mind, he believes that he can "build" the project in so many hours, and he does. However, what the boss is really asking is how long it will take to discuss, design, code, test, review, change, discuss again, redesign, re-code, test again, package it up, document it if necessary, and deliver it to the client. These are all the factors that I consider when asked "how long?" which is why my time estimations are so much larger than my colleague's.

OK, so why then do I overestimate and he underestimates? Again, there are several factors. He underestimates because he is again answering the question literally. My colleague suspects it will take him a certain amount of hours to complete a project, and he answers the question as if he were going to devote said hours to the project. However, what the boss is really asking is, "When can you finish this new project while still contributing to all your current projects and compensating for forthcoming interruptions such as impromptu meetings, phone calls, and cleaning the latest Outlook virus off my system?" These are the reasons that I overestimate; I anticipate a plethora of distractions, and when I'm lucky enough to get an hour or two of solid uninterrupted work done, I manage to deliver projects ahead of my anticipated schedule.

The real humor in all this is we've been iterating this cycle for so many years now the big boss man, admittedly, interpolates our time estimates and expects projects in half the time I declare and twice as long as my colleague proclaims.

Exception Obfuscation: Hiding what really went wrong

In my last entry, I discussed my colleague's plea to pollute [sic] our precious four-year-old over-one-hundred-thousand-lines JDK-1.1-compliant code base with some fancy schmancy new collections APIs. Well, this week he's at it again with what he calls "exception chaining" and I call "exception obfuscation."

Exception

public Exception(Throwable cause)

Constructs a new exception with the specified cause and a detail message of (cause==null ? null : cause.toString()) (which typically contains the class and detail message of cause). This constructor is useful for exceptions that are little more than wrappers for other throwables (for example, PrivilegedActionException).

Parameters:

cause - the cause (which is saved for later retrieval by the Throwable.getCause() method). (A null value is permitted, and indicates that the cause is nonexistent or unknown.)

Since:

1.4

http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Exception.html#Exception(java.lang.Throwable)

Yup, it's another whiz bang new feature in the JDK 1.4 API. You can take a perfectly good exception and obfuscate it inside another exception. And what for pray tell does my colleague want to use this shiny new bauble? He wants to take all those pesky methods that declare multiple exceptions, like IOException and SocketException and SQLException and wrap them up into a neutral exception like SomethingBrokeException so that the calling classes don't have to declare catch blocks for all the possible types of exceptions; they can just catch the one generic exception.

"Well," you might ask, "why doesn't he just catch Exception? That will satisfy all the declared exceptions." Yes, but it will also encompass some undeclared exceptions, and when they get thrown, we want them to propagate, because they might indicate the system is in an unstable state, and in some cases it's better to crash than to keep processing and possibly cause even more damage.

OK, so he's got a valid complaint and a viable solution. What's the problem? The problem is what about calling classes (either present or future) that care about the type of exception thrown? What if the calling class wants/needs/knows-how to handle IOExceptions differently from SQLExceptions? Now it's catching a generic SomethingBrokeException, but it cares about what broke!

"Well," my colleague argues, "if the calling class really cares about what type of exception was the root of the problem, it can call the new getCause() function." So the point is that we can replace a try/catch block with an if/else block, and the calling classes that don't care about the problem can just catch the generic exception and thus save the programmer a little typing. Poppycock, I say!

I'm pretty big on analogies. Being a CTO, I commonly have to explain complex technological concepts to non-technical people (like my clients, and the CEO), and I always do so through analogies. So allow me to address this issue likewise: It's like you're a general contractor and I've hired you to build a new room onto my mansion. You've fitted the new room with contemporary five-prong electrical outlets so I can take advantage of the next generation of appliances. But since I don't have any such appliances, you're also offering to sell me some new-five-prong-to-old-three-prong adapters. Gee, thanks a lot! Now I've got a house with two different types of electrical outlets, and I have to keep a box full of adapters in the closet for special occasions. You've taken something that worked just fine, didn't make it work any better, and just made it harder for me to use with my existing tools, so that it might be easier to use with future tools.

If it ain't broke, don't fix it

Last week I had a little debate with my right hand man. We finally took the plunge and switched from Microsoft J++ to Eclipse as our primary IDE. This move required a massive change to the source tree due to a bug in Eclipse where it refuses to compile if you have a subdirectory with the same name as a class; but I digress. This isn't another blog rant about what's good and bad about Eclipse. We both put all our other work aside and spent the entire day performing a massive reorganization of the code tree... and this was a major pain in the ass thanks to Microsoft Visual Source Safe; but again I am straying from the main topic. We made Eclipse happy, and it returned the favor.

So back to the debate. J++, being the dinosaur that it is, compiles and conforms to the 1.1 version of the JDK. Eclipse, on the other hand, supports the 1.4 version of the JDK. So my right hand man sends me an instant message telling me that he's going to start using the newer 1.4 version of the collections API. That set off the klaxons in my head.

What's the harm in using the newer collections API? Well, before I answer that question, let me first ask what's wrong with the older versions?

1. Are the older collections broken? Nope.
   
2. Do the newer virtual machines support the older collections? Of course.
   
3. I am aware that the newer collections offer [theoretical] performance gains by sacrificing thread safety, but our system has no performance problems, so I'll refer you to the quote by Joseph Newcomer in my prior blog post.
   

So there's certainly no need to switch to the new API, but why wouldn't we? Well, here are my reasons:

1. Inconsistency within our code base. We have over 115,000 lines of Java code in our product. It's over four years old. If we start using the newer collection API now, there's inevitably going to be a situation where we are returned an old collection from and older object and have to convert it into a newer collection in order to pass it into a newer object. That means we'll need to build, or find and possibly license, a set of utility classes for converting from one collection type to another. This is discussed in an old article from Sun entitled Converting Between Old and New Collections. This headache can be avoided by continuing to use the older collections.
   Side note: I have another mantra about not passing generic collections in and out of functions, but that's a future rant.
   
2. Portability. Yeah, the major platforms have ports of the latest JDK versions, but there's always the possibility that we'll have to port to a platform that's a little behind the curve. If that scenario ever surfaces, we'll have to re-code all the classes that take advantage of the shiny new toys in the later versions off the JDKs.
   

So I had to exercise my seniority and put the kibosh on the new collections.

Quotes to Code By

Whenever I come across a quote that strikes me as insightful or witty, I print it out in a large font and stick it on my wall for visitors to appreciate. Here are some of my favorites...

Inspiration...

Good, better, best, never let it rest, until your good is better, and your better best.
-- Shepparton-based steel manufacturer J. Furphy & Sons

Second place is the first loser.
-- Anonymous

I want to achieve as much as I can in this sport, tactically outwitting the opposition to win. I want to time trial as fast as I can physically go. I want to be a key member of a strong team that can ride aggressively and win and make other riders suffer in pursuit.
-- Emma James, April 2002

Wisdom...

We can take the time to fix it now, or we can have this discussion again next month.
-- Thomas E. Davis

There is never time to do it right, but always time to do it over.
-- Anonymous

The trouble with doing something right the first time is that nobody appreciates how difficult it was.
-- Anonymous

With each passing year I realize that the prior year I didn't know jack shit.
-- Thomas E. Davis

Growth for the sake of growth is the ideology of a cancer cell.
-- Edward Abbey

...too many cooks working on code in the early days causes bad architecture. Software development works best when a single person creates the overall architecture and only later parcels out modules to different developers. And if you add developers too fast, development screeches to a halt, a phenomenon well understood since 1975.
-- Joel Spolsky (referring to Brooks’ Mythical Man Month)

Think about the design decisions you made a year ago.
Think about how ignorant they seem in retrospect.
Think about the decisions you are making today.
Think about how they will seem a year from now.
-- Thomas E. Davis

Comprimises...

The cost of flexibility is complexity. Every time you put extra stuff into your code to make it more flexible, you are usually adding more complexity. If your guess about the flexibility needs of your software is wrong, you've only added complexity that makes it more difficult to change your software. You're obviously not getting the payback. The alternative is to use the Extreme Programming approach and not put the flexibility in at all. Extreme Programming says, since most of the time we get it wrong, just don't put the flexibility in there. If you strive to keep your design as simple as possible by avoiding speculative flexibility, then it's easier to change the code because you have less complication to deal with. The code is easier to understand and easier to change. As a result, you can make changes much more quickly.
-- Martin Fowler

Simplicity is about acknowledging the tricks exist but not using them.
-- Kent Beck

Optimization matters only when it matters. When it matters, it matters a lot, but until you know that it matters, don't waste a lot of time doing it. Even if you know it matters, you need to know where it matters. Without performance data, you won't know what to optimize, and you'll probably optimize the wrong thing. The result will be obscure, hard to write, hard to debug, and hard to maintain code that doesn't solve your problem. Thus it has the dual disadvantage of (a) increasing software development and software maintenance costs, and (b) having no performance effect at all.
-- Joseph Newcomer

Good. Fast. Cheap.
Pick any two!
-- Anonymous

I have worked with people who thought 80 hours a week made them better programmers, but from my perspective, they were so worn out that they got less done. Managers saw the long hours and were impressed by their dedication and loyalty, but all I saw was people spending hours on trivial problems because their brains were so fogged they were incapable of the five minutes of thought that would have pointed out a better solution.
-- Anonymous

Miscellany...

If you call its code,
it's an API.
If it calls your code,
it's a framework.
-- Simon Brunning

...while databases are slower, in many cases much slower, than procedural code, they have an important property: they can be used to answer unanticipated questions acceptably quickly. How quickly is acceptably quickly? Well, if the database can come back with an answer faster than it takes a skilled programmer to come up with a special purpose program to answer the question, it has done its job.
-- Anonymous post to slashdot.org

A disgruntled coworker...

"All this could be much better" should be our company motto.