Corporate Coder

"Can you make that hidden button always visible?"

2009-09-02T06:58:00.001-07:00

What an easy request. Right? Just change the CSS to display:inline. An easy fix. In fact what could be easier?

Well maybe not so easy. What if that button does nothing until certain criteria are met (which is why it was hidden)? Often users do not understand that cosmetic changes have functional implications. Worse than that, changes are often made to design without asking about the functional implications.

For example, "What does it mean that this button is visible?","What now happens if someone clicks on a button which shouldn't be clicked under these cirsumstances?", "Should we throw an alert error?","How about an inline error?"

Why should you show a button which cannot by definition of the current spec even do anything at this point in time?

It is getting to the point where I just say "Yes".

Creating solutions to non-existent problems

2009-09-02T06:43:00.000-07:00

People love Excel. People hate Excel.

People want a solution which is not Excel. Let's face it, there is very little color aside from green. It is a pain to format cells. It gets boring. We had hoped to move on by using Powerpoint for everything, but we can't expect a painter to do the job of an engineer. So what do we do?

We ask for a custom applications to be built. AHA! We will escape the doldrums of empty cells. We will have buttons, and color, and, and...well what do you have?

If your business process is to manipulate large swaths of data, data that can be altered by a large group of people seldom at the same time, perhaps Excel isn't so bad.

"Time to get to the point," you say? It is simply this: If there is a problem with a business process, creating custom applications is not going to help. If you are tied down by the randomness of excel manipulations and want to have some structure to your process, you can't then demand the ability to use Excel. Excel solves certain types of issues. If it is inadequate, that's fine. It often is. But the alternative is not to complicate the randomness of Excel with the restrictions of a custom application. Applications are built according to processes and workflows, something that is inherently missing from Excel.

In those cases where custom Excel addons are created or custom workbook templates are created, the business process is well defined before development begins. Specs change, features grow, but the over reaching arc should remain consistent.

Most users do not understand this. But if the converse were true well, you wouldn't have any users. A simple example would be selling someone an application in development that would cure cancer, but when you actually delivered the product it ended up making ice cream. It is the same project, but through lack of thoroughness and direction, the outcome varied vastly from the initial proposal.

Since user have little idea of the process of software development it is hard for them to see their little changes as having big impacts. Even in non-coupled, highly modularized code this is an issue if now the separate components are logically contradictory.

VS2005 SP1

2009-09-02T06:24:00.000-07:00

Why does it take over an hour to install a service pack?

Outsourcing interpretation

2009-09-02T06:21:00.000-07:00

You would think spending two weeks in india would really fill me with the desire to blog aoubt my experiences. Both here and my non-work related blog. Surprisingly, two weeks in a foreign country didn't have the impact one might think. But since my return one constant thought has been stuck with me.

As much as software development cannot be compared to assembly line production, outsourcing development cannot be compared to manufacturing outsourcing.

Maybe more on this. I don't know.

C# and Xml Literals

2009-04-08T05:54:00.000-07:00

How do you use XML Literals with C#?

You don't.

You choose a language based on history, need, efficacy, etc. Not just because you feel like it.

And Boom.....

2009-04-07T11:20:00.000-07:00

So I learned today that from now we will no longer code in vb.net and all new projects will be in c#. Why? Well. Because, that's why.

God is an agile developer

2009-04-02T11:28:00.000-07:00

One thing that greatly bothers me in my current position is that no matter how much work I do each day, nothing is ever accomplished. Every project is kept in perpetual development and feature creep is more common than well...anything that is very common. I code. I honestly do. This is why I haven't written much lately. I have been head down barreling forward. Today I have a reprieve. And I also have feature creep.

First of all I am no starry-eyed dream and I realize that feature creep is to be expected in any project. In fact it should be planned for. For most applications there is neither the time nor the knowledge to flush out every possible area of development. However, planning feature creep should be differentiated from scope and purpose changes. Feature creep should be seen as a way to maintain the vitality of an application while also maintaining its core purpose. If the purpose of an application needs to change, a new project should be rededicated. Salvage what code you can and start over. Sometimes it is easier to jump into the sea than to change the course of a sinking ship. This is because feature creep often devolves into a change in the scope of the application itself, often rendering initial planning moot. Design often follows purpose, even if loosely. So such a change can destroy the initial premises of the design of your application. Of course this may not be the case which is why good employees should be not being let out the door. Thoughtful developers will take the time to decide to salvage or ditch.

Feature creep does not have to be a bad thing if it is incremental and time is allowed for each iteration to run its course. However, feature creep becomes detrimental when it becomes a hindrance to the release of the software. Often at my company, each iteration of a feature results in modification of that feature and the addition of new features. This happens in a purely development, sometimes theoretical place. Unfortunately the view of software is in its totality. Only the perfect idea of the software can be released. This waiting for perfection often adds two things to the project - an invisible deadline and failure.

The deadline vanishes since each new request pushes the software date out further and further. Failure then arises from the inevitable waterfall approach that ends up becoming the de facto development model. So much time goes between project launch and project end that several things happen:
1 - People wonder why this software took so long when it seems so simple
2 - People wonder why things do not behave exactly as they expect

The longer a project goes without business user mediation, the more likely it is that the same users will forget why they have asked for something as well as forget how much they asked for.

What becomes necessary and what becomes auxiliary is blurred since now both are merged into this one application. Unfortunately there is no way to get this time back. Everyone is unhappy and nothing gets done.

Oh and the title? Read the creation story in the bible, if you still don't get it. I can't help you.

Posting absence

2009-04-02T10:53:00.000-07:00

It is hard to continue writing about something that's not all that important unless you are an egotistical maniac. But I am not dead. The work I have been doing has required little comment since it is mostly fiddlying with a 3rd party app.. I bascially am configuring our new search engine. So, um, yeah.

It's all a numbers game

2009-03-11T11:56:00.001-07:00

At what level does tagging a document in a hierarchy become meaningless. A break down of our term usages ranges from using the term "United States" 585 times which is 2.178 percent of our total documents to "Fund Strategy - SWF" used once for 0.004 percent. Now, this is all the terms against all the documents. This list of 1639 terms is usually broken into several hierarchies. Classification can range from 20 percent to less than a hundreth of a percent.

What is tragic and comic all in once is that in the tagging for case information the most frequent tagging is "No Tag" with 23% for capability hierarchy and 24% for industry hierarchy.

If such a high percentage of documents can go untagged, is it then necesary to tag cases at the opposite extreme. There is one case tagged "Enterprise ASP" for 0.003 percent. What makes it worse is that if such a tag is used so infrequently, what is the likelihood that someone will A-Search for the term or B-Know what the abbreviation is even for.

If these were internal use only taxonomies used to maintain directories and file manipulation, then such organization might matter. Unfortunately, these are customer facing choices and complexities.

What is getting confused is a taxonomical search vs keyword search. At the level where terms are arbitrarily added onto documents, you have moved from a purposeful taxonomy into a scattered keyword. While it does not matter how you find the document from a user's perspective, different strategies should be used for tagging data when filling out taxonomies vs keywords. We expect rigidity in taxonomies and fluidity in keywords. Right now we have neither.

And then there were 3

2009-03-11T11:21:00.000-07:00

As we move forward implementing a new search engine, it is becoming increasingly clear that organization of the data to be indexed is quite a mess. Although the documents themselves are fine, how metadata is attached and ordered is scatterbrained to say the least.

One of the areas which needs the most improvement is in classification of documents. Each document contains metadata for several hierarchies. In order to discuss this in a simpler fashion I will work on one hierarchy -- Industry. The industry hierarchy helps classify documents according to which industry they belong to and at which level. There is a possible depth of 6 in the hierarchy. So, a document can be tagged in the following manners:

Industry: Consumer Products
Industry: Consumer Products\Food
Industry: Consumer Products\Food\Beverages
Industry: Consumer Products\Food\Beverages\SoftDrinks

It is not mandatory that each document goes to the 6th level because articles can be more or less specific regarding an industry. The previous post regarding searching has already pointed out the problems with our current taxonomy. This post will expand on that one by referring to real current problems.

When you look at the distribution of the number of terms at each level of the hierarchy it breaks down like this:

Hierarchy	1	2	3	4	5	6
Industry:	14	96	275	380	412	421
Capability:	11	62	164	215	215	215
Cases:	15	99	292	412	456	464

In each of these levels, the higher levels are included. These numbers are the count of terms in each level. In each hierarchy you can see that level three (or possibly four) is the climax of term creation. After that, the number of documents actually tagged 4,5, and 6 levels is very low. The documents with that level of tagging doeas not even reach a tenth of one percent in the total number of documents. It is highly unlikely that a user is searching through navigation to get to that level of granularity. In addition the UI for such granularity creates a nightmare for navigation. Finally, this granularity creates a user base which tries to be a specific as possible thus reducing recall (the number of hits returned from search).

It is that last issue which our current system faces. In order to increase recall, expansive dictionaries have been created in order to increase recall by expanding the query. Of course, this has a negative effect on precision. And while recall is now increased, the drop in precision means more sifting for the user. Add to this poor relevancy ranking and search result becomes next to meaningless.

Perhaps it is best to leave heirarchy tagging only to 3 levels. After 3 levels the hierarchy taxonomy serves no purpose.

If a document needs more information not contained in the document itself or in the metadata already gathered, it would be possible to add another field to the document such as "Industry Tags". This could be free form modifiers and text supplied by authors and knowledge managers. Currently only the knowledge managers tag and classify the documents. This leaves alot of room for interpretation in terms of specificity. And as we mentioned before, the taxonomy is out of control. A knowedge manager can add new terms if that person deems it necessary, and hen dealing with very low levels of granularity, that situation is likely to arise.

Instead, an Indutry Tag field is used (mostly by authors and users) to help define the document. Users working in the same area with the same businesses are more likely to define documents in a similar manner. Authors are better able to summarize or keyword-ize their own works, at least better than a disinterested 3rd party Knowledge Manager.

Stopping at 3 or 4 levels creates a better search experience since now, the navigation terms contain fewer terms and more documents per term. The industry tags, now set by actual consumers of the text can then narrow down the results within that more general hierarchy.

A Mystery

2009-03-11T05:48:00.000-07:00

I don't know why, but this makes me feel all warm and fuzzy inside. I haven't posted in a while. I have been trying to get started with JQuery. Trying to come up with an Ajax solution, I remember the great XDocument and the ability of VB to include literals in code.

Public Function GetEmployee(ByVal employee_code As String) As XDocument
   Dim query As XDocument = <Root>
     <%= From e In New EmployeeDataContext().employees Where e.employee_code = employee_code Select <name><%= e.last_name %></name> %>
     </Root>
   Return query
End Function

First Responders

2009-03-04T07:18:00.000-08:00

So you are invited to a birthday part of someone you don't know. You show up late, and as you open the door the guest of honor is blowing out the candles. From about 6 feet away, all you see is the end of the blowing, but notice the candles are not going out. Why?
This is a very simple example, but i use it to highlight something else.
When we encounter a problem in programming, what is our first response? What is our first gut reaction versus what are the first steps we take to resolve the issue?

Our reaction is heavily dependant on our own conditioning and experience both with development in general but also with the technology in particular.

This topic has been in my mind for quite a while for several reasons. The most prominent is this:

Who do you blame when something isn't working?

Essentially there are 3 main sources of potential "bugs" when trying to debug one's code:

1 - Your code
2 - The other person's code
3 - The technology

Unless you are dealign with beta, CTP or RC software, you should look in the order above when trying to fix a bug. This is especially true in my position where most of our code is at a high level where syntax is more likely a problem than stumbling upon some obscure bug for some corner case.

Being a small shop that develops internal busniess applications we are relatively free to use newer technologies in order to brighten an otherwise dark and mundane day. Not all developers are on board with such an approach. Feeling relatively safe in their position doing things they have always done, they are reluctant to move to newer technologies and platforms. For them something new is more work not less. For them typing out 1000 lines of code is not an issue since there is no ultimate goal.

Me, I have youtue to check, people and articles to write, code to explore. I embrace technologies that allow me more free time with a better guarentee of performance. for me, the learning curve is built into project development timelines.

These other corporate coders can't see the ultimate payoff since for them, as each day passes, any learning curve becomes too steep to climb. Learning new technologies is not much different than running. If you run a mile everyday, that mile becomes easier and easier. If you never run, a quarter mile is daunting and the end might as well be 1000 miles away.

Learning new technologies not only helps you learn the technology, but gives insight on how to approach other newer technologies. You apply lessons learned form the learning itself.

The "meta" learning is what is at stake. The corporate coder who rarely ventures beyond his drag and drop world is quick to blame new technologies when they do not behave exactly as expected. They are quick to yell out "Well it doesn't work and there are alot of bugs." These developers know their little world so well, they project that level of knowledge to absolutely everything outside that world. If they can't figure something out, that thing must be broken or too complicated. It is important to prevent that stagnation.

What is the outcome of such a developer on a given project?

I recently had to work with such a developer and here were some of the outcomes:

The project used .Net table adapters for the data access layer. Without going into the pros and cons of the technology 3 of the 4 developers agreed to use this technology. The fourth developer eventually signed on (in words) to use table adpaters to access the data in the logic layer.

First Sign

The very first issue came when in a meeting he said what we were trying to accomplish was impossible, he had never seen it doone and doubt we could do it. What was the task? Create an XML files from the table adapter, read the XML in to a table adapter, insert into the database from the table adapter. This is far from even a difficult task nevermind an impossible one. Within 30 minutes I had a working prototye for him using our project. I had actually created prodeuciton level code in a matter of minues. I am not a genius, that is what table adpaters are used for. Serialized data can be easily transformed into strongly typed datatables, generic datatables or XML.

Second Sign

Everything was going fine. For most people, this is not a sign, but I knew if I wasn't hearing anything, aomething was going wrong. The developer then tells me there is no way to set the connection string at runtime. He had not done any research, he had simply not been able to find a first glance ay of doing it. He had always set the connection string in the same way. Since it apparently couldnot be done in the same manner it was impossible. Without even looking at the code I offered several possible ways to set the connection string. I was able to do this, NOT because I had mastered the technology, but because I had taken time to learn how Microsoft does things. There were just some basic technieus which were probably in use in the table adapters.

Third Sign

One of the strength of the strongly typed datatables is that hey allow direct access to the data columns as row properties. This increased performance by elminating looking up columns by string values. However, when I looked into this developers code he was still using row.item("columnName") as opposed to row.ColumnName. By refactoring out the old way and using the new notation, I increased performance 400%. He just didn't understand how there could be a difference between an in memory object and a lookup. Since he never had to learn, he just didn't bother to learn.

The final sign

This developer had one section of code where he had to do his own data manipulation. And guess what, he reverted to what he knew. I didn't bother refactoring this code because i didn't have time, I just added an abnstracted layer to make it work within the general framework. Sometimes you just don't have enough time.

To Be Continued....(this was getting too long)

Sometimes it's the simple things

2009-02-27T06:59:00.000-08:00

Sometimes the solution is so obvious:

Func<Document, bool> predicate = a => a.FileDate >start_date && a.FileDate <end_date;

BuildAndAddRows(table, predicate);

//inside the function

var query = (from d in this.DocumentList orderby d.VehicleName select d).Where(predicate).Select(a => new { a.VehicleName, a.VehicleID });

Searching for Bibby Fisjer

2009-02-26T07:45:00.000-08:00

I am ramping up to start our search project again. The search engine we are using is a 3rd party tool, so most of the work of the development team is going to lie in coding the front end.

But there is much more involved that just coding to an API. There is the actual business implementation of the search as well as the migration from our old system to the new one.

I plan to document as much of this process as possible here as this project should provide me if not the reader with hours of enjoyment. Here is problem #1:

Jargon

The following refers only to navigational searching, not full-text or keyword searching
Suppose there exists a standardized list of industries which the majority of large businesses use to catalog customers. Let's call it an ISO Industry List. For argument sake, let's say this list has 5 industries listed:
1 - Finance
2 - Tourism
3 - Healthcare
4 - Technology
5 - Manufacturing
These are common industries which are likely to exist into the foreseeable future. They are distinct enough so that two will not merge and become a hybrid, even if aspects of one incorporates features of another.

Let us also asuume a company exists which prefers to use its own jargon to label documents.* For whatever reason, whether valid or not, the ISO codes are not acceptable to this company and so they start creating their own list of industries. Let us assume the decision to do so was motivated by two main reasons:
1 - The ISO codes do not divide sectors appropriate for the business
2 - They just don't like the names in the ISO List.

As a result the company begins producing a list of its own based on what it perceives as industries pertinent to its business. Not having the resources to survey and create their own ISO type list, people being just adding to the list industries they think are valid. They create first pass:
1 - Banking
2 - Tourism
3 - Healthcare
4 - Medical Devices
5 - Manufacturing

For now this appears to do the trick. Then the company gets a new client which does not fit into this list and a new category is create:
6 - Private Equity
And then another:
7- Biotechnology
And another:
8- Pharmaceuticals

But then someone things that a client actually belons in two industries and creates:
9 - Biotech & Pharma.

Each of these categories has been attached to a document in our search engine. But before we get to the actualy documents, let's look how we already create a mess.

Industries are not atomistic

In our list of industries, even before we get to #9, we have created a list of industries which are neither atomistic nor reside on the same level of any reasonable heirarch. Ideally a list like the one above should have each leaf of the tree on the same level. Biotechnology is far more specific than Manufacturing. Creating this disjunction leads to confusion when adding new items since the person adding the new category is uncertain as to the level of specificity they should use.

Ever Changing Titles

Having this confusion leads to categories which more reflect jargon than concrete types. The list starts to grow each time a new category is added. The more industries are added the harder it becomes to determine which are valid and which are invalid. Adding industries for corner cases leads us to create industires for only one maybe two clients.

The documents are tainted

All of this above would not be such an issue if this list were not actually ties to document searching. Creating a list based on such whim creates the following issues when trying to search on the documents:
1 - If the jargon industry is changed, the document must be changed and re-indexed. In a system of thousands of documents there is a large overhead in terms of management and time.
2 - If the jargon industry represents corner cases of clients, the likelihood that someone will search on the word is slim. Although the precision on such a search is high, the recall for the document is low. In addition, documents are classified as different levels of a tree. When recall is high the specifity of the document as it relates to the search may not be clear.

Is tagging hierarchy the answer?

One might think the way to resolve the above issue is to tag each document with a heirarchy. But, besides the overhead of creating that much more data to index it does not solve the underlying problem of non-atomistic industries. A hierarchy can suffer from all of the pitfalls of a jargon induced list.

A way out

One way to solve this issue is to use the ISO list to tag documents. In fact that is the only way out, short of creating a new list formed in the same manner and with the same rigid standards as ISO. What we can then do is create a translation dictionary to translate jargon industries to ISO industries at search time. This allows us to maintain a certain sense of identity to the user while preserving the integrity of the documents. We can then create heirarchies either using the ISO lists or by creating our own without affecting the location of that document or having to re-index. We use these hierarchies to direct the search, but use the levels of the hierarchy to do the actual searching.
This solution solves the two reasons for creating one's own list above. first we can divide the industries by using our custom mapping diciotnary which will allow us to rename the categories. However, at the base level we retain our rigid atomistic separation. What we gain hoever is much more valuable. By using a standardized set of industries we are much less likely to end up with corner cases. An ISO type list has already gone through the rigid scrutiny to a level most businesses can't.

But how can we map the categories?

All of the above sounds great in theory, but the technical part is not so clear. Here is what I propose. Using an ISO list of industries and their hierarchies, we tag each document with the highest level and the lowest level that document refers to. For example:
Document #1
Title: Manufacturing in a Port Modern Era
High Industry: Manufacturing
Low Industry: Manufacturing
This document deals with general aspects of manufacturing so we tag the highest level and the lowest level the same.

Document #2
Title: Computer Chip Processing
High Industry: Manufacturing
Low Industry: Computer Components
This document deals with a specific type of manufacturing so we label it as this level

With our mapping dicitonary we can map over "Chip Processing" to "Computer Components" if all the computer components we ever deal with are microchips. If the company changes direction, we can expand our dictionary or shrink it as necessary. The dictionary allows for many to many relationships and can be used to expand or shrink our search as well. Again we must remember that this is for directed navigational searching and not key word searching. Whereas keywords and full-text searching is a clouded nebulous search, navigational searches should be uniform and atomistic.

*a document is any piece of content we wish to add to our search system

Day or Reckoning

2009-02-26T07:22:00.001-08:00

The Ideal World

The time has come yet again for our annual review cycle. Once a year we have a 360 degree review of peers and managers in hopes of making us all better people. I would say in hope of making better workers, but you will shortly see why this is not the case. At least not for me.

To start with, let me say that in an ideal world, annual reviews are a great thing. Sometimes we get so wrapped up in our work and we fail to see the bigger picture. Sometimes we continue to make the same mistakes or fall into the same patterns because that is what people do. A little heads up a little light on the path is a good thing.

In addition, being able to give feedback on/to your manager is also a good thing. Managers as well as employees fall into the same ruts and mediocre decisions as everyone. Peers are the same way. We can relate issues to peers but at review time we have the opportuntity to track overall trends. If one person tells you something it might be a fluke, but if the consensus comes back with that same issue, you might actually need to reconsider how you do things.

Reviews also provide opportunity for meritocracy to shine through compensation reviews and bonuses.

And all this is great isn't it?

The Real World

But this is not where I live. And if I had to venture a guess, most of the world doesn't live here either.

Let me revisit the points I just made but in the opposite order.

Compensation

Regardless of the state of the current economy, reviews in most companies arrive at the time of cost of living adjustments, not raises. For many reasons, most of them petty and bogus, real raises based on performance can't be given out. Why?
1 - Everyone will know when someone makes more in a current cycle. This causes resentment. People will wat to know who sabotaged them. Even if they are a horrible employee.
2 - Often managers just don't have the ability to determine % increase.
3 - Managers will attempt to be fair offering everyone the same %.

Manager Input

People also aren't going to tell their manager if something is wrong. People leave jobs, they usually don't try to work things out. Persistant and permeating institutionalized issues are not the something the average employee has the will to take on. Instead the the whole thing becomes a farce. After all we all can't be Erin Brokovich.

Peer input

Either they like you or they don't. There is no such thing as an objective review. What can I say about my colleague when I have never seen their work. As a developer most of our value is in the code we write and the applicaitons we produce, but since I neither see the code nor use the application, I am at best guessing.

My Case

And here is the rub and the totality of it all. In may case, there is no review of the actualy work I have done in the last year. No one looks at my code, reviews my scheduling, the level of bugs in my code, etc. No mention of working late, or my sick days. No code reviews.

So what does the annual review become? It becomes a personaliy test.

The review takes on the feel of a counselling session much more than how to become a better developer. Without the proper infrastructure in place, reviewers cling to the only thing they can - how you make them feel. What makes it much worse on the peer side that so many people need to provide input that have either no idea what you do, or have never actually seen your work. Our teams are so small and separated from business users, the actual human interaction performance (the measure of the review at this point) is not actually contributed to by people I have worked with.

Many people I have talked to have been told they need to participate more in company social events, but very few, if any, have been told they need to start brushing up on their skills.

In this senario, those who are more vocal and social are then promoted with greater responsibilities than those who actually perform. This does nothing for morale since those who are upset with the outcome usually don't stick around for the next one.

My darkest fear...

2009-02-23T13:06:00.001-08:00

GUID collision.

I AM NOT A BOBBLEHEAD

2009-02-09T07:02:00.000-08:00

A Dark and Dreary Winter Day

Corporate coders don't take ownership. The one thing that allows them to remain stoic, well obtuse, is the fact that each line of code is like the one before it, the one after it and every other one in their repository. There is no context, no concern. They just don't care.

Today I found out officially that I am not longer on a project that I did a ton of work for a little over a year ago. This annoys me not only because of the time invested, but also because this was a project I had a lot of passion for, had a educational background in, and was more than just fitting data into grids.

Why am I annoyed

The paragraph aboves gives good indication as to why this annoyes me on one very personal level, but it exposes a larger issue here:

We as developers are treated as totally and uniformly interchangeable

At some level, each developer should understand the basics and the standards of their company. But at a deeper level we are each individuals who have different interests,strengths and weaknesses. Failing to recognize these traits in your developers will only lead to disatisfaction and demoralization of the very people who create great applications. But the failure to recognize the individuality of the developers and then failing to rcognize the fallout is maddening.

But these things happen when the 80/20 rule has 80% corporate coders and 20% individual contributors.

I can't document that...

It is one thing to do a handoff of a project that is basically data in and data out. It is an entirely different thing to hand off concepts and understanding. This project required me to understanding search engines in general, our new product in particular. These concepts are deeper than just settings and flags in a UI. In addition I worked on creating API calls for custom front end searching, paging, and navigation. Python had to be learned, so that dictionaries could be compiled and merged at indexing time. These are not just step one, two, three type of operations. There has to be a careful overview and understanding as a whole, not only of the technology, but also of human use of language. And this is just the search side, forget about talking about querying the search engine.

Repetition and waste

Two of my biggerst pet peeves are having to repeat myself and negligent waste. Trying to teach someone what took me several months to learn and assimilate, which was assimilated over a certain educational background, is nearly impossible. More time would be spent re-iterating all that had happened than would be needed to actually develop the applicaiton.

But surely developer one can come right in and take over the work of developer 2 right? But now since developer one is no longer on the project, he has to help developer 2. But if the goal was to keep developer one on another project, what is the point? His time will still be spent on the old project just in a different role.

Does any of this make any sense?

....this is so incoherent.....

Lost in Abstraction

2009-02-06T05:49:00.001-08:00

There are a million and one post as to what makes a good programmer/developer. Knowing how to find answers is one of those top criteria. I used to think (though shallowly) that being able to use Google was a good place to start judging. The more I thought about how we use tools, however led me to believe that there are two underlying competencies: Abstraction and Translation

Abstraction

Making objects and classes and decoupling and all the jazz is fine once you get to the stage of actually designing an application. That it not the abstraction I am talking about. The abstraction I refer to is the ability to pull out general principles from specific instances. For example, when you search for something in Google which does not appear in the first two pages, how confident are you that you can find what you are looking for? Are you able to abstract out the bigger picture and re-scope your search based on your initial findings or do you give up? As someone without a degree in computer science, I am often at a loss for the precise technical word for a general set of concepts.

Intelligence is being able to traverse the tree up and down in terms of generality and specificity. In essence to be able to cut away what is unnecessary, take what is left and reapply the core to something else. If all of this is too vague, let me give an example.

Let’s say have a co-worker. I mean, I do have a co-worker, but this is just an example. We have a meeting to discuss how we should implement a business rule. If your first reaction is to talk about web control functionality, you are starting on the wrong level. During one such meeting a co-worker asked “How are they going to key in the values?”. This might seem like it has nothing to do with abstraction, but it does.

Some people are so tied to a technology, or what is in front of them that they are unable to talk about the abstract

The reason why this skill is important is because it allows you to interact with the second necessary competency for quality development which is Translation.

Translation

I purposely chose the word translation over interpretation for this very reason:

Every translation is also an interpretation, but every interpretation does not include a translation

Translation is a more complicated competency. Here is a simple example. Looking at a photo of Man Ray’s, you can interpret the photo and ascribe to it a meaning wholly of your own. It can be totally internal without reference to the creator, sometimes with little reference to the work itself. Translation requires that one’s interpretation takes place within a framework that has a verifiable connection to the origin of the phenomenon. While there is seldom, if ever, a right or wrong translation, there are better and worse translation. We then settle on the best translation available. Referring to my earlier post: Translation is an ethics.

In development, translation occurs on several levels which is why it is vital to quality development. Translation occurs first in the gathering business requirements and then turning those requirements into business rules. As with translating natural human languages, there is seldom a one to one ratio of input to possible outputs. We take requirements and filter out the sediment to examine what are the essential rules.

Those business rules then become translated again into models and architecture. Good database design is based on translating actual business needs into properly designed schemas. Classes are then used to translate rules and data into applications. I realize this is all very simplified, but the concept is still there.

Put Together

Abstraction and translation then allow us together to move away from previous experiences (i.e. previous applications, meetings,etc) to gain insight on the current project. Quality developers know when to stop abstracting out, they know when they have reached proper abstraction and are then able to translate and integrate that abstraction with the current application parameters. Frameworks are an example of how this is put to work. The .Net framework is built basically on the most common set of classes. Microsoft has abstracted out the majority of functions necessary to program applications and has translated/compiled them into a framework. While some may argue that there is not enough abstracted out, that again is subjective argument, based on better or worse translations. Both C# and VB are translation of the very same basic concepts.

So now what?

As developers we need to develop these skills constantly. It is easy to become comfortable and stop abstracting and translating. In the worst case scenario we rebuild the same application over and over again. We make the same mistakes because we never take a higher view of our work. In less noticeable ways, we succumb to boredom and malaise by becoming lazy at re-examination of our work. If we consider our work done as soon as it works we will eventually end up in the worst case scenario. Once something actually works we need to re-examine what we did to make it work, see if it can be refactored for better performance and see how the whole works together. What can be applied to the next application, what can be used to maintain existing applications.

For me, this all comes before being able to write the fanciest hashtable.

Are you a Textual Minimalist or a Preacher of the Word

2009-01-28T06:56:00.000-08:00

Choose a side

Of course the answer to almost every dichotomy is to find a solution in the middle which is a compromise. Whether you subscribe to the progress of history as Thesis, Antithesis, Synthesis like Hegel and his cronies or whether you believe society is a wave which only appears to make progess and never moves, the reality is that we need to make choices between opposites.

Continuing off a previous post regarding building user expectations, one area which needs further examining is the use of signs as replacements for text. Or rather replacing the sinage of letters with signs which are more readily digested and understood.

Linguistic researched has already shown that words are better representations of signs than their individual letters. There are two phenomenon which represent this. One test concluded that participants can more quickly identifiy whole words than individual letters, and that you can remove vowels from words in a context and still interprete the meaning with very little loss in terms of speed. This is why yo ucan review an article you have written and miss the same misspellings over and over again. Although you think you are viewling and reading each letter and each word, your brain is filling in gaps in order to speed processing.

Ok enough background. What good is this?

Please shut up

Well I am a Textual Minimalist. I would prefer to have simpler signs to designate meaning rather than text. However, in my company alot of people seem to balk at this and prefer to have alot of text. Like over-analysis/under-construction in the design phase, over textualization can often run into the same issues.

First over textualization wastes time be deciding the correct verbage to use. It has to be clear to the user not only in relation to the application but also to previous examples or text. If the text is being added in design mode, it is without context with regards to the application. But, once added to an application over textualization lives in two contexts, one that is the application into which it is inserted, the other which is the everday language we use to communicate.Since the specialized nature of business applications requires verbage which differs from our everyday usage, using alot of text on the screen often requires some sort of jargon interpreter.

A solution to reducing this complexity is to use sing simple signs and color codings. And as in the previous post regarding building expectations, these signs should be conventions regarding first the application, and secondly the business processes. The more abstract the signs are the more likely the user will be able to assign the sign with the action or functionality of the application without using textual interpreters. The business process sets the ocntext fo the application, and the application sets the context for the signs and texts which it contains. This is why building expectations though clear business rules is important for application development. Trying to create verbage in the application to backfill in business logic will almost always yield high resistance to application implementation.

The Difference

I think it is time for an example. Suppose we have two grids, one utilizing text, the other an image.

While functionally the same (clicking either the image or the button), these two cells do not necessarily convey the same message. Using my company as an example, we never delete data. I am not sure why, that is just 'the way it is'. Rows marked as 'deleted' are seldom used for reporting and once marked as such are even less likely to ever again make an appearance. But the point isn't what we do with the data, it is how the user feels as they interact with the application. We need to remember that although developers interact with data, end users interact with applications. Very few have the conceptual framework of data management. One row in a grid is one row in a grid even though the underlying structures may represent several tables and an aggregation of hundreds of rows.

The phenomenon

What happens when you click the image is more closely associated to that image than when you click on the button (whose text and column read 'Delete'). I will say this as fact. The text on the button says 'Delete' but in fact the data is not getting deleted. Would it be better to say "Make data eternally invisible?" Surely this is not an option. Having only the image present reduces the confusion and complexity since it merely indicates a 'going awayness' of the row. The user realizes that after clicking the button the row is irretrievable. Whether or not the data is actually deleted or just hidden is not of concern to the user for their real world usability. In a rare application that is does matter, signs can be used to signify the difference in other ways. In those cases, it would probably be even more problemmatic to have one button that says 'Delete' and one that says 'Hide', compounded even more if there is no 'Unhide' option.

But I have something really really important to tell you right now about this very important thing here onthe screen

Sometimes over textualization can actually inhibit workflow in addition to adding conceptual complexity. The most insidious form of over textualization is the modal popup when used for validation or informational purposes as opposed to stoppping workflow to indicate an error or the need to take corrective actions. How many times during work can I have a box popup that gives me information that I do not really need. And what is in the box? Text.

One symptom of over textualization is not only the sheer volume text, but rather the forcing of this text on the user. Instead of having popups, modal ones at that, disrupt the workflow of a user, or having gigantic panels taking up real estate with instrucitons consider the following:

Validation errors handled with highlighted violators. Empty textboxes can become color shaded. Invalid values can either have text added next to the offender inline with the entry point or submit point or the form can surface a tool tip.

Tooltips are a very useful tool as long as they don't fall prey to over textualization. Tooltips should not be used in place of training documents. If your application has so many tooltips that the user is afraid to stop their mouse, then your application needs to design and usuability re-working.

Informational issues concerning instructions can be put into a helpfile if need be (i.e. 'How to fill out this form'). Better yet, help icons can be surfaced in the place of a modal popup. If the user feels the need to see the information, they can choose to click on the icon of continue working.

The worst for last

Let me give you an example of how the use of over textualization has been abused at my company.

We have a very simple gridlike application for entering Time worked for the week. For Thanksgiving break, they automatically enter the time for you and nicely shade the box indicating that the values cannot be changed for that cost code for those days. All very nice. However, when I tried to enter time for Thanksgiving day, a modal popup well..popped up telling me basically that there was already some time entered for the thanksgiving day vacation and if I needed to change any of the values other than those for that day I could or I could keep the hours I added to Thanksgiving day. Bascially, the message told me to keep on going, don't mind me. It was the worse kind of over textualization, not only did it disupt my workflow (several times actually since the code that triggered the popup kept on triggering somehow), but it also did not give me any real information. It did not say that what I was doing was incorrect. It did not provide a better way of doing anything. It was basically a squirrel that just ran infront of my car forcing me to put the brakes on.

Nothing is wrong with words as long as they are meaningful. And you can quote me on that. Just don't quote me in your next applications.

You might be a corporate coder if...

2009-01-28T06:51:00.000-08:00

you'd rather code than play in the snow.*

*(This assumes you are stuck in the office working on work)

Linq Madness

2009-01-27T08:01:00.000-08:00

Dim query = From cm In db.case_masters _
Where cm.start_date > Now.AddMonths(-36) _
And cm.case_type_code = 1 _
Group Join cr In db.case_relationships On cr.case_code Equals cm.case_code And cr.client_code Equals cm.client_code _
Into bpGroup = Group _
From bp In bpGroup.DefaultIfEmpty() _
Group Join cr In db.case_relationships On cr.case_code Equals cm.case_code And cr.client_code Equals cm.client_code _
Into cmGroup = Group _
From cp In cmGroup.DefaultIfEmpty() _
Join co In db.case_offices On cm.case_code Equals co.case_code And cm.client_code Equals co.client_code _
Where bp.relationship_id = "BV" _
And cp.relationship_id = "CM" _
And co.office_type = 1 _
And cm.case_type_code = 1 _
And cm.client_code = basis_client_code _
Select cm.case_name, billing_region = db.fnOfficeHierarchy_GetEntityNameByApp(office_code, "RG", "DEF"), _
billing_partner = bp.employee.first_name + ", " + bp.employee.last_name, _
billing_partner_home_office = bp.employee.employee_offices.Where(Function(d) d.office_type = 4).First.office.office_name, _
case_manager = cp.employee.first_name + ", " + cp.employee.last_name, _
industry = cm.gxc_case_terms.Where(Function(d) d.relationship_strength = 0.8 And d.gxc_term.term_type_code = 6).First.gxc_term.term, _
industry_practice_area = cm.gxc_case_terms.Where(Function(d) d.relationship_strength = 0.8).First.gxc_term.industry_hierarchies.First.practice_area, _
capability = cm.gxc_case_terms.Where(Function(d) d.relationship_strength = 0.8 And d.gxc_term.term_type_code = 4).First.gxc_term.term, _
capability_practice_area = cm.gxc_case_terms.Where(Function(d) d.relationship_strength = 0.8).First.gxc_term.capability_hierarchies.First.practice_area, _
peg_client = IIf(cm.class_code = "S", True, False), _
portfolio_client = IIf(cm.case_private_equity_portfolios.Count = 0, False, True)

Linq: Projecting Query into New Object VB.Net

2009-01-27T07:57:00.001-08:00

Sometimes it is useful to create a new object from data not stored in relational format in your database. In these cases, you can create a custom class and project linq to sql properties into it.

Here we have created an OrderDetail class (not shown) and projected into it properties from our linq to sql query. This will return a type of IQueryable(Of OrderDetail)

Dim query = from p in product _
join o in orders on p.product_id equals o.product_id _
select new OrderDetail with {
.ProductName = p.name,_
.Quantity = o.qty}

Building user expectations vs managins user experience

2009-01-27T07:31:00.000-08:00

I see this over and over again with the applications I build. Each project, while professing to be different from last time, inevitably falls into the same old pattern of development. One area which falls victim to this more than anything is standardization.

For the most part our applications get data and display data. In those rare occassions where data is actually manipulated, it is most often done in some sort of grid format. And while my peers may grumble at the humble excel like structures, it is a format which is reasonably familar to the average user.

Sit an average worker in front of excel and they can do something with it. It isn't rocket surgery. Word and Excel manage to thrive because as functionality is added over time, the core product remains the same. Buttons, behavior and layout remain virtually untouched in the majority of functionality facing a user. Microsoft has done an amazing job building user expectations. When the next version of Word or Excel comes out, I will jump right into it creating pivot tables and writing long-winded diatribes.

This is boring. But this is how we ultimately get things done. Less time is spent learning relearning and training and more time working.

But there is another counter trend which appears from time to time, especdially at our company. Doing things in the same way is no longer acceptable. In each project a tiny group of business product owners and even smaller group of developers decide how THIS project should behave and how THIS grid should act. The 'THISNESS' of the project acts blindly towards previous projects as well as typical usability conventions. Why is this?

One potent motivator is the need for people to feel as though they are involved in the project. Everytime someone opens the application and the first cell of the grid is red, they can smile and say "I pushed for that". Whenever columns are read-only for no apparent reason, someone can point proudly to the fact that users are being saved time by not wasting valuable writing time. When it then becomes impossible to enter a cell because upon entrance the current cell becomes the cell to the right, well thank the Lord because the user does not have to even enter a cell to try to type in it.

Overanalysis is the second motivator in the example seen above. It is like a snake in the middle of your path. You carefully walk by the snake. It doesn't move. You feel the need to be certain, so you go back and prod it with a stick. You get bit. The problem is that someone already told you the snake was alive and indeed very poisonous.

Analysis is not bad. But the focus of the anlysis should be clear. There is a big difference when analyzing "What should be done vs How to do it". Often more time spent on detailing out the objective of the application will decrease the time spent figuring out how to do it. There is a big difference in thinking regarding the following two sentences:

"We need a way to highlight vacation times greater than one week"
AND
"We need to highlight vacation time greater than one week with a red underline"

One way of thinking is generic and adaptable. The other is much more rigid. In addition there are far more assumptions made in the second statment. Why red? Will the user know what a red underline means? Alot of time can be spent over analyzing such issues while more important business objectives fall to the wayside. This is even more true if there are no standards across development. And this is still only in the planning phase.

Strangely placed buttons, wierd acting grids and the like aim to manage user expectations. Development teams (including process owners) often see this as a way to save time and create something unique. However, people work much faster if they have new tools which imitate existing tools. Imagine if everytime you typed in a certain application, the letter C came out as a P. But with another application C came out as N. I could almost guarentee there would be no good reason for doing this, but it is not out of the realm of the impossible. Far greater assumptions have been made while developing applications.

Business applications should strive to build user expectations by building on standard behavior and presentation as well as standard business processes. Product owners should review existing applications when modeling future ones. Similar naming conventions, color coding and language goes a long way towards productivity and can cut down on development time by reusing code and cutting out the need to cumbersome help files to document not only how this works, but even to a less visible degree, how this applicaiton works as opposed to another internally developed application.

But I am just one man...well it is time to get back to my grid, which has readonly columns without indication, blinking, twirling and other useless features. Don't worry, it will save your data...well some of it....well eventually.

Linq Select Subquery

2009-01-27T07:23:00.001-08:00

Dim db as new DataBaseDataContext

Dim query = from p in db.products _
select .category_name = (from c in db.category where p.category_id = p.category_id select c.category_name).first

Morality, Aesthetics and Denormalization (pt1. Moving towards an ethics of denormalization)

2009-01-09T12:18:00.001-08:00

A Computer Problem

So I had this problem. I have a list of names (B). From this list of names I need to produce another list of names (N). Both B and N have data attached to them. B always exists, N's existence relies on B's existence.

We can picture the list of B as as the lowercase alphabet and the list of N as single digits (0-9)

Both B and N contain additional data.
N is created from B but it is not static.
For example, we can create a relation between B and N several ways.
One B can be made into several N (d-->2 && d-->3)
Many B can be made into one N (a,q,c-->1)

In addition at any point in time the relation NB can change. What was once (a,q,c-->1) is now (a,q-->1). In this case, 'c' is removed from N and is no longer relevant in the relation.

The purpose of this relation BN is to ultimately get data about this relation. So BN is only relevant once data (W) has been gathered on BN. W as a set of data includes datetime information about BN as well as a series of questions about BN. W is a context in which something can be said about BN.

The first pass at creating this structure was a standard march towards Normalization. We already have B, we can create N. Now we need a table which maps B to N. So far so good. But oops, we forgot about the point in timeness. We could create a history table with an audit trail, or create date range fields on the mappings and create new mappings whe needed. Since the relation BN could change at any time and we need to track BN, the relation BN is also immutable. This would be alot of work to normalize and maintain in an application.
Let us digress just a bit...

A Moral Issue

In the above example, B represent a morality. It is unchanging. What is B has always been B and is unable and unaffected by its relation to N. The data tied to B underlies many programs. It is a sort of primary key across applications. It is either B or not B. (using the references above 'a' is 'a' or not 'a') That is all it can be. N on the other hand represents something completely different.

N by itself has no meaning. It is a fleeting event, a concept, an idea. It is an aesthetic component. It is totally arbitrary. There is no value judgement placed on N. For this manner N can actually contain 'a' and not 'a'

Here is where the car is going....
B is 'Thou shall not kill'
N is the death of some man

Viewed in this manner the relation BN which takes place within the context of W (or in fact IS THE CONTEXT of W). W itself is the ascribing of meaning B to an event N. If we picture two scenarios using B and N above we can see how the this works.

SCENARIO A: A man walks down the street and is robbed and killed by a complete stranger.

SCENARIO B: A man walks down the street and is robbed and stabbed by a complete stranger. In the course of the struggle, the victim overpowers his assailent and kills him.

In both of these cases B and N is the same. One shouldn't kill (or murder) and in each scenario someone dies. However, in the relation between the underlying morality and the event, we end up with a relation BN. The judgement we make on that relation is W. W is valid only for that point in time in which the relation BN exists. To fully flush this out is beyond the scope of this post. Of course there are questions as to the completeness of facts and other intangibles. W becomes history which then can inform and shape morality but remains something completely different.

Extending W beyond the BN relationship

It is important to understand the BNW relationship in both a computing and noncomputing environment. From the computing environment, we could store every BN we needed to within a defined set of lists, assign each one an id and totally normalize the data. We could be insane and create millions of two column tables. However, if the relations are never used, then we have created alot of noise with no return or value. This is made worse still if we had to then account for new BN relations. W allows us to create a context in which to gather data.

In terms of W, is it important to note that while W may represent a context, it ONLY represents a context. W informs us about an event (N) taking place against a set of pre-defined rules (B). W cannot inform us about another W. If we need to reanalyze W, we do so by converting W into an N. Our new knowledge becomes the context (W) which we can say something about. But, since B is contained in each and every W, B can be influenced in light of W at the occurrence of N. The opposite cannot be true since the N which is contained within W is never the same N. For even though it may appear to be the same, viewing it as such is a matter of convenience. Trying to say something about every instance of N through W results in the naturalistic fallacy. And I think that is a good place to stop.

What makes a corporate coder?

2009-01-09T12:09:00.000-08:00

This will be a growing list.

Shows up to work

Codes for N hours a day

Thinks company events rock

Thinks free pizza is better than profit sharing

New inspiration

2009-01-09T12:07:00.000-08:00

I grabbed the title of this blog from a presentation by Zed Shaw hosted at http://www.vimeo.com/2723800.