Passionate about data

Data and its implications on software design and development.

Lessons From Re-Learning

I have been working on a project that I had worked in 2005, trying to get a handle on what I had done about 3 years back. Exploring code and the database has been fun, also discovering the data layout and building new set of data for production has been extremely entertaining. What I learnt from this whole experience was this <blockquote>if your code(application or other wise) is not expecting data, this data should not be provided or even considered valid by the database, the database should be designed such that it does not even allow invalid combinations of the data.</blockquote>

Here is an example. We have a base table named as SETTINGS, that can be changed by the user by overriding the value in the extension table known as SETTINGSEXTENSION, table structure is show below

KEY (BusinessKey)
VALUE (Value for the Key)
OVERRIDDENVALUE (Value overriding the value in the SETTINGS table)
BYUSERID (Value over ridden for user)

in this scheme the same user can create extensions for the same base key causing the application to barf. If we introduce unique index on SETTINGSEXTENSION TABLE(SETTINGSID, BYUSERID) we ensure that the database does not allow this data and makes sure that the application will not barf.

There are many more other things I learnt.. those will follow..

Cannot Make It Out to XP Day London

After a lot of frustration about my schedule, I have had to come to this conclusion, that I cannot physically make it to London, XP day.

I’m going to miss it. Nick Ashley is going to take up my spot and I know he will do a great job.

Why Do Evolutionary Design

Why do Evolutionary Design or Iterative Design or Incremental Design? Everyone who has not worked in an evolutionary manner asks this? My answer, if you think the system you designed is NOT GOING TO CHANGE EVER then sure you can do design once and deploy once and you are done, move on to next project. But tell me one project you have been on, that does not have any changes in requirements, changes in technology, changes in look and feel etc after it was deployed.

ThoughtWorks at Oracle Open World

ThoughtWorks is going to be at Oracle Open World. I’m excited about this especially since it will give ThoughtWorks and Me to talk about software practices and how to apply these software practices to the database development world, off course I will talk about my books Refactoring Databases and Continuous Database Integration. ThoughtWorks will have a booth at “343 Moscone South” and I will be there on Nov 14.

Frameworks Should Not Constrain Your Table Design

Recently talking to someone about a persistence framework that they are using, this framework creates a rows in the database table with just Primary Key value and then later on updates the table row with values for other columns. Because of this framework, the tables cannot have any NOT-NULL constraints defined.

Lesson Learnt

On any machine running network related services like in my case Oracle Listener and Dispatch Services. Don’t rebuild/stop and restart the firewall stuff like iptables.

Had to spend a lot of time, figuring out what was going on. So hard lesson learnt don’t mess with iptables when you are running oracle listener/dispatcher

When Does Evolutionary Design Happen?

A question I get, mostly related to the evolutionary database design and development. When the pair (team) gets a new feature (story) to work on, the team looks at the existing table/database design and sees if the current design is enough to implement the feature they are working on. If the currency database design does support the feature they are trying to implement, then they do not have to change the database at all, they will move on to implement the feature and change the application code as necessary.

Parsing Mapping Files for Usage Information

Currently working on a legacy application, thats been in production for a long time now. I wanted to find out what are the Tables and Columns being used by the application. Since we could see that some table columns where not being used. We are using a Object Relational mapping framework on the project, so we decided to write some code that would parse all the mapping files and gives us a list of table names and columns. We used this list to create rows in a table with two columns tablename and columnname. Once the table had this data, we just ran one more SELECT against the metadata of the database and our table which pretty much gave us a list of Table and Columns that we are not using

Long Running Data Migrations During Database Refactorings.

When you are refactoring large databases, you will have certain tables that have millions of rows, so lets say we are doing the Move Column refactoring, moving the TaxAmount column from Charge table which has millions of rows to TaxCharge table. Create the TaxAmount column in the TaxCharge table. Then have to move the data from the TaxAmount column in the Charge table to the TaxAmount column you created in the TaxCharge table.