I have written a couple of posts about 7-8 months back (here and here) on how to call stored procedures from a rails application. After using stored procedures in my app for a while I started to think about whether we really need them. So over a couple of months I discussed it (mainly short discussions about stored procedures not whether to use them or not) with my colleagues, friends, other people  and gave some thought to it. If you are new to stored procedures then here is a sort of textbookish definition.

Stored procedures are available to applications accessing an RDBMS, they are similar to constructs seen in other programming languages and stored in database data dictionary. Stored procedures remove the compilation overhead required for inline queries as they can be pre-compiled, hence they run faster in some cases. However, now most RDBMS, as they evolved, implement statement caches to avoid repetitive compilation of SQL statements.

Why people think we need them?

People, especially with database background or DBAs, are of the view that we should use stored procedures for most of the tasks involving database because they offer following advantages:

1. Stored procedures separate server side function from the client side, i.e, put all your business logic in stored procedures.
2. Stored procedures are modular and offer advantages compared to embedded queries buried within the GUI code.
3. They are faster, secure and maintainable
4. Reduced network usage between clients and servers

Why people or I think we don’t need them?

1. If the application in question even has a remote chance of running on more than one RDBMS then I would just kick stored procedures out of the door because the exact implementation of stored procedure varies from one database system to another and I certainly won’t like to tie my app to a particular database.
2. Having business logic in stored procedures is not only breaking every rule of good programming but also not following the tried and trusted way of architecting n-Tier applications, i.e. separating presentation, business logic and storage/data access. Do not mix storage/data access with logic because stored procedures are just an extension of data access tier. As they say, use database as a very efficient file cabinet. The only concern of a database should be to store data or persistent objects and the business logic should reside in the application code. Database in itself is not an application you are developing, it’s part of the layer/tier that stores data. We use SQL in RDBMS and SQL is simply a data manipulation language hence very limited in its scope. A programmer can do everything in the application code that can be done in a stored procedure but the opposite of this is definitely not possible. Hence, databases are not programmable enough to include all the business logic. If you somehow manage doing that using temp tables, inserting redundant rows into the tables, by spending ‘n’ number of extra hours or other hacks then you are gonna face performance/scalability issues and surely maintenance problems later in the development life cycle, i.e. a nightmarish, incomprehensible & complex POS (piece of shit). These issues will come back persistently to haunt you and your team. Hence, the advantage 1 and part of advantage 3 mentioned above is not valid anymore.
3. Storing business logic in the stored procedures also leads to more work for testing compared to having them in (Object Relational Mapping) OR / domain model class and stored procedures/triggers make test driven development less productive. Also, if I am using ORM then I do not need to use any SQL in my application code because the ORM will generate optimised queries for me on the fly and will also bring simplification and consistency in the coding of whole team. Hence, advantage 2 and part of advantage 3 mentioned above is not valid any more.
4. Stored procedures can very easily get out of synch with the application code. For instance, if I have different versions of my app in the source code repository and in case something goes wrong with my latest version then I can very easily switch back to an older version by simply checking out / exporting stuff from the repository. But these facilities do not exist for stored procedures / triggers. It’s troublesome to have business logic residing in stored procedures because they are in the database and once you alter an existing procedure the old one is gone, hence no going back. Now some of you may argue to keep each stored procedure/trigger in some text file and commit it to the repository with the app but that’s again creating additional work for your team which means you are also adding chances of error/mistakes. However, if you do this then you still have to run your stored procedures/triggers with every update that’s again prone to mistake/errors.
5. When something goes wrong then stored procedures don’t provide meaningful feedback unless you code all the exception handling within the procedure itself.
6. You can’t pass objects to stored procedures. If you are using an ORM, and you have a class that is mapping to a table then you can simply pass the object of that class and you are good but with stored procedures you have to pass each field or object attribute as parameters and you may end up passing lots of them, may be 20, 30,40 or even more. Hence, you end up writing more code which essentially is more error prone and if you get a ‘bad call’ error then go figure what just happened. That’s just creating one object what if you need to add a record to 4 tables simultaneously using transaction and let’s say each of them have 10 fields then you need to pass 40 parameters. Change is the only constant in life, especially in the business environment.  Lets say, due to some business requirement your database schema changes and you have to add/remove columns to/from those tables. If you are using stored procedures then you atleast need to take these steps to make sure that your application is still working fine:
   • Add new data vaildation rules for CRUD operation.
   • Change the way data presented to users and your form fields where you allow users to create/update records.
   • Add or remove new parameters to the stored procedure calls from your front end
   • Add or remove new parameters to your stored procedure definition
   • Add or remove new parameters to all other stored procedures or triggers from where you are calling your affected stored procedure(s).

   If you are using ORM instead of stored procedures then you need to take these steps to make sure that your application is OK:

   • Add new data vaildation rules for CRUD operation.
   • Change the way data presented to users and your form fields where you allow users to create/update records.

   That’s it and you will also have your whole app properly version controlled so that you can switch back and forth seamlessly. So why do you want to do something that involves more work leading to delays, increased complexity, less productivity and has more chances of error? Hence, stored procedures/triggers are actually a constraint in your system and according to Theory of Constraints; generally, in any system or process there is one constraint at a time. To improve overall system or process performance first find the constraint then improve it or eliminate the constraint entirely. Do I need to say more?

If you think you can use stored procedures to hide business logic,  then you are definitely using stored procedure what it is not good or designed for. I neither like putting SQL queries in my code nor have any problem using stored procedures only if they can offer me/my team/employer some measurable benefits but in my opinion an OR model is much better in terms of developer time, code, productivity and maintenance. Using stored procedures indicates adding another layer, another language, complexity and losing database portability. If I have an application running on ‘n’ number of different clients then it will be much easier for me to keep my application code synchronized and maintainable rather than the stored procedures and triggers. Triggers pose another problem area because application does not know whether they exist hence do not even have a clue whether they have run or not. What if you have some logic in one of your trigger and that itself may be calling another stored procedure(s) (again some logic on important data) and due to one or the other problem trigger didn’t fire, you are screwed because neither you nor your application will know whether the trigger was fired or not unless your application users see some discrepancy in their data or reports, report them back to you and then you run multiple sql queries and check everything manually. How kewl is that? Not for me atleast.

Having said all that, I don’t mean the stored procedures were never useful or the advantages of stored procedures cited by DBAs were never true, yes they were true until 6/7 years ago. But with time, new technologies come into existence, some lose their advantages and some become obsolete. There is no point in sticking to something for which you have other better options available.

I received few requests in the past ( personally and recently a comment on my post describing how to use will_paginate plugin ) about how to format pagination links while using will_paginate plugin.

So here it is:

When you add

<%= will_paginate @obj_instance %>

in your views to show your pagination links then it also creates an element (div) and wraps the whole thing under the class name pagination. The one with the current page will have the class name current.

Here is the html code generated by will_paginate


<div class="pagination">
<a href="/page_path?page=2">« Previous</a>
<a href="/page_path">1</a>
<a href="/page_path?page=2">2</a>
<span class="current">3</span>
<a href="/page_path?page=4">4</a>
<a href="/page_path?page=5">5</a>
<a href="/page_path?page=4">Next »</a>
</div>


Now, it is totally upto your imagination how you want to style it using CSS.

But, if due to some reason you don’t want to use the class pagination and add your own then do something like this


<%= will_paginate @obj_instance, :class => 'my_class' %>


Similarly, if you want to add an element id then

<%= will_paginate @obj_instance, :id => 'my_element' %>


Let’s say if you have done both like so:

<%= will_paginate @obj_instance, :id => 'my_element', :class => 'my_class' %>


then the new html generated by will_paginate will be

<div class="my_class" id="my_element">
<a href="/page_path?page=2">« Previous</a>
<a href="/page_path">1</a>
<a href="/page_path?page=2">2</a>
<span class="current">3</span>
<a href="/page_path?page=4">4</a>
<a href="/page_path?page=5">5</a>
<a href="/page_path?page=4">Next »</a>
</div>
Yes, it is that simple.

SPhred was relaunched last week (1st week of April, 2008) with RESTful interface, new codebase and a new design. Here is a summary of the changes:

1. Now we have OpenID authentication on SPhred, i.e., new users do not need to Sign Up to use SPhred if they have an Open ID account.
2. In the past, some users have suggested that they do not want their full name to be shown on their profile page, so now we do not show any user’s full name anywhere because we have added nicknames or user names.
3. I thought its a bit tedious to type your whole email address to login, hence added an option of using the Sign In name, ie. you can either use your Sign In name or the email address. If you do not know your Sign In name yet then Sign In using your email address and you will see it once you are signed in.
4. If you have done some activity on SPhred like adding tags to your profile, SPhredding, connected with other people then SPhred can also recommend a network of people with similar interests on SPhred. Simply go to your profile and click on the ‘More Like you’ link to check this out.
5. You can also send short messages to other users by using the ‘Buzz’ link on a user profile.
6. I think it is important to bring the old posts back to users’ attention once in a while even if it has gone past 10th page and hence have implemented a ‘Random Picks’ widget which is shown on almost every page. Go to the widgets page http://www.sphred.com/widgets and check out the other widgets.
7. I received lots of spam in the past, hence now we have captcha implemented on almost every page where users contribute to SPhred.
8. Previously, every post on SPhred had a URL like http://www.sphred.com/sphred_details/site_name, this url scheme is still kept for backward compatibility but now we have better and more meaningful (& RESTful) urls like http://www.sphred.com/categories/1-New-Launches/websites/site_name.NOTE: The old url scheme will be phased out eventually hence can I request you change the old urls on your blogs/websites to the new ones.
9. There are many more behind the scenes changes and some you will notice in time.
10. And yes, this is how SPhred looked in the past http://www.sphred.com/images/sphred.png

If you have any feedback and suggestion then do send it to me, have fun!

When it comes to using database views then most of us think that it is difficult to use them with Rails or we may not be able to use ActiveRecord with it. Since I experimented with it in the past so I thought why not share it with you folks.

There are several situations when we would like to use database views with our Rails application like:

Situation 1:

If we have a complex SQL query that we don’t want to code in your Rails app due to performance related or any other issues.

Situation 2:

When our database guys have already spent a great amount of time writing useful views that can be used in the current app

Situation 3:

When we have some repeated query to run in every select statement to fetch data from every table. For instance
Select * from table_name where created_on < today;

In that case we would prefer to have a view with this query and call that view.

In all the above situations, it is pretty easy to use our database views with Rails.

How?

Simply create your model as you would for a table and it will work fine just like your model for a database table.

Interesting thing is, apart from viewing data using views I can even update, delete and insert data in my database tables using my views with all those ActiveRecord/ORM niceties (in Sybase) as long as each database view consists of only one table. If you are using more than one table in your database views then you can only view data but cannot do delete, update and insert operation.

I think this is really useful if we have a database that do not conform to Rails conventions but we would like to follow those conventions without changing the current database if other applications depend on it as well. Simply create views for each table in the database and use those views for our models instead of actual tables.

Those single table views (in other words updatable views) will allow us to follow Rails database conventions without changing our existing database tables while also keeping our database guys happy ;o)

(Though we also have other options if we are not following rails conventions)

NOTE: I have tried updatable views only with Sybase and would love to know about your experience with other databases.

Continued from Stored Procedures and Rails – Part 1

Now we know how to call our good and reliable stored procedures but there is a difference when we get records from database using Rails ORM/ActiveRecord compared to a stored procedure. This requires a different approach to access records returned from a stored procedure.

Lets say we have a table orders and a corresponding model Order. We all know, how to access data using ActiveRecord, eg.

@orders = Order.find(:all, :conditions => ["some simple condition"])

then what is returned to us is an array of order objects which we can access by either using a block or a for loop. For instance,

for order in @orders
  puts order.order_name
end

but this is not possible in case if we use a stored procedure to get data on orders because we don’t get back an array of order objects instead we simply get back an array of hashes, ie. an array of records with their attributes and values. Hence, the above code will not work.
Lets say you call your stored procedure, like this

@orders = CallStoredProcedure.fetch_records("exec procedure_name 'args1', 'args2'")

as mentioned in the previous post.
Then you need to access records with a little modification within your for loop

for order in @orders
  puts order['order_name']
end

I thought this might save sometime who are new to using stored procedures.

NOTE: The only downside of using stored procedures in Rails currently is; you cannot use the Object Relational Mapping which you might have been used to by know, ie. picking all children of an order by simply typing

order.children

The best thing to do is to use stored procedures judiciously and mainly for complex logic/calculation in your database but when you want to do something simple with your data like simply showing something or a quick update then use ORM over stored procedures.

There is no documentation in Rails about how to use stored procedures or atleast it is difficult to find where to look at. And according to DHH stored procedures is not the Rails Way to do things. DHH mentioned in his post

I want a single layer of cleverness: My domain model. Object-orientation is all about encapsulating clever. Letting it sieve half ways through to the database is a terrible violation of those fine intentions. And I want no part of it.

I somewhat agree to what DHH has to say. And that’s all fine but what if we don’t have any other option apart from using stored procedure and our stored procedure has some complex logic which we don’t want to put in our Ruby code then here is a little hack to it

ActiveRecord::Base.connection.execute("execute procedure_name")

But the above line will not return anything it will only execute your procedure, considering everything went OK.

Moreover, it’s not DRY if everytime you have to write that line of code just to call a procedure.

Here is a better way to do it:

1) Create a class and let’s call it

Class CallStoredProcedures
  your methods here .......
end

2) Create a class method within it

def self.fetch_db_records(proc_name_with_parameters)
  ActiveRecord::Base.connection.execute(proc_name_with_parameters)
end

Then simply call it as

CallStoredProcedures.fetch_db_records("exec procedure_name 'args1', 'args2', 'and so on' ")

Now you might say, that this will only execute a procedure (as I mentioned above) what if I want to return records or values from stored procedures. Then I will prefer class methods and this is how I would do it,

class CallStoredProcedures < ActiveRecord::Base

  def self.fetch_db_records(proc_name_with_parameters)
    connection.select_all(proc_name_with_parameters)
  end

  def self.insert_update_delete_calculate(proc_name_with_parameters)
    connection.execute(proc_name_with_parameters)
  end

  def self.fetch_val_from_sp(proc_name_with_parameters)
    connection.select_value(proc_name_with_parameters)
  end

end

And then simply call it like

CallStoredProcedures.method_name("exec procedure_name 'args1', 'args2', 'and so on' ")

Sweet…

If you know a better way to do it then I would love to know your views.

Stored Procedures and Rails – Part 2

The Feedtools library is a very comprehensive and powerful Ruby library for handling rss, atom, etc as well as caching. It makes creating, consuming and manipulating feeds a piece of cake.

It’s ideal for parsing RSS feeds in Ruby on Rails applications.

Assuming that you already have installed Ruby and Rubygem, simply run

gem install feedtools

to install the FeedTools gem

or download it from the FeedTools project page and put the untar/unzipped folder in the vendor/plugins directory of your app.

To do a very quick test, create a file called testfeed.rb and add the below code in it:

    require "RubyGems"
    require "feed_tools"

    feedurls = 'http://www.sphred.com/combined_feed'
    # If you want to fetch more than one feed then comment the above feedurls variable and   uncomment the below one.
    #feedurls = %w(http://feeds.feedburner.com/Sphred_top_10_feeds   http://feeds.feedburner.com/Sphred_site_only http://feeds.feedburner.com/Sphred_site_feature)
    my_feeds = FeedTools::build_merged_feed feedurls

    my_feeds.title = 'Sphred.com Feed'
    my_feeds.copyright = 'SPhred'
    my_feeds.author = 'Nasir '
    my_feeds.id = "http://www.sphred.com/combined_feed"

    File.open('./my_feeds.xml', 'w') do |file|
      file.puts my_feeds.build_xml()
    end

    puts "Checkout Sphred.com while I fetct feeds for you"

Running this file from IRB will create a my_feeds.xml file in your current directory with all the feed contents.

To show feeds on a website, you need to do a few things:
1) Put

require "feed_tools"

at the top of your controller
2) Create an action within this controller to show your feeds, lets call that action user_data

    def user_data
      @feed = FeedTools::Feed.open(params[:feed_url])
      # You can first test it with a static feed url like this
      #@feed = FeedTools::Feed.open('http://www.sphred.com/combined_feed')
    end

3) In the corresponding view add this code

  <div class="feeds">
    <h3>
      <a href="<%= h @feed.link %>">
      <%= @feed.title %></a>
    </h3>
    <p><%= @feed.description %></p>
    <% for feed in @feed.items %>
      <div class="feed_item">
        <h4>
          <a href="<%= h feed.link %>">
          <%= feed.title %></a>
        </h4>
        <p><%= feed.description %></p>
     </div>
   <% end %>
 </div>

This is all good for development environment or for small apps but once you go to production environment then you have to think of caching to avoid hitting the feed server every time before displaying feed contents.

I prepared a mind map based on Kent Beck’s book. Extreme Programming – mind map.

I will be updating this post with a new mind map and some write up in the next few days, so check back soon.

First, A big thanks to all the guys out there for registering and sending invitation to their friends/relatives to try out sphred.com

SPhred is currently in very early stages, like what other sites say ‘in beta’.

A few things are added on SPhred in the last couple of days:

1) Earlier you first had to create a post and then could add tags to it but now you can add tags while creating a post.

2) Now you can put a face to your name i.e. add a picture. This functionality is available by going to your profile page. The option is given just below “Profile for your name”. You can access your profile page after signing in by clicking on your email address part of ‘Signed in as your email address‘.

3) Page to view users who sphredded an item has been redone and looks much better now (atleast I think so). Have a look by clicking on the number of users who sphreaded an item. Or you can directly go here http://sphred.com/users_sphredding/9-tutsbuzzcom

4) Search is functional now though currently it only looks for the search term in a Sphred Item’s title or description.

6) Comments look much better now as you can see the picture of people who posted comments.

I would love to hear your feedback or suggestions i.e. what you would like to have on sphred and what you think isn’t working or won’t work.

Once again thank you all for using sphred.

In Rails, we use collection_select helper to display values from a particular model using a drop down but it is generally tied to a particular model, i.e. if we look at the generated html then we see something like

select name="customer[customer_search]" id="customer_customer_search"

What if we want something like:

select name="customer_search" id="customer_search"

Then we can simply use select_tag helper and get the values from our model like this

customers = Customer.find(:all, rder => 'first_name').map{|x| [x.full_name] + [x.id]}
select_tag(:customer_search, options_for_select(customers))

If we don’t want to display the values from model and would like to feed in something simple and manually, then

customers = [['text_to_display1', 'value1'], ['text_to_display2', 'value2']]
select_tag(:customer_search, options_for_select(customers))

If text_to_display and its value both are same then we can use something as simple as:

customers = ['wee', 'asdf']
select_tag(:customer_search, options_for_select(customers))