Url Shortener gem – bit.ly api wrapper in ruby

26 11 2009

Url shortener gem came out of my need to tweet rss feeds to twitter every 5 minutes and shortening the url is the first thing required to do that. I looked at different url shortening services and decided to use bit.ly‘s api as it appeared to be mature, comprehensive and more stable compared to other url shortening services.

The url_shortener gem allows to shorten/expand urls, retrieve stats and other information about them.

How to use

Since it is an API wrapper hence an account with bit.ly i.e. login and api key is required to use the url_shortener gem. bit.ly gives an API key to every user so if you have an account with bit.ly then you already have the key otherwise you can get it easily by signing up with bit.ly

Then the usual drill to install the gem

gem sources -a http://gemcutter.org
sudo gem install url_shortener

For a quick test, start an irb session and


require 'rubygems'
require 'url_shortener'

Before sending any request to bit.ly you need to initialize the Authorize and Client classes with bit.ly credentials:

authorize = UrlShortener::Authorize.new 'your_bitly_login', 'your_bitly_api_key'

If you don’t have a bit.ly account then you can use the bitlyapidemo key to try this

authorize = UrlShortener::Authorize.new 'bitlyapidemo', 'R_0da49e0a9118ff35f52f629d2d71bf07'
client = UrlShortener::Client.new(authorize)

bit.ly provides following four main end points or resources:

I suggest to check the xml outputs by clicking on the above end points to familiarize with the results returned.

Shorten Urls
Once you have the client object then call the shorten method on it with the url as the parameter.

shorten = client.shorten('http://www.yahoo.com') # => UrlShortener::Response::Shorten object

This will return a UrlShortener::Response::Shorten object which has several useful methods to parse results and access xml elements

shorten.result # => returns a hash of all data returned from bitly
shorten.urls # => returns the short url

The shorten method accepts a single or multiple url as parameters to shorten the urls

shorten = client.shorten('http://www.yahoo.com', 'http://www.google.com', 'http://github.com')

apart from the #urls and #result method you can also pass
#urls_with_long_url_keys, #hashes_with_long_url_keys, #hash and any xml element name within results->nodeKeyVal element

shorten.urls_with_long_url_keys
#etc

Expand Urls
To expand urls, either the hash or short url hash can be passed to the expand method which returns UrlShortener::Response::Expand object

expand = client.expand(:shortUrl => 'http://bit.ly/1RmnUT') # => UrlShortener::Response::Expand object
#OR
expand = client.expand(:hash => '1RmnUT') # => UrlShortener::Response::Expand object

Similar to the shorten method, you can call #result method to get a hash of xml or #url/#long_url to get the expanded long url

expand.result # => returns a hash of all data returned from bitly
expand.url # => returns a long url string

Stats

stats = client.stats(:shortUrl => 'http://bit.ly/1RmnUT') # => UrlShortener::Response::Stat object
#OR
stats = client.stats(:hash => '1RmnUT') # => UrlShortener::Response::Stat object
stats.result # => returns a hash of all data returned from bitly

There is a #referrers and #user_referrers method in the UrlShortener::Response::Stat class that does the xml parsing to return all referrers and as usual any xml element name can be passed as the method name to the stats response object like

stats.hash # => returns url hash
stats.referrers # => returns an array of referrers
stats.user_referrers # => returns an arrays of referrers
stats.clicks # => returns number of clicks

Info

info = client.info(:shortUrl => 'http://bit.ly/1RmnUT') # => UrlShortener::Response::Info object
#OR
info = client.info(:hash => '1RmnUT') # => UrlShortener::Response::Info object
info.result # => returns a hash of all data returned from bitly

All elements within doc element are available as methods to the info response object.
If you are a ruby purist and don’t want to use camelized method calls then you can use the underscore, for e.g.

info.htmlMetaDescription
#OR
info.html_meta_description

will work seamlessly and give the same results. Checkout the results return by bitly’s info resource to see what other methods are available to info response object.

url_shortener gem is available on github.





Design Pattern Relationships

17 11 2009

Useful diagram showing relationship between different design patterns from GoF.

Design Pattern Relationship





Ruby gem for stock / finance data

28 10 2009

A post on the yahoo finance / stock gem had been overdue for a long time. Last week I added the xml support for the result formatting and published a new version of the gem. So I thought its a good time to write a bit about how I got started with it, implementation details, usage, etc.

Problem

My mother has a portfolio of stocks and it became a bit difficult for her to keep an eye on their performance either by using online or print medium. So she beautifully used her powers of delegation and asked me to check them regularly :). As usual, first thought, use google or yahoo finance for this kinda stuff but soon I wanted to generate more data for the stocks and I found that they lack them. I said, what the hell and built a rails application that would go to a stock exchange site and scrape the pages. Then I thought, an app like this could be useful for other people like me but the problem was, the site site I was using for scraping was country specific. I decided to make it a bit more general purpose before putting on the github and started using ScrAPI gem (ScrAPI uses CSS selectors for scraping) to scrape data from a site that has stocks from more than just one country. ScrAPI was working fine but wasn’t consistent in some cases. So I started looking for finance APIs, obviously google has a finance API but you need an account with google finance to use it. I think that’s a bit inconvenient for user’s to first have a google finance account to use a library though they may already have but that’s still a hassle. So started looking elsewhere and found yahoo finance. Best thing about yahoo finance is you don’t need any account to use it. And yahoo provides data as CSVs and its easy to use YQL (Yahoo Query Language). Hence, I decided to use Yahoo finance for the data purposes and write a ruby gem that can be used in any ruby or rails application. On another note, I met a couple of awesome yahoo guys at barcamplondon7 who showed me some cool APIs; Developer Apps like, open tables, YAP, blueprint etc,. Check them out here

This is how I got started on nas-yahoo_stock gem.

Implementation

The tests for the gem code are written using rspec 1.2.2 and it has cucumber features for integration testing.

There are different interface classes (based on their responsibilities) that are derived from the base interface class. For example, the quote interface provides quotes for multiple companies in a single request with variable parameters, symbol interface submits request to a yahoo page and scrapes data using some regex rules, etc.

Then we have different result classes to easily present result data in the format we want them. This has been achieved by implementing the Strategy Pattern to switch easily between different result formats. Benefit: a single result interface for the YahooStock::Quote / YahooStock::History / YahooStock::ScripSymbol classes for getting and presenting data into different formats (the examples are given in the Usage section).

I kept one thing into consideration while building the gem was to send requests to yahoo only when it is absolutely necessary. For instance, the gem sends a request to yahoo only when the results method is called on an object like YahooStock::Quote / YahooStock::History / YahooStock::ScripSymbol and not during any other operation. Which is fine but what about calling results multiple times for the same data and format or to generate different formats for the same data? An easy solution for that is to implement the Observer Pattern. The observers in nas-yahoo_stock gem keep an eye on the data requested whenever the results method is called on the quote, symbol or history objects and if the data requested currently is different from the previous request then only the gem hits yahoo else it returns the previous result without hitting yahoo.

The source code for the gem is available on github -> nas-yahoo_stock

Usage

To use the library, start by:

gem sources -a http://gemcutter.org
sudo gem install nas-yahoo_stock

Once the gem is installed, we can go to the irb and

require 'rubygems'
require 'yahoo_stock'

Before finding the history or quote information about a company stock we will need the stock symbol for that company. YahooStock::ScripSymbol class gives us all the possible stock symbols for a company. For instance, if we want to find the stock symbol for Yahoo.
symbol = YahooStock::ScripSymbol.new(‘Yahoo’)
To get the results,
symbol.results.output
the above method call give us a string of all the values.
If we want to store the result in a file and come back to it later, then we can
symbol.results.store('file_name')
The results can be easily presented into different formats like
1. to return an array instead of a string use:
symbol.results(:to_array).output
2. to return a hash, use:
symbol.results(:to_hash).output
3. to return xml, use:
quote.results(:to_xml).output

Once we have the stock symbol, then the quote or history information about that stock can be retrieved easily. Lets get to the history part first.

To get a stock’s history, the YahooStock::History class needs to be initialized with a hash of stock symbol, start and the end date for which we need the stock data

history = YahooStock::History.new(:stock_symbol => ‘yhoo’, :start_date => Date.today-20, :end_date => Date.today -2)
Then we can call all the same methods on the history object that we have called on the symbol object to get results, like
#to get history as an array
history.results(:to_array).output
#to return a hash, use:
history.results(:to_hash).output
#to return xml, use:
history.results(:to_xml).output
#to store results in a file, use:
history.results.store('filename')

To get quote information about a stock
1. Initialize quote object
quote = YahooStock::Quote.new(:stock_symbols => [‘YHOO’, ‘GOOG’])

If we call the results method on the quote object then it will give us only two values for the above two stocks which are included by default on object initialization. But a number of parameters can be passed to the quote object to get information on and these parameters can be viewed by sending valid_parameters message to the quote object.
quote.valid_parameters

To view the current parameters being used in the quote object
quote.current_parameters

The parameters are also categorised in realtime, standard and extended categories. To use parameters for these categories just pass one of the category names to the quote object and it will use those parameters, for e.g.
quote.realtime

To use all parameters just use:
quote.use_all_parameters

To view the current stock symbols used
quote.current_symbols

To add more stocks to the list
quote.add_symbols(‘MSFT’, ‘AAPL’)

To remove stocks from list
quote.remove_symbols(‘MSFT’, ‘AAPL’)

Then we can call all the same methods on the quote object that we have called on the symbol and history objects to get results, like
#to get quote as an array
quote.results(:to_array).output
#to return a hash, use:
quote.results(:to_hash).output
#to return xml, use:
quote.results(:to_xml).output
#to store results in a file, use:
quote.results.store('filename')

If you need the results in some other format apart from the formats provided by the gem then you can simply pass in a block to the result object like
quote.results{YourSpecialFormat.method_name}





Sphred.com – off internet

5 10 2009

Sphred.com is off internet as I am not able to devote much time due to my other commitments.

But I still have all the data and code, so if someone wants to rerun it then email me or I just may put it on github when I have sometime.





Pretty rdocs for local gems

22 09 2009

Bdoc gem provides a nice interface to move between different gems. To view rdocs for installed gems


gem install manalang-bdoc
bdoc

the second step will open a browser window to view rdocs for them.

This will show the docs for all version of the gem installed.

To make it look a bit prettier and do some gotapi kind of ajaxy stuff then install the hanna rdoc template gem.
Note: Just so you know gotapi doesn’t work at all if you have javascript turned off.
gem install mislav-hanna
open ~/.gemrc and add

:rdoc: --inline-source --line-numbers --format=html --template=hanna

rebuild all your gems

gem rdoc --all --no-ri
That will take sometime depending upon the number of gems are installed locally on the machine. Once all the gems are rebuilt, then run
bdoc
on command line to view rdoc using new templates.





Do I need Stored Procedures? Maybe or Maybe not.

31 07 2008

I have written a couple of posts about 7-8 months back (here and here) on how to call stored procedures from a rails application. After using stored procedures in my app for a while I started to think about whether we really need them. So over a couple of months I discussed it (mainly short discussions about stored procedures not whether to use them or not) with my colleagues, friends, other people  and gave some thought to it. If you are new to stored procedures then here is a sort of textbookish definition.

Stored procedures are available to applications accessing an RDBMS, they are similar to constructs seen in other programming languages and stored in database data dictionary. Stored procedures remove the compilation overhead required for inline queries as they can be pre-compiled, hence they run faster in some cases. However, now most RDBMS, as they evolved, implement statement caches to avoid repetitive compilation of SQL statements.

Why people think we need them?

People, especially with database background or DBAs, are of the view that we should use stored procedures for most of the tasks involving database because they offer following advantages:

  1. Stored procedures separate server side function from the client side, i.e, put all your business logic in stored procedures.
  2. Stored procedures are modular and offer advantages compared to embedded queries buried within the GUI code.
  3. They are faster, secure and maintainable
  4. Reduced network usage between clients and servers

Why people or I think we don’t need them?

  1. If the application in question even has a remote chance of running on more than one RDBMS then I would just kick stored procedures out of the door because the exact implementation of stored procedure varies from one database system to another and I certainly won’t like to tie my app to a particular database.
  2. Having business logic in stored procedures is not only breaking every rule of good programming but also not following the tried and trusted way of architecting n-Tier applications, i.e. separating presentation, business logic and storage/data access. Do not mix storage/data access with logic because stored procedures are just an extension of data access tier. As they say, use database as a very efficient file cabinet. The only concern of a database should be to store data or persistent objects and the business logic should reside in the application code. Database in itself is not an application you are developing, it’s part of the layer/tier that stores data. We use SQL in RDBMS and SQL is simply a data manipulation language hence very limited in its scope. A programmer can do everything in the application code that can be done in a stored procedure but the opposite of this is definitely not possible. Hence, databases are not programmable enough to include all the business logic. If you somehow manage doing that using temp tables, inserting redundant rows into the tables, by spending ‘n’ number of extra hours or other hacks then you are gonna face performance/scalability issues and surely maintenance problems later in the development life cycle, i.e. a nightmarish, incomprehensible & complex POS (piece of shit). These issues will come back persistently to haunt you and your team. Hence, the advantage 1 and part of advantage 3 mentioned above is not valid anymore.
  3. Storing business logic in the stored procedures also leads to more work for testing compared to having them in (Object Relational Mapping) OR / domain model class and stored procedures/triggers make test driven development less productive. Also, if I am using ORM then I do not need to use any SQL in my application code because the ORM will generate optimised queries for me on the fly and will also bring simplification and consistency in the coding of whole team. Hence, advantage 2 and part of advantage 3 mentioned above is not valid any more.
  4. Stored procedures can very easily get out of synch with the application code. For instance, if I have different versions of my app in the source code repository and in case something goes wrong with my latest version then I can very easily switch back to an older version by simply checking out / exporting stuff from the repository. But these facilities do not exist for stored procedures / triggers. It’s troublesome to have business logic residing in stored procedures because they are in the database and once you alter an existing procedure the old one is gone, hence no going back. Now some of you may argue to keep each stored procedure/trigger in some text file and commit it to the repository with the app but that’s again creating additional work for your team which means you are also adding chances of error/mistakes. However, if you do this then you still have to run your stored procedures/triggers with every update that’s again prone to mistake/errors.
  5. When something goes wrong then stored procedures don’t provide meaningful feedback unless you code all the exception handling within the procedure itself.
  6. You can’t pass objects to stored procedures. If you are using an ORM, and you have a class that is mapping to a table then you can simply pass the object of that class and you are good but with stored procedures you have to pass each field or object attribute as parameters and you may end up passing lots of them, may be 20, 30,40 or even more. Hence, you end up writing more code which essentially is more error prone and if you get a ‘bad call’ error then go figure what just happened. That’s just creating one object what if you need to add a record to 4 tables simultaneously using transaction and let’s say each of them have 10 fields then you need to pass 40 parameters. Change is the only constant in life, especially in the business environment.  Lets say, due to some business requirement your database schema changes and you have to add/remove columns to/from those tables. If you are using stored procedures then you atleast need to take these steps to make sure that your application is still working fine:
    • Add new data vaildation rules for CRUD operation.
    • Change the way data presented to users and your form fields where you allow users to create/update records.
    • Add or remove new parameters to the stored procedure calls from your front end
    • Add or remove new parameters to your stored procedure definition
    • Add or remove new parameters to all other stored procedures or triggers from where you are calling your affected stored procedure(s).

    If you are using ORM instead of stored procedures then you need to take these steps to make sure that your application is OK:

    • Add new data vaildation rules for CRUD operation.
    • Change the way data presented to users and your form fields where you allow users to create/update records.

    That’s it and you will also have your whole app properly version controlled so that you can switch back and forth seamlessly. So why do you want to do something that involves more work leading to delays, increased complexity, less productivity and has more chances of error? Hence, stored procedures/triggers are actually a constraint in your system and according to Theory of Constraints; generally, in any system or process there is one constraint at a time. To improve overall system or process performance first find the constraint then improve it or eliminate the constraint entirely. Do I need to say more?

If you think you can use stored procedures to hide business logic,  then you are definitely using stored procedure what it is not good or designed for. I neither like putting SQL queries in my code nor have any problem using stored procedures only if they can offer me/my team/employer some measurable benefits but in my opinion an OR model is much better in terms of developer time, code, productivity and maintenance. Using stored procedures indicates adding another layer, another language, complexity and losing database portability. If I have an application running on ‘n’ number of different clients then it will be much easier for me to keep my application code synchronized and maintainable rather than the stored procedures and triggers. Triggers pose another problem area because application does not know whether they exist hence do not even have a clue whether they have run or not. What if you have some logic in one of your trigger and that itself may be calling another stored procedure(s) (again some logic on important data) and due to one or the other problem trigger didn’t fire, you are screwed because neither you nor your application will know whether the trigger was fired or not unless your application users see some discrepancy in their data or reports, report them back to you and then you run multiple sql queries and check everything manually. How kewl is that? Not for me atleast.

Having said all that, I don’t mean the stored procedures were never useful or the advantages of stored procedures cited by DBAs were never true, yes they were true until 6/7 years ago. But with time, new technologies come into existence, some lose their advantages and some become obsolete. There is no point in sticking to something for which you have other better options available.





Formatting/styling will_paginate

16 04 2008


I received few requests in the past ( personally and recently a comment on my post describing how to use will_paginate plugin ) about how to format pagination links while using will_paginate plugin.

So here it is:

When you add

<%= will_paginate @obj_instance %>

in your views to show your pagination links then it also creates an element (div) and wraps the whole thing under the class name pagination. The one with the current page will have the class name current.

Here is the html code generated by will_paginate


<div class="pagination">
<a href="/page_path?page=2">« Previous</a>
<a href="/page_path">1</a>
<a href="/page_path?page=2">2</a>
<span class="current">3</span>
<a href="/page_path?page=4">4</a>
<a href="/page_path?page=5">5</a>
<a href="/page_path?page=4">Next »</a>
</div>

Now, it is totally upto your imagination how you want to style it using CSS.

But, if due to some reason you don’t want to use the class pagination and add your own then do something like this


<%= will_paginate @obj_instance, :class => 'my_class' %>

Similarly, if you want to add an element id then

<%= will_paginate @obj_instance, :id => 'my_element' %>

Let’s say if you have done both like so:

<%= will_paginate @obj_instance, :id => 'my_element', :class => 'my_class' %>

then the new html generated by will_paginate will be

<div class="my_class" id="my_element">
<a href="/page_path?page=2">« Previous</a>
<a href="/page_path">1</a>
<a href="/page_path?page=2">2</a>
<span class="current">3</span>
<a href="/page_path?page=4">4</a>
<a href="/page_path?page=5">5</a>
<a href="/page_path?page=4">Next »</a>
</div>

Yes, it is that simple.








Follow

Get every new post delivered to your Inbox.