I had a little adventure today.  I purchased orlandopostgresql.com so that I could have a domain for the Orlando Meet-Up group that I run here in town for PostgreSQL professionals.  Our previous domain orlandopg.com was allowed to expire. Unfortunately I was not its owner.  At any rate as I went to add the necessary MX records and test that the email address I set up worked I happened to notice that my gmail account was reporting that it was unable to retrieve email from my home mail server ( I will post about how I use Gmail for my various domains, yet still run my own mail server as well  in a later post).  Curious I looked and found that postfix was unable to connect to my Pg server due to "Too many connections".  Believe it or not I was pleasantly surprised.  The fact that I was running out of connections meant that I am actually getting traffic to my website.  A quick verification with lsof -i | grep postgres revealed that Apache had eaten up all my connections which were still at the default 100.  This is a perfect opportunity to set up PgBouncer which is my favorite connection pooler for PostgreSQL.  Its config file is very simple and easy to follow, and 10 minutes later the site is back up and running faster than ever and I won't have to worry about running out of connections again.

I have decided to document my upgrade to Lion on my Mac Pro 1,1.  First of all I must say that I am quite happy with my computer.  It's about 5 years old and is the best investment ever.  The cost at purchase was about 5k and frankly it has LASTED.  Which is really unheard of for a PC.  I have had one HDD failure and that has been the only hardware issue.  I started with 4 GB of RAM and have upgraded to 8GB.  2 Dual Core Xeon processors are still kicking today.  I do not do a log of gaming or video editing so they rock for me. 

So anyway I want to upgrade to Lion.  In my eyes I still have plenty of RAM.  If I need to I can get all the way up to 64 GB, but I doubt I will have the need for that.  My biggest bottle necks in performance is the disk IO and graphics.  With Lion's emphasis on animations I am going to try and get the Sapphire Radeon 4870 I bought a while back to work again. I never got it to work in SL, but I am hoping Lion may support it.  Addressing the disk io there is really only adding an SSD drive.  I have discovered that my onboard SATA controller only supports 3gb/sec transfer rates, so there is no point in getting a 6gb/sec drive.  I also think I will only use it for a boot drive, but what size to get ?  I need to know how much space Lion takes up and account for that plus the swap file to get maximum OS performance.  I would get my home directory on my spindles, but what about the Applications dir ?  That is by far my largest directory outside of home.  Should I get a ssd drive big enough for both the OS and the Applications ? or should I move applications to the spindles ?  Does the OS even support moving around these system directories ?  Here is my current boot drive layout:

/System : 4.98 GB

/Library : 148.8 GB

/Applications : 41.04 GB

/Users : 804.54 GB

/Developer : 10.36 GB

/opt : 16.64 GB

Plus I have 102.01 GB of data on the 500 GB drive that the SSD would replace.  Gotta say I wonder if the speed is worth the drop in size.

Well I am off to do some research as I need to know what besides /System HAS to be on the boot volume.  This will determine the size (and cost) of the SSD I would have to get. 

UPDATE: I have decided to get the Crucial M4 CT128M4SSD2CCA 2.5" 128GB SATA III MLC Internal Solid State Drive.  I feel this is the fastest drive in my price range.  At only 128GB though I am not willing to replace one of my 500 GB drives.  So I'm going to stick in my lower optical bay and lose a Super Drive.  This will allow me to stripe my remaining drives and use it as a 2 TB volume.  I'll post more as I accomplish my goals.

So I started using Google+ the other day, and I must say I like it much better than Facebook.  I feel more in control of who gets to see what, plus it integrates into all the other services Google provides like Gmail and Reader seamlessly.  I even stuck a statistics widget on my side bar.  The friend race is on (at least until I get tired of it ).

Here is a link to the recap of the 6th OrlandoPg Meet-Up group. 

http://www.meetup.com/OrlandoPg/messages/13171818/

Ketema.Net is now being served from Nginx.  I have learned how to set up and configure Nginx on Gentoo and OS X.  Its fast!

Today was the third meeting of the OrlandoPg Meetup group that I host.  A member Steven wanted to talk about how he could interact with PostgreSQL using Lua.  I informed the group about PostgreSQL's architecture that allows stored procedures to be written in nearly any programming language.  We also talked about how pretty much every language I can think of has a database interaction library of some sort.



That led to some quick googling and we spent the rest of the meeting in a coding dojo like environment to see if we could set up and work with PL/Lua and luasql-postgres.  This post will detail how to set up the environment, create a simple stored procedure in PL/Lua, and write a program using the luasql-postgres package or "rock".


The set up: I'm using a mac for development.  OS 10.6.6 to be exact.



To install the Lua programming language I used Mac-Ports.
"sudo port install luarocks +curl openssl"

is the command I used.  I actually already had lua installed, so I just installed the luarocks package manager.  If you do not have lua installed, this command will install it as well.



Next I needed to get the lua package (rock) luasql-postgres.
"sudo luarocks install --from=http://luarocks.org/repositories/rocks-cvs/ luasql-postgres POSTGRES_DIR=/usr/local/pgsql"

makes that happen.  This assumes you already have PostgreSQL installed. If you don't use macports to install that and adjust your POSTGRES_DIR accordingly.  I always install PostgreSQL from source myself. 



Next we need to get the PL/Lua extension for PostgreSQL.  It can be downloaded from: PgFoundry Once you have it un-tar it and edit the Makefile.  Adjust the lines:



LUAINC = -I/opt/local/include 

LUALIB = -L/opt/local/lib -llua



Save the file the run
"sudo make && make install"


You now need to install the language into your database.  Head over to your contrib directory, mine is:

"/usr/local/pgsql/share/contrib/"
and run

"psql -f pllua.sql DATABASENAME".

This registers the language handler within your database.



Now you can create a stored procedure using Lua. Example:

create or replace function hello(_name text) returns text as
$$
return string.format("Hello, %s!", _name)
$$ Language pllua;


Now you can create a simple lua script to call this function.  Example:

require "luasql.postgres"

env = assert ( luasql.postgres() )

con = assert ( env:connect( "DBNAME","USERNAME","PASS","DBHOST" ) )

cur = assert ( con:execute( "SELECT * FROM hello('World!');" ) )

row = assert ( cur:fetch() )

print(row)



One non-standard thing I had to do to get this script to "see" the luasql.postgres extension was to make a symlink:



"sudo ln -s /opt/local/var/luarocks/lib/lua/5.1/luasql/postgres.so /opt/local/lib/lua/5.1/luasql.so" 



This is necessary because MacPorts and LuaRocks are not really working together.



You can execute the lua script with the command ""lua SCRIPTNAME" and get the expected output of:

References:

http://www.nessie.de/mroth/lunit/

http://www.lua.org/manual/5.1/manual.html

http://pllua.projects.postgresql.org/

http://www.keplerproject.org/luasql/index.html

St my new job I have been exposed to Git version control.  I had always used SVN, but never really used any of the advanced features like branching or tagging.  I now like git.  It has taken a little trial and error and some reading http://progit.org/book/

This post is about automating commits so I don't have to worry about it and I get an automatic "go back in time" versioning system.

Reference: http://www.spheredev.org/wiki/Git_for_the_lazy Excerpt:

Work in bits

When dealing with git, it's best to work in small bits. Rule of thumb: if you can't summarise it in a sentence, you've gone too long without committing.

Well I wanted to have the bits part, but not having to add a message every time.  I usually code for a single specific purpose at any given time.  But in the process of "getting there" there are many changes that happen and that I may want to go back to in the evolution. So here is what I did:  I made the following cron job:

#!/bin/bash

pushd ROOTPATHOFREPOSITORY

/FULLPATH/git commit -a -m "auto commit"

The key part here is the pushd command.  It was a pain figuring that one out.  executing from cron makes git not "know" where it is so I kept getting "fatal: Not a git repository (or any of the parent directories): .git" errors

Hopefully this little pushd command is useful to someone else. 

Today I decided to get rid of a small thing that has been annoying me for a while.  On Mac OS X, because it is BSD UNIX based it has a local mail delivery agent, specifically Postfix.  Whenever your computer has messages it needs to send to you it delivers them locally to /var/mail/SHORTUSERNAME  This is all well and good, however unless your a Terminal user you'd never know you had any mail, and even if you happened to open the terminal and got the prompt "You have mail"

Once I took a second to think about it I did the following:  change the file /private/etc/postfix aliases and include an entry for your local account shortname and enter the external email address you wish your mail to goto.  run postalias hash:/private/etc/postfix/aliases then newaliases

Thats it.  now mail destined for your local account will get sent to your normal email inbox. 

The purpose of this entry is to demonstrate an important RDBMS concept and how understanding the fundamental concept can be applied to different systems.

Background: For the past 3 to 4 years I have used the PostgreSQL ORDMS extensively in web application development and consider myself an expert.  I have studied database concepts extensively and used them in real world applications.  Because I was free to choose over the last few years I chose to use PostgreSQL. One of the main reasons being cost, it's free to use.  However I wish to illustrate that just because I used the particular system my skills in other systems, particularly MS SQL Server have not been diminished, but actually enhanced from years of practical applied database theory, and those theories can be quickly transferred between systems.

The concept: Partitioning is a global database strategy for dealing with large unwieldy tables.  Over time a heavily inserted table can grow so massive that working with it just becomes slow due to size despite correctly written queries.  What I intend to show is how I understand partitioning and how I have used it in the past with Pg, and then show how in less than 6 hours I was able to translate that understanding into a working implementation using MS SQL Server 2008.

Skills demonstrated: Understanding of database concepts, virtual machine proficiency, software installation of Windows Server 2003, software installation of MS SQL Server 2008, research skill, coding proficiency in Transact SQL and PL/PgSQL and technical writing.

 The demonstration background:  One of my personal projects is a VOIP company: www.tonerouter.com, as well as in my last position our central technology was processing phone calls.  All in all I have dealt with hundreds of millions if not billions of records representing phone calls.  The application environment is high insert and typically summary reads over a known time period.  Meaning when pulling data from this table it is typically for reporting on say the current months worth of calls.   Over time we saw that older calls were hardly ever accessed except for large group reporting for example to see call trends over a past year.  This scenario is perfect for partitioning.

Pg implementation: In Pg partitioning is accomplished through inheritance and use of the rule and or trigger systems.  A parent table is created and then rules or triggers redirect inserts destined to the parent table into the proper child table based on a column criteria.  Integrity is maintained through check constraints.  Queries need only reference the parent table name and the constraint exclusion feature allows the planner to choose the proper child table based on the where clause.  That's the theory anyway.  In practical application I have found that Pg does not handle complex or "sliding" partition parameters very well, such as dates and time.  For simple comparisons though it works great.  To work around the limitation I developed the following strategy: I chose to create partition column that is a computed column based on the month and year of the record being inserted.  The data type is varchar and the values are simply strings in the format "month_year". A string constant lends itself well to Pg's constraint checking.  I used the trigger system to manage creation of the child tables so that they are created automatically upon insert and all the necessary rules are added to the parent table dynamically.  The code to do this is here.

SQL Server Implementation:  Converting this technique to SQL Server was not very difficult.  Because the concept and desired effect were known I simply needed to find the implementation details.  The resources I used to do this were MSDN and specifically the following document: MSSQL Partitioning Strategies.docx.  At first read I was tempted to use the Partitioned View strategy and it on the surface is very similar to what Pg was doing:  Horizontal partitioning with multiple tables.  However upon further reading it became apparent that technique would not be the best.  Instead SQL Server 2008 has the ability to internally manage partitions within a single table.  This technique is actually closer to what Pg's technique is doing because it allows the queries to reference one table name, handles inserts automatically and places them in the right place and allows the planner to select the proper partition based on a where clause.  The steps needed to accomplish this were:

1) Create a partitioning function.
2) Create a partition scheme.
3) Create the table and place it on the partition scheme.

The following code samples and screen shots illustrate the above steps.

CREATE PARTITION FUNCTION cdr_by_month ( varchar(14) )

AS RANGE LEFT

FOR VALUES ( 'november_2009', 'december_2009', 

'january_2010', 'february_2010','march_2010', 'april_2010', 'may_2010', 'june_2010', 'july_2010', 

'august_2010','september_2010', 'october_2010', 'november_2010', 'december_2010' ) ;

CREATE PARTITION SCHEME cdr_monthly

AS PARTITION cdr_by_month ALL TO ([PRIMARY]) ;

One thing I did not go into, but is extremely important is the disk management aspect of partitioning.
Knowing the access and usage patterns of your queries and knowing that you need to put your highly accessed tables on fast
disks is extremely important in db design. In Pg this is accomplished through Tablespaces, in MS SQL Server it is through
file groups. Different terms for the same concept which is knowing where to place your tables to best take advantage of fast
disks.

Final Thoughts: Partitioning is an important database concept. All major relational database systems implement it in some
form. Understanding the concept from a design point of view is important and being able to implement the strategy in
different systems is also important. You never know what tool you will be called upon to get a job done. I hope to have shown
how I am able to use and work with SQL Server 2008 because of my past experience with older SQL Server editions, as well as my ability to take a concept, research it for a different platform, and then implement it in that platform.

Table Definition
Partition String Function