Wednesday, May 18, 2011

The Recursive CTE

(Yet another boring org chart example – except this one has multiple roots)

One of my apps has a hierarchial org chart in its database.  We don’t use the hierarchyid data type for this table.  Instead, each row in the table just has a unique org ID and the org ID of the parent org which should exist in the same table.  The ParentOrgID field is NULL in the rows for the root orgs.  Notice I said root orgs – plural.  We have an odd situation with this database where there’s actually two different companies’ trees stored in this single table.  I recently had a requirement to pull out all the orgs for both companies and say what the root was, in addition to presenting how deep in the hierarchy it lived.  Here’s how I did it.

--Example source table

DECLARE @Orgs AS TABLE (

      OrgID INT PRIMARY KEY,

      ParentOrgID INT NULL,

      OrgName VARCHAR(20) NOT NULL

);

 

--Set up some sample data

INSERT INTO @Orgs (OrgID, ParentOrgID, OrgName) VALUES

      (1,NULL,'Company A (Root)'),

      (2,1,'Marketing'),

      (3,1,'Finance'),

      (4,3,'Accounting'),

      (5,2,'Sales'),

      (6,1,'Research'),

      (7,NULL,'Company B (Root)'),

      (8,7,'Production'),

      (9,7,'Development'),

      (10,7,'Processing'),

      (11,10,'Shipping'),

      (12,3,'IT'),

      (13,8,'Industrial Relations'),

      (14,2,'Branding'),

      (15,11,'Intl Shipping'),

      (16,12,'DBAs'),

      (17,12,'Server Ops'),

      (18,12,'Desktop Support');   

 

--Declare some constants pointed at my root orgs (could also use a config table)

DECLARE @ARoot INT;

DECLARE @BRoot INT;

SELECT @ARoot = OrgID FROM @Orgs WHERE OrgName = 'Company A (Root)';

SELECT @BRoot = OrgID FROM @Orgs WHERE OrgName = 'Company B (Root)';

 

WITH AllOrgs AS  --setting up the CTE

(SELECT OrgID,

      CASE WHEN OrgID = @ARoot then 'A'  --This case block is my way of dealing with

            WHEN OrgID = @BRoot then 'B'  --aliasing the company names as a code

            ELSE '' END as [RootCompany],

            1 as [OrgLevel],             --C.S. students would put a 0 here...

            OrgName, ParentOrgID FROM @Orgs

      Where OrgID = @ARoot OR OrgID = @BRoot --This is the query for the first row

            UNION ALL

      SELECT c.OrgID, p.RootCompany, p.OrgLevel + 1, c.OrgName, c.ParentOrgID

            from AllOrgs p

            INNER JOIN @Orgs c

            on c.ParentOrgID = p.OrgID  --query for remaining rows

)

SELECT OrgID, RootCompany, OrgLevel,Orgname, ParentOrgID

FROM AllOrgs Order by RootCompany, OrgLevel, OrgName; --returns the CTE data

 

Inside the CTE are two SELECT statements married by a UNION ALL.  The first one is basically for the first line of data to return (or first lines in this case, since there are two possible roots).

The second query returns the data for each successive row, recursively.  SQL Server will automatically quit recursion when a row is null so you don’t have to worry about closing the loop yourself somehow as long as you do the join correctly and there are no circular references.  In the second query, “p” aliases the CTE itself which allows us to read the parent row’s data and do some basic operations such as adding one to its OrgLevel value.  Note that as with all UNION statements in SQL Server, you only have to alias the column names in the top query – column names in later queries are ignored.

Finally, I select the data I want with my desired sorting.  CTEs self-destruct after use (note the semicolon after the select, but not after the closing parenthesis after the CTE setup), so if you need to do multiple operations on the results, insert them into a temp table first.

Thursday, April 28, 2011

How to determine the current SSIS runtime environment

If you want to test if you’re currently debugging an SSIS package via BIDS/Visual Studio, or running normally via DTExec, you can use the following utility functions inside a Script task.

    Public Function SSIS_IsBIDS() As Boolean

        Return (SSIS_Environment() = "DtsDebugHost")

    End Function

    Public Function SSIS_IsDTExec() As Boolean

        Return (SSIS_Environment() = "DTExec")

    End Function

    Public Function SSIS_Environment() As String

        Return System.Diagnostics.Process.GetCurrentProcess.ProcessName.ToString

    End Function

Friday, February 18, 2011

Renumbering Rows in a Table Variable (or Table/Temp Table)

I was recently working on a data cleanup problem where I had to do lots of comparisons of one row to the next row and I was trying to do my best to avoid using cursor for this.  I was using the old trick of having an IDENTITY() field and doing a self-join where the identity field in the main table = the identity field in the “comparison” table –1.  This was working great until I had to do a second set of deletes from my temp table and realized that some of my rows were now missing and therefore my IDENTITY() numbers weren’t always sequential anymore.

I found a great trick from MSDN (here) on how to get around this.  Below I’m providing some sample code to demonstrate this trick.  I believe this should work on SQL Server 2005 or higher.  The trick is by the comment called “Renumber the RowIDs”.  It involves using a CTE and the ROW_NUMBER() window function as part of an UPDATE statement.

Be sure to use semicolons after your statements when you start using CTEs or MERGE statements as SQL Server can start getting confused if the code is ambiguous.

--Setup a table variable with some data that has some definite duplicates and some

--“not so sure” duplicates – this happens to be someone’s job history.

DECLARE @SomeData TABLE

       (RowID INT UNIQUE,

        EmpID VARCHAR(8),

        PositionTitle VARCHAR(40),

        PositionEffDate DATETIME,

        PositionID VARCHAR(8));

        

INSERT INTO @SomeData VALUES

       (1,'00000012','Manager','1/1/2000','ABC123'),

       (2,'00000012','Manager','1/1/2000','ABC123ZZ'),

       (3,'00000012','Sr. Manager','1/1/2002','ABC125'),

       (4,'00000012','Sr. Manager','1/1/2002','ABC125'),

       (5,'00000012','Sr. Manager','1/1/2002','ABC125ZZ'),

       (6,'00000012','Director','1/1/2006','ABC126'),

       (7,'00000012','Director','1/1/2006','ABC126'),

       (8,'00000012','Sr. Director','1/1/2009','ABC129ZZ');

      

SELECT 'Has Dups and possibly bad rows' as [Description],

       * FROM @SomeData;

 

 

--this gets rid of rows that are certainly dups - same position title,

--  effective date, and position ID.

DELETE FROM @SomeData WHERE RowID IN (

       SELECT compare.RowID

              FROM @SomeData main

                     LEFT OUTER JOIN @SomeData compare on main.EmpID = compare.EmpID

                           AND main.RowID = compare.RowID -1

                     where main.PositionTitle = compare.PositionTitle

                           AND main.PositionID = compare.PositionID

                           AND main.PositionEffDate = compare.PositionEffDate);

                                        

SELECT 'Pure dups removed, some possibly bad rows, split row ids' as [Description],

       * FROM @SomeData;

 

--Renumber the RowIDs

WITH newPH AS(

       SELECT RowID, ROW_NUMBER() OVER(Order By RowID ASC) as [newRowID] FROM @SomeData)

UPDATE newPH

       SET RowID = newRowID;

 

SELECT 'Pure dups removed, some possibly bad rows, fixed row ids' as [Description],

       * FROM @SomeData;

--this gets rid of rows that are still dups - same position title,

--  effective date, keeping the first row that matches

DELETE FROM @SomeData WHERE RowID IN (

       SELECT compare.RowID

              FROM @SomeData main

                     LEFT OUTER JOIN @SomeData compare on main.EmpID = compare.EmpID

                           AND main.RowID = compare.RowID -1

                     where main.PositionTitle = compare.PositionTitle

                           AND main.PositionEffDate = compare.PositionEffDate);

 

--See that the "Sr. Director" row is kept since that has a different title than the

-- prior row.  The second Sr. Manager row (with the ZZ code in position ID) is removed.

SELECT 'Fixed up' as [Description],

       * FROM @SomeData;

Thursday, January 27, 2011

Creating a deep hierarchy with FOR XML

I have two tables in an application that I support.  One is called [Report], and the other is [ReportParameter]. 

      --SAMPLE DATA
      DECLARE @Report TABLE (
            ReportID INT PRIMARY KEY,
            Title VARCHAR(30),
            TitleHelp VARCHAR(100),
            ReportName VARCHAR(50),
            ShowSearchScreen BIT,
            AdminsOnly BIT
      )
      DECLARE @ReportParameter TABLE (
            ReportID INT NOT NULL,
            ParameterName VARCHAR(30) NOT NULL,
            [Type] VARCHAR(15) NOT NULL,
            PRIMARY KEY (ReportID, ParameterName)
      )
     
      INSERT INTO @Report VALUES
            (1,'Report #1','First report...','Report1.rpt',1,0),
            (2,'Report #2','Second report (no selection screen)','Report2.rpt',0,1)

      INSERT INTO @ReportParameter VALUES
            (1,'@StartDate','datetime'),
            (1,'@EndDate','datetime'),
            (2,'@ProductCode','string')

      SELECT * FROM @Report
      SELECT * FROM @ReportParameter
My goal is to convert these two tables into an XML file.  There are a few tricky things about the destination schema:
  1. Specific Name for the root element and all child elements.
  2. Mixed levels of hierarchies.
  3. Some denormalization of the data is required because the new schema uses a “tag” format rather than allowing some custom fields like my “ShowSearchScreen” field.
The first part is pretty easy.  I can write this query to get the below formatted XML output.
SELECT ReportID as [@ID], Title, TitleHelp as [Description],
            ReportName as [FileName]
      FROM @Report r
            FOR XML PATH('Report'), ROOT('Reports');
Results in:
<Reports>
  <Report ID="1">
    <Title>Report #1</Title>
    <Description>First report...</Description>
    <FileName>Report1.rpt</FileName>
  </Report>
  <Report ID="2">
    <Title>Report #2</Title>
    <Description>Second report (no selection screen)...</Description>
    <FileName>Report2.rpt</FileName>
  </Report>
</Reports>
Note the FOR XML PATH(‘Report’) which identifies what I want each data node to be called, and the ROOT(‘Reports’) option which lets me name the root node.  So far so good.  Now for adding my parameters.  I need to create the Parameters node by doing a subselect, and I can get a column to show as an attribute by prefixing the name with an @ sign:
      SELECT ReportID as [@ID], Title, TitleHelp as [Description],
            ReportName as [FileName],
            (
                  SELECT ParameterName as [@Name],
                        [Type] as [@Type]
                        FROM @ReportParameter rp
                        WHERE rp.ReportID = r.ReportID
                        FOR XML PATH('Parameter'), TYPE
             ) as [Parameters]
      FROM @Report r
            FOR XML PATH('Report'), ROOT('Reports');
This results in the following XML:
<Reports>
  <Report ID="1">
    <Title>Report #1</Title>
    <Description>First report...</Description>
    <FileName>Report1.rpt</FileName>
    <Parameters>
      <Parameter Name="@EndDate" Type="datetime" />
      <Parameter Name="@StartDate" Type="datetime" />
    </Parameters>
  </Report>
  ...
</Reports>

The “TYPE” keyword tells the main query that the child query will be passing up XML nodes and that it shouldn’t try to escape it as text.

Almost there.  Now I just want to expose my “ShowSearchScreen” field.  However, my schema doesn’t have a field for it. Instead, the schema supports arbitrary tags with an ID=”” atrribute and node value.  This means I have to take advantage of a special function called data() which I will assign as the name of the column I want to show up as the node’s value.
SELECT ReportID as [@ID], Title, TitleHelp as [Description],
            ReportName as [FileName] ,
             (
                  SELECT 'ShowSearchScreen' as [@ID],
                              rss.ShowSearchScreen as [data()]
                        FROM @Report rss where rss.ReportID = r.ReportID
                        FOR XML PATH('Tag'), TYPE
            ) as [Tags],
            (
                  SELECT ParameterName as [@Name], [Type] as [@Type]
                        FROM @ReportParameter rp where rp.ReportID = r.ReportID
                        FOR XML PATH('Parameter'), TYPE
             ) as [Parameters]
            from @Report r
            FOR XML PATH('Report'), ROOT('Reports');

This results in our final XML document which meets our specs.

<Reports>
  <Report ID="1">
    <Title>Report #1</Title>
    <Description>First report...</Description>
    <FileName>Report1.rpt</FileName>
    <Tags>
      <Tag ID="ShowSearchScreen">1</Tag>
    </Tags>
    <Parameters>
      <Parameter Name="@EndDate" Type="datetime" />
      <Parameter Name="@StartDate" Type="datetime" />
    </Parameters>
  </Report>
  ...
</Reports>
Bonus tip – how to do a pivot: If you wanted to get the “AdminsOnly” field to show up as a second tag, you could extract it by changing the query to something like this:
SELECT ReportID as [@ID], Title, TitleHelp as [Description],
      ReportName as [FileName] ,
       (
            SELECT * FROM (SELECT 'ShowSearchScreen' as [@ID],
                        rss.ShowSearchScreen as [data()]
                  FROM @Report rss where rss.ReportID = r.ReportID
            UNION ALL
                  SELECT 'AdminsOnly' as [@ID],
                        rss.AdminsOnly as [data()]
                  FROM @Report rss where rss.ReportID = r.ReportID)
                  AS tags FOR XML PATH('Tag'), TYPE
      ) as [Tags],
      (
            SELECT ParameterName as [@Name], [Type] as [@Type]
                  FROM @ReportParameter rp where rp.ReportID = r.ReportID
                  FOR XML PATH('Parameter'), TYPE
       ) as [Parameters]
      from @Report r
      FOR XML PATH('Report'), ROOT('Reports');
This is a pretty standard  SELECT * FROM (<my subselect query>) with the FOR XML stuff on the outside.  I could add as many tags as I wished to do so by adding more UNION ALL statements.  This allows for pivoting the fields into the tag node list and will generate something like this:
    <Tags>
      <Tag ID="ShowSearchScreen">1</Tag>
      <Tag ID="AdminsOnly">0</Tag>
    </Tags>