Scott on Writing

Musings on technical writing...

Maintaining an Audit Trail

Have you ever worked on a data-driven application where it was essential all changes to the system's underlying data could be examined by an auditor at some later point in time?  In medical, university, governmental, and, increasingly, corporate settings, it is becoming increasingly important for legal reasons that one can go back into the system and see precisely how the data changed over time and (most importantly) by whom.

If you've ever needed to create such an auditing application, what techniques have you used?  When I've needed such functionality I've always done the following:

  1. For each database table X that requires an audit trail, create a table named XAudit that has the exact same schema as X but with four additional fields:
    1. XAuditID - a primary key, IDENTITY field
    2. Deleted - a bit field that defaults to 0 that indicates if the record has been deleted.
    3. CreatedOn - a datetime field that defaults to the current date and time (getdate()).
    4. CreatedBy - a field that records who changed the data.  If you are having the users log directly onto the database to perform the edits, you can use the SUSER_SNAME() function to get the username.  If you are having all users come through some standard account, you'll need to record this information in the X table so that it translates over smoothly to the audit table.

      Note: you may already be tracking the last time and user that changed a record's data, and therefore these last two fields would be captured in table X and therefore not need to be additional fields added to the audit table.
  2. Create a trigger on X for UPDATE and INSERT that inserts the contents from the inserted table into XAudit.
  3. Create a trigger on X for DELETE that inserts the contents from the deleted table into XAudit, putting in a value of 1 into the audit table's Deleted field.

That's all there is to it!  At that point you have a complete audit trail showing creation, editing, and deletion of all records, including when the change was made and by whom.

If you keep an audit trail using a different technique, or if there's a better way than how I do it, I'd be interested in learning more - leave a comment.

posted on Friday, June 17, 2005 12:26 PM

Feedback

# re: Maintaining an Audit Trail 6/17/2005 12:42 PM Scott S.

I haven't had to do much of this, but I might need to soon. I'm curious about one thing, how do you deal with foriegn keys? Do you actually define a relationship that enforces referential integrity between XAudit and a lookup table?

My guess is that you would not define relationships from XAudit to lookup tables. That should work because for a given date, you could look at the audit table for the lookup table to see what the value in the lookup was at the time the record was entered in XAudit. I may have answered my own question, but any comments would be appreciated.

# re: Maintaining an Audit Trail 6/17/2005 1:29 PM Scott Mitchell

The way I handle FKs is I divide tables into two classes: I have those that might be changed and therefore I would have to audit those FKed tables, too. That's one class. The other class are "lookup" tables, ones that might get new stuff added to it but should never be (majorly) edited or ever have items deleted. To those I'll add FK constraints in the audit table, so if someone does try to delete a lookup table row, it will fail.

# re: Maintaining an Audit Trail 6/17/2005 4:16 PM steve

Why don't you create some sort of Event Log, sort of like the Windows Event Viewer? Create one table, a column can be the table name, another can be a CLOB of the XML version of the table record...you get the idea.

In my experience, audit logs are *never* looked at. They're just something you do to make managers feel better.

# re: Maintaining an Audit Trail 6/17/2005 5:16 PM Mike Swaim

At my last job, we had a single audit table, using a technique like Steve mentioned. Only the audit triggers listed what changed in more or less human readable format. The triggers that populated the table were a pain. I'm using the audit technique that you mentioned, except they're called shadow tables.
And I have had people look at our audit tables. People have gotten in trouble based on our audit tables.

# Link Listing - June 17, 2005 6/17/2005 8:33 PM Christopher Steen

Link Listing - June 17, 2005

# re: Maintaining an Audit Trail 6/17/2005 8:34 PM Scott Mitchell

Steve, audit trails are *legally required* in certain scenarios, regardless if they're looked at or not! :-)

While not exactly a data-drive Web application example, consider the law suits companies have been getting into because of not keeping an email audit trail.

# what do u think of steve's? 6/19/2005 12:58 AM Emad

Scott, Steve proposed a nice way for maintaining audit trail, can you comment on his post? what do you think about it

# re: Maintaining an Audit Trail 6/19/2005 5:53 PM Joel Hendrickson

In my last project I did pretty much exactly what you describe, but I found this SQL Server add-in recently that looks extremely promising if I have to do it again:

http://hindsight.generalentropy.com/Index.aspx

Maintaining the audit db sucks during development, and especially for me because I have a different db for each company (its a hosted solution)

# re: Maintaining an Audit Trail 6/20/2005 8:07 AM Raterus

http://apexsql.com/apex_sql_audit.htm

I got my company to cough up the money and we purchased this product. I'm very satisfied with the product and how the company approaches support.

Basically the product allows you to create trigger templates that you can apply on a per-table, per-field basis. It will create the needed trigger code in T-SQL, which you can then run in SQL Server. All the audited data goes into two tables for the entire database, and you can view the data either through queries, or the interface you provide. Also, you have the ability to "undo" if need be. It's very handy and has saved our company tons of time from creating a home-grown, tedious solution by hand for every table. It also makes the auditors smile when they come around every year, and that is definitely a good thing! :-)

# re: Maintaining an Audit Trail 6/20/2005 6:18 PM Cheng Yuan Yap Ye

Both hindsight and apexsql look promising. However, sometimes management has to look up reports without the help of programmers in which case management may not know how to create the queries. I have no experience with either of these products but at first glance it seems like it's more of a tool for programmers / DBAs. Perhaps using Crystal Reports could help create more user-friendly reports? But it looks like both tools will help save a chunk of time in creating and maintaining triggers and audit tables.

I use the same audit method as Scott Mitchell does. I did not like the method Steve proposed because data lumped together in a column means you have to parse the data to get a specific value. Of course, XML makes it easier but then storing the data in the original schema gives you the same benefit minus the XPath navigation. The other problem is you have to build a string to store all that data into one column. How do you know how many characters you will need? And you may need to escape certain characters in the string.

One important thing to note is when using triggers to record inserts, updates and deletes remember that the inserted and deleted tables could contain multiple records, not just one.

# re: Maintaining an Audit Trail 6/27/2005 8:28 AM Jamie

Anyone have a solution to auditing table with text columns?

# re: Maintaining an Audit Trail 7/8/2005 4:23 AM Phil Short

Please forgive a newbie (to .NET) asking a dumb question! How do you "create a trigger on X for ..." whatever? Thats may be all there is to it when you know how, but I was left none the wiser!

Thanks!

# re: Maintaining an Audit Trail 7/8/2005 11:34 AM Scott Mitchell

Phil, a trigger can be created through SQL Enterprise Manager or through using T-SQL syntax. There are gobs of articles/FAQs describing how to do this, just Google away: http://www.google.com/search?q=sql%20trigger

# re: Maintaining an Audit Trail 7/19/2005 8:44 AM Jamie

I figure I might give my request one more shot. Has anyone come up with a (graceful) way of handling an audit trail for tables with text column data types? I can think of possibly trying Instead Of triggers, since you have access to to the text fields this way, but I would think you would have to basically be repeating the functional (insert\update\delete) code from your stored proc inside of the trigger in conditional logic (ugh). I was hoping someone had bumped up against this and came up with something. Of course, this looks like it won't be a problem in SQL 2005 ;)

# re: Maintaining an Audit Trail 9/13/2005 9:10 AM Laurie

I am using a trigger to insert a record into an audit log and it is causing the source application to break. Here's what is happening. Somehow, the identity value on the audit log table is being used as the foreign key for the subsequent updates to the source system. I am using a recordset in VB6. My work around was to set the primary key's identity = no and on the trigger set the sid to max(sid) + 1. But my question is, why would the foreign key be set to the sid of the audit log table?

# re: Maintaining an Audit Trail 9/18/2005 9:44 AM Patty

We just installed http://hindsight.generalentropy.com/Index.aspx
for testing and it worked right out of the box. We bought it because of the snapshots. You can just make a normal query and put a date/time after it to get the results at that point in time.

# re: Maintaining an Audit Trail 2/28/2006 9:56 AM Lora

I am in the process of creating an audit trail as noted above and am very new to database design and administration.
I am developing electronic case report forms for a clinical trial that must meet ICH and CFR Regulations.
It has been noted that the audit trail also needs an explanation or reason for each change. I am unsure of how to incorporate this into the procedure????

# re: Maintaining an Audit Trail 3/13/2006 4:32 AM Michal Boleslav Měchura

I don't have any answers but I would like to share a gripe, if I may.

In the database I am developing, originally I did audit trails in a single table which simply stored human-readable, machine-unreadable text stamps of deleted and updated data from every other table in the database. This was fine until people started asking "Oh, this is cool, can we have an undo button as well?"

A couple of meetings later and I'm having to redo my audit mechanism totally now. I am thinking of using triggers along with the "one audit table per each real table" approach, plus widgets in the UI to allow people to look at the audit tables and to re-INSERT and re-UPDATE deleted and updated data.

It seems like this "feature request" has just doubled my workload. It's probably worth it because it's something people are asking for, but the irony is that, had I never implemented any audit trail in the first place, my users probably wouldn't have realized that this something they CAN ask for! Oh wel... :-)

# re: Maintaining an Audit Trail 6/9/2006 10:47 AM Gurb

How about never updating the data and only inserting a new row each time a change is made. Triggers can be problematic espcially when you need more than one trigger on a table.

Title:  
Name:  
Url:
Protected by Clearscreen.SharpHIPEnter the code you see:
Comments   

My Links

Ads Via DevMavens

Archives

Post Categories

 

I am a Microsoft MVP for ASP.NET.
I am an ASPInsider.
<March 2010>
SMTWTFS
28123456
78910111213
14151617181920
21222324252627
28293031123
45678910

Comment Stats

DayTotal% of Total
Sunday 2056.8%
Monday 42514.1%
Tuesday 51917.2%
Wednesday 55618.4%
Thursday 58019.2%
Friday 54718.1%
Saturday 1886.2%
Total 3020100.0%

Hour1Total% of Total
12:00 AM 782.6%
1:00 AM 812.7%
2:00 AM 682.3%
3:00 AM 822.7%
4:00 AM 692.3%
5:00 AM 1264.2%
6:00 AM 1193.9%
7:00 AM 1816.0%
8:00 AM 1926.4%
9:00 AM 1585.2%
10:00 AM 1886.2%
11:00 AM 1936.4%
12:00 PM 2016.7%
1:00 PM 1846.1%
2:00 PM 1695.6%
3:00 PM 1354.5%
4:00 PM 1153.8%
5:00 PM 1073.5%
6:00 PM 1013.3%
7:00 PM 1073.5%
8:00 PM 923.0%
9:00 PM 882.9%
10:00 PM 913.0%
11:00 PM 953.1%
Total 3020100.0%

Comments by Blog Entry Date/Time

Day Entry MadeAvg.Total
Sunday 5.00160
Monday 4.80384
Tuesday 4.04477
Wednesday 7.39680
Thursday 6.26676
Friday 5.07466
Saturday 4.78177
Total 5.403020

Hour1 Entry MadeAvg.Total
12:00 AM 5.2937
1:00 AM 1.002
5:00 AM 0.000
7:00 AM 3.8550
8:00 AM 3.72134
9:00 AM 6.06297
10:00 AM 5.63276
11:00 AM 4.22194
12:00 PM 6.16351
1:00 PM 3.09133
2:00 PM 4.89230
3:00 PM 7.67322
4:00 PM 4.00108
5:00 PM 6.07170
6:00 PM 4.64116
7:00 PM 8.95188
8:00 PM 8.63164
9:00 PM 5.00115
10:00 PM 6.31101
11:00 PM 4.5732
Total 5.403020

Learn More About Comment Stats
1 - All times GMT -8...


Blog Stats

Favorite Web Sites

My Books

My MSDN Articles