{"id":12046,"date":"2009-12-09T15:44:53","date_gmt":"2009-12-09T13:44:53","guid":{"rendered":"https:\/\/mamchenkov.net\/wordpress\/?p=12046"},"modified":"2009-12-09T15:44:53","modified_gmt":"2009-12-09T13:44:53","slug":"web-statistics-and-visitor-tracking-things-you-need-to-know","status":"publish","type":"post","link":"https:\/\/mamchenkov.net\/wordpress\/2009\/12\/09\/web-statistics-and-visitor-tracking-things-you-need-to-know\/","title":{"rendered":"Web statistics and visitor tracking : things you need to know"},"content":{"rendered":"<!-- google_ad_section_start -->\n<p>First of all, just to make it clear, I don&#8217;t recommend writing your own web statistics \/ analytics \/ tracking application. \u00a0<a href=\"http:\/\/www.google.com\/analytics\/\"><strong>Google Analytics<\/strong><\/a><strong> can track and report pretty much everything you will ever need.<\/strong> Period. If you think it can&#8217;t do it, chances are you just don&#8217;t know how. \u00a0That&#8217;s much easier to correct than to write your own tracking \/ reporting application. \u00a0I promise. \u00a0In case though, Google Analytics doesn&#8217;t do something that you need, grab one of those Open Source applications and modify it to suit. \u00a0While not as easy as learning Google Analytics, that would still be much easier than doing your own thing from scratch.<\/p>\n<p>However, if you still decide to roll out your own tracker, here are a few things that you need to know.<\/p>\n<ul>\n<li><strong>Use the bicycle, don&#8217;t reinvent it.<\/strong> Most of the tracking applications that I&#8217;ve seen use some form of JavaScript, which is appended right before the end of the page markup. \u00a0Said JavaScript collects as much statistics as you need and generates a request to an image on the remote server (your tracking application), passing gathered statistics as parameters to the image. \u00a0On the server side, your tracking application gathers sent parameters, merges them with whatever else you can get from the server side, and saves in the database or in your data storage of choice.<\/li>\n<li><strong>Keep ad blocking applications in mind.<\/strong> Many ad blocking plugins for different browsers block 1&#215;1 pixel images from remote servers. \u00a0Be a bit more creative &#8211; use a 2&#215;1 or a 1&#215;2 pixel image. \u00a0If it is a transparent GIF at the bottom of the page, nobody will notice it anyway.<\/li>\n<li><strong>Gather as much as you can from the server side.<\/strong> It&#8217;s simpler, and you minimize the chances of breaking things with an URL which is too long (your GET request for the image with all parameters can run pretty long, especially if you pass current page and referring page URLs).<\/li>\n<li><strong>Minimize the length of your parameter names and values when you pass them to image GET request.<\/strong> Again, this is to avoid extremely long URLs. \u00a0You can sacrifice readability in your JavaScript and instead document parameters in the server side tracker application.<\/li>\n<li><strong>Record both client&#8217;s IP address and possible proxy server&#8217;s IP address.<\/strong> That is available for you in the request headers ($_SERVER[&#8216;HTTP_X_FORWARDED_FOR&#8217;] in PHP for example). \u00a0Once you got the IP addresses, use <a href=\"http:\/\/www.maxmind.com\/app\/city\">GeoIP<\/a> to lookup the country, region, city, coordinates, etc. \u00a0It&#8217;s better to do so at the time you record the data. \u00a0There is a free GeoIP service as well, but it will give you much less information. \u00a0The commercial one is not that expensive.<\/li>\n<li><strong>Record client&#8217;s browser information.<\/strong> <a href=\"http:\/\/php.net\/manual\/en\/function.get-browser.php\">Browsercap<\/a> is very useful for that. \u00a0However, it&#8217;s better to parse user agent string with browsercap at the report \/ export time, not at the request recording time. \u00a0This will guarantee that you always have the most correct information about the browser in your report. \u00a0Browsercap gets updated with new signatures pretty often.<\/li>\n<li><strong>If you are tracking a secure site (HTTPS), chances are you won&#8217;t have referrer information available to you<\/strong>. \u00a0Apparently, that&#8217;s a security feature.<\/li>\n<li>If you use both JavaScript and PHP to figure out the referrer, keep in mind that JavaScript uses <em><strong>document.referrer<\/strong><\/em>, while PHP uses <em><strong>$_SERVER[&#8216;HTTP_REFERER&#8217;]<\/strong><\/em>. \u00a0Notice that one is spelled with two Rs, while the other &#8211; with one. \u00a0That might save you some troubleshooting time.<\/li>\n<li>It&#8217;s better to use the same JavaScript code snippet across all your sites. \u00a0<strong>To avoid SSL-related security warnings, your JavaScript need to figure out if it&#8217;s in HTTPS web site or in plain HTTP one.<\/strong> See Google Analytics example on how to actually do that. \u00a0 It doesn&#8217;t hurt to have a signed SSL certificate for the HTTPS hosting of your tracker application.<\/li>\n<li><strong>Don&#8217;t forget about HTML and URL escaping \/ encoding.<\/strong> Check that everything works properly for you in different browsers. \u00a0JavaScript is still hard to nail right sometimes.<\/li>\n<li><strong>Keep the version of tracker application in every request log entry.<\/strong> This will much simplify your migrations later. \u00a0One of the ways to keep this automated is to use tags \/ keyword substitutions in your version control software (here is <a href=\"http:\/\/svnbook.red-bean.com\/en\/1.4\/svn.advanced.props.special.keywords.html\">how to do this in Subversion<\/a>).<\/li>\n<li><strong>Make sure your tracker spits out that transparent image no matter what.<\/strong> Broken image icons are very visible and you don&#8217;t want those on your site just because your tracker database went down temporarily.<\/li>\n<li>For the best cross-site tracking, start tracker session, which will remain the same when visitor will go from one of your tracked web sites to another. \u00a0<strong>If your tracked web sites use sessions, pass their IDs to tracker, so that both tracked and tracker session IDs could be logged in the same request.<\/strong> This will help you link stats from several sites together, as well as do all sorts of drill-downs into site-specific stats straight from the bird-view reports.<\/li>\n<li><strong>Don&#8217;t be evil!<\/strong> There is a lot that you can collect about your visitors. \u00a0Make sure that you tell them exactly what you are collecting and how you are using it. \u00a0Aggregate and anonymize your logs to prevent negative consequences. \u00a0I&#8217;m sure you know what I mean.<\/li>\n<\/ul>\n<p>Once again, think really good before you decide to do one yourself. \u00a0It&#8217;s not an easy job. \u00a0And even if you grab all the data you want and save it in your database, there is an incomparably bigger issue to solve yet &#8211; reports, graphs, export, and overall visualization and analytics part of that data. \u00a0Why would you even want to go into that?<\/p>\n<!-- google_ad_section_end -->\n","protected":false},"excerpt":{"rendered":"<!-- google_ad_section_start -->\n<p>First of all, just to make it clear, I don&#8217;t recommend writing your own web statistics \/ analytics \/ tracking application. \u00a0Google Analytics can track and report pretty much everything you will ever need. Period. If you think it can&#8217;t do it, chances are you just don&#8217;t know how. \u00a0That&#8217;s much easier to correct than &hellip; <a href=\"https:\/\/mamchenkov.net\/wordpress\/2009\/12\/09\/web-statistics-and-visitor-tracking-things-you-need-to-know\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Web statistics and visitor tracking : things you need to know<\/span><\/a><\/p>\n<!-- google_ad_section_end -->\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_links_to":"","_links_to_target":""},"categories":[1,18,62],"tags":[2231,1139,38,1041,2158],"keyring_services":[],"class_list":["post-12046","post","type-post","status-publish","format-standard","hentry","category-general","category-programming","category-technology","tag-google-analytics","tag-javascript","tag-php","tag-statistics","tag-subversion"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":11840,"url":"https:\/\/mamchenkov.net\/wordpress\/2009\/09\/30\/a-glimpse-into-google-analytics-power\/","url_meta":{"origin":12046,"position":0},"title":"A glimpse into Google Analytics power","author":"Leonid Mamchenkov","date":"September 30, 2009","format":false,"excerpt":"For a few years now I always recommend Google Analytics to anyone who is looking for a statistical \/ analytical package for their web site.\u00a0 While there are a few alternatives, I think that almost none of them can match Google Analytics in both ease of use and analytical power.\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2009\/09\/google_analytics_help.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":12160,"url":"https:\/\/mamchenkov.net\/wordpress\/2010\/02\/18\/how-accurate-is-google-analytics\/","url_meta":{"origin":12046,"position":1},"title":"How accurate is Google Analytics?","author":"Leonid Mamchenkov","date":"February 18, 2010","format":false,"excerpt":"That's the question that I was asked recently by one of the co-workers. \u00a0 It is simple and not so simple at the same time. \u00a0It really depends on what you are looking for, what is the acceptable accuracy, and what is that you are comparing Google Analytics with. For\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":13386,"url":"https:\/\/mamchenkov.net\/wordpress\/2010\/09\/14\/urchin-google-analytics-in-a-box\/","url_meta":{"origin":12046,"position":2},"title":"Urchin &#8211; Google Analytics in a box","author":"Leonid Mamchenkov","date":"September 14, 2010","format":false,"excerpt":"Google Analytics has proven itself over and over again as an extremely valuable tool for pretty much everyone interested in website statistics. \u00a0But as awesome as it is, Google Analytics has a number of limitations. \u00a0These don't come handy when you need to analyze non-public websites, such as intranets or\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2010\/09\/urchin_7-500x351.gif?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":28481,"url":"https:\/\/mamchenkov.net\/wordpress\/2018\/04\/18\/what-is-gtag-js-with-google-analytics-and-do-i-need-it\/","url_meta":{"origin":12046,"position":3},"title":"What Is gtag.js with Google Analytics and Do I Need It?","author":"Leonid Mamchenkov","date":"April 18, 2018","format":false,"excerpt":"If you are using Google Analytics, or any other Google marketing tool on your website, make sure to read through the \"What is gtag.js with Google Analytics and do I need it?\" article.\u00a0 It explains the change in the Google Analytics tracking scripts which are slowly rolling out, and provides\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":22560,"url":"https:\/\/mamchenkov.net\/wordpress\/2014\/09\/22\/tracking-token-stripper-addon-for-google-chrome\/","url_meta":{"origin":12046,"position":4},"title":"Tracking Token Stripper addon for Google Chrome","author":"Leonid Mamchenkov","date":"September 22, 2014","format":"link","excerpt":"Tracking Token Stripper addon for Google Chrome removes UTM variables from the URL.","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"tracking token stripper","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2014\/09\/tracking-token-stripper-500x316.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":9680,"url":"https:\/\/mamchenkov.net\/wordpress\/2005\/11\/17\/signed-up-for-google-analytics\/","url_meta":{"origin":12046,"position":5},"title":"Signed up for Google Analytics","author":"Leonid Mamchenkov","date":"November 17, 2005","format":false,"excerpt":"Everyone and their brother is talking about the new Google service - Google Analytics. Basically, this is a smart way of getting website statistics. Instead of installing and configuring a local web log analyzer, you just sign up for the Analytics, insert some JavaScript code into your website and have\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_sharing_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/12046","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/comments?post=12046"}],"version-history":[{"count":0,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/12046\/revisions"}],"wp:attachment":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/media?parent=12046"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/categories?post=12046"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/tags?post=12046"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/keyring_services?post=12046"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}