LION Searching, AJAX Coding With jQuery and ASMX, Jaxer Tinkering, QAM HDTV on VMC, COD4 Gaming, CacheFile Forums, and Jabber Services

by Jon Davis 24. February 2008 23:49

I haven't blogged much lately but there are reasons for that. Lots of little things going on. Lots of things needing closure.

Lots of things, though, that have recently enjoyed closure, and lots of new beginnings.

Blogging about blogging is a big no-no, but let me summarize some of the kewl things I've been either up to or else observing:

  • At work, our internal Lucene server that I built had been flaking out. I finally fixed the glitch, and I was able to continue improvements to it. In the process of trying to figure out the problem, I checked out Lucene Solr a bit. When basic unit tests failed, likely due to learning curve issues, we decided to stick with our own. 
    Ours also has a new name: LION (Lucene Index[ing] On/Over .NET). I'd like to open source it someday but being as it's becoming an integral part of our web site foundations, that isn't likely to happen soon.
    It uses WCF to be accessible over a variety of communications protocols including HTTP (SOAP), TCP, and Named Pipes.
    It uses a custom Query object that supports condition trees, field selections, ordering .. basically all the stuff supported in Lucene query but generic to Lucene (so a SQL Server or a Solr query handler can also be built). A few things I am looking into for enhancing LION are:
    • Faceted searching - This is a term CNET seems to have coined with Solr, but it's a commonplace trend.
      The idea is basically to get a list of "narrow-down" hyperlinks in the margin of search results, along with a count for each; for instance, at eBay, when you search for "dell", in your search results, on the left sidebar you'll get a list of clickable filters, along with quantity of hits, such as "Laptops (8270)". When you click on it, you get more hyperlinks that are more specific as they fall under the filter you clicked. 
      In engineering terms, in the case of Solr, when you perform a search, the search result are cached, and each hit is evaluated againt a list of several known categories. Each hit is flagged for each match with a BitSet (somethng .NET developers are not typically keen to, think List<bool> but more performant). From the BitSet, counting and filtering can be performed quickly.
      The hassle is that faceted searching has several components. You have to consider a) what fields are considered facet categories and b) defined by whom, the document schema or the query, c) filtering the search on the facet, and d) returning the facet counts with the search results.
      I can't move forward yet until I feel confident that I haven't missed a lot here. This is just where I'm at in my understanding of faceted searching thus far.
    • XML-defined schemas - Right now, Lucene document structures are defined in non-serializeable C# classes; that is, they "serialize" by way of manual propogation of property values to a Lucene Fields collection. Ugh. This is an area where when I saw Solr I slapped myself on my forehead and moaned, oh brother, what have I done?!
    • A QueryBuilder object - This is what I typed up yesterday as a design reference (the first sample will be built, not the second, but the second is sort of a psuedo-TSQL explanation of the first):
      Query q = new QueryBuilder(   // most members return a created instance of this
          lionHandler)              // allow the handler to make proprietary query adjustments         
        .Select("IdxA.Field", "A")
          .And("IdxB.FieldB", "B")
          "IdxB.D < 299",           // conditioned facet counting
          "Dingbats $0 - $299"      // facet renaming
          "IdxB.D >= 300",
          "Dingbats at least $300"
        .From("IndexA", "IdxA")     // index selection, 2nd param is alias
          .And("IndexB", "IdxB")
        .Where(                     // returns a ConditionBuilder object with a parent QueryBuilder
          "A < B"                   // condition compares against another field
        .And("B LIKE \"ASDF\"~5)    // parsing
        .AndWhere                   // nests a BooleanCondition
          .Where("A > 4")
          .Or("A < 2")
          .Escape                   // escape nest; optional
        .AndWhere                   // nests another boolean condition, sibling to prior
          .Where("IdxA.C < 50")     // this one excludes a range
          .Or("IdxA.C > 500") 
        .AndThen                    // returns the parent QueryBuilder instance
          // todo; see FacetCount for syntax
        .Order("A", "B")            // sort the results
        .Page(2, 20)                // page 2, 20 hits per page
        .Query;                     // generate the query object
        // equivalent:
      Query query = QueryParser.Parse(
        lionHandler, @"
          IdxA.Field AS A,
          IdxB.FieldB AS B
          IdxB.D >= 300
        )   AS " + "\"Dingbats $0 - $299\"" + @"
          IdxB.D >= 300
            AS " + "\"Dingbats at least $300\"" + @"
          IndexA AS IdxA,
          IndexB AS IdxB
          A < B
          AND B LIKE " + "\"ASDF\"~5" + @"
            A > 4 OR A < 2
            IdxA.C < 50 OR IdxA.C > 500
          --todo, see FACETCOUNT for syntax
        ORDER BY A, B
        PAGE 2
        HITSPERPAGE 20";
      As is evident, there are a LOT of new features to be implemented in the actual service in order to pull a query like this off, such as comparing field against another field (a feature SQL users are spoiled on).
    • Index joining - I would love to be able to filter out the results of two distinct queries and join on a field value of each, i.e. an INNER JOIN. That's much further down the road, though.
  • A co-worker has been having a lot of fun with jQuery lately .. actually, both of the co-workers on each side of me have been. But on Friday the fellow engineer to my left had fun getting jQuery to do AJAX/JSON calls to ASMX and SubSonic, where the ASP.NET web app had a reference to System.Web.Extensions.dll (from GAC-deployed ASP.NET AJAX; this DLL is the dependency for JSON-enabled ASMX). He had to add [ScriptMethod] attributes to the method signatures, and the MIME type in the AJAX request from Javascript *HAD* to be application/json or else the ASMX would refuse to serve JSON. This required a custom getAsmxJson() function for jQuery, but fortunately jQuery, thanks to Javascript itself, is rediculously easily extensible.
    In the end, he was able to build a framework where he could draw up a simple HTML form, a quick and dirty SubSonic object definition, and a simple ASMX handler, to end up with a completely AJAX-controlled form management system. 
    I suppose in retrospect he could've just used a ScriptManager and WebForms for even more hands-off ease. I think the long-term goal for all of us, though, is to move away from WebForms and move towards a web services oriented web architecture.
  • The same fellow also took a look at Jaxer. I knew he'd get into this.
    I didn't realize it but Jaxer being an Apache mod it will support working with PHP-generated output. Nor did I realize that the discussion of wanting an ISAPI module for IIS would not mean an alternate ISAPI handler from ASP.NET, but rather a complementary one, just as it already works with PHP on Apache.
    The discussions, then, are valid, that a) <script runat="server"> won't work in an ASP.NET environment combined with Jaxer, and b) .NET-enabled Jaxer is otherwise feasible with the creation of an ISAPI module. Just needs to happen already.
    These are exciting times. Jaxer is cool stuff. There are a lot of concerns, though, that people have that Jaxer, having the Mozilla DOM on the server (unique to each reqeust and/or user session), will not scale well. Fortunately, it can be argued that server hardware capacity is now continuing to grow at a faster rate than user demand. I don't know how true that argument is, though.
  • At home, I obtained a Silicon Dust HDHomeRun unit to enjoy QAM digital cable over my analog cable subscription, piped to Vista Media Center as though it were an ATSC broadcast. Vista Media Center [still] does not support QAM (unencrypted, open television over cable) broadcasts for whatever reason, but HDHomeRun is one of only a couple known workarounds. It has two tuners, both support either QAM (cable) or ATSC (antenna). It not only works, but it works surprisingly well. The signal quality and quantity of ATSC (antenna) broadcasts is remarkable, it's tragic that you can't combine QAM and ATSC on Vista Media Center (due to the fact that VMC doesn't know that the two tuners are different). But the tie-in with VMC was otherwise nearly seamless. I am now recording shows like American Idol, Nature, The Tonight Show, etc., in amazing high definition detail. 
  • I am also finding myself firing up my Xbox 360 a lot less lately. The majority of my Xbox 360 time before had been for the Vista Media Center Extender, to watch live and recorded TV over the network from my PC. Now that I have an HTPC right in my living room, there's rarely any need for the Extender; I only need it now if either I'm already in the Xbox "desktop" or else I'm finding the PC overwhelmed with other tasks or something, such as if audio and video are out-of-sync, which happens less when viewing from the Xbox.
    I also installed Call of Duty 4 on the HTPC today, and I downloaded (and purchased) Pinnacle Game Profiler, which makes COD4 on the PC work pretty much just like COD4 on the Xbox, and so far it seems to work great!! I also got it to all work with Half-Life 2, which required me to reconfigure it slightly so that it didn't always sprint when fully forward, and I mapped sprint to left joystick button and flashlight to right joystick button.
  • I finally restored forums for at I also upgraded to phpBB 3; hopefully its CAPTCHA is better; I may end up doing some home-grown CAPTCHA.
    I had shut it down because of the spam that was posted there every day, most of which was pornography, and some of which included actual porn images in the posts. It's just awful, as doesn't target an age group at all.
  • Over the last couple weeks, I also got e-mail services working on the dedicated server that hosts You can reach me at jon@ or at stimpy77@ at that domain name. Forum posts will now propogate announcement e-mails properly now, too.
  • Speaking of, I have been keeping it up-to-date. Updates have been about every two weeks or so now. These updates were part of the motivation to get the forums back online, though; I thought about adding a blog instead, but I figure, nah, discusson forum is better. We just need to get the spam under control.
  • This evening I set up Jabber services at (Feel free to sign yourself up. My account there is stimpy77 -AT-, with proper treatment of "-AT-" of course.)
  • There's more I've been up to, but these are all I have time to bring up now. I wish I could blog in detail about all of these things, but at least I mentioned each of them.

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5


Add comment

(Will show your Gravatar icon)  

  Country flag

  • Comment
  • Preview


Powered by BlogEngine.NET
Theme by Mads Kristensen

About the author

Jon Davis (aka "stimpy77") has been a programmer, developer, and consultant for web and Windows software solutions professionally since 1997, with experience ranging from OS and hardware support to DHTML programming to IIS/ASP web apps to Java network programming to Visual Basic applications to C# desktop apps.
Software in all forms is also his sole hobby, whether playing PC games or tinkering with programming them. "I was playing Defender on the Commodore 64," he reminisces, "when I decided at the age of 12 or so that I want to be a computer programmer when I grow up."

Jon was previously employed as a senior .NET developer at a very well-known Internet services company whom you're more likely than not to have directly done business with. However, this blog and all of have no affiliation with, and are not representative of, his former employer in any way.

Contact Me 

Tag cloud


<<  May 2021  >>

View posts in large calendar