Monday, December 8, 2014

The news RSS Webjob

For the news.sleepysecurity.ninja I decided to build a webjob that would handle the reading of the RSS feeds and putting that data into a database that I could then mine.  Being relatively new to using the SyndicationFeed class in .net, and to webjobs, my code went through a few iterations.

After creating my solution and subsequent website project, I added a webjob using the built in mechanism to Visual Studio 2013, by right clicking on the web project, going to add, and selecting New Azure Webjob Project.

New Azure WebJob Project menu selection

 After running through a small wizard, a project is created for you with a 2 main code files.

class Program
{
  static void Main(){
    var host = new JobHost();
    host.Call(typeof(Functions).GetMethod("ManualTrigger"),new {value = 20});
  }
}
 
The other one is the functions file that contains a definition for ManualTrigger.  The JobHost is part of the Azure WebJobs SDK and provides the following services out of the box:
  1. Get azure account strings from your app.config  (the strings can also be passed in directly to the ctor).
  2. Reflect over your code to find C# methods with the SimpleBatch attributes. (much like how WebAPI discovers controllers)
  3. Listen for new blobs that match the [BlobInput] pattern.
  4. when a blob is found, invoke the function
  5. automatically log the invocation so that you can view the results in a separate dashboard. 
So basically, although you can create and run almost any program as a WebJob, by creating one using the JobHost, it provides some special helpers to deal with interacting with other Azure services.  Pretty neat.  The last point above (the dashboard) is super handy for monitoring the webjobs and getting some quick debugging information.  I will get to that later.

One key thing to note is that I had to add two connection strings to my website config in order to get the dashboard to work correctly.



Another interesting thing is the way that the function is called.  It's a static method that is called via reflection.  Interesting.  This makes it hard to do DI or anything like that.  I guess the design pattern here is that a webjob should really do only one thing, and therefore, you shouldn't need to reuse a bunch of the code.

Okay moving on, since I was just fooling around and not following TDD, this is the first version of code that I wrote.  Ewwwww.  I didn't even want to show it.

I know the above is a picture.  Don't copy it, it sucks.  Firstly, I built a webjob that runs on a schedule.  This is different to a continuous webjob, or one that is called via external trigger.  In order for the JobHost to find this method properly, you have to annotate it with the [NoAutomaticTrigger] function.  I figured that I would update my data every hour, so I set that to be the initial schedule.

There are several things bad about the above code, but most importantly, it isn't testable.  I need this function to follow some logic:

  • Read feed and get the latest info
  • Read the database for all posts from that feed
  • If the feed contains new posts, add them, else, continue on

Although it seems easy to read the above, I want to be able to make it testable.  Furthermore, I don't like how this functions class knows about the database, etc. After some refactoring, here is what the code looks like.


        [NoAutomaticTrigger]
        public static void GetFeedData(TextWriter log, IDataService dataService, IRssService rssService)
        {
            var feeds = dataService.GetAllFeeds();
            foreach (var feed in feeds)
            {
                Console.WriteLine("Processing " + feed.Name);
                var posts = dataService.GetAllPostsFor(feed);

                var newPosts = rssService.ReadRssFeed(feed);

                var postsToAdd = newPosts.Where(x => !posts.Any(y => y.PostId.Equals(x.PostId))).ToList();

                postsToAdd.ForEach(x => dataService.InsertPost(Post.CreateFrom(x, feed)));
                
            }
        }

After some refactoring and moving some core components out into services, I can now test the code.  I used Moq in another test project (which I won't show here).  One interesting thing about webjobs, is that any Console.Writeline messages actually get piped to the output and easily viewed.


Pretty cool.  The last thing I needed to do is add in some error handling and event logging.  It turns out that websites and webjobs use the built in Trace mechanism and log to a log source of your choice automatically.  You can find out more about what those options are at this link

Thursday, December 4, 2014

news.sleepysecurity.ninja launched

Sooo, I love reading up on the latest security news.  I try and do this every morning.  Lately, the content aggregator I was visiting seemed to be under maintenance.  I wanted to spend some more time with the Azure PaaS offering, and decided this was probably a good time to do so.

I have big plans for this site (at least until I get caught up with something else bright and shiny) and hope to continue working on this over the next couple of months.

Right now the architecture is pretty simple.  I used this site to get a good list of security blogs to follow.  I created a webjob to handle the parsing of the feeds and storing them into a database so that I can display.  I then created an MVC5 app to read the data and display it.  Pretty simple right now, but I am hoping to expand on it a bit.

For example, I want to run the parsing of the feeds in parallel.  At this point in time, I'm thinking that I would have one webjob push data onto a queue, and then another webjob pick it up to process.  There are some issues with this architecture (webjobs only scale with the websites, and not independently) but I'm hoping to learn lots by working thought it.

In any event, if you want to check it out, you can do so by clicking here.


Monday, October 20, 2014

Azure IaaS Management and Desired Service Configuration

In a previous post, I started to talk about using Azure for training environments.  At Hitachi, we have been using Azure to host AX training for clients.  It has been going pretty well and I have been working on an internal script that we use to help deploy the training environment.

During a recent podcast I caught on to Desired Service Configuration (DSC) for PowerShell. Released last year, DSC is a management platform that "enables deploying and managing configuration data for software services and managing the environment in which these services run."  Sounds pretty neat!

You can read more about DSC here.  The base modules that come with DSC are pretty good.  They get the job done.  Please note that you will have to install WMF 4.0 to get any functionality.  In any event, what is really exciting is the modules that are currently in "experimental" mode, which are due for release with WMF 5.0.

You can see what is currently underway here.

There are some pretty handy modules in there, including one to manage Azure deployments.  Many times in my own lab, I have blown away everything and started again.  It will be nice to finally have a script that I can use to quickly rebuild a base configuration.

So here we go.

First thing, you will want to have windows 8 or better.  With WMF 5.0 still in preview, there is no support for the older operating systems.  This took most of my time as I had to download a Windows 8 image to play around with.

Once you have your windows 8 image configured (Install WMF 5.0 with the link above, Install Azure Powershell, etc), you can head to DSC Resource Kit to download the latest zip.  The install instructions are fairly easy, you just have to unzip into the modules directory (generally c:\program files\windowspowershell\modules).

This module is written a little bit differently then the other ones.  Basically, instead of calling out to various nodes, you call localhost with the azure configuration.  This then runs what you have configured.

One note is that you will have to allow WinRM to run.  An easy way to configure this is to run

winrm quickconfig

from an elevated powershell command line.  Please note that I tried running this from the ISE command window only to have it fail to complete. From reading through the options, it basically enables the service to run, and creates an exception in the firewall allowing WinRM connections.  There are probably some security considerations to be had when running the quickconfig, but I'm just running this on my local, so .....

DSC supports a separation of intent and configuration, which is a nice touch.  You can create a configuration element that is generic in nature (create storage account, etc) and then combine it with configuration that is user/task specific.

The DSC Resource Kit contains a ton of example code that you can filter through.  Mine is very similar to the xAzure example for creating a VM in the cloud.

As usual, you can grab a copy of my script in github.

As this technology continues to advance, I could see how you would be able to create a script that fully deploys an environment (AD, SQL, WebServer, etc).  Furthermore, you can use a central server to control all deployments and prevent drift.

Have fun!