RavenDB Import / Export in code

The recommended way of doing backups automatically is using Smuggler on the server in a scheduled job or if you do manual backups and restores it’s through the RavenDB management studio but what happens if you’ve not got the ability to do either of those? It’s time to build in import/export functionality into your application.

This was a requirement that I was looking at a while back and then when I did some further investigation more recently I had found a thread on the RavenDB Google group and the associated pull request to allow for smuggling functionality without the http server running; problem solved! So how can it be used?

The changes that were made extracted out an interface ISmugglerApi to allow for a common mechanism for both the server/http version and the internal embedded implementations.

So how do we backup? This is an implementation from an MVC application perspective. Context is a way to determine the difference in implementation if you require to develop and/or deploy onto the different platforms.

public async Task<ActionResult> Backup()
{
    SmugglerOptions smugglerOptions = new SmugglerOptions { BackupPath = Server.MapPath(ServerMapPath) };
    switch (Context)
    {
        case Context.Embedded:
            DataDumper dumper = new DataDumper(((EmbeddableDocumentStore)MvcApplication.Store).DocumentDatabase, smugglerOptions);
            var embeddedExport = dumper.ExportData(null, smugglerOptions, false);
            await embeddedExport;
            break;

        case Context.Server:
            var connectionStringOptions = new RavenConnectionStringOptions
            {
                ApiKey = “insert ApiKey”,
                DefaultDatabase = “db_name,
                Url = “http://localhost:8080”
            };

            var smugglerApi = new SmugglerApi(smugglerOptions, connectionStringOptions);
            var serverExport = smugglerApi.ExportData(null, smugglerOptions, false);
            await serverExport;
            break;
    }

    return File(Server.MapPath(ServerMapPath), "application/binary", "backup.dump");
}

For the embedded version you specify the mapped server path on the local machine at the location the server will write the file to, initiate a new instance of the DataDumper class and call ExportData. There is no need for a stream or to get the incremental flag for this basic usage.

The hosted server version you have to specify they api key, database name and the url of the server to use. These are all bit of information you should know if you or can get access to if you are developing in this way. Once this is established you can call ExportData in the same way as the DataDumper call.

Restoring the backed up file is very similar but in reverse.

public async Task<ActionResult> Restore()
{
    SmugglerOptions smugglerOptions = new SmugglerOptions { BackupPath = Server.MapPath(ServerMapPath) };
    switch (Context)
    {
        case Context.Embedded:
            DataDumper dumper = new DataDumper(((EmbeddableDocumentStore)MvcApplication.Store).DocumentDatabase, smugglerOptions);
            var embeddedImport = dumper.ImportData(smugglerOptions);
            await embeddedImport;
            break;

        case Context.Server:
            var connectionStringOptions = new RavenConnectionStringOptions
            {
                ApiKey = “insert ApiKey”,
                DefaultDatabase = “db_name”,
                Url = “http://localhost:8080”
            };

            var smugglerApi = new SmugglerApi(smugglerOptions, connectionStringOptions);
            var serverImport = smugglerApi.ImportData(smugglerOptions);
            await serverImport;
            break;
    }

    return View("Index");
}

There are many other options which can be used but the above is one of the simplest implementations.

When I was doing some research on how to use this mechanism there were no examples I could find in the blogosphere however reading the the code and looking at the unit tests of the RavenDB source it was enough of an example to work with. I’d highly recommend reading the unit tests and the source of open source if you are struggling to find examples of how to use it; it is good documentation especially if the mechanism you are trying to use is used in the software itself.

Tournament – Results and Calculating Scores with RavenDB Indexes

This is the 4th part of a series that is following my progress writing a sporting tournament results website. The previous posts in this series can be found at:

• Part 1 – My Journey with Asp.Net MVC and RavenDB – Introduction to Tournament
• Part 2 – Tournament – Project Setup
• Part 3 – Tournament – Learning io CRUD

In the previous post I explained the basic CRUD operations and interesting routing issues which I found to get the setup I was looking for.

What do I want to achieve?

By the end of this post I will have shown you how I have started the process off of saving match data, how I have chosen a jQuery plugin to aid with the user interface when entering results and my first run at a calculated index to calculate the scores of the matches so far.

Match up details

The arguably most important part of a result stats website is recording the match ups between players and the outcomes. Once this is done the accumulation of the result scores can be implemented. How they are linked to Events is a small issue compared to recording the actual results.

Let’s start with the first part and see how it goes.

So let’s take a look at some of the potential scenarios which could happen and how we will deal with them.

1. Single and Doubles match ups with home and away players
2. Players could play for the other team due to number match ups
3. Players could move club between events
4. Ringers could play for any club at any time

To address these issues for each match up there needs to a be a collection of home players and away players. These will be the player records at the point in time at which the match up takes place; the assumption here is the match details won’t be changed after any player changes are made for future events. As we’re using a document database for this and the root is the match then each match will have an instance of the player record at that time point when it occurred.

public class Match : RootAggregate
{
    // todo: link to leg
    public Classification Classification { get; set; }
    public Result Result { get; set; }
    public Team WinningTeam { get; set; }
    public Team HomeTeam { get; set; }
    public Team AwayTeam { get; set; }
    public ICollection<Player> HomePlayers { get; set; }
    public ICollection<Player> AwayPlayers { get; set; }
    public ICollection<Comment> Comments { get; set; }
}

As we can see above we also want a reference to both the home and away teams as well as the winning team. With these details by themselves it would work fine and store the data as required. I have decided to include a Classification enum to make it easier to distinguish between Singles or Doubles match ups (the doubles classification will also be used for 3 on 2 match ups). Due to the issue of 3 on 2 scenarios I’ve kept the players as collections and not explicitly player 1 and player 2. In addition to this I’ve added in an associated “Result” enum value. 

public enum Result
{
    Incomplete = 0,
    Draw,
    HomeWin,
    AwayWin
}

Having this result value will aid with determining the scores without having to do comparisons between different properties (and potential sub properties) with the aim to make the score index calculations cleaner to read.

Chosen

Chosen is a jQuery plugin to aid with making dropdowns and multi selects more user friendly. I first came across this about 12 months back when it was introduced at work to aid with large dropdown selectors. I decided to use this because I would be selecting individual teams from dropdowns but mainly I would be selecting multiple players from a multi select form input and wanted to make it keyboard friendly to speed up result entry.

As you can see from the small screen shot above it aids with single selection and makes the multi select look cleaner by hiding the “noise” around a default vanilla multi select input form control.

I’m not completely sold this will be the final solution for this issue however it performs as required for now. When I get to working on the UI then I will decide if Chosen is the best way to go or to find an alternative solution.

Score calculations

I had read multiple blog entries on how to do a Map / Reduce index in Raven DB. The general concept of this is to map from a document collection into a simple form with a counter set to 1 which can then be collated and grouped on a property and the counter value can be aggregated to get the full result. This would be fine however I want to count Wins, Losses, Draws and eventually Extras per player in one index. I knew there had to be a way but what was it?

This is when I discovered AbstractMultiMapIndexCreationTask. It works in a similar way to the more basic AbstractIndexCreationTask<T> however instead of having a specific type it “maps” - as defined by T – it allows for a generic method called AddMap<T>. AddMap allows the document type to be defined and then to map into a separate result type (as per the AbstractIndexCreationTask) however you can call this as many times as you like (although I’d imagine there was a limit somewhere either implementation wise or coding standards wise) on any number of document collection types.

Using the AddMap method I added in a “map” for Home wins / losses, Away wins / losses and Draws irrespective of if the player was playing at home or away. I will look to build on this in the future to do more stats for points such as “performs better on home soil” type scenarios.

AddMap<Match>(matches => from match in matches
                    from player in match.HomePlayers.Select(x => x)
                    where match.Result == Enumerations.Result.HomeWin
                         select new Result
                         {
                              PlayerId = player.Id,
                              Wins = 1,
                              Losses = 0,
                              Draws = 0
                         }
                );

Above is one of the AddMap calls. Looking at this you can see how we can determine, using a combination of HomePlayers and Result value, that the home players have a win. After another internal code review I have noticed that I didn’t need to do the Select(x => x) however this was left over from earlier development when I was looking at extracting out just the player id and will be updated.

I now had the “mapped” part of the implementation. At this point I have a collection of Result objects with each entry being the result of an individual match per player in that match.

The Reduce part is a simple aggregation using the PlayerId as the group key. This can be easily established using the code below.

Reduce = results => from result in results
                    group result by result.PlayerId
                    into r
                    select new Result
                        {
                            PlayerId = r.Key,
                            Wins = r.Sum(x => x.Wins),
                            Losses = r.Sum(x => x.Losses),
                            Draws = r.Sum(x => x.Draws)
                        };

This will give you a result record per player, who has played at least one match, and their “record” for all events.

At this point it got me the data I wanted but I also wanted to display the player information as well as the results. I first started off looking to do a “join” between the index results and a Player query although I realised I was still thinking too “relational” and needed to find another cleaner option; enter TransformResults.

TransformResults is another function on an index which you can access much like AddMap and Reduce. The tooltip description is a bit cryptic “The result translator definition” but if you allow Visual Studio to work its magic to set out the signature it makes a bit more sense …

TransformResults = (database, results) =>

What this gives you is a delegate in the form of a function expression which takes in IClientSideDatabase and the IEnumerable<T> of your results set which is the outcome of the Reduce we did earlier. The IClientSideDatabase interface, and it’s associated implementation which gets passed in by RavenDB at run time, is purely a lightweight data accessor. As per the documentation the methods are used purely for loading documents during result transformations; perfect! Using this mechanism I was able to Load the player record for each of the result values to be able to access the additional information.

TransformResults = (database, results) => from result in results
                            let player = database.Load<Player>(result.PlayerId)
                            select new Result
                                          {
                                              PlayerId = result.PlayerId,
                                              Player = player,
                                              Wins = result.Wins,
                                              Losses = result.Losses,
                                              Draws = result.Draws
                                          };

Conclusion

In this post I have gone over how I’ve decided to model match information, used a jQuery plugin called Chosen to aid with the user experience when adding / editing a match data and how I used a AbstractMultiMapIndexCreationTask derived index in Raven DB to calculate the scores for each player, returning their scores along with their player data.

As always you can follow the progress of the project on GitHub. Any pointers, comments, suggestions please let me know via Twitter or leave a comment on this blog.

Tournament – Learning to CRUD

This is the 3rd part of a series that is following my progress writing a sporting tournament results website. The previous posts in this series can be found at:

• Part 1 – My Journey with Asp.Net MVC and RavenDB – Introduction to Tournament
• Part 2 – Tournament – Project Setup

In the previous post I explained the setup that I will be using for my project. In this post I will go through the basic CRUD operations and interesting routing issues which I found to get the setup I was looking for.

What do I want to achieve?

By the end of this post I will show you how I have managed to sort out the CRUD operations for a simple Entity object including working with RavenDB to get auto incrementing identifiers. I will also describe the tweaks to the default routing definitions to allow for the url pattern I want to continue to work with for the public urls.

Should your Entity be your ViewModel?

After my post the other day about your view driving data structure I decided to put in ViewModel / Entity mappings after originally writing the CRUD setup straight onto both my Team and Player entities. I didn’t like the fact that there were DataAnnotations in my data model so I refactored it to use ViewModels instead to avoid this “clutter” and keep the separation between View and Data. To aid in the mapping between the two types I Nuget’ed down AutoMapper.

To keep with the default pattern of everything which is required by the application being setup in Global.ascx.cs in Application_Start but done through calling static methods into specific function classes in App_Start I created an AutoMapperConfig class with a static register method (see below).

public class AutoMapperConfig
{
    public static void RegisterMappings()
    {
        // Team
        Mapper.CreateMap<Team, TeamViewModel>();
        Mapper.CreateMap<TeamViewModel, Team>();

        // Location
        Mapper.CreateMap<Location, LocationViewModel>();
        Mapper.CreateMap<LocationViewModel, Location>();

        // Player
        Mapper.CreateMap<Player, PlayerViewModel>();
        Mapper.CreateMap<PlayerViewModel, Player>();
    }
}

With doing this I can keep all my mappings in a single place and it won’t clutter the Global.ascx.cs with unnecessary mess.

Auto incrementing Ids

The default setup for storing documents in RavenDB is using a HiLow algorithm. However this does not give the auto incrementing id functionality I’d like. Why will this not do? Well the id makes up part of the url route linking to the individual entity and I want to keep that as sequential and not dot about. The default ID creation would be fine for Ids which aren’t exposed to the public but they will be in this application.

Thanks to the pointers from Itamar and Kim on Twitter I was able to achieve the desired effect without much trouble at all. In the create method on the TeamController when storing the data I had to call a different override to the Store method and specify the collection part of the identifier to use.

RavenSession.Store(team, "teams/");

And that was it.

Routing setup

There was two issues when it came to routing; accessing the record for admin functions and accessing the details of a specific entity to keep with the routing in the current site. Default RavenDB ids are made up of the collection name of the document and the id eg “teams/2”. To get Asp.net MVC to understand these the controller actions which accept an id need to be changed from int to string

public ActionResult Edit(string id) { .. }

And the default route gets updated as follows:

routes.MapRoute(
name: "Default",
url: "{controller}/{action}/{*id}",
defaults: new { controller = "Home", action = "Index", id = UrlParameter.Optional }
);

On a brief look it might not look like anything has changed however the {id} part of the url has been updated to {*id} to allow for anything before the id part – this will allow the Raven DB default collection based indicators to match eg. teams/2

The next part was the public routes to see the details of the different entities. Let me explain this a bit further. The default url to access teams is /team and for a specific team I want to have a url which follows the url pattern of /team/{id}/{slug} eg /team/1/team-name-slug-here. This is to partially aid with SEO but also make the urls more human readable when posted on forums, twitter etc.

This can be established using the following route:

routes.MapRoute(
    name: "EntityDetail",
    url: "{controller}/{id}/{slug}",
    defaults: new { action = "detail", slug = UrlParameter.Optional },
    constraints: new { id = @"^[0-9]+$" }
    );

The constraint is to limit the matching to integer values only – remember routes match from specific to general. We want to limit the restriction to just this url pattern only.

The next hurdle to over come was the issue with the raven DB identifier which includes the collection name (eg teams/2 or players/5), all I want is the integer part to make up the url and hence match the url route pattern which we defined above. To do this I wrote a small extension method which works on a string. It’s very basic, see below. 

public static string Id(this string ravenId)
        {
            var split = ravenId.Split('/');
            return (split.Count() == 2) ? split[1] : split[0];
        }

This will allow me to reference the identifier properties on my view models however be able to access the integer part of it to setup the route for the entity specific none admin views. So how do we use this?

<viewModel>.Id.Id()

Conclusion

In this post I have shown how to setup the basic mappings between entities and view models. In addition to this I have shown how I made Raven DB play nice with sequential incrementing identifiers. I have also discussed the current routing requirements and how to extract out the integer identifier part of a Raven DB collection identifier to be able to set these as required.

In the next post I will briefly discuss how I am planning on saving match results and take a look at the first attempt at using indexes in Raven DB to calculate each players results.

As always you can follow the progress of the project on GitHub. Any pointers, comments, suggestions please let me know via Twitter or leave a comment on this blog.

Tournament – Project Setup

For the first few iterations of this project it will be based in a single Asp.Net MVC 4 Internet Web Application. This can be found through the regular File > New process in Visual Studio. I may at some point decide to break it up into different projects but at the moment I don’t see an immediate need and don’t want to over complicate it at the beginning.

Once Visual Studio has setup the project and is happy then it’s time to add in the required packages to use RavenDB. Most of the development will be done using just the client RavenDB functionality as I have a RavenDB server running locally however when I get close to deployment I will create it so that it can either run with a dedicated server instance or in an embedded way.

The NuGet packages which I have installed are the following:

<package id="RavenDB.Client" version="2.0.2261" targetFramework="net40" />
<package id="RavenDB.Client.MvcIntegration" version="2.0.2261" targetFramework="net40" />

These give us the basics required to get the data access rolling as well as a way to start profiling what is going on.

I’ve decided to go for the setup of having the DocumentStore instance setup in the Global.ascx.cs on Application_Start and then create a session on each action call via a BaseController and finalise at the end of the call of each action. After some Googling this seems to be a simple and common way to setup a RavenDB session, avoiding DRY situations and I read somewhere (can’t remember where) that creating a session is relatively inexpensive processing wise. If this changes or I find an issue with performance then I may need to come back to it.

The base controller is very simple …

public class BaseController : Controller
{
    public IDocumentSession RavenSession { get; protected set; }

    protected override void OnActionExecuting(ActionExecutingContext filterContext)
    {
        RavenSession = MvcApplication.Store.OpenSession();
    }

    protected override void OnActionExecuted(ActionExecutedContext filterContext)
    {
        if (filterContext.IsChildAction)
            return;

        using (RavenSession)
        {
            if (filterContext.Exception != null)
                return;

            if (RavenSession != null)
                RavenSession.SaveChanges();
        }
    }
}

Now we’re ready to roll!

As always you can follow the progress of the project on GitHub. Any pointers, comments, suggestions please let me know via Twitter or leave a comment on this blog.

My Journey with Asp.Net MVC and RavenDB – Introduction to Tournament

The idea of this journey is based on a number of reasons. The main two are to learn how to use and work with RavenDB, focusing the usage in an Asp.Net MVC context and the other is to re-work a site which has been live for the past 4 years which has had minimal change over that time and needs a refresh and admin functionality.

I have chosen to use RavenDB as the backend of this system due to the nature of how the data is stored. The site is a sporting results site so once an event has taken place the results will not change. They may need some tweaking while entering them but once in and confirmed they won’t change. Due to this a relational database I feel isn’t the best fit for the data. Reading and speed of reading the data is the main function. The current site calculates scores on the fly all the time where as this isn’t required due to the previous reason of once the results are in they’re in.

I have decided to post the code onto GitHub during the development so the world can see how I’m progressing. I will also be posting about what I have learnt, what issues I’ve come across and how I’ve over come them. Also I believe it’s a good way to show case of what I am capable of. This series won’t be a step by step walk through but I hope with looking at the code and the description of the main parts in the blog it should aid others in how to get up and running.

Any pointers, comments, suggestions please let me know via Twitter or leave a comment on this blog.

Let’s start the journey ….