My top 3 favorite Windows productivity tools!

There are way more tools I love and use every day, but the following 3 are my must-haves on every Windows machine I use, so I wanted to share them with you. And they are completely free!

(Ok, there are actually 4 – the first one, Chocolatey is a pre-requisite to the others. Chocolatey is a package manager, like apt-get, but made with Windows in mind. If you haven’t already, install this first! I’ll wait.)

#1 – Rapid Environment Editor (RapidEE)

C:\> choco install Devbox-RapidEE

RapidEE is an editor for the Windows environment variables! No more being stuck squinting in the unresizeable Windows dialog, this tool will not only let you add, search and modify envvars easily, it will also show invalid paths in red, letting you find dead entries quickly!

#2 – Everything

C:\> choco install Everything

Imagine a list view, with a search box at the top, and nothing else, which finds any file in your computer immediately – you have just imagined Everything. This lightweight search engine indexes your entire hard drive(s) in seconds after installing, letting you then find any file, anywhere! My preference is to bind the shortcut for a New Window to Ctrl-Shift-Z, so I can pop open Everything everywhere with this single chord. Set it under Tools – Options – Keyboard.

#3 – Ditto

C:\> choco install ditto

Ditto is a clipboard manager for Windows. If you ever used ReSharper’s Clipboard Manager you will appreciate this little tool – it remembers every single Ctrl-C you’ve ever done, allowing you to paste back any item at any time! The default shortcut for Ditto is Ctrl-`, and it is packed with features, such as searching, exporting and removing entries, with full unicode support and ability to sync clipboards between machines. What’s more, it supports “smart paste”, allowing you to paste entries in places such as the Windows Command Prompt, which will only support proper copying and pasting only in Windows 10.

Regular Expression enhancements in ReSharper 9

Yesterday, JetBrains announced the first public EAP of ReSharper 9! While I’m sure they will dedicate whole blog posts about the new (and truly amazing!) features of ReSharper 9, I wanted to beat JetBrains to the punch, and let you know about one incredible feature (which I consider a “killer feature” of ReSharper 9). I am talking about the improvements in dealing with Regular Expressions (RegEx)!

There were always 2 ways to write regular expressions in C# code – either inside one of the methods of Regex class, e.g. Regex.IsMatch, or defining a string literal as a constant, and using it in a Regex class later.

ReSharper 9 has Syntax Highlighting and Code Completion for regex literals! When you start typing a regex into a Regex class, you’ll see a list of suggestions, which is being narrowed down as you type:

There’s also a special syntax highlighting for the complete regex – as well as automatic checking for errors:

However, the “killer feature” is the ability to validate and test the regular expressions straight from Visual Studio! Pressing Alt-Enter anywhere inside the expression will bring the Quick Actions menu, with a new option to Validate regular expression:

Which opens a new window that lets you input some text to validate your regular expression!

But the best part of these new enhancements is the ability to transform any string literal into a regular expression! Simply press Alt-Enter inside the string literal, and select Make regular expression here:

Will turn into:

And you get all the regex features (validation, intellisense, etc) inside your regex string literals!

Amazing! Smile

Debug symbols, dotPeek and long paths in Visual Studio

In my previous post, I explained how to use the symbol server in dotPeek 1.2 to debug any assembly in Visual Studio, allowing you to set breakpoints and step into any method (provided it was decompiled by dotPeek).

While this is great, I noticed that there was one particular method I couldn’t step into – the moment I tried I got the sadly familiar Source Not Found page:

Clicking the Browse and find… link did nothing, and the Source search information dropdown appeared below. Expanding it, I could see where Visual Studio attempted to load the source file from:

Locating source for 'C:\Users\Igal\AppData\Local\JetBrains\dotPeek\v1.2\SymbolCache\CSharp\Microsoft.VisualStudio.ProjectSystem.VS.Implementation.pdb\538887009A094E419882756878C69B2A1\Microsoft.VisualStudio.ProjectSystem.VS.Implementation\VisualStudio\ProjectSystem\VS\Implementation\Package\Automation\OAProjectItems.cs'. (No checksum.)
The file 'C:\Users\Igal\AppData\Local\JetBrains\dotPeek\v1.2\SymbolCache\CSharp\Microsoft.VisualStudio.ProjectSystem.VS.Implementation.pdb\538887009A094E419882756878C69B2A1\Microsoft.VisualStudio.ProjectSystem.VS.Implementation\VisualStudio\ProjectSystem\VS\Implementation\Package\Automation\OAProjectItems.cs' does not exist.
Looking in script documents for 'C:\Users\Igal\AppData\Local\JetBrains\dotPeek\v1.2\SymbolCache\CSharp\Microsoft.VisualStudio.ProjectSystem.VS.Implementation.pdb\538887009A094E419882756878C69B2A1\Microsoft.VisualStudio.ProjectSystem.VS.Implementation\VisualStudio\ProjectSystem\VS\Implementation\Package\Automation\OAProjectItems.cs'...
Looking in the projects for 'C:\Users\Igal\AppData\Local\JetBrains\dotPeek\v1.2\SymbolCache\CSharp\Microsoft.VisualStudio.ProjectSystem.VS.Implementation.pdb\538887009A094E419882756878C69B2A1\Microsoft.VisualStudio.ProjectSystem.VS.Implementation\VisualStudio\ProjectSystem\VS\Implementation\Package\Automation\OAProjectItems.cs'.
The file was not found in a project.
...

And so on. Quick search for OAProjectItems.cs using my most favorite tool, Everything, revealed that it was indeed present in that location, so why couldn’t Visual Studio open it? I decided to open the file manually by pasting its full path into the Start – Run dialog (Win-R), but then I got the following error:

Finally, I tried to go to the file location using the cmd, and I got my answer – the path was simply too long for Windows (and therefore, Visual Studio) to handle!

Windows has an unfortunate MAX_PATH limitation at 260 characters is the source of great pain, and I hope Microsoft fixes it one day. Meanwhile, here’s how to “work around” this particular issue: dotPeek generates debug symbols under %LOCALAPPDATA%\JetBrains\dotPeek\v1.2\SymbolCache\CSharp for C# code, and unfortunately, it isn’t possible to configure. Luckily, it is possible to configure where Visual Studio looks for debug symbols. This is a little-known page in the Solution properties called Debug Source Files:

What I did was copy the entire contents of dotPeek’s SymbolCache\CSharp directory into a local directory d:\sym, and added it to the search list (pictured above). Also, I made sure to delete everything from the bottom list (Do not look for these source files) – if Visual Studio is unable, for any reason, to open a source file, it will add it to this blacklist. It was filled with the files I needed, so I removed them from the list.

After doing this, I could go back to debugging, was was able to step into methods that were previously unavailable!

As the comments below mention, this is also possible to do without copying, by creating a (symbolic) link between the directories! From an elevator command shell, run:

mklink /D d:\sym %localappdata%\JetBrains\dotPeek\v1.2\SymbolCache\CSharp

Thanks guys!

Happy debugging!

How to debug anything with Visual Studio and JetBrains dotPeek v1.2!

Sometimes, we wish we could just step into some 3rd party library, to figure out how it works, but we either don’t have the source code, or otherwise just can’t. Fortunately, this is made possible by dotPeek v1.2 that was just released, which can act as a symbol server for decompiled assemblies!

So let’s suppose we want to put a breakpoint inside Console.WriteLine (or any other method in any other assembly). Here’s what we need to do:

  • Open dotPeek, add the required assemblies to the Assembly Explorer, and press the Start Symbol Server button.

You can configure the port and the symbol generation settings in Tools – Options. The default address is http://localhost:33417/

  • In Visual Studio, go to Tools – Options, then navigate to Debugging – Symbols. Add the location of the dotPeek symbol server.

In addition, make sure that Just My Code (in General) is unchecked, and press OK. Some symbols will be loaded, this might take a few moments.

  • Next, we need to set a breakpoint inside the method which we’re interested in. This can be done with a little-known Visual Studio trick, allowing you to create a breakpoint at any function. Go to Debug – New Breakpoint – Break at Function, and in the dialog enter the fully qualified method name, e.g. System.Console.WriteLine.

After pressing OK, you’ll get a message saying “IntelliSense could not find the specified location. Do you still want to set the breakpoint?”. It’s fine – press Yes.

  • Finally, start your application with the debugger (F5) and you will stop at the breakpoint! You can use all familiar debugging options, such as stepping over/into, watch, autos and the datatip.

Happy Debugging!

How I Improved Sharing My Blog Posts On Twitter With This One Simple Trick!

(I’m so sorry for writing this title! They made me do it, I swear! I won’t ever do it again!)

I came across a wonderful blog post the other day, titled Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Couchbase vs Neo4j vs Hypertable vs ElasticSearch vs Accumulo vs VoltDB vs Scalaris comparison, written by Kristof Kovacs (@kkovacs). Being the twitter addict I am, I immediately felt the need to share this post, and I was met with the eternal twitter dilemma – which part of the title/credit I omit, due to the 140 character limit. I ended up doing this:

Which is what most people would do (chop enough of the title to fit, use a URL shortener for the link, the twitter way). Which didn’t leave room to give credit to the original author of the post (a fact pointed to me by @omervk, thank you very much!)

I was wondering if there was a way to encode this information (author’s twitter handle) within the post itself, using html meta tags. Then I realized that most online tech blogs have this feature available to them – any time you paste a link from, say, Engadget or The Verge on twitter, a preview of the article is visible in the twitter stream.

Turns out, this is available not just for tech blogs, but to everyone! Enter Twitter Cards.

Twitter Cards is exactly what it sounds like – you can embed media and other information to your own tweets! There are many types of cards to choose from, but for my purposes I wanted the Summary card – a simple card including a title, description, thumbnail and the twitter account information. This requires adding various html meta tags to your page, all detailed in the URL above.

To enable Twitter Cards, you will need to Apply for the service, after adding the meta tags on your blog/site. The application process supposedly takes up to 72 hours, but in my case I was approved within minutes! Your mileage may vary.

Here is what I did to enable this for my WordPress blog:
  1. Install Twitter Cards Meta plugin
  2. After installing and activating, go to Settings – Twitter Cards Meta
  3. Fill in your information (e.g. your twitter username)
  4. Go to the Twitter Cards Validation page, and log in with your twitter account
  5. Select desired card type (I selected Summary)
  6. Click the Validate & Apply tab, and paste a link to any one of your blog posts
  7. If you’re satisfied with the results, press Go and after seeing the preview, apply for approval

It might take up to 72 hours to get approved, but usually this happens within minutes (as it was in my case). You will get an email from twitter when you’re approved:

And voilà! Now, every time someone shares a link to a blog post I wrote, it will contain a preview, and my twitter handle!

Turning on Twitter Cards will also give you much better Twitter Analytics – with much more detailed information about clicks and retweets of your cards!

Migrating from TFS to git to Visual Studio Online–the survival guide

This is a step-by-step recount of my attempt to migrate an existing 3-year old TFS (TFVC) repository to git, while keeping all the history, and then moving it to Visual Studio Online, a TFS in the cloud. This wasn’t an easy task, as there is no direct export-import built into either tools. I also ran into some problems during this lengthy process, and I describe the steps to solve them. Set aside a few hours of your time, brew some coffee (or tea), and let’s get started!

Step 1: TFS to git

We first need to export the entire TFS repository to git. This is achieved by “cloning” the entire TFS repository with git-tfs or git-TF, both open source tools. While designed to do the same, mainly, providing a bridge between git and TFS, the former is an older, more mature project, and the latter is a tool created by Microsoft for the same purpose. I initially tried to use Microsoft’s git-TF, but after more than 24 hours of waiting for the clone to end, it died with a Java exception (the tool by Microsoft is written in Java).

First, install git-tfs. It’s best installed with Chocolatey by using cinst gittfs. Next, we need to get the exact name of the project we want to clone. Assuming your TFS server is https://tfs.contoso.com, first, run the following command to list all the branches:

git tfs list-remote-branches https://tfs.contoso.com/tfs/DefaultCollection

This will output all the branches that exist in the DefaultCollection. Branches marked with [*] are the root branches, and these are what we want to clone. Run the following command to clone the required branch, e.g.:

git tfs clone https://tfs.contoso.com/tfs/DefaultCollection $/Project/MyApp/Dev [<directory>]

This will clone the Dev branch (in this case) to a new directory specified in [<directory>]. If [<directory>] is not specified, a new directory called Dev will be created.

The clone operation might take up to several hours, depending on the size of your TFS repository. The cloning operation with git-tfs will pull each individual check-in, and will apply it as a series of git operations, recreating the history exactly as it happened! Git-tfs will also run a git-gc operation every 100 commits, so the new repository size will be kept in check, and obsolete files will be removed.

After the clone operation is complete, you will have a git representation of the TFS repository on your local disk.

Step 2: git to Visual Studio Online

After setting up a free account in Visual Studio Online (VSO), we first need to create a new project, and make sure Git is selected as the source control. After the project had been created, we can go to the CODE tab, and be presented with instructions to either clone an empty repository, or push an existing one. We want to use the second option, so first, set the remote URL in our cloned git repository to point to VSO.

In a command prompt (or use any one of the 3rd-party visual Git clients), go to the cloned git directory and paste the first command, e.g.:

git remote add origin https://contoso.visualstudio.com/DefaultCollection/_git/MyProj

This will set the origin (the remote repository address) to point to our git repository in VSO.

The most important step is next: unfortunately VSO has a timeout limit of 1 hour for any connection, meaning that an attempt to push the entire repository might fail, if the repository is too big! What’s worse, pushing is an atomic operation, it cannot be resumed in case of a timeout, you will have to start from the beginning.

Luckily, there’s a workaround I was managed to find at the bottom of a similar issue in MSDN forums. Basically, we can push our repository in smaller chunks, thus minimizing the chance of a timeout. This is not the ultimate solution, but it worked for me. Our problem is currently being able to push, subsequent cloning should be much faster, since download bandwidth is almost always faster than the upload.

Disclaimer: This is a somewhat advanced git usage, and I must admit, I don’t quite understand it myself fully. Below is a series of commands I ran, based on the mentioned issue, and it worked in my case. YMMV. If you can explain this, please leave a comment below!

First, the issue tells us, we need to get an idea of the number of commits we have, by running the following:

git rev-list --all --count

In my case, the number was 4012, which I assume is the total number of commits.

Next, however, the issue says to run this:

git rev-list master --first-parent --count

Which is the “depth of the ‘first-parent’ lineage of master”, according to the post author. Not quite sure what this meant (the documentation wasn’t very helpful, either), running this command produced a number which was about half the previous one: 2512.

Not knowing exactly what the numbers meant, I decided to try and follow the post’s advice, and split that number into 5 sections, of 500 commits each, and I pushed them in the following order:

git push origin master~2500:refs/heads/master
git push origin master~2000:refs/heads/master
git push origin master~1500:refs/heads/master
git push origin master~1000:refs/heads/master
git push origin master~500:refs/heads/master

And finally:

git push -u origin master

To push the remaining commits, and create a remote tracking branch for master. To make extra sure, I ran this last command again, and got “Everything up-to date” message.

Refreshing the CODE tab in VSO revealed that all my source code was uploaded successfully! I cloned the repository locally, and ran a diff between the two directories in Beyond Compare, just to make absolutely certain that everything was copied properly. It was!

Step 3: VSO to FogBugz

Few more things left to migrate, and those are all the issues on FogBugz to VSO. In the meantime, though, we need to keep our source control integration with FogBugz. Turns out this is also not a simple task. But, this is a blog post for another day. Stay tuned!

When good permissions gone bad–a case of a failed build

I was called over to see if I could help solve a strange issue – every time the build script (Ant) for the client’s Android app ran – certain files that were modified by the build script (a .properties file, few others), were suddenly inaccessible to other people logging to the machine – only the user who initiated the build could still write to the files. Looking at the file permissions tab proved as much: only the current user and the Administrators group could access the file!

My initial investigation into Ant’s build.xml led me to an interesting discovery – all the files that lost their permissions were modified using the ReplaceRegExp task for Ant – a task that could replace text in a file using regular expressions. Quick Google search for “replaceregexp ant file permissions” led me to this similar issue, which was, unfortunately, closed as wontfix.

imageI decided to investigate for myself. Using my go-to tool for this task, Process Monitor, I decided to trace all activity of java.exe (which runs the Ant tasks), looking for anything that has to do with setting permissions or writing the file, filtering just the file I was interested in (project.properties). After running the specific Ant task that did a simple regex replace in the file, it was indeed changed, and its permissions were changed as well.

However, looking at the File Summary window of Process Monitor (Tools – File Summary), I saw that not only there were no new ACL permissions set (Set ACL was called 0 times), no actual bytes were written!

So how was it possible that the file was modified, but nothing was written? Looking back at the events in Process Monitor I also could not see any calls to WriteFile.

Feeling confused, I then started looking for anything that could appear relevant, until one entry in particular caught my eye: a call to SetDispositionInformationFile with a flag Delete: True.

I checked who was calling this API from the Stack tab of this event’s properties, and saw that it was a call to deleteFile, originating in java.dll. This confirmed my suspicion that the file was not written directly at all – but replaced with another file, possibly from a temp directory.

My suspicions proved to be correct – looking at the source code for ReplaceRegExp task I saw that it was exactly how it did it: by using

FILE_UTILS.createTempFile("replace", ".txt", null, true, true)

to create a temporary file, and then later renaming that temporary file with the original file name!

FILE_UTILS turned out to be just a wrapper calling Java’s File.createTempFile. The 3rd argument, which was null in this case, tells createTempFile where to create the file. If null is passed, it will use the default %TEMP% environment variable.

Which ended up explaining the problem exactly – by default, it uses the local user’s %TEMP% directory to create the temp file that replaces the original one. The default per-user %TEMP% directory located in the %LOCALAPPDATA% directory of the user’s profile – meaning only the current user (and local Administrators) can access it! This means, that any file created in this directory will inherit the same permissions! In our case, ReplaceRegExp’s implementation caused the per-user temp file to overwrite the file that was in a public folder, causing it to lose all permissions!

A quick workaround for the problem was to set the TEMP directory to c:\temp, temporarily, during the build.

Next time your Ant script causes issues with file permissions – make sure your files are not replaced under your nose.

Code, music and single malt whisky!